[C++/Python] Kernel for SetItem(BooleanArray, values)

[asfimport]

We should have a kernel that allows overriding the values of an array by supplying a boolean mask and a scalar or an array of equal length.

Reporter: Uwe Korn / @xhochy
Assignee: David Li / @lidavidm

Related issues:

• [C++/Python] Kernel for SetItem(IntegerArray, values) (is related to)
• [C++] Add "replace" kernel (is related to)
• [C++/Python] Support necessary functionality to have an Arrow-string type in pandas (is depended upon by)

Original Issue Attachments:

• image-2021-11-22-14-40-02-692.png

PRs and other links:

• GitHub Pull Request #10412

_{Note: This issue was originally created as ARROW-9430. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
This would produce a new array, if that aligns with your expectations?

[asfimport]

Uwe Korn / @xhochy:
Yes, I'm aware of Arrow's immutable nature ;)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I am not sure this is actually a duplicate of ARROW-11044. At least my intent of a "replace" kernel was something else (provide a mapping with the replacement values), and ARROW-11044. turned into an "ifelse" kernel based on the PR discussion.

[asfimport]

Niranda Perera / @nirandaperera:
@jorisvandenbossche I agree. I think ARROW-11044 only resolves the 'scalar replacement'.

[asfimport]

David Li / @lidavidm:
Is the intended result here something like

Setitem(array, mask, values) -> new_array

SetItem([1, 2, 3, 4], [true, false, true, null], [4, 5]) -> [4, 2, 5, null]

?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
And to summarize the behaviour of the last example of @lidavidm:

• This kernel sets values with a boolean mask

• Nulls in the mask propagate as nulls in the result

• The length of the replacement values equals the number of true values in the mask (the case where those replacement values are of equal length as the input array / the mask is covered by "if_else" kernel)

  This provides the equivalent behaviour of numpy's array[mask] = values (except not modifying array in place, but returning a new array)

[asfimport]

Niranda Perera / @nirandaperera:
I think this can be done using ARROW-10640 (if_else function), by calling

if_else(mask, values, array)

Isn't it?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I am not sure that's the best way to implement it. "if_else" is a scalar kernel that works element-wise, where mask/values/array need to have the same length (or is a scalar that is broadcasted). While for the setitem operation here, the "values" to set (or replace) based on the mask have a different length (matching the number of true values in the mask). In principle we could first convert that array to an array of the correctly length, and then use "if_else", but not sure how efficient that would be compared to a custom implementation.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Note that @lidavidm alraedy has an open pull request: #10412. So maybe we can further discuss implementation details there.

[asfimport]

Niranda Perera / @nirandaperera:
@jorisvandenbossche Fair point! Thanks

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 10412
#10412

[asfimport]

Alvin Chunga Mamani / @AlvinJ15:
Hi @lidavidm,  I'm working in ARROW-1699 and using the method ReplaceWithArrayMask for replace the nulls using a generated mask, but the method gave me unexpected results when the input Array contains nulls and this nulls are replaced with values.
When testing this special Test on vector_test_replace_test.cc

this->Assert(ReplaceWithMask, this->array("[1, null, 1]"),
   this->mask("[false, true, false]"),
   this->array("[7]"),
   this->array("[1, 7, 1]"));

The result is:

Actual:
[
1,
null,
1
]

And  debugging the code, the output values is correct, but the null_bitmap isn't updated and keep the original null_bitmap
In the file vector_replace.cc  on line 202, the bitmap is updated only for turnoff the null replacement values

This is and expected behavior or an issue on ReplaceWithArrayMask ? 

[asfimport]

David Li / @lidavidm:
No, can you file a new JIRA? Probably the output bitmap needs to be a combination of the original array's bitmap, the mask, and the replacement array's bitmap.