Describe the bug
Dictionary-encoded values nested inside a struct or list change xxhash64's result, so the same data hashes differently depending on its encoding. When the first field of a struct, or any list element, is dictionary-encoded, the hash accumulated so far is thrown away. Earlier arguments, and earlier list elements, then have no effect on the result.
To Reproduce
-- Dictionary-encoded field inside a struct: the first argument is dropped
SELECT xxhash64(7, named_struct('d', arrow_cast('a', 'Dictionary(Int32, Utf8)'))) AS dict_7,
xxhash64(8, named_struct('d', arrow_cast('a', 'Dictionary(Int32, Utf8)'))) AS dict_8,
xxhash64(7, named_struct('d', 'a')) AS plain_7,
xxhash64(8, named_struct('d', 'a')) AS plain_8;
-- Dictionary-encoded list elements: only the last element counts
SELECT xxhash64(make_array(arrow_cast('a', 'Dictionary(Int32, Utf8)'),
arrow_cast('b', 'Dictionary(Int32, Utf8)'))) AS dict_list,
xxhash64(make_array('a', 'b')) AS plain_list,
xxhash64(make_array('b')) AS plain_last_only;
$ cargo run --bin datafusion-cli -- --spark -f repro.sql
+----------------------+----------------------+---------------------+---------------------+
| dict_7 | dict_8 | plain_7 | plain_8 |
+----------------------+----------------------+---------------------+---------------------+
| -8582455328737087284 | -8582455328737087284 | 2701516948731497016 | -694621655434011405 |
+----------------------+----------------------+---------------------+---------------------+
+----------------------+----------------------+----------------------+
| dict_list | plain_list | plain_last_only |
+----------------------+----------------------+----------------------+
| -6391946315847899181 | -7357372313250070212 | -6391946315847899181 |
+----------------------+----------------------+----------------------+
Expected behavior
Dictionary encoding should not change the hash: dict_7/dict_8 should equal plain_7/plain_8, and dict_list should equal plain_list.
Additional context
Bug fixed in Comet already: apache/datafusion-comet#5757
Describe the bug
Dictionary-encoded values nested inside a struct or list change xxhash64's result, so the same data hashes differently depending on its encoding. When the first field of a struct, or any list element, is dictionary-encoded, the hash accumulated so far is thrown away. Earlier arguments, and earlier list elements, then have no effect on the result.
To Reproduce
Expected behavior
Dictionary encoding should not change the hash: dict_7/dict_8 should equal plain_7/plain_8, and dict_list should equal plain_list.
Additional context
Bug fixed in Comet already: apache/datafusion-comet#5757