Repository navigation
Tuple Sketches: Deserialization improvement + bug fix - #76
Conversation
|
Thanks for your contribution! I will take a review today. |
|
no problem @tisonkun, thank you for taking the time! Please let me know if there is anything I can help with. |
A Python SerDe can retain a slice of its input as the decoded value. A view created from a raw pointer does not retain the serialized sketch and can outlive its storage. Carry the original bytes through each sketch reader and build the view from that object. Translate the native cursor to an offset in the image so retained slices stay valid without copying the remaining input for every item.
Existing from_bytes implementations use bytes methods such as decode and index. Pass the original bytes object and the offset of each item directly to the callback so these readers work without copying the remaining input.
Read the four-byte little-endian length written by PyStringsSerDe and check that both the prefix and payload fit from the supplied offset. Python slices silently truncate, so a short payload could otherwise be returned as a complete string. Exercise a 16 MiB string and truncated records directly. These cases replace the repeated tuple fixtures and their string update policy.
tisonkun
left a comment
There was a problem hiding this comment.
I pushed three follow-up commits to keep retained summary views valid, preserve the bytes API for custom SerDes, and validate string lengths. I also simplified the tests around those cases.
Looks good to me. Please take another look at the follow-ups when you have a chance.
cboumalh
left a comment
There was a problem hiding this comment.
Just reviewed, thanks for simplifying this further!
|
Thanks for your contribution @cboumalh! Although, I have no idea when the next release will be scheduled. I would appreciate it if you could share your use case and see whether a recent release can help in your case. |
|
@tisonkun Thanks for the review :) We're deserializing large tuple sketches at scale through PySpark jobs at Amazon. On the AWS service we use, we don't have any other options besides using the library directly since the service hasn't adopted Spark 4.2 (supports tuple sketches natively), and we can only use PySpark. Currently we have to write the deserializer from scratch to overcome the bottleneck caused by the bug we just fixed. The sooner this gets released the better! If that is possible. |
Switches the per-entry deserialization buffer from an allocated nb::bytes (which copied the full remaining serialized blob on every retained entry, producing O(N²) copy work on large sketches) to a zero-copy memoryview over the C++ buffer. Updates the PyObjectSerDe.from_bytes contract from bytes to a buffer-protocol object and adjusts PyStringsSerDe accordingly. Also fixing a 3-byte length-prefix read that corrupted string summaries.
default performance:
performance with memory view:
bench_tuple_serde.py