Specify storage inheritance for FILE self-references - #603
Conversation
alkis
changed the title
| | OffsetIndex | yes | yes (7) | yes | yes | no | no | | ||
| | BloomFilter Header | yes | yes (8) | yes | yes | no | no | | ||
| | BloomFilter Bitset | yes | yes (9) | yes | yes | no | no | | ||
| | Self-Reference | yes | yes (10) | yes | yes | no | yes | |
There was a problem hiding this comment.
Why Page ordinal is not used here? When only a few pages are required to read, we still need to count number of values of skipped pages in order to decrypt a self-referenced value?
There was a problem hiding this comment.
You are right that a derived ordinal is wrong for exactly this reason. I switched the AAD to key on the self-reference offset instead. The offset is a field of the FILE value being read, so the reader has it in hand and never counts values in pages it skips.
Page ordinal would also work for a reader — you know the ordinal of the page you are decoding — but it ties the out-of-band range to page layout, which the rest of this PR deliberately avoids: compression and encryption are properties of the column chunk, not of the page a value happens to land in. Keying on offset keeps the stored representation independent of page boundaries, and the offset already uniquely identifies it within the file.
Key the self-reference AAD on `offset` rather than a derived ordinal, so a reader can resolve a self-reference without decoding the pages it skips. State that the encryption buffer of a self-reference must fit the 4-byte length field, and that larger values must use an external reference. State that the decompressed size is not stored and how readers obtain it. State that an encrypted stored representation is bound to a single column chunk and must not be shared between column chunks.
Align the wording with the validation rules, which state the requirement normatively. Co-authored-by: Isaac
Describe self-reference storage as being anywhere in the file rather than in a page, since the referenced bytes are outside the page data. Drop the forward-looking note about a possible decompressed size field. State that `offset` and `size` cover the encrypted module for an encrypted self-reference, as already stated for the unencrypted case. Describe rather than mandate how the stored bytes are interpreted. Co-authored-by: Isaac
The previous wording described `size` as the size after compression and encryption, which omits the length prefix and nonce that the range also covers. Reference the layout in Encryption.md so the extent is defined in one place. Co-authored-by: Isaac
5 participants
Specify storage inheritance semantics for
FILEself-references:inlinefield for schemas permitting self-referencesinlinecolumn chunkoffsetandsizeover the stored representationDiscussion: https://lists.apache.org/thread/zrzc7t9fccg92rx3h4fw3ndw3bdo5xr7