[diskann-garnet] Update data design doc - #1328
Conversation
There was a problem hiding this comment.
Pull request overview
Updates the diskann-garnet data design documentation to correct term descriptions and to provide more explicit key/value sizing for the data stored in Garnet.
Changes:
- Clarifies key prefixing/scratch-space expectations for Garnet operations.
- Documents key/value sizing for major term types (vectors, neighbor lists, quantized vectors, attributes, metadata, ID mappings).
- Updates internal-term descriptions (start point behavior, FSM representation, quantizer state storage).
Suppressed comments (2)
diskann-garnet/docs/data-design.md:129
- In the External ID Mapping section, "Interal" is misspelled and the note says "this key" when it refers to the value size. Also, describing IDs as a "bitstring" is misleading since these are byte strings.
*Key*: External ID bytes; this key is a variable length bitstring that the user assigned.
*Value*: Interal ID as bytes; this key is always 4 bytes in length.
diskann-garnet/docs/data-design.md:120
- Similar to other sections, this describes the external ID as a "bitstring". Since external IDs are arbitrary byte strings, using "byte string" is clearer and consistent with the rest of the document.
*Value*: External ID; this is variable length bitstring that the user assigned.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| ### Key Data Prefixing | ||
|
|
||
| In order to reduce allocations in the data access path in Garnet, Garnet needs some place to scribble state into during operations. It uses a single byte immediately preceding the first key byte for this purpose. This means that any key pointer given to Garnet access methods must contain valid space preceding the real key. For this reason, key data pointers are `* mut` and not `* const` and care must be taken to ensure the memory preceding that pointer is valid. | ||
| In order to reduce allocations in the data access path in Garnet, Garnet needs some place to scribble state into during operations. It uses a single byte immediately preceding the first key byte for this purpose. This means that any key pointer given to Garnet access methods must contain valid space preceding the real key. For this reason, key data pointers are `* mut` and not `* const` and care must be taken to ensure the memory preceding that pointer is valid. In diskann-garnet, we precede the key data with at least 4 bytes of scratch space. |
| Neighbor lists are stored as a fixed size of `(max_neighbors + 1) * mem::size_of::<u32>()` where `max_neighbors` accounts for the graph slack factor. The final entry is the true length of the neighbor list. | ||
|
|
||
| For example, in a graph where the degree is 16 and the graph slack factor is 1.3, the size of a neighbor list would be `((16 * 1.3) as u32 + 1) * 4` bytes long. Using fixed size lists this way means that all neighbor list allocations are the same size. | ||
| For example, in a graph where the `M` value is given as 16, the size of a neighbor list would be `(16 + 1) * 4 = 68` bytes long. Using fixed size lists this way means that all neighbor list allocations are the same size. |
| *Key*: Internal ID as bytes; this key is always 4 bytes. | ||
| *Value*: Quantized bits for the vector. This varies by quantizer. For BIN-family quantizers, this is 1-bit per dimension plus a fixed overhead of up to 6 bytes. For the Q8 quantizer, this is 20 bytes + 1 byte per dimension. | ||
|
|
||
| Quantized vectors a similar to full precision vectors in that they are fixed size and read/written as a whole, although they will often have a more complex representation that just an array of quantized elements. |
| ### Attributes | ||
|
|
||
| *Key*: Internal ID as bytes | ||
| *Key*: External ID as bytes; this is variable length bitstring that the user assigned. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1328 +/- ##
=======================================
Coverage 91.55% 91.56%
=======================================
Files 522 522
Lines 99541 99541
=======================================
+ Hits 91139 91143 +4
+ Misses 8402 8398 -4
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
Copilot please check if there are any remaining discrepancies between the updated design document and code |
Co-authored-by: harsha-simhadri <5590673+harsha-simhadri@users.noreply.github.com>
I checked the doc against the code. One genuine discrepancy: the Neighbor Lists section said Everything else matches the code: 4-byte internal-ID keys and the |
| ### Key Data Prefixing | ||
|
|
||
| In order to reduce allocations in the data access path in Garnet, Garnet needs some place to scribble state into during operations. It uses a single byte immediately preceding the first key byte for this purpose. This means that any key pointer given to Garnet access methods must contain valid space preceding the real key. For this reason, key data pointers are `* mut` and not `* const` and care must be taken to ensure the memory preceding that pointer is valid. | ||
| In order to reduce allocations in the data access path in Garnet, Garnet needs some place to scribble state into during operations. It uses a single byte immediately preceding the first key byte for this purpose. This means that any key pointer given to Garnet access methods must contain valid space preceding the real key. For this reason, key data pointers are `* mut` and not `* const` and care must be taken to ensure the memory preceding that pointer is valid. In diskann-garnet, we precede the key data with at least 4 bytes of scratch space. |
There was a problem hiding this comment.
Anything to note regarding pointer alignment?
| *Value*: `[u32; max_neighbors + 1]` stored as bytes. Every value in the index will have this fixed size of `(max_neighbors + 1) * 4` bytes. | ||
|
|
||
| Neighbor lists are stored as a fixed size of `(max_neighbors + 1) * mem::size_of::<u32>()` where `max_neighbors` accounts for the graph slack factor. The final entry is the true length of the neighbor list. | ||
| Neighbor lists are stored as a fixed size of `(max_neighbors + 1) * mem::size_of::<u32>()` where `max_neighbors` is the configured maximum degree `M`. Unlike core DiskANN, this is the true maximum degree that a neighbor list can reach; the graph slack factor is applied to lower the build target degree rather than to inflate this stored maximum. The final entry is the true length of the neighbor list. |
There was a problem hiding this comment.
Note that core diskann has moved away from the magic slack in favor of max-degree and pruned-degree.
This update corrects some errors and gives specific key and value sizes for every term stored in Garnet.