IPIP-550: PBNode field ordering by achingbrain · Pull Request #550 · ipfs/specs

achingbrain · 2026-08-24T13:20:35Z

Having Link messages before the Data message in PBNode messages means for HAMT shards you have to read every Link before you can process any of them, since the hash type/fanout values used to calculate the prefix each directory entry has is stored in the Data field at the end of the message.

This IPIP suggests allowing writers to encode the Data field first, this allows readers to use streaming parsers that can skip out on reading links if they have already reached the directory entry they are interested in.

Further discussion can be found in #533

github-actions · 2026-08-24T13:22:06Z

🚀 Build Preview on IPFS ready

🔎 Commit: 30f42c7
🔏 CID bafybeia5rsmh2crukvx63qa7qkcbsotv6hxirkwyeopdx2moafvowxldea
📦 Preview:
- inbrowser.link

Having `Link` messages before the `Data` message in `PBNode` messages means for HAMT shards you have to read every `Link` before you can process any of them, since the hash type/fanout values used to calculate the prefix each directory entry has is stored in the `Data` field at the end of the message. This IPIP suggests allowing writers to encode the `Data` field first, this allows readers to use streaming parsers that can skip out on reading links if they have already reached the directory entry they are interested in. Further discussion can be found in #533

lidel

Thanks, left some quick first-pass drive-by comments below.

Writing the `Links` first makes reading HAMT data more expensive since, if you are only trying to read a certain path, you have to read all of the `Links` before you can process any of them, as the config needed to calculate the hash prefix length is stored in the `Data` field at the end of the message. Swap the fields round so `Data` can be read before `Links`. Further discussion: ipfs/specs#533 IPIP: ipfs/specs#550

* feat: tests for IPIP-550 PBNode field ordering Gateways must resolve UnixFS content regardless of PBNode field order: the legacy Links-first encoding and the Data-first order proposed for the unixfs-v1-2026 profile. - fixtures: pbnode-field-orders.car, single-entry Directory and HAMTShard in both orders, byte-identical to the IPIP table - tests: path resolution through all four roots, raw block responses byte-exact for all five blocks Refs ipfs/specs#550 * chore: changelog entry under v0.14.1

Make the Data-first ordering opt-in per profile and align the text with DAG-PB codec behavior. - unixfs.md: Profiles registry section with names implementations SHOULD use; both PBNode orderings with reader/writer rules and UnixFS precedence over the legacy DAG-PB codec spec; data-first test vectors in the appendix - ipip-0550: accurate motivation and compatibility, PBNode field order profile parameter (links-first default), two-way security notes, alternatives, fixtures relabeled and linked to tagged gateway-conformance v0.14.1 CARs, working code links

lidel · 2026-08-27T20:01:11Z

Pushed 17444ef and b707207: the ordering is now defined per profile. Does this look sensible @achingbrain?

unixfs.md gains a Profiles registry (unixfs-v0-2015, unixfs-v1-2025, unixfs-v1-2026) showing both PBNode orderings with reader/writer rules, and Compatibility now matches DAG-PB codec behavior: decoders already accept both orders, and for UnixFS data this spec takes precedence on the write side.

Working code in Go:

gateway-conformance v0.14.1 ships this IPIP's fixtures (linked from the Test fixtures section) and tests that gateways serve Directory and HAMTShard nodes in both field orders (feat: tests for IPIP-550 PBNode field ordering gateway-conformance#304).
Reads already work both ways in Go (go-codec-dagpb accepts either order), so only the write side needed code.
test(unixfs): lock in PBNode field order behavior (IPIP-550) boxo#1212 and test(unixfs): lock in CID profile behavior (IPIP-550) kubo#11439 add opt-in write support: the unixfs-v1-2026 profile name defined here plus a dedicated low-level option, verified byte-exact against the fixtures table. Defaults and existing profiles keep their bytes and CIDs.

That makes ratification relatively low-risk: whoever needs the streaming-friendly order can lock it in via the profile, and everyone else, legacy DAGs and libraries included, keeps preexisting behavior.

…-ordering # Conflicts: # src/unixfs.md

achingbrain · 2026-08-28T10:04:32Z

Thanks for the updates, all looks good to me.

lidel · 2026-08-31T14:53:06Z

I've opened:

feat: opt-in Data-first PBNode field order ipld/go-codec-dagpb#103 but its a nice to have, optional, test(unixfs): lock in PBNode field order behavior (IPIP-550) boxo#1212 does not need it, it uses self-contained encoder.
feat: noncanonical fixtures (PBNode field order) ipld/codec-fixtures#212 with regression tests to ensure read-interop with Rust etc continues to work. Also optional, boxo and gateway-conformance are enough for GO-JS interop.

A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. PBNodeFieldOrder stays as a documented low-level opt-in; UnixFS_v0_2015 and UnixFS_v1_2025 now pin PBNodeLinksFirst explicitly. Tests derive data-first from UnixFS_v1_2025 plus the knob and keep asserting the same IPIP-550 fixture bytes and CIDs. Refs ipfs/specs#550

No new profile: a dated successor to unixfs-v1-2025 reads as "the recommended latest" and invites unintentional adoption of a new de facto CIDv1 default. Data-first stays an explicit low-level opt-in. - ipip-0499: explicit "PBNode field order: links-first" row in both profile tables, protecting them from unintended change - ipip-0550: defines the parameter instead of a profile and records the dropped profile under Alternatives - unixfs.md: links-first canonical, data-first opt-in; profile registry keeps 2015/2025 only

A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. Import.UnixFSPBNodeFieldOrder stays as the documented low-level opt-in; unixfs-v0-2015 and unixfs-v1-2025 now pin links-first explicitly. - deps: boxo bump to the ipfs/boxo#1212 commit that drops UnixFS_v1_2026 - test/cli: field order exercised via the config knob, same IPIP-550 fixture bytes and CIDs Refs ipfs/specs#550

lidel · 2026-09-02T13:00:23Z

To keep the scope in check, pushed d99d0ee: dropped the unixfs-v1-2026 profile from this IPIP.

A dated successor to unixfs-v1-2025 reads as "the recommended latest" and invites unintentional adoption, effectively a new CIDv1 default, and this IPIP should not change any defaults. Instead, PBNode field order is now a regular profile parameter (links-first when absent). We should not touch ratified IPIPs, but just for clarity the IPIP-0499 tables state links-first explicitly so the preexisting profiles can't drift, and data-first writing stays a documented low-level opt-in that no profile enables (ipfs/boxo#1212, ipfs/kubo#11439, ipld/js-dag-pb#111).

The read side is unchanged and remains the priority: both orders formalized and covered by tests (gateway-conformance v0.14.1). Rationale is recorded under Alternatives;

A future IPIP can still mint a data-first profile if real multi-implementation demand appears, and that review deserves to be slow. We just won't do it right now in this IPIP.

Motivation now spells out the primary goal: both orderings exist in the wild, reading either is the guarantee, and the opt-in writer keeps that read path exercised. Cites the robustness principle from the architecture specs.

To allow efficient reading of HAMT directory contents, encode Data messages before Links. This would be a breaking change as CIDs would change, so add an option to allow users to opt-in to this new behaviour. Refs: - ipld/ipld#383 - ipfs/specs#550

Adds support for writing `Data` fields before `Links` in PBNode messages for better HAMT traversal performance. Refs: - ipfs/specs#550 - ipld/ipld#383

* feat(unixfs): unixfs-v1-2026 profile (IPIP-550) Opt-in Data-first PBNode field ordering behind the new io.UnixFS_v1_2026 profile, per IPIP-550. Default output is unchanged: all existing profiles keep the legacy Links-first order and their CIDs. - ipld/merkledag: PBNodeFieldOrder global and a Data-first encoder used when a profile opts in (candidate for upstreaming to go-codec-dagpb) - ipld/unixfs/io: PBNodeFieldOrder profile parameter and UnixFS_v1_2026, wired through ApplyGlobals - tests assert byte-exact fixtures from the IPIP table Refs ipfs/specs#550 * ci: run gateway-conformance from ipip-550 branch Temporary pin so the PBNode field ordering tests from ipfs/gateway-conformance#304 run against boxo gateway backends. Switch back to a tagged release once one ships. * ci: gateway-conformance back to v0.14 v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again. * fix(merkledag): rotate dagpb output for data-first Derive PBNodeDataFirst bytes from dagpb.AppendEncode by moving the trailing Data field to the front, instead of a second hand-written encoder. One encoder owns link sorting and field presence, so a decoded block with unsorted links now re-encodes sorted under both orders, and byte parity with links-first holds by construction. Unknown PBNodeFieldOrder values return an error instead of silently encoding links-first. - fieldorder.go: moveDataFirst; godoc spells out the process-wide nature of the setting, the set-once-at-startup constraint, and cites the DAG-PB strictness section - coding.go: switch on the order after AppendEncode, error on unknown - fieldorder_test.go: unknown-order test; property test over random nodes against a protowire-based links-first oracle, covering unsorted links, nil and empty Data, multi-byte length prefixes, and CIDv0, identity, and CIDv1 link hashes * test(unixfs): pin unixfs-v1-2026 CIDs end to end Build a three-chunk file and a sharded directory from fixed inputs under UnixFS_v1_2025 and UnixFS_v1_2026 and pin the root CIDs; the file CIDs match `ipfs add --chunker=size-1000` output under each profile. Every dag-pb block in both DAGs must lead with the profile's first field, and the two DAGs must decode to the same nodes. - profile.go: UnixFS_v1_2026 written as a full literal so the test asserts each parameter instead of reconstructing the copy - profile_test.go: per-field asserts for all three profiles; ApplyGlobals subtest checks all six globals; saveAndRestoreGlobals also restores chunk.DefaultBlockSize and helpers.DefaultLinksPerBlock so applied profiles no longer leak into later tests * docs(unixfs): which CIDs unixfs-v1-2026 changes Every dag-pb node with both Data and Links gets a new CID, files larger than one chunk included; single-chunk raw-leaf files keep theirs. Existing links-first directories are re-encoded when reopened through the directory API (MFS directories on their next access), a sharded root first and each child shard as it is loaded. The godocs also state why the setting is a process-wide global and that it must be applied once at startup. - profile.go: field, profile and ApplyGlobals godoc - doc.go: UnixFS_v1_2026 in the profile list, Global Settings section - CHANGELOG.md: scope, MFS re-encode, ✨ marker, PR link * style(unixfs): gofumpt profile_test.go * refactor: drop UnixFS_v1_2026, keep opt-in knob A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. PBNodeFieldOrder stays as a documented low-level opt-in; UnixFS_v0_2015 and UnixFS_v1_2025 now pin PBNodeLinksFirst explicitly. Tests derive data-first from UnixFS_v1_2025 plus the knob and keep asserting the same IPIP-550 fixture bytes and CIDs. Refs ipfs/specs#550

* feat: opt-in unixfs-v1-2026 profile (IPIP-550) Opt-in Data-first PBNode field ordering via the new unixfs-v1-2026 config profile, per IPIP-550. Defaults and the preexisting unixfs-v0-2015 and unixfs-v1-2025 profiles are unchanged and keep their CIDs. - config: Import.UnixFSPBNodeFieldOrder (links-first default, data-first) and the unixfs-v1-2026 profile applying unixfs-v1-2025 settings plus data-first - core/node: wires merkledag.DefaultPBNodeFieldOrder from config - deps: boxo bump to the ipfs/boxo#1212 encoder commit - test/cli: byte-exact fixtures from the IPIP-550 table; pinned CIDs for existing profiles unchanged Refs ipfs/specs#550 * ci: run gateway-conformance from ipip-550 commit Temporary pin to the ipfs/gateway-conformance#304 head so the PBNode field ordering tests run against kubo. Switch back to a tagged release once one ships. * ci: gateway-conformance back to v0.14 v0.14.1 shipped the ipfs/gateway-conformance#304 tests, so the moving v0.14 tag covers them again. * docs: unixfs-v1-2026 scope and MFS re-encode State what the profile actually changes: every dag-pb node with both Data and Links gets a new CID, files larger than one chunk included, and data already in MFS is upgraded to the new order on first read (plain `ipfs files ls` or `stat` included), a sharded root before its child shards; the MFS root is re-encoded by any command that starts a node. - config/profile.go, docs/config.md: profile description and the unixfs-v1-2026 section - docs/changelogs/v0.43.md: highlight leads with the need (readers get the HAMT layout before links) and the upgrade-on-first-use behavior * chore: update boxo to 04a079ec27b1 Pins the ipfs/boxo#1212 branch tip: data-first bytes derive from dagpb.AppendEncode so link sorting is inherited, unknown field order values return an error, and pinned end-to-end CIDs cover the profile. * refactor: drop unixfs-v1-2026, keep opt-in knob A dated successor profile invites unintentional adoption and a de facto new CIDv1 default. Import.UnixFSPBNodeFieldOrder stays as the documented low-level opt-in; unixfs-v0-2015 and unixfs-v1-2025 now pin links-first explicitly. - deps: boxo bump to the ipfs/boxo#1212 commit that drops UnixFS_v1_2026 - test/cli: field order exercised via the config knob, same IPIP-550 fixture bytes and CIDs Refs ipfs/specs#550 * chore: update boxo to 02026ddcf262 Squash-merge of ipfs/boxo#1212 on main. * docs: tighten v0.43.1 changelog - tighter CID profile entry; MFS caveat now says re-encode happens on rewrite, not read - deps: boxo pseudo-version with ipfs/boxo#1212, gateway-conformance v0.14.1 * docs: tighten Import.UnixFSPBNodeFieldOrder docs

achingbrain requested a review from a team as a code owner August 24, 2026 13:20

achingbrain force-pushed the ipip-550-pbnode-field-ordering branch from 37fdb60 to 51f16e3 Compare August 24, 2026 13:48

lidel reviewed Aug 24, 2026

View reviewed changes

Comment thread src/ipips/ipip-0550.md Outdated

Comment thread src/unixfs.md Outdated

Comment thread src/ipips/ipip-0550.md Outdated

Comment thread src/ipips/ipip-0550.md

achingbrain mentioned this pull request Aug 25, 2026

Allow data-first field order in dag-pb PBNode message ipld/ipld#383

Open

achingbrain added 3 commits August 25, 2026 09:56

chore: make writing legacy format optional

94edd62

chore: add security considerations

d36b608

chore: add profile

213b326

achingbrain mentioned this pull request Aug 26, 2026

feat: allow encoding Data fields before Links in PBNode messages ipld/js-dag-pb#111

Merged

achingbrain added 2 commits August 27, 2026 09:34

chore: add fixtures

546204b

chore: add legacy fixtures

4fc886e

This was referenced Aug 27, 2026

feat: tests for IPIP-550 PBNode field ordering ipfs/gateway-conformance#304

Merged

test(unixfs): lock in PBNode field order behavior (IPIP-550) ipfs/boxo#1212

Merged

test(unixfs): lock in CID profile behavior (IPIP-550) ipfs/kubo#11439

Merged

lidel added 2 commits August 27, 2026 22:08

Merge remote-tracking branch 'origin/main' into ipip-550-pbnode-field…

f54834e

…-ordering # Conflicts: # src/unixfs.md

docs: full unixfs-v1-2026 parameter table

b707207

achingbrain mentioned this pull request Aug 28, 2026

feat: support IPIP-550 data-first field order ipfs/js-ipfs-unixfs#486

Merged

lidel mentioned this pull request Aug 31, 2026

feat: opt-in Data-first PBNode field order ipld/go-codec-dagpb#103

Open

achingbrain mentioned this pull request Sep 1, 2026

feat: add data-first dag-pb block ipld/codec-fixtures#211

Draft

lidel mentioned this pull request Sep 1, 2026

feat: noncanonical fixtures (PBNode field order) ipld/codec-fixtures#212

Open

vmx mentioned this pull request Sep 1, 2026

Allow Data field before Link field ipld/rust-ipld-dagpb#12

Open

docs: read-side motivation and credits

30f42c7

Motivation now spells out the primary goal: both orderings exist in the wild, reading either is the guarantee, and the opt-in writer keeps that read path exercised. Cites the robustness principle from the architecture specs.

Navigation Menu

Uh oh!

IPIP-550: PBNode field ordering - #550

IPIP-550: PBNode field ordering#550
achingbrain wants to merge 11 commits into
mainfrom
ipip-550-pbnode-field-ordering

achingbrain commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026 •

edited

Loading

Uh oh!

lidel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lidel commented Aug 27, 2026 •

edited

Loading

Uh oh!

achingbrain commented Aug 28, 2026

Uh oh!

lidel commented Aug 31, 2026 •

edited

Loading

Uh oh!

lidel commented Sep 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

achingbrain commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Build Preview on IPFS ready

Uh oh!

lidel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lidel commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

achingbrain commented Aug 28, 2026

Uh oh!

lidel commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lidel commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Aug 24, 2026 •

edited

Loading

lidel commented Aug 27, 2026 •

edited

Loading

lidel commented Aug 31, 2026 •

edited

Loading

lidel commented Sep 2, 2026 •

edited

Loading