colbin

A binary serialization format faster than and as compact as Protocol Buffers. No schema compilation, no dependencies. Go, Rust and JS.

The Go module, the Rust crate and the JavaScript (WebAssembly) package read and write the same bytes. An optional self-describing mode, colbin/JSON, makes a message convertible to JSON with nothing else on hand.

No precompilation step. The Go struct is the schema. There is no .proto file, no protoc and no generated types; cb field tags are optional.
No dependencies. The Go module's go.mod has no require block.
Protobuf-level size and speed. In the measurements, on TPC-H, TPC-DS, Public BI and SchemaStore, colbin is smaller than vtprotobuf and msgpack on every dataset. On slices of structs it is several times smaller, because it writes them as compressed columns. On tables it encodes 1.1–1.2× faster than vtprotobuf and decodes 1.6–2.0× faster; on small documents generated code still encodes faster.
Self-describing mode (colbin/JSON). The schema section travels inside the message, so ToJSON renders it without the Go type. The section adds under 1% to a table or a document.

data, err := colbin.Marshal(&order)            // a struct, no tags required
err = colbin.Unmarshal(data, &back)

doc, err := colbin.MarshalSelfDescribing(&order) // colbin/JSON
text, err := colbin.ToJSON(nil, doc)             // JSON, no schema or type needed

The problem
Design
Usage
Measurements
Limitations
Implementations
Where it is used
Documents

The problem

The target is bandwidth between services, storage and clients, without changing how the code that produces and consumes the data is written.

JSON costs no setup. But every record carries its field names, and every number is decimal text. A list of a thousand records repeats every key a thousand times.
Protocol buffers remove the names and encode numbers in binary. They need a .proto file, a compiler step and generated types kept in sync with the application's own. They also encode a list of records row by row: each row repeats every field tag, and each element pays a tag and a length.
Varints, used by protobuf for keys, lengths and integers, are compact but decoded byte by byte, in a loop whose trip count depends on the data.

colbin is designed around four constraints:

The Go struct is the schema. Field ids come from optional cb tags, or are derived from field names when there are none.
One format for every shape. A 30-byte message, a table of thousands of rows and a nested document of hundreds of kilobytes use the same encoding. The layout adapts per field, not per message.
A message can be read without the producer's type. A schema section, sent out of band or embedded in the message, is enough to render it as JSON.
Decoding never loops on data. Every size is either in a header or in a field whose width the header names.

Design

Each decision below was kept because it measured better than the alternative. RATIONALE.md records the measurements, including the ideas that were reverted. INTERNALS.md specifies the format bit by bit.

Messages are key runs, at one of two key widths

A message is a root descriptor byte followed by a key run — or, for a struct at four key bits with no schema or dictionary, the key run alone: a sequence of [key][descriptor][payload] fields, terminated by the end of the enclosing length. The run's key width is chosen per run and recorded in the descriptor that opens it:

	layout	fields	can skip an unknown field
K4 (narrow)	`[key:4][desc:4]`, one byte	ids 1–16	yes: the descriptor says how long the field is; the type comes from the schema
K8 (wide)	`[key:8][desc:8]`, two bytes	ids 1–255	yes: the descriptor names a class with a known size rule

A nested struct opens its own run, so a narrow struct can hold a wide one and the reverse. A type with more than 255 fields is split into pages of 255. Key 255 of each page links to the next, up to id 4 080. (INTERNALS §3–4, §6.5)

Field ids: tags are optional

A field's id comes from one of two places.

From a tag. cb:"N" makes the field id N, and its wire key is N−1. A type whose ids all fit in 1–16 uses the one-byte K4 field header.

From the field name. A field without a number takes fnv8 of its name (FNV-1a folded to one byte). If that key is taken, it moves to the next free one upward. Fields are assigned in declaration order, after all tagged fields. The Rust and JavaScript implementations use the same hash, so an untagged Go struct is readable from both.

type Reading struct {          // no tags: ids from the names, K8 keys
    SensorID  uint64
    Timestamp int64
    Value     float64
}

type Reading struct {          // tags: explicit ids, K4 keys
    SensorID  uint64  `cb:"1"`
    Timestamp int64   `cb:"2"`
    Value     float64 `cb:"3"`
}

Untagged types stay compatible as long as fields keep their names and their relative order:

New fields go at the end.
A rename keeps the old id if the old name is put in the tag: cb:"SensorID".
Reordering is only unsafe when two names hash to the same byte, because declaration order decides which one moves.

A field's name comes from its json tag. The name in the schema, in the JSON view, and the one an unnumbered field hashes is the json tag's name when there is one, then a name in the cb tag, then the Go name. The cb number only places the field on the wire. So ID int64 `json:"id" cb:"1"` is id 1, called id. json:"-" does not drop a field; cb:"-" does.

What an untagged type costs, measured on a six-field record: derived keys span 0–255, so the type always uses K8. That is one extra byte per present field: 33 B instead of 27, and 44 ns instead of 38 to encode. Decode time is the same. Both widths skip a field the reader does not know, so producer and consumer can add or drop fields independently either way.

Tags are therefore an optimization for size and for the narrow fast path, not a requirement. In a table, keys are written per column rather than per row, so the key width hardly matters there.

Scalars are byte-aligned, and sizes are not varints

Zero values are omitted. A field holding 0, "", false, or an empty slice or map is not written at all.
Integers are sign-and-magnitude, with the byte count in the descriptor:
- A value takes 1 to 8 bytes, as many as it needs.
- In K4 the descriptor is a byte count first: 0–3 carry no payload, 4–11 carry 1–8 bytes, 12–15 a length. The values 1–4 (1–3 and −1 for a signed field) are the descriptor itself, so a bool, a status code or a small count is one byte, key included. A negative value other than −1 takes the length form, one byte more.
- K8 adds a varint form, which the encoder writes only when it is shorter: 3.34 B against 3.60 per field on ids 0–1000.
Floats are written byte-reversed. The zero bytes of an IEEE-754 value are the low mantissa bytes, and reversing them lets the byte count drop them: 1.0 is 2 bytes, and a float64 holding an exact float32 is 5.
Strings and blobs of up to 8 bytes carry no size in K4: the descriptor's byte count is the size. Longer ones carry a length of 1, 3 or 5 bytes. A decoded string is a sub-slice of the message, not a copy.
No size is a varint. Every read of a size is a single load of a width the header already gave, so decoding never runs a loop whose trip count is data.

(INTERNALS §3.1–3.2, §5)

Composites carry a byte length

Nested structs, lists, tables and maps are written with a backpatched byte length:

In K8, that length is what lets a reader skip a composite without knowing its type.
In K4, a composite is the length form of the descriptor ([key:4][1 1 0 f], the flag a struct's key width or a table), and the class comes from the schema. Composites therefore do not force the wide width.

Pointers distinguish "absent" from "zero":

A nil pointer is omitted.
A pointer to a zero value writes an explicit zero, 2 bytes. This is the only value the format writes just to mark presence.
A pointer to a slice or a map is encoded as the slice or the map.
A slice of pointers to structs, []*T, is encoded as the []T it points at: the same bytes, a table past 8 rows. A nil element is written as an empty struct.

Map keys are strings, integers or bools. Map values may be scalars, strings, blobs, structs, any, or slices of structs or of scalars: map[K][]Struct writes each value as a list, or as a table past 8 rows. map[K]*Struct and map[K]*[]T are encoded as their pointees, a nil value as an empty one. A type holding a map of slices uses K8.

Slices of structs become tables

A []Struct whose element fields are all columnable switches layout at 8 rows:

Below 8 rows it is a list: each element is a key run.
From 8 rows up it is a table: one key per column instead of one per field per row.

The choice is made per field, at encode time, and the two layouts have different descriptor classes, so the reader dispatches on what it finds.

A table of more than 4 096 rows is written in groups of 4 096, each its own set of columns. The encoder then transposes one group at a time rather than the whole table, which keeps its memory to a group's worth and its columns in cache: TPC-H's normalized tables encode 18–26% faster for 0.04% more bytes.

six-field row	bytes per row
list of structs, 7 rows	16.9
table, 1 000 rows	11.0

Each integer column goes through the column codec:

First, a transform chosen per column: raw, delta, frame-of-reference or constant.
Then the residuals are bit-packed in blocks of 128 values, at a width chosen per block. 128 values at w bits is exactly 16w bytes, so every block is byte-aligned and no state crosses a block boundary.

256 × int64	raw	encoded
monotonic ids	2 048 B	171 B
timestamps	2 048 B	235 B
constant zeros	2 048 B	3 B
random	2 048 B	2 051 B

Decode runs at 1.3 ns per value and encode at 3.5. A struct containing a nested struct or a slice cannot be a column, and it stays a list at any length. (INTERNALS §6.2–6.3, §7)

packed5 (opt-in)

colbin.SetPacked5(true) packs strings drawn from upper-case letters and digits at about 5 bits per character:

The encoder decides per string and falls back to plain bytes when packing would not be shorter, so turning it on never grows a message.
The choice is recorded in each string's descriptor, so the reader needs no configuration.
It costs a pass over every string on both sides, which is why it is off by default.

(INTERNALS §8)

The schema section, and colbin/JSON

The wire carries no names and no types. The schema section is the type's encoding plan serialized: a key, a name and an op code per field. Nested and recursive types are hoisted into an indexed struct table, so a recursive type describes itself in finite space. It can travel two ways:

Out of band, once per connection. Messages stay exactly the size Marshal produces.
Embedded (MarshalSelfDescribing). This is colbin/JSON, the self-describing mode: root byte 0xD4/0xDC, then the section, then the body. The body is byte for byte what Marshal writes. Unmarshal skips a section that is the Go type's own, and reads the message through any other one, matching fields by name (see below). ToJSON(nil, msg) renders the message with no schema and no Go type available.

A browser can therefore send a payload to a Go backend with no schema of its own: codec.marshal(value).standalone goes into colbin.Unmarshal and lands in the typed Go value. The browser numbers its fields 0, 1, 2 and types every integer as int64, so the decoder matches each field to the Go one by name (json tag first, then the Go name), skips fields the Go type does not have, leaves fields the message does not have at zero, and converts numbers to the field's type — refusing, with the field's name, a value that does not fit (int64 70000 into an int16) and a float into an integer. Send .standalone, not .message: a message with no section has nothing to match by and is read by the Go type's own field ids. A foreign decode of 500 products costs about 1.3× the same records written by Go and about a tenth of ToJSON followed by encoding/json.

The section is 38 to 448 bytes for the types measured here. On a table or a document that is under 1% of the message. On a single small record it is most of the message, which is why a stream of small messages should send the section once instead.

ToJSON's output has the same content as encoding/json's, with three differences:

Fields a record omitted are rendered after the ones it holds.
Empty slices and maps render as null.
NaN and infinity are refused.

(INTERNALS §10, §12)

Dynamic values

any, []any and map[string]any are carried anywhere a typed field can go. A dynamic value writes its own type on the wire: one descriptor byte naming integer, float, string, blob, list, map, null, true or false.

An array of records inside a dynamic value is written behind a tag that refers to a struct in the section. The rows are then the same table a typed field would write. A thousand five-field records take:

26 024 B as a typed []User;
26 067 B as map[string]any{"rows": users};
78 196 B in encoding/json.

(INTERNALS §9)

Usage

go get github.com/ivanjoz/colbin

type SaleLine struct {
    ProductID uint32
    Quantity  uint32
    UnitCents int64
}

type Sale struct {
    ID         uint64
    CreatedAt  int64
    TotalCents int64
    Lines      []SaleLine // a table from 8 lines up
}

data, err := colbin.Marshal(&sale)
err = colbin.Unmarshal(data, &back)

A handle resolves the type's plan once. After that, encode and decode allocate only what the decoded value itself needs:

var saleCodec = colbin.MustCodec[Sale]()

buf, err = saleCodec.Append(buf[:0], &sale)
err = saleCodec.Unmarshal(buf, &back)

Generated code: codec.Generate emits the straight-line calls against the wire package that one would write by hand.

ten-field record	encode	decode
hand-written against `wire`	7.6 ns	22.4 ns
`codec.Generate`	6.9 ns	22.9 ns
`Codec[T]` handle	23.5 ns	27.6 ns
`Marshal` / `Unmarshal`	52.7 ns	48.5 ns

Reading without the type:

schema, _ := colbin.SchemaFor[Sale]()
section := schema.Bytes()                      // send once

parsed, _ := colbin.ParseSchema(section)       // on the reader
text, _ := colbin.ToJSON(parsed, message)
value, _ := colbin.DecodeAny(parsed, message)  // map[string]any

standalone, _ := colbin.MarshalSelfDescribing(&sale)   // colbin/JSON
text, _ = colbin.ToJSON(nil, standalone)

Measurements

Against msgpack and vtprotobuf

On four datasets: TPC-H, TPC-DS, Public BI and SchemaStore. colbin is the smallest on every one: 1.6–4.2× under msgpack and 1.1–1.6× under vtprotobuf (Lambda x86_64 run; the bytes have not changed since).

Speed is a local run on an i7-1355U, Go 1.27.0, colbinbench in colbin-benchmarks, median of 5 samples, cores 0 and 2, against msgpack (tinylib/msgp, generated) and vtprotobuf. Each cell is the other format's time divided by colbin's, as a geometric mean over the dataset's cases: above 1 colbin is faster.

dataset (cases)	encode msgpack	encode vtprotobuf	decode msgpack	decode vtprotobuf
TPC-H (10)	1.22×	1.23×	1.74×	1.97×
TPC-DS (24)	1.29×	1.18×	1.45×	1.58×
Public BI (46)	1.21×	1.13×	1.70×	1.61×
SchemaStore (21)	0.71×	0.76×	0.88×	1.09×

Decode allocates a fresh value per call. SchemaStore counts the 21 of its 27 documents that have a Go struct. The dynamic path (map[string]any) is not shown: protobuf has no equivalent of it, so it is not comparable.

It loses on small documents: SchemaStore encode is 0.71× msgpack and 0.76× vtprotobuf, because a generated encoder (about 5 ns for a 7-byte message) beats an interpreted plan (about 20 ns). It also loses on tables of 1 to 25 rows (TrainsUK1, income_band, region, nation) and on the float-heavy Redfin1–Redfin4 encode (0.81–0.86× msgpack), where colbin does decimal compression that msgpack does not. Every number is in PERFORMANCE.md.

In the browser

The npm package decodes a table into a flat typed buffer in WebAssembly. A row builder, generated per field signature and cached, then turns that buffer into objects, so no JSON text is produced or parsed. Measured on 1 000 product records under Node 24 (bun run bench in js/):

	wire	gzip	to objects
JSON + `JSON.parse`	106 670 B	12 124 B	0.32 ms
colbin + `codec.unmarshal`	30 413 B	3 182 B	0.16 ms

Reproducing

git clone https://github.com/ivanjoz/colbin-benchmarks && cd colbin-benchmarks
datasets/fetch-all.sh                        # download or generate the datasets, checked by SHA-256
go run ./cmd/colbinbench -cases all -formats colbin,msgpack,vtprotobuf \
  -ops encode,decode-fresh,decode-reuse -count 5 -cores 0,2

Comparative benchmarks live in that repository so that colbin's go.mod stays free of protobuf and msgpack.

Limitations

A field's type is not on the K4 wire. A reader steps over a field it does not know, but a field whose type changed under the same id is read as the new type. Give a changed field a new id.
Untagged ids depend on field names and declaration order, as described under Field ids.
Not supported yet: fixed-size arrays, slices of slices other than [][]byte, interfaces with methods (a protobuf oneof), map[any]T, map[bool]any, map[K]*scalar and map values nested deeper than a slice (map[K][][]T, map[K][]*T). int and uint are encoded as 64-bit.
An empty slice and a nil slice are the same on the wire. Both decode as nil, and a pointer to an empty slice decodes as a nil pointer.
No compatibility promise across minor versions while on 0.x. Every side must run the same version, and a mismatch shows up as corrupt data rather than a version error.
Known implementation gaps are listed in INTERNALS §20. For example, the Rust derive does not write maps of structs or of slices.
Field ids stop at 4080. A protobuf field number above that has to be renumbered.

Implementations

	location	notes
Go	the repository root	the reference implementation: reflection codec, `Codec[T]`, code generator, schema section, JSON rendering
Rust	`rust/`	`#[derive(Colbin)]` emits straight-line encode and decode; JSON → colbin encoder with schema inference (`rust/ENCODER.md`)
JavaScript	`js/`, npm `colbin`	the Rust crate compiled to WebAssembly; decodes to objects, columns or JSON text, encodes from JSON

#[derive(Colbin)]
struct Charge {
    #[cb(1)] company_id: u32,
    #[cb(2)] note: String,
}
let message = charge.encode();
let back = Charge::decode(&message)?;

import { Codec } from 'colbin'
const codec = await Codec.open()
codec.setSchema(section)
const rows = codec.unmarshal(message)

The implementations are pinned to each other with test corpora that Go writes:

Rust decodes Go's bytes, encodes the same bytes from the same values, and renders every document walk to Go's exact JSON.
JavaScript does the same for every type in its corpus, and refuses every single-byte corruption that Go refuses.
CI regenerates each corpus and fails on any difference.

(INTERNALS §18)

Where it is used

colbin is used in Genix, a self-hostable ERP and e-commerce platform:

Backups. Tenant data exports, one .colbin.zstd entry per batch of table rows.
genix-orm. Struct, slice and map columns stored as colbin blobs in ScyllaDB, and whole records in DynamoDB.
Configuration and sessions. The per-company configuration document, and the session token.
Next: responses from the Genix backend to its frontend, through the npm package.

Two more projects use it:

fareward, inside Genix: framing between the Go client and the Rust services (credits, locks, budgets, the request log). Go uses Codec[T] and Rust uses #[derive(Colbin)].
genix-search: query requests and responses.

Documents


`INTERNALS.md`	the format bit by bit, the Go codec, the Rust crate, the WebAssembly module, the npm package, cross-language testing and release
`RATIONALE.md`	the decisions and the measurements behind them, including reverted ones
`rust/README.md`	the Rust crate and its derive
`rust/ENCODER.md`	encoding JSON to colbin: inference, refusals, warnings
`js/README.md`	the npm package
colbin-benchmarks	the comparative benchmarks and the pinned datasets

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
codec		codec
column		column
corpus		corpus
internal/wirefmt		internal/wirefmt
js		js
packed5		packed5
rust		rust
web		web
wire		wire
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
INTERNALS.md		INTERNALS.md
LICENSE		LICENSE
PERFORMANCE.md		PERFORMANCE.md
RATIONALE.md		RATIONALE.md
README.md		README.md
bun.lock		bun.lock
colbin.go		colbin.go
colbin_test.go		colbin_test.go
doc.go		doc.go
example_slice_test.go		example_slice_test.go
go.mod		go.mod
package.json		package.json
slice.go		slice.go
slice_bench_test.go		slice_bench_test.go
slice_test.go		slice_test.go

Navigation Menu

Repository navigation

About

Repository files navigation

colbin

The problem

Design

Messages are key runs, at one of two key widths

Field ids: tags are optional

Scalars are byte-aligned, and sizes are not varints

Composites carry a byte length

Slices of structs become tables

packed5 (opt-in)

The schema section, and colbin/JSON

Dynamic values

Usage

Measurements

Against msgpack and vtprotobuf

In the browser

Reproducing

Limitations

Implementations

Where it is used

Documents

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

colbin

The problem

Design

Messages are key runs, at one of two key widths

Field ids: tags are optional

Scalars are byte-aligned, and sizes are not varints

Composites carry a byte length

Slices of structs become tables

packed5 (opt-in)

The schema section, and colbin/JSON

Dynamic values

Usage

Measurements

Against msgpack and vtprotobuf

In the browser

Reproducing

Limitations

Implementations

Where it is used

Documents

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages