Skip to content

Make the README a short, uniform guide for newcomers #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
liran merged 3 commits into from
Nov 4, 2026
Merged

Make the README a short, uniform guide for newcomers #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
02e7703
📝 docs: move reference detail from the README to docs
liran Nov 4, 2026
7892cdf
📝 docs: restructure the README around a runnable quick start
liran Nov 4, 2026
458413f
📝 docs: install v0.17.1 and link each package's API reference
liran Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ go vet ./...
```

Default tests use in-memory protocol data. To regenerate code, install the
[exact tool versions](README.md#generate), then run `./scripts/generate.sh` and
[exact tool versions](docs/development.md#generate), then run `./scripts/generate.sh` and
`git diff --exit-code`.

## Prepare a pull request
Expand Down
299 changes: 104 additions & 195 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,217 +2,126 @@

[![CI](https://github.com/batchstream/weir-protocol/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/batchstream/weir-protocol/actions/workflows/ci.yml) [![Go Reference](https://pkg.go.dev/badge/github.com/batchstream/weir-protocol.svg)](https://pkg.go.dev/github.com/batchstream/weir-protocol) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

Shared public schemas, generated Go types and validation for Weir servers and
clients. Requires Go 1.27.3.
The public gRPC schema, generated Go types and validation shared by Weir servers
and clients.

```sh
go get github.com/batchstream/weir-protocol@v0.17.0
```
## Overview

Applications do not need this module directly. Use the
[Go SDK](https://github.com/batchstream/weir-go). Use this module to write another
client or to change the protocol.

### Contents

| Path | Contents |
| --- | --- |
| `api/weir/v1/store.proto` | The public schema |
| `api/weir/v1` | Generated Go types for the `weir.v1.StoreService.Execute` RPC |
| `api/protocol` | Validation of requests and responses: envelopes, Store names, resources, records and Native |

### Terms

| Term | Meaning |
| --- | --- |
| Store | A named backend connection on a Weir server. Every request names a Store. |
| Resource | The path of one record in a Store, made of percent-encoded segments. `s:hello` is the string key `hello`. |
| Execute | The only RPC. It is a bidirectional stream. One stream selects one Store and one operation kind. |
| Record | One Read or Mutate request. The first record index is 1. Each later record adds 1. |
| Result | The response to one record. Results arrive in completion order, so match them by index. |
| Native | One opaque backend command, such as a query or a scan. It does not run record batching. |

### Dependency direction

The dependency direction is one-way: Weir and the Go SDK depend on this module.
This module depends on neither repository, including through test dependencies.
It contains no server lifecycle, backend adapter, or SDK.

- `api/weir/v1`: the bidirectional `weir.v1.StoreService.Execute` RPC for Read,
Mutate, and Native. Clients send it to a Weir server that hosts the Store; any
other server fails the RPC with NOT_FOUND.
- `api/protocol`: common envelope, Store name, resource, record, and Native
validation.

The public schema lives in `api/weir/v1/store.proto`. Servers and SDKs consume
the module's generated Go types.

## Streaming semantics

Execute selects one Store and operation kind for its lifetime. Each Read or
Mutate request carries one record. Servers aggregate admitted records into
database batches; clients send and consume incrementally without collecting a
wire batch. The protocol imposes no item or payload-size limit on business calls
or individual records; servers may bound the size of one message. Backend
protocols govern accepted documents and commands. Native carries a required
opaque Document. NativeHead carries optional opaque metadata and a content type
for subsequent body chunks. Request and metadata bytes are opaque to the public
protocol. Their content types select adapter-owned formats; new adapters need no
protobuf fields or schema changes. Empty request data is permitted. Unknown
valid content types are left to the selected adapter.

Each gRPC message, `ExecuteRequests` or `ExecuteResponses`, carries one or more
requests or responses in stream order. A peer handles each exactly as if it came
in a message of its own, so a sender packs everything it has ready into one
message instead of paying for a message per record. `ValidateExecuteRequests`
and `ValidateExecuteResponses` check a message's own envelope; its requests and
responses are validated one at a time as they are applied.

Each request is validated before execution. A record the server cannot admit,
such as one its adapter rejects or one sent while the Store drains, receives its
own failed result, NOT_STARTED for a mutation, and the stream continues. A
request that fails `ValidateExecuteRequest`, changes the Store or kind, breaks
the index sequence or cancels a record not yet sent fails the RPC with
INVALID_ARGUMENT after the results of the records admitted before it. A record
without a result when the RPC fails may still have effects. The first record
index is 1; every later record advances it by one. UINT64_MAX is invalid. Each
record receives at most one ExecuteResponse with its ordinal and typed result,
in completion order, so consumers match results by index. One stream can
therefore carry the records of independent callers without a slow or failed
record holding back the others. A client that shares a stream validates each
record before sending it, since an invalid request ends the stream for every
caller on it.

`timeout_micros` optionally bounds one Read or Mutate record from the moment
the server reads it; time spent in transport buffers is not counted, so clients
still enforce their own deadline. A record still queued when it expires fails
with DEADLINE_EXCEEDED, NOT_STARTED for a mutation; an executing record reports
the evidence its cancellation leaves. Zero sets no record deadline. Native
requires zero and uses the RPC deadline. `RecordTimeout` and `TimeoutMicros`
convert between the field and Go durations without shortening or disabling a
caller's budget.

A request that sets only `store_name` and `cancel_index` cancels an earlier
record of the same stream. It does not advance the index or receive a response.
The canceled record still receives its own result: a CANCELLED failure,
NOT_STARTED for a mutation, while it is still queued; the evidence its
cancellation leaves while it executes; and its ordinary result once it has
completed. Cancel requests let a stream shared by several callers release the
work of one caller who stops waiting. Native streams use RPC cancellation
instead.

Each stream has room for `RecordWindow` (32) records without results, whatever
the Store queue holds. A client that never has more than 32 records without
results is never held back by the server's window, so its cancel requests take
effect as soon as they arrive; a further record waits until a result frees room.

Servers and clients enforce Store/kind consistency and index sequences. Stateless
validators check each envelope and nested fields. Requests and responses reject unknown
fields throughout the message tree. Responses require declared result/event
variants, enum values and valid completion evidence.

ReadResult.missing confirms document absence after a successful read.
TARGET_NOT_FOUND reports a required adapter-owned target that does not exist; it is
an individual Failure, not a missing document result.
Clients can consume results incrementally; SDK convenience collectors may retain
results at the caller's request. Each response confirms one record.

Store names and resource paths are separate fields. Resource paths use canonical percent-encoded segments without a scheme
or leading slash. `ParseRelativeResource` validates and decodes these paths;
`EncodeSegment` constructs individual segments. NativeRequest permits an empty
resource for backend-root operations. Nonempty Native resources are canonical
prefixes; the request bytes select the backend command, path and parameters.
Weir imposes no Native operation allowlist; backend authorization applies.

A mutation stream is not a transaction. Mutations to the same resource execute in
input order across requests, including after a failed item; different resources may
run in parallel. Ordering across streams follows the database semantics. APPLIED also includes satisfied no-op semantics such as Lua keep or deleting an
already missing document. An APPLIED result may also
carry a subsequent acknowledgement failure; other outcomes require a Failure.
An individually received MutationResult retains its application evidence if the
stream later fails. A submitted mutation without a result may have applied.
Clients must never automatically replay unacknowledged mutations.

`MutateRequest.options` carries `MutationOptions` uniformly for every action.
Its `visibility` uses `MutationVisibility`: absent options or the zero value
APPLICATION_ACKNOWLEDGED require application acknowledgement. QUERY_VISIBLE
additionally confirms visibility to subsequent normal queries when APPLIED has no
Failure. APPLIED with a Failure retains application evidence but does not confirm
visibility, so replay is still unsafe. Search adapters wait for backend refresh;
adapters without deferred query visibility satisfy this when the mutation is
acknowledged. No-op mutations must also establish the requested visibility.
Unknown option fields and undeclared visibility values are rejected.

LuaTransform supplies Source and optional Input with Lua 5.4 semantics and no
runtime selector. The entry point, return actions, helper set and document
conversion rules are stable parts of the contract.
Source returns exactly one function, called as `function(current, incoming)`.
Missing current or omitted Input is nil; documents are ordinary Lua tables.
The callback returns exactly one object to create/replace, or an explicit
`weir.keep()`, `weir.delete()` or `weir.reject(message)` action. Keep/delete on
missing are successful no-ops; reject is PRECONDITION_FAILED/NOT_APPLIED.
Nil, missing/multiple returns, arrays and scalar results are invalid. A callback
error or invalid result never writes. Explicit null is
`weir.null()`. See the server's [Lua guide](https://github.com/batchstream/weir/blob/main/docs/lua.md).

## Native backend commands

Native accepts exactly one Command at index 1, followed by client half-close.
Every event uses index 1. A complete response emits NativeHead, zero or more
nonempty body chunks, and NativeEnd with RESPONSE_COMPLETE; its body may be empty.
A failure may emit NativeEnd before any head. Body chunks always follow a head.
NativeHead contains optional opaque metadata and the body's content type.
NativeEnd reports response transport completion, never whether a mutation was
applied. Its evidence may be retained when a later RPC error occurs. Backend
error replies remain native response bodies; clients interpret backend status
and application semantics. Clients must not automatically replay Native calls
whose effects are unknown.

Queries, scans, and counts use Native backend commands. The client owns native
filters, projection, sorting, page sizes, offsets, cursors, and counting semantics.
For MongoDB, send ordered BSON commands such as find, aggregate, getMore, and
count. For Search, send native HTTP requests such as _search, _count, and PIT
requests. Responses retain the complete native reply, including cursor, hit, and
count metadata; Weir does not extract documents or synthesize a page checkpoint.
Clients manage continuation requests and backend cursor/PIT cleanup. Native does
not promise cross-call session affinity, cursor portability, a snapshot, or a
Weir continuation-token format. Backend protocols and permissions apply.

Read and Mutate provide record batching, per-resource ordering, and application
evidence. Native is the direct opaque command path; it does not run record
batching or Lua transforms. This keeps query execution and pagination policy in
the backend and its client.

## Validation rules

Requests and responses reject unknown fields throughout the message tree.
FailureCode accepts only declared nonzero values. MutationOutcome and
NativeCompletion require known evidence values. A known Failure never discards
APPLIED evidence or establishes retry safety.

Native content types and backend expressions retain adapter-owned
semantics. Each adapter documents its accepted formats and limits; opaque bytes
do not waive that contract. Lua helpers, conversions and execution semantics
follow the Lua contract above.

## Validate
The server and the SDK depend on this module. This module depends on neither.

## Requirements

| Item | Requirement |
| --- | --- |
| Go | 1.27.3 |

## Quick start

This encodes a resource path and validates it.

**Step 1.** Create a module and add the protocol.

```sh
go test -race ./...
go vet ./...
mkdir hello
cd hello
go mod init example.com/hello
go get github.com/batchstream/weir-protocol/api/protocol@v0.17.1
```

Default tests use in-memory protocol data. They do not contact external
endpoints or start databases. Descriptor tests protect the public RPC and field
contract. CI also checks that the module graph contains neither the server nor
the SDK and that `go.mod` replaces no module.
**Step 2.** Save this as `main.go`.

## Generate
```go
package main

Supply these exact tools externally; they are not stored in the repository:
import (
"fmt"
"log"

- protoc 33.4
- protoc-gen-go 1.36.11
- protoc-gen-go-grpc 1.5.1
"github.com/batchstream/weir-protocol/api/protocol"
)

func main() {
resource := protocol.EncodeSegment("my db") + "/records/s:hello"
fmt.Println(resource)

segments, err := protocol.ParseRelativeResource(resource)
if err != nil {
log.Fatal(err)
}
fmt.Println(segments)
}
```

**Step 3.** Run it.

```sh
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.5.1
# Place the official protoc 33.4 compiler on PATH.
./scripts/generate.sh
git diff --exit-code
go run .
```

Expected output:

```text
my%20db/records/s:hello
[my db records s:hello]
```

Use `PROTOC`, `PROTOC_GEN_GO`, `PROTOC_GEN_GO_GRPC`, and `GOFMT` to select explicit
external tool paths. The script checks compiler/plugin versions and preserves the
project's named struct style in generated code. It removes unused compatibility
aliases and unimplemented server scaffolding; implementations provide the complete
service interface. Regeneration is byte reproducible.
## Documentation

## Publish a stable version
| Guide | Contents |
| --- | --- |
| [Semantics](docs/semantics.md) | Streams, deadlines, cancellation, mutations, Lua, Native and validation rules |
| [Development](docs/development.md) | Module boundaries, code generation and releases |
| [`api/protocol` reference](https://pkg.go.dev/github.com/batchstream/weir-protocol/api/protocol) | Every validation function |
| [`api/weir/v1` reference](https://pkg.go.dev/github.com/batchstream/weir-protocol/api/weir/v1) | Every generated type |

After the reviewed change is merged, an authorized maintainer can publish:
## Development

```sh
gh workflow run release.yml --ref main -f version=vX.Y.Z
go test -race ./...
go vet ./...
```

The workflow accepts only stable `vMAJOR.MINOR.PATCH` versions, runs CI on the
`main` commit, then creates the version tag at that commit and its GitHub release.
It refuses a version already tagged at another commit. Protocol consumers should
pin the published version rather than a branch name.
Default tests use in-memory data. To change the schema, edit
`api/weir/v1/store.proto` and follow [Generate](docs/development.md#generate).

- Contributing: [CONTRIBUTING.md](CONTRIBUTING.md)
- Security reports: [SECURITY.md](SECURITY.md)

## Related repositories

| Repository | Purpose |
| --- | --- |
| [weir](https://github.com/batchstream/weir) | Server |
| [weir-go](https://github.com/batchstream/weir-go) | Go client SDK |
| **[weir-protocol](https://github.com/batchstream/weir-protocol)** | Public gRPC schema and validation (this repository) |
| [weir-charts](https://github.com/batchstream/weir-charts) | Helm chart |
| [weir-tests](https://github.com/batchstream/weir-tests) | System tests and benchmarks |

## License

MIT. See [LICENSE](LICENSE).
56 changes: 56 additions & 0 deletions docs/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Development

How to validate the module, regenerate code and publish a version.

## Module boundaries

The dependency direction is one-way: Weir and the Go SDK depend on this module.
This module depends on neither repository, including through test dependencies.
It contains no server lifecycle, backend adapter, or SDK.

## Validate

```sh
go test -race ./...
go vet ./...
```

Default tests use in-memory protocol data. They do not contact external
endpoints or start databases. Descriptor tests protect the public RPC and field
contract. CI also checks that the module graph contains neither the server nor
the SDK and that `go.mod` replaces no module.

## Generate

Supply these exact tools externally; they are not stored in the repository:

- protoc 33.4
- protoc-gen-go 1.36.11
- protoc-gen-go-grpc 1.5.1

```sh
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.5.1
# Place the official protoc 33.4 compiler on PATH.
./scripts/generate.sh
git diff --exit-code
```

Use `PROTOC`, `PROTOC_GEN_GO`, `PROTOC_GEN_GO_GRPC`, and `GOFMT` to select explicit
external tool paths. The script checks compiler/plugin versions and preserves the
project's named struct style in generated code. It removes unused compatibility
aliases and unimplemented server scaffolding; implementations provide the complete
service interface. Regeneration is byte reproducible.

## Publish a stable version

After the reviewed change is merged, an authorized maintainer can publish:

```sh
gh workflow run release.yml --ref main -f version=vX.Y.Z
```

The workflow accepts only stable `vMAJOR.MINOR.PATCH` versions, runs CI on the
`main` commit, then creates the version tag at that commit and its GitHub release.
It refuses a version already tagged at another commit. Protocol consumers should
pin the published version rather than a branch name.
Loading