VGI for .NET

Add your own functions and tables to DuckDB with C# and Apache Arrow.
Built by 🚜 Query.Farm

A VGI worker is a small .NET program that DuckDB talks to over Apache Arrow IPC. It can expose scalar / table / table-in-out / table-buffering / aggregate functions and whole catalogs (schemas, tables, views, macros) that behave like native DuckDB objects. DuckDB launches your worker for you when a query needs it — you never run a server by hand.

This repo is the C# worker SDK (QueryFarm.Vgi). It is wire-compatible with the canonical Python SDK and the Go/Rust/Java/TypeScript ports, so a C# worker drops in behind the same ATTACH ... (TYPE vgi). Built on vgi-rpc-csharp; targets .NET 10. The package selects its patched Arrow assemblies automatically, including in transitive consumers; no manual exclusions for DuckDB.NET's optional official Arrow assemblies are required.

This branch uses VGI wire 3.2.0 under vgi.v3 and requires a native VGI client speaking wire 3.2.x. Upgrade clients and workers together; wire 2.x, 3.0.x and 3.1.x clients are incompatible. See catalog metadata and publication.

Qualification: the DuckDB 2.0 HTTP lane passed 351 canonical SQLLogic files, with 28 explicit environment or capability skips; writable-catalog files are excluded from this HTTP count. The same shared suite tests the other SDKs. See docs/roadmap.md for the milestone history.

Why a worker instead of a C++ extension?

Traditional DuckDB extension	VGI worker
Written in C/C++, compiled and linked against DuckDB	Written in C#, one standalone worker process
Must be rebuilt for each DuckDB version	Version independent
Complex build / signing / release cycle	dotnet build, ship the executable
Runs in-process	Process isolation

Reach for it when you want to: call REST APIs or external services from SQL, run ML inference (ML.NET, ONNX Runtime, etc.), expose an external database/API/filesystem as a queryable catalog, or ship domain-specific functions to your team as one binary.

Start a new worker

The fastest way to a new C# worker is the vgi-starter-csharp template: a complete, tested worker with a scalar, a table function and a paginated REST API example, recipes for common data-engineering tasks, SQL and unit tests, metadata that scores 100 on vgi-lint, CI, Docker, signed release workflows, and an AGENTS.md for coding agents.

gh repo create my-worker --template Query-farm/vgi-starter-csharp --private --clone
cd my-worker
make init NAME=my_worker   # rename the catalog, package and binary
make check                 # everything CI runs

Or click Use this template on GitHub. See starter templates for details. The rest of this README covers the SDK itself.

Your first worker

1. Add the package:

dotnet add package QueryFarm.Vgi

2. Write a function and serve it:

using Apache.Arrow;
using Apache.Arrow.Types;
using QueryFarm.Vgi;
using QueryFarm.Vgi.Attributes;
using QueryFarm.Vgi.Scalar;

var worker = new Worker()
    .CatalogName("example")
    .DefaultSchema("main")
    .RegisterScalar(new UpperCaseFunction());

await worker.RunFromArgsAsync(args);

public sealed class UpperCaseFunction : ScalarFn
{
    public override string Name => "upper_case";

    private void Compute([Param] StringArray value, StringArray.Builder result)
    {
        for (var i = 0; i < value.Length; i++)
        {
            if (value.IsNull(i)) { result.AppendNull(); continue; }
            result.Append(value.GetString(i).ToUpperInvariant());
        }
    }
}

ScalarFn reflects Compute's parameters once per subclass and dispatches per batch — no manual Arrow-schema bookkeeping needed for the common case. Compute may be an instance or static method of any accessibility, and may be inherited from an intermediate base class; mark one method [Compute] to choose among overloads or to use another name. ScalarFn also has virtual Comment, Tags, Categories and Examples to override for catalog metadata.

Scalar functions can override ArgumentMonotonicity with an IReadOnlyList<ArgumentMonotonicity>. The optional list is in ArgumentsSchema declaration order and must have exactly one entry per field. Fixed, defaulted, and constant arguments each occupy one slot; a vararg declaration occupies one slot regardless of call-time expansion. Named SQL invocation order does not reorder the claims; null makes no claims.

3. Build it (dotnet build -c Release), then call it from a DuckDB engine that has the vgi extension. The vgi extension currently ships with Query Farm's Haybarn DuckDB distribution, which starts with no install via uvx haybarn-cli. Stock duckdb works too via INSTALL vgi FROM community.

INSTALL vgi FROM community;
LOAD vgi;

-- LOCATION is the command DuckDB runs to launch the worker; the first ATTACH argument names
-- the catalog it appears under (independent of what the worker itself calls itself).
ATTACH 'example' AS example (TYPE vgi, LOCATION './my-worker');

SELECT example.upper_case('hello'); -- => 'HELLO'

Troubleshooting

• ATTACH can't find the worker — LOCATION is resolved relative to DuckDB's working directory, not your project. Use an absolute path if in doubt.
• Catalog Error: ... does not exist — qualify with the attach alias (example.upper_case) or run USE example;.
• Runtime / type errors — exceptions thrown from Bind/Compute (and bind-time type-bound checks) surface directly in DuckDB's error message.

Arguments, constants, secrets and filters

Everything a function reads from a call is exposed through public types in QueryFarm.Vgi.Arguments — no QueryFarm.Vgi.Internal import is needed:

Where	Property	Type
Table / table-in-out / buffering / aggregate params	Arguments	TableArguments — Get<long>(0), Get<string>("format"), GetOrDefault("limit", 100L)
ScalarBindParams, ScalarProcessParams	Constants	ConstantArguments — [ConstParam] values by position or name: Get<long>("precision")
TableBindParams, TableInOutBindParams	SecretValues	SecretValues — Get(type, scope:, name:) (drives the two-phase lookup), Require (the same, but a missing secret fails the bind), FindByType, AllOfType, ForScope; each Secret has GetString(field)
init/process/scalar/aggregate params	SecretValues	the same, read-only
TableInitParams, TableBufferingFinalizeParams	Filters	FilterSet? — Matches(batch, row), Matches(rowDictionary), Predicates

public void Bind(TableBindParams p)
{
    var bucket = p.Arguments.Get<string>(0);
    var secret = p.SecretValues.Get("s3", scope: bucket);   // null on the first attempt: DuckDB
    if (secret is null) return;                              // resolves it and the bind is retried
    var keyId = secret.GetRequiredString("key_id");
}

Secrets arrive keyed by their DuckDB name (an unnamed CREATE SECRET (TYPE s3, ...) is __default_s3), never by type, so select them with the selectors above rather than Resolved["s3"]: FindByType/Get(type) match each secret's type field (preferring an unscoped one), Get(type, name:) also checks the type, and ForScope(path, type) takes the longest scope prefix of path, then an unscoped secret, then none. Secrets a function always needs are declared statically with RequiredSecrets => [new RequiredSecret { SecretType = "s3" }] (one entry per type/scope/name; an entry without a SecretType fails discovery rather than being skipped).

Argument constraints are declared once and both advertised (vgi_function_arguments()) and enforced at bind, with an error naming the argument: [ConstParam(Ge = 0, Le = 10)], [ConstParam(Choices = new object[] { "csv", "json" })], [ConstParam(Pattern = "^[a-z]+$")] for scalar constants, and TableArgFields.PositionalWithRange(...) / TableArgFields.Constrained(field, ge:, gt:, le:, lt:, choices:, pattern:) for table-function positional and named arguments. Patterns are unanchored searches (Regex.IsMatch); anchor with ^...$ for a full match. A SQL NULL value skips the checks.

Every function interface also has an Examples member (FunctionExample with Sql, Description, ExpectedOutput) surfaced through the catalog.

Function shapes

Shape	Interface	Base class	Use case
Scalar	IScalarFunction	ScalarFn	1:1 row mapping
Table (producer)	ITableFunction	—	row generator, no streamed input
Table-in-out	ITableInOutFunction	—	stream input rows → output rows, one turn at a time
Table-buffering	ITableBufferingFunction	—	sort/aggregate/join-style: see every input row before producing any output
Aggregate	IAggregateFunction	—	cumulative state + final emit

Each raw interface is a small, direct implementation surface (see any fixture under fixtures/QueryFarm.Vgi.ExampleWorker/ for real examples); ScalarFn is the one convenience base class with attribute-driven parameter binding ([Param], [ConstParam], [Setting], [OutputLength] — see "Your first worker" above). Projection/filter pushdown (including genuine expression/spatial-predicate pushdown, evaluated via an embedded DuckDB engine — see Internal/ExpressionFilterEvaluator.cs), ORDER BY/TABLESAMPLE hints, settings, secrets, splits, and cross-process state storage are all handled by the framework, not something each function reimplements.

A table-in-out function can answer one input batch with several replies (one upstream API page each, say): declare ContinuesOutput, emit every reply but the last with output.Emit(batch, metadata, more: true), and implement ITableInOutProcessor.Resume to emit the next one, keeping the position on the processor. A blended (InputFromArgs) function may also declare InputRowsPerCall to cap the input rows one call receives. See ContinuedOutput and fixtures/QueryFarm.Vgi.ExampleWorker/TableInOut/PagedFunctions.cs.

Filter-capable table functions advertise FilterSemanticProfiles (the default is vgi.duckdb.standard.v1 when FilterPushdown is true), plus optional versioned extension-function, runtime-filter, and evaluation-context capabilities. Protocol 2.0 filter payloads use only the strict vgi.filters.v2 snapshot/delta encoding: literals remain typed Arrow payload fields, field_ref can nest to any depth, and IN sets can be inline Arrow lists or externally indexed join-key batches. TableInitParams.Filters (a FilterSet, decoded against the unprojected bind output schema) rejects index/name mismatches, malformed payloads, or legacy encodings atomically; FilterSet.Matches applies required predicates exactly and may ignore a complete advisory predicate it cannot evaluate. FilterSet.Decode decodes a dynamic filter payload the same way.

Beyond functions: full catalogs

A worker can expose more than bare functions — a complete catalog of schemas, function-backed tables, views, and macros that behave like native DuckDB objects:

var worker = new Worker()
    .CatalogName("example")
    .DefaultSchema("main")
    .RegisterScalar(new UpperCaseFunction())      // ScalarFn, as above
    .RegisterTable(new MyGeneratorFunction())     // ITableFunction — see Function shapes above
    .RegisterSchema("data", comment: "Reference tables")
    .RegisterCatalogTable(myTable, identity: "data");

ATTACH 'external_db' (TYPE vgi, LOCATION './my-catalog-worker');

SELECT * FROM external_db.data.users;            -- a catalog table
SELECT * FROM external_db.main.upper_case(name)  -- a function
FROM (VALUES ('alice')) t(name);

identity scopes a registration to a specific catalog identity when a worker serves more than one logical catalog from the same process (see Worker.RegisterCatalog); most workers only need the default.

Attach options and credentials

A catalog can declare typed ATTACH-time options (ATTACH ... (TYPE vgi, LOCATION ..., region 'eu')) with AttachOptions.Spec (namespace QueryFarm.Vgi.Catalog), passed as the CatalogInfo.AttachOptionSpecs it registers via Worker.RegisterCatalog, and read the values in an OnAttach handler with AttachOptions.Read, and in every function and transaction hook of the attach with the params' AttachOptions property (AttachOptions.FromAttachData(bytes) from raw bytes):

worker
    .RegisterCatalog(new CatalogInfo
    {
        Name = "sales",
        AttachOptionSpecs =
        [
            AttachOptions.Spec("api_key", "API key", StringType.Default, required: true, secret: true),
            AttachOptions.Spec("region", "Region", StringType.Default, defaultValue: "us-east-1"),
        ],
    })
    .OnAttach(request =>
    {
        var options = AttachOptions.Read(request);   // GetString / GetBoolean / GetInt64 / ...
        if (options.GetString("api_key") is not { Length: > 0 })
            throw new InvalidOperationException("sales needs api_key");
        return null;
    });

// in any function (table, in-out, buffering, scalar, aggregate) or TransactionHooks:
var apiKey = initParams.AttachOptions.GetString("api_key");   // secret options included

The worker keeps an attach's options in its attach_opaque_data, which every call of the attach carries back. Secret options are encrypted there (XChaCha20-Poly1305, bound to the attach's random id, under an HKDF-SHA256 subkey of the worker's key): Worker.AttachSecretOptionsKey(...) if set, else on HTTP the signing key (VGI_SIGNING_KEY, or the key minted at startup), else on stdio/unix a host-local key file shared by the user's VGI workers on that host ($VGI_ATTACH_SECRET_KEY_FILE, else $XDG_RUNTIME_DIR/vgi/attach-secret-options.key, else <temp>/vgi-<user>/attach-secret-options.key; 32 random bytes, 0600, created on first use). So a secret never appears in plaintext in a value the client holds, on any transport (sealing rule 5), and catalog code never copies it anywhere itself. Every worker process serving the catalog over HTTP needs the same signing key (or secret-options key) to read it. A value that does not open (moved into another attach, tampered with) fails with attach_opaque_data not recognized.

AttachData.Parse(bytes) splits the value into the routing Identity, the per-attach random AttachId, the Extra bytes the handler returned (AttachData.ExtraOf(bytes) for just those) and Options. (AttachOptionSpecBuilder.Build in QueryFarm.Vgi.Internal still works; AttachOptions.Spec wraps it.)

Credential options (API keys, tokens, passwords) MUST be declared secret: true. The caller passes the value inline as an ordinary attach option. The DuckDB extension redacts it from duckdb_databases(), keeps only a salted hash of it in its cache key, and never logs it; clients mask it and keep it out of exported configuration. To keep the credential out of the SQL text, pass it as an expression:

ATTACH 'sales' (TYPE vgi, LOCATION 'https://sales.example.com', api_key getenv('SALES_API_KEY'));

Never copy a secret option into ExtraOpaqueData (on stdio and unix sockets that part reaches the client in plaintext) — read it through AttachOptions instead. A catalog that needs only to recognize a credential, not use it, can also keep just a digest. fixtures/QueryFarm.Vgi.WritableTxWorker reads its secret token option only through the accessor.

secret combines with required. A secret option may declare a default, but normally shouldn't. required is advertised, not enforced by the extension: reject a missing required option in your OnAttach handler.

Multi-branch tables and filtered scan planning

A CatalogTable can redirect its scan to one or more branches instead of a single ScanFunction: Branches lists ScanBranchSpecs (a VGI or native DuckDB function such as read_parquet, a companion catalog table, or a format + locations), which the client stitches together with UNION ALL. Protocol 3.2 adds filtered planning (spec):

• ScanPlanning = ScanPlanningModes.Filtered makes the client resolve the branches once per scan, after it has collected the query's predicates on the table. A filtered table cannot inline its scan (ScanFunction with InlineScanFunction = true is refused).
• PlanScanBranches is called for every catalog_table_scan_branches_get with those predicates (request.Filters, null when none were collected). Return a ScanBranchesPlan with fewer branches or files, or null to answer from Branches/ScanFunction unchanged. Every predicate is advisory, so return any superset of what could match; zero branches means "no rows". ScanPlanningFilters.MayMatch(stats) tells you whether a candidate described by min/max statistics can match.
• ScanBranchSpec.RowCount and ScanBranchSpec.ColumnStatistics (BranchColumnStatistics per column) let the client drop a branch whose statistics contradict the query. They are conservative claims about every row of the branch, and the defaults are the safe claim: bounds unknown, HasNull = true, HasNotNull = true. Narrow a claim (e.g. HasNull = false) only when it holds for every row, or leave the column out; a wrong claim silently drops rows. HasNotNull = false (entirely NULL) cannot be combined with Min/Max.

new CatalogTable
{
    Name = "events",
    Columns = columns,                       // event_date DATE, n BIGINT
    ScanPlanning = ScanPlanningModes.Filtered,
    RequiredExtensions = ["parquet"],
    PlanScanBranches = request => new ScanBranchesPlan
    {
        Branches = [.. manifest
            .Where(file => request.Filters?.MayMatch(file.Stats) ?? true)
            .Select(file => new ScanBranchSpec
            {
                FunctionName = "read_parquet",
                PositionalArguments = [file.Path],
                RowCount = file.Rows,
                ColumnStatistics = file.Stats,  // e.g. ["event_date"] = new() { Min = day, Max = day, HasNull = false }
            })],
    },
};

Writable tables

A CatalogTable becomes writable by naming an ordinary ITableInOutFunction per operation (InsertFunction, UpdateFunction, DeleteFunction) and its maximum result mode per operation in WriteResultModes (WriteOperation.* → WriteResultMode.Count / Rows / Changes). The helpers in QueryFarm.Vgi.Catalog.WriteResults build what a write function needs:

var columns = new Schema([idField, nameField, WriteResults.RowIdField()], null);  // rowid BIGINT, is_row_id
var user = WriteResults.UserColumns(columns);                                      // without the row id

// in the write function:
public Schema ResolveOutputSchema(TableInOutBindParams bind) =>
    WriteResults.Schema(WriteRequest.From(bind.Arguments).ResultMode, user);
// per input batch (UPDATE: the SET columns + the row id last):
output.Emit(WriteResults.Batch(mode, user, oldRows, newRows));

UPDATE and DELETE need a row-id column (CatalogTable.RowIdColumn), which DuckDB reads as a virtual column, so the table's ScanFunction must declare ProjectionPushdown => true and emit TableInitParams.ProjectedSchema; registration refuses a table that names a row id over a scan without it. A writable table that also carries Constraints makes its attach negotiate vgi.constraint.enforce.v1 automatically — its write functions then promise to keep the ENFORCED constraints (e.g. refuse to change a key). A write function whose processor awaits I/O can implement IAsyncTableInOutProcessor (and a scan, IAsyncTableFunctionProducer).

Transactions

DuckDB opens a worker transaction for every SQL transaction (autocommit statements included) on an attach whose result advertises supports_transactions. Choose that per attach in the OnAttach handler and handle the lifecycle with TransactionHooks:

sealed class Hooks : TransactionHooks
{
    public override Task<byte[]?> BeginAsync(TransactionContext c) => Task.FromResult<byte[]?>(null); // null: SDK mints 16 random bytes
    public override Task CommitAsync(TransactionContext c) => ApplyStaged(c.Attach, c.TransactionOpaqueData); // throw to fail COMMIT
    public override Task RollbackAsync(TransactionContext c) => DiscardStaged(c.TransactionOpaqueData);
}

worker
    .OnAttach(request => new AttachContext
    {
        SupportsTransactions = AttachOptions.Read(request).GetBoolean("transactional") ?? false,
    })
    .TransactionHooks(new Hooks());

Functions see the same plaintext id as TransactionOpaqueData on TableBindParams, TableInitParams, TableInOutBindParams and TableInOutInitParams (on HTTP the SDK seals it on the wire and opens it before either runs), so a write function can stage changes under it and the commit hook can apply them. Calls of one transaction may reach different worker processes; keep staged state where all of them can see it. fixtures/QueryFarm.Vgi.WritableTxWorker is a complete example (vgi docs/writable_tx_fixtures.md).

Transports

await worker.RunStdioAsync();                          // default — DuckDB's plain LOCATION
await worker.RunUnixSocketAsync("/tmp/my-worker.sock"); // AF_UNIX, for the launch: pool
await worker.RunFromArgsAsync(args);                    // parses --unix/--idle-timeout/etc. from argv

LOCATION also accepts http://…/https://… for an HTTP worker, or a launch:<argv> prefix for the pooled AF_UNIX launcher transport (a worker process reused across every DuckDB connection that shares the same (argv, cwd, VGI_RPC_*-env) identity, rather than cold-spawned per ATTACH).

Critical rule: stdout is the wire channel for stdio-transport workers. Every diagnostic/log line must go to Console.Error, never plain Console.WriteLine — a stray stdout write corrupts the Arrow IPC stream.

Hosting additional protocols

A worker can host further application protocols beside vgi.v3 — a reporting protocol, a shared fixture — through one hook, called exactly once when the server is built:

new Worker()
    .RegisterScalar(...)
    .HostedProtocols(() => [HostedProtocol.For<IReports>(new Reports(config))])
    .RunFromArgsAsync(args);

The protocols are hosted, in the order returned, on every transport (stdio, unix, the Iroh raw upstream, HTTP), and listed by vgi_rpc.Reflection.v1 after vgi.v3. They cannot change vgi.v3: every request is routed on its vgi_rpc.protocol key with no fallback, so DuckDB dispatches exactly as before. Each interface needs its own [ProtocolName] (not vgi.v3, not under the reserved vgi_rpc. prefix); a bad entry fails startup with an error naming the hook.

Token introspection (vgi_rpc.Identity.v2)

A worker may opt into hosting vgi_rpc.Identity.v2 over HTTP, for a reverse proxy that must resolve an opaque bearer credential to a principal:

worker
    .Identity(
        resolveToken: token => apiKeys.Lookup(token) is { } row
            ? new TokenIdentity(row.Principal, row.Label)
            : null,                                  // "the store answered: unknown"
        introspectPrincipals: ["proxy@example.com"]) // or VGI_INTROSPECT_PRINCIPALS
    .HttpAuthenticate(myAuthenticate);               // who the caller is

Only the methods whose hooks you supply are hosted (resolveToken → introspect_token, mintGrant → issue_grant); with neither, the protocol is absent. A worker that supplies resolveToken without an introspector allowlist refuses to start: authenticating and introspecting are different capabilities, and "any authenticated caller" would let any user resolve any other user's credential to its owner.

For a transient failure — the store is down, a timeout, a 5xx — throw QueryFarm.VgiRpc.Errors.AuthUnavailableException("store unreachable", retryAfterSeconds: 10), the same error an HTTP authenticate delegate throws to get a 503 with Retry-After. The framework reports it as identity_unavailable with your retry hint, which callers know not to negative-cache. Never throw an ArgumentException (or return null) for an outage: that reads as "this credential is unknown".

Grants as bearer credentials

issue_grant mints a credential for automation to present later as an ordinary bearer, and the HTTP worker accepts it back:

worker.SealedGrants(GrantKeys.Parse([Environment.GetEnvironmentVariable("MY_GRANT_KEY")!]));
// or: --grant-key KEY (repeatable, first mints), or VGI_RPC_GRANT_KEYS=key1,key2

With grant keys configured (SealedGrants(...), --grant-key, or VGI_RPC_GRANT_KEYS with the optional VGI_RPC_GRANT_AUDIENCE / VGI_RPC_GRANT_MAX_TTL_SECONDS, default 7 days), the HTTP worker hosts vgi_rpc.Identity.v2 even without hooks, mints sealed vgig1. grants through issue_grant (unless Identity(mintGrant: ...) supplies a minter), and accepts them as Authorization: Bearer credentials on every call, as the grant's owner. Keys are standard base64 of exactly 32 bytes; a malformed one stops the worker at startup. No keys, no change. A caller must have authenticated recently (auth_time) to mint; a grant carries none, so grants never mint grants. Sealed grants are not individually revocable: keep the lifetime short, and rotate by adding the new key first, then removing the old one after its grants expire.

A worker that supplies resolveToken also has it consulted for bearers the earlier authenticators did not accept. Order: your HttpAuthenticate delegate (throw AuthFailure for a credential that is not yours), then sealed grants, then resolveToken. A bad vgig1. token is a 401 that never reaches the resolver; a resolver outage (AuthUnavailableException) is a 503. Behind the Iroh bridge the bearers are checked inside the peer-identity policy, never beside it.

Attach tickets (vgi.attach_tickets.v1)

An attach ticket lets a runner reattach a user's catalog later, as that user, without ever seeing the user's attach options. While the user is attached and logged in, the client calls seal_attach. The worker seals the catalog name, the options (secret ones included) and the version specs into a vgia1. ticket that only this worker can open. Later a runner presents Authorization: Bearer <grant> plus a single option:

ATTACH 'ticket_probe' (TYPE vgi, LOCATION 'https://…', vgi_attach_ticket 'vgia1.…');

catalog_attach redeems the ticket before any catalog code runs. Any other option beside it is invalid_request. The ticket opens only under the caller's principal (attach_ticket_invalid otherwise) and only within its lifetime (attach_ticket_expired). The sealed request then replaces the incoming one, so your OnAttach handler sees exactly what the user attached with, and the name the runner typed is ignored. A ticket carries no authority of its own: without a grant or login for the same principal, it attaches nothing.

worker
    .SigningKey(Encoding.UTF8.GetBytes(Environment.GetEnvironmentVariable("MY_SIGNING_KEY")!)) // or VGI_SIGNING_KEY
    .SealedGrants(grantKeys);                                                             // or VGI_RPC_GRANT_KEYS

The protocol is hosted on HTTP only, and only when the signing key is configured explicitly (SigningKey(...) or a non-empty VGI_SIGNING_KEY) and the worker can issue grants (grant keys, or Identity(mintGrant: ...)). Otherwise it is absent, so a client learns from reflection that it is unavailable. Tickets expire at the worker's grant maximum (ttl_seconds = 0 asks for the maximum); with no maximum they don't expire. seal_attach returns an AttachTicket whose ExpiresAt is a DateTimeOffset? (timestamp[us, tz="UTC"] on the wire, protocol version 2.0.0), null when the ticket never expires. Rotating the signing key invalidates every ticket. vgi_attach_ticket is a reserved attach-option name, so declaring it in any letter case fails at definition or at RegisterCatalog. Neither the ticket nor a restored option is ever logged. Spec: vgi-python docs/protocol/vgi-attach-tickets.md. The format lives in AttachTickets, checked byte for byte against the cross-SDK vectors in test/QueryFarm.Vgi.Tests/Vectors/.

The example worker serves the cross-SDK ticket_probe catalog. It has options region (default 'us-east-1') and api_key (required, secret), and table main.probe returns region and the first 12 hex characters of sha256(api_key). Over HTTP the worker accepts the test bearers vgi-test-alice and vgi-test-bob as fresh logins, so a client can mint grants. It hosts tickets when VGI_SIGNING_KEY and VGI_RPC_GRANT_KEYS are both set.

Protocol overview

VGI uses vgi_rpc, an Apache Arrow IPC-based RPC framework, for all client-worker communication — you don't write to this directly (Worker/ScalarFn/the function-kind interfaces handle it), but here's what happens per query:

DuckDB (client)                      VGI worker
  │──── bind(request) ─────────────▶ │  function name, args, input schema
  │◀─── BindResponse ───────────────  │  output schema (your Bind/ResolveOutputSchema)
  │──── init(request) ─────────────▶ │  start the processing stream
  │◀─── stream header ──────────────  │  execution_id, max_workers
  │──── exchange/tick(batch) ──────▶ │
  │◀─── output batch ───────────────  │  your Compute/Produce
  │──── [stream close] ────────────▶ │

See docs/roadmap.md and inline doc comments in Internal/VgiServiceImpl.cs for the full RPC surface (catalog DDL, transactions, splits, secrets, etc.) beyond this per-query happy path.

Repo layout

src/QueryFarm.Vgi/                    the published package
  Attributes/                         [Param]/[ConstParam]/[Setting]/[OutputLength]/[Compute]
  Arguments/                          TableArguments, ConstantArguments, SecretValues, FilterSet
  Scalar/ Table/ TableInOut/          per-function-kind interfaces + ScalarFn
  Buffering/ Aggregate/
  Catalog/                            CatalogTable/CatalogView/CatalogMacro
  Protocol/                           IVgiService + wire DTOs (Generated/, from vgi-python)
  Internal/                           implementation detail (not public API): VgiServiceImpl,
                                       codecs, evaluators, storage
fixtures/QueryFarm.Vgi.ExampleWorker/ the ~170-function conformance-driving fixture worker
fixtures/QueryFarm.Vgi.SimpleWritableWorker/  writable-catalog write-path fixture
fixtures/QueryFarm.Vgi.WritableTxWorker/      `txfix`: writable table + transaction hooks (public API only)
fixtures/QueryFarm.Vgi.BadProtocolWorker/     malformed-protocol negative-test fixture
examples/01-minimal-scalar-worker/    "Your first worker" above, as a buildable project
test/QueryFarm.Vgi.Tests/             xUnit unit tests
scripts/run_tests.sh                  fast local sqllogictest runner (see CLAUDE.md)
ci/                                   GitHub Actions integration-test harness

Read fixtures/QueryFarm.Vgi.ExampleWorker/ for a working example of every function kind and catalog feature — it's the fixture the full sqllogictest suite is graded against.

Testing your own worker

The fastest check is to call your function from a DuckDB session (see "Your first worker" above). For automated tests, drive the worker directly with QueryFarm.VgiRpc's client, or shell out to a DuckDB session from your test harness. test/QueryFarm.Vgi.Tests/ shows the former pattern for this SDK's own unit tests (schema derivation, dispatch, codecs, storage).

Build & test

make build                # dotnet build vgi-csharp.slnx
make test                 # unit tests (test/QueryFarm.Vgi.Tests)
make format_check         # dotnet format --verify-no-changes
make test_integration      # full sqllogictest suite against $(VGI_DIR), default ../vgi (launcher transport)

See CLAUDE.md for the full local-development workflow, including the fast sqllogictest iteration loop and the wire-protocol conventions worth knowing before touching Protocol/.

Architecture notes

• No IDL — RPC method dispatch and versioning ride as vgi_rpc.* custom metadata on Arrow IPC batches, not a schema-defined wire format. The protocol's request/response types are generated from the canonical vgi-python dataclasses (make regen_protocol), and a conformance test checks what this port serializes against the protocol's schemas.
• Two-tier dataclass rule: a method's own top-level parameter/return type embeds as IPC inside a binary field; a property nested inside another dataclass is a native Arrow struct.
• Positional vs. name-based decoding: request types (C++ → worker) decode positionally — property declaration order must exactly match the C++ generated schema's field order. Response types (worker → C++) are validated with a strict arrow::Schema::Equals against the C++ extension's generated schema factories.
• Cross-process storage: table-buffering and per-transaction state must survive landing on a different worker process than the call that wrote it (the worker-pool/launcher owns process lifetime, not the caller) — see IFunctionStorage's doc comment for the durable, execution-id/transaction-id-scoped storage contract this requires.

See inline doc comments throughout src/QueryFarm.Vgi/ and fixtures/QueryFarm.Vgi.ExampleWorker/ for the deeper "why" behind specific design choices — most non-obvious decisions are documented at the point of use, cross-referencing the specific sqllogictest file(s) they exist to satisfy.

See Iroh operations for native clients and bridge-ready raw or HTTP workers.

License

Copyright 2025, 2026 Query Farm LLC.

Licensed under the Query Farm Source-Available License, Version 1.0 — see LICENSE for the full terms. In brief, you may use, modify, and redistribute the software freely for non-production use, and for production use except where it would constitute a Competing Offering or a Commercial Marketplace as defined in the license. Each version converts to the Apache License, Version 2.0 on the tenth anniversary of its public release.

For uses not permitted under this license, contact hello@query.farm for a commercial license.

Browser catalog

HTTP workers serve the shared browser catalog at their HTTP root, with vgi-client.js alongside it. The page discovers schemas, tables, views, and functions through the normal VGI RPC endpoints. ?format=json returns worker identity instead. Both assets use the worker's authentication callback and support conditional requests and HEAD.

Applications hosting their own ASP.NET Core server can call app.MapVgiLandingPage(name, serverId, prefix, authenticate: authenticate) alongside app.MapVgiRpc(...), using the same prefix and authentication.