Symgliph

Bound the context. Verify the source.

Project site · SCGP-0.1 specification · Public study

Important

Symgliph is an experimental research protocol and Rust reference implementation. SCGP-0.1 is a draft. Protocol conformance verifies artifact construction and source integrity; it does not verify retrieval relevance or model-answer correctness.

Symgliph is a source-verifiable context compiler. It gives an included file corpus a deterministic identity, derives an analyzer-versioned semantic index, selects candidate evidence for a task, and emits ordinary text beneath an explicit model-token budget. Before any selected excerpt reaches a model, Symgliph verifies the current source file against the recorded corpus.

question -> candidate retrieval -> semantic expansion -> source verification -> bounded prompt

The model does not read a glyph, hash, or private symbolic language. Symgliph can render corpus identity as a deterministic SVG fingerprint, but that visual artifact is an optional representation of the indexed corpus—not the mechanism that reduces tokens.

What is guaranteed

For a declared corpus, analyzer profile, query, and budget, the reference implementation checks:

• deterministic corpus and semantic roots;
• exact file and excerpt digests for selected evidence;
• portable source paths and half-open byte ranges;
• a hard o200k_base prompt-token ceiling; and
• fail-closed behavior when indexed source has changed.

These guarantees cover identity, provenance, and packet construction. Candidate selection can still omit relevant evidence, and a model can still produce an incorrect answer.

Public result

The first real-corpus study compares four context strategies on 30 pinned Multi-SWE-bench Rust maintenance tasks using one fixed file-localization model and one response per task and strategy.

Strategy	Mean prompt-token savings	Exact fix-path recall	Recorded API spend
All-files control	0.00%	53.32%	$4.5755
Lexical BM25	95.82%	46.87%	$0.0974
Embedding cosine	95.78%	43.34%	$0.3237
Symgliph compiled	97.58%	41.76%	$0.0668

Symgliph substantially reduced prompt tokens and provider-reported API spend in this run, but it did not preserve quality parity: exact fix-path recall was 11.56 percentage points below the all-files control and 5.11 points below lexical BM25. The study supports the efficiency hypothesis and identifies retrieval quality as the next research problem.

Adaptive context experiment

The experimental SACP-0.2 profile addresses that gap by walking an ascending token and semantic-node ladder. It stops at the first packet that passes a declared evidence-sufficiency heuristic, when candidates are exhausted, or when hard limits are reached. Every attempted context digest and stopping condition is retained for evaluation.

query -> retrieve -> pack -> estimate sufficiency
                           ├── pass: emit packet
                           └── fail: expand budget and evidence

The built-in estimator measures query-term coverage, evidence count, and optional excerpt completeness. It is deliberately named estimated_sufficient: passing it does not establish retrieval recall or answer quality. A follow-up Multi-SWE-bench run tested the policy against full context, BM25, and the fixed compiled arm.

The provisional 50% query-term threshold was selected by an exploratory replay of the existing 30-task traces. It preserved the fixed compiled arm's evidence recall while reducing selected context, but it is not independent validation. See the calibration record.

In that follow-up, adaptive context retained the fixed arm's 51.85% evidence recall while reducing mean prompt tokens from 5,138 to 4,345 and improving exact fix-path recall from 41.76% to 45.09%. It still trailed BM25 and full context. Read the adaptive study and claim boundary.

Read the experimental Adaptive Context Compilation 0.2 profile.

Read the methodology and claim boundary, inspect the raw report, or recompute it without API access:

scripts/eceb-study-check.sh

Quick start

# Compile corpus identity and semantic records.
cargo run -- build ./my-corpus

# Emit source-verified context under a 4,000-token ceiling.
cargo run -- pack ./my-corpus \
  --query "where is authorization checked?" \
  --max-tokens 4000

# Start small and expand until the declared evidence estimator passes.
cargo run -- adaptive-pack ./my-corpus \
  --query "where is authorization checked?" \
  --min-tokens 1000 \
  --max-tokens 8000 \
  --require-sufficient

# Recompute the corpus and semantic roots against current source.
cargo run -- verify ./my-corpus

build writes:

my-corpus/.symglyph/
├── manifest.json       # inspectable corpus and semantic records
├── manifest.symglyph   # checksummed binary artifact
└── glyph.svg           # optional visual fingerprint

Hidden paths are excluded by default. Use --include-hidden only with an output directory outside the corpus.

Library API

use symgliph::{ContextEngine, ContextRequest, CorpusBuilder};

let manifest = CorpusBuilder::new("./my-corpus").build()?;
let engine = ContextEngine::new("./my-corpus", manifest);
let packet = engine.pack(&ContextRequest::new(
    "where is authorization checked?",
))?;

assert!(packet.prompt_tokens <= packet.max_tokens);
// Send packet.prompt to the model; retain packet.evidence for provenance.
# Ok::<(), symgliph::Error>(())

Every evidence record identifies its semantic node, source path, byte range, full-file digest, and excerpt digest. If a selected file changed after compilation, packing fails rather than emitting stale evidence.

ContextEngine::pack_adaptive accepts an AdaptiveContextRequest and returns the selected packet plus the complete expansion trace. A failed estimate still returns a valid verified packet unless the caller elects fail-closed behavior.

For local model and agent bridges, symgliph serve exposes context.describe, context.pack, context.adaptive, and context.proof over newline-delimited JSON-RPC:

{"jsonrpc":"2.0","id":1,"method":"context.pack","params":{"query":"authorization checks","max_tokens":2000,"max_nodes":8}}

See docs/context.md for the packet contract, threat model, measured experiments, and limitations.

Research protocol

The draft Symgliph Context Glyph Protocol 0.1 defines four conformance capabilities over one deterministic semantic core:

Class	Capability	What it establishes
I	Identity	File records, corpus root, and verification
S	Semantics	Versioned analyzers, nodes, references, and semantic root
G	Glyph	Deterministic SVG fingerprint of corpus and semantic identity
C	Context	Source-verifiable text packets under a token budget

Glyph and Context are sibling profiles. Context production does not depend on SVG rendering. Conformance proves deterministic protocol behavior, not retrieval or answer quality.

Run the independent gates:

spec/validate.sh                 # SCGP artifact and packet conformance
scripts/eceb-conformance.sh      # ECEB policy, provenance, and budget gates
scripts/eceb-study-check.sh      # replay the public comparative study
scripts/eceb-adaptive-study-check.sh # replay the adaptive follow-up

Artifact model

For a corpus X, the implementation produces:

A(X) = (manifest, corpus root, semantic graph, semantic root, source index)
                                          ├── context packet
                                          └── optional SVG fingerprint

The corpus root commits to the inclusion policy, sorted paths, sizes, and exact file digests. The semantic root separately commits to analyzer versions, definitions, references, resolved edges, and source ranges. This separation allows analyzer behavior to evolve without pretending the underlying source bytes changed.

The SVG embeds both roots and structural metrics. It is useful for recognizable identity and inspection, but exact expansion still requires the manifest and original source. GlyphRenderer is the optional Class G implementation.

Current analysis coverage

The built-in Tree-sitter Rust analyzer extracts functions, types, traits, modules, implementations, macros, imports, containment, and unambiguous local references. The Markdown analyzer creates document and heading-delimited section nodes. Other file formats can participate in corpus identity without automatically receiving equivalent semantic structure.

When a corpus contains Cargo.toml, Symgliph uses cargo metadata --no-deps to namespace workspace symbols and local dependencies. It does not execute project code or build scripts. Additional analyzers can implement the Analyzer trait.

Research map

• Source-verifiable context experiments: local proof, multi-model A/B runs, Fabric instruction recovery, and blind semantic discovery.
• ECEB-0.1: a quality-, policy-, provenance-, and budget-aware measurement protocol.
• SACP-0.2: deterministic expansion, explicit stopping semantics, and auditable evidence-sufficiency estimates.
• Adaptive calibration: threshold sweep, negative findings, and the preregistered next-run setting.
• Adaptive ECEB study: the prospective model run, complete traces, measured improvement, and remaining quality gap.
• Multi-SWE-bench Rust study: the current four-arm public result and raw observations.
• Semantic-routing golden set: linked corpora, queries, qrels, evidence, hard negatives, and review status.
• Million-line scale gate: deterministic construction, round-trip, query, memory, and size measurements.

Some experiments use OpenRouter and spend API credit. Credential parsers accept only the documented environment keys and do not write secrets into reports.

Enterprise direction

Symgliph is not yet an enterprise context gateway. The protocol is designed so that future systems can add access policy, budget controls, provenance, repeatable evaluation, and auditable provider spend at the boundary between organizational knowledge and AI models. Establishing that product claim will require broader private-corpus studies, repeated model trials, quality noninferiority, and operational policy testing.

Design principles

1. Precise claims: integrity guarantees are not relevance guarantees.
2. Deterministic artifacts: unchanged included bytes and policy produce one corpus root.
3. Fail-closed provenance: selected source must match its recorded digest.
4. Explicit budgets: packet construction never silently exceeds its declared token ceiling.
5. Inspectable evidence: schemas, raw runs, and negative gates remain public.
6. Optional visualization: SVG identifies an artifact; it does not replace source or model-readable context.

License

Licensed under either Apache License 2.0 or the MIT license, at your option.

Community and security

Contributions are welcome through CONTRIBUTING.md. Please use private GitHub Security Advisories for vulnerabilities as described in SECURITY.md. Participation is governed by the CODE_OF_CONDUCT.md.