LineageWeave

Reconstructs git-branch-style lineage DAGs from scattered short records -- turns a flat pile of loosely-grouped, timestamped items into a browsable set of branching threads, without any explicit "this follows from that" link existing in the source data.

group A-100
  rec-001  Initial site visit and project scope discussion
      └─ rec-002  Pricing renegotiation follow-up
           ├─ rec-003  Pricing renegotiation: revised quote sent
           └─ rec-004  Delivery schedule question raised
                └─ rec-005  Delivery schedule confirmed with logistics
  rec-006  Unrelated: annual account review   (own root -- no forced match)

This is a demo prototype: it ships with synthetic sample data only (lineageweave/fixtures.py) and no connection to any real dataset or organization.

Why

Given a pile of records with no native cross-record link, no single cheap signal reliably tells you which record continues which -- see docs/lineage-bi-research-notes.md for the validation numbers and the literature this design follows. LineageWeave fuses several independent, individually-weak signals (temporal proximity, a shared grouping key, text similarity, and an optional LLM judgment) instead of trusting any one of them alone.

How it fits with the rest of the ecosystem

LineageWeave is a thin orchestration/BI layer. It does not do its own psychometric or statistical estimation -- that stays inside TEPP (Rust), consumed here purely through TEPP's own published wire contract (lineageweave/tepp_client.py, AnalysisRunRequest v1), never by reading TEPP's tables or reimplementing TEPP's model. See ARCHITECTURE.md for why the "computation layer must be Rust + GPU/CPU multithreaded" rule that applies to TEPP does not apply to this repo.

The optional LLM-adjudication channel calls contextual-orchestrator (lineageweave/adjudication_client.py). Tree assembly reuses ThreadWeave (JWZ message threading) and channel fusion reuses RankWeave (weighted score fusion) -- both real dependencies, not reimplemented here.

Run it

pip install -e .
python -m lineageweave.server
# -> http://127.0.0.1:8420

Or use the library directly:

from lineageweave import reconstruct
from lineageweave.fixtures import sample_records

trees = reconstruct(sample_records())
for tree in trees:
    print(tree.group_key, "branch points:", tree.branch_points())

Bring your own data

Map your records into lineageweave.Record (see lineageweave/models.py for the field docs) and call reconstruct() directly -- nothing in this package assumes any particular source schema.

To turn on the embedding or LLM channels, pass a real client instead of the Null* defaults:

from lineageweave import reconstruct
from lineageweave.adjudication_client import ContextualOrchestratorAdjudicationClient

llm = ContextualOrchestratorAdjudicationClient(base_url="http://localhost:8000", api_key="...")
trees = reconstruct(my_records, llm=llm)

Test

pip install -e ".[dev]"
pytest

Local product stack (Docker Compose)

The reconstruction library above is being wrapped in a real product (see ARCHITECTURE.md and ADR 0001). Phase 1's infrastructure -- PostgreSQL, Valkey, and a real Keycloak OIDC realm seeded with synthetic demo accounts -- runs via Docker Compose:

make up      # docker compose up -d: postgres, valkey, keycloak
make smoke   # real login as the synthetic demo user + JWT signature
             # verification against Keycloak's live JWKS -- proves the
             # OIDC round-trip actually works, not just that containers
             # started
make down

Postgres and Keycloak are built (docker/postgres-init/, docker/keycloak/) rather than bind-mounted, so the keycloak database's init script and the realm seed ship inside the images themselves -- portable to any Docker host or CI runner, no assumption about a shared local filesystem layout.

Demo accounts (docker/keycloak/realm-export.json) are synthetic: demo.analyst / demo.admin, password lineageweave-demo-only, each carrying corp_code / pu_code as token claims -- these are throwaway local-dev credentials in a locally-run realm, never the org's real Keyverse tenant (see ADR 0001 for why).

Host ports (15432, 16379, 18080, 18420) deliberately avoid each service's own default -- a dev machine commonly already runs its own Postgres/Redis/local server on those. Override via .env (copy .env.example) or inline if even those collide, e.g. KEYCLOAK_PORT=28080 make up.

Postgres's POSTGRES_DB (the "app" database) is migrated automatically on first boot -- docker/postgres-init/Dockerfile bakes in the exact same migrations/0001_initial_schema.sql file tests/test_schema.py applies, no re-typed copy.

backend/ is a FastAPI app talking directly to that database (asyncpg, no ORM, no file DB) and to Keycloak's live JWKS for OIDC verification:

make up
make seed   # scripts/seed_demo_data.py: inserts synthetic corp/account/post
            # rows keyed to the *real* Keycloak demo users' subject ids,
            # plus Valkey ticket_created events so Activity is not empty
curl http://localhost:18420/healthz

GET /api/posts, GET /api/posts/{post_id}, GET /api/posts/{post_id}/keymen, GET /api/keymen/{person_id}/related, GET /api/posts/{post_id}/affiliate-tree, GET /api/posts/{post_id}/voc-evidence, and POST /api/posts/{post_id}/extract-keymen require a real bearer token (RBAC: the account's role must grant post_read; ABAC: a private post is only visible to accounts affiliated with its owning corporate entity -- backend/app/main.py). A Keyman who is only mentioned on a post the account cannot see is 403, same deny path. backend/tests/test_api.py proves both the allow and the deny path against a live Keycloak + throwaway Postgres database, including that a private post scoped to a different corporate entity is excluded from the list and 403s on direct fetch.

frontend/ (React + Vite + TypeScript, docker compose's fourth service) is a real client, not mocked or static: react-oidc-context drives an actual Authorization Code redirect through Keycloak, the home page draws the reconstructed lineage as a git-branch SVG (GET /api/lineage; post_admin can rebuild), and the post list / detail popup call the FastAPI backend over real fetch() with the token Keycloak issued.

make up
make seed
cd frontend && cp .env.example .env.local && pnpm install && pnpm run dev
# -> http://localhost:5173, click "Log in", redirects through the real
#    Keycloak login page for demo.analyst / lineageweave-demo-only

docker compose up also builds and serves the frontend itself (nginx, frontend/Dockerfile) at http://localhost:15173 -- the VITE_* build args are wired from the same .env ports as every other service. frontend/src/App.test.tsx covers the login-redirect and fetch-then-render-popup paths (react-oidc-context's useAuth mocked -- the real OIDC round-trip is what scripts/smoke_test_oidc.py and backend/tests/test_api.py already prove against a live Keycloak).

Modular / standalone

This repo runs standalone (own server, own tests, own CI) and is equally usable as a library module (import lineageweave) inside a larger service -- no global state, no required environment variables, every external dependency (embeddings, LLM adjudication, TEPP) is injected, not hardcoded.

License

MIT -- see LICENSE.