Project Ledger

Project Ledger is a reusable reference implementation for building a production-oriented SEC/XBRL data platform. It was extracted from a broader engineering implementation and generalized so another engineer can run, study, and extend the architecture without inheriting private infrastructure, personal identifiers, credentials, or project-specific history.

The repository demonstrates how to move from deterministic local development toward managed cloud execution while preserving stable ingestion, storage, modeling, testing, and delivery boundaries.

What this project demonstrates

• fixture-first API integration and deterministic CI
• governed issuer selection and CIK validation
• raw-source preservation before transformation
• normalization of semi-structured XBRL observations
• repeatable and idempotent pipeline behavior
• dbt staging, analytical grain definition, and deduplication
• strict typing, linting, unit tests, data tests, and smoke tests
• Airflow orchestration boundaries
• Docker packaging
• parameterized GCP infrastructure as code
• separation of code, configuration, identity, and secrets
• extraction of a project-specific implementation into a reusable engineering template

Architecture

SEC Company Facts API or fixtures
            |
            v
Python ingestion and validation
            |
       +----+--------------------+
       |                         |
       v                         v
raw JSON per issuer       normalized JSONL facts
       |                         |
       +------------+------------+
                    v
             dbt + DuckDB
                    |
                    v
             analytical marts

Optional managed path:

GitHub Actions -> container image -> Cloud Run Job -> GCS/BigQuery
                                      ^
                                      |
                               Airflow/Composer

The local implementation is intentionally runnable with DuckDB and committed fixtures. The cloud layer is parameterized so storage, execution, IAM, and orchestration can be replaced without rewriting the core ingestion contract.

See Architecture for component responsibilities and extension points.

Engineering scope

The reusable template includes:

• Python SEC Company Facts ingestion
• live and fixture-backed clients
• raw JSON and normalized JSONL outputs
• atomic local writes
• dbt/DuckDB staging and mart models
• Python and dbt tests
• GitHub Actions validation
• Docker packaging
• an Airflow DAG example
• Terraform for a GCP deployment skeleton

The broader implementation from which this repository was extracted also explored managed GCP execution, BigQuery serving, Composer or Airflow orchestration, workload identity, monitoring, market-price enrichment, machine-learning extensions, language-model-assisted analysis, and earnings-event processing. Those later extensions are documented as integration directions rather than represented here as completed template features.

Key design decisions

1. Determinism before live integration. The complete transformation path runs from committed fixtures.
2. Raw preservation before normalization. Original payloads remain available for replay and debugging.
3. Configuration over identity. Issuers, paths, cloud resources, and runtime contacts are externalized.
4. Stable local-to-cloud contracts. Core ingestion and modeling boundaries remain consistent as infrastructure changes.
5. Validation across layers. CI checks application code, pipeline behavior, analytical models, and infrastructure syntax.

See Engineering Decisions for the rationale and trade-offs behind these choices.

Quick start

1. Install prerequisites

Install Python 3.12 and uv.

2. Create local configuration

cp .env.example .env
cp config/ledger.example.yml config/ledger.yml

Set a descriptive SEC user agent in .env:

LEDGER_SEC_USER_AGENT=YourCompany Data Engineering your-team@example.com

3. Install and validate

uv sync --group dev
uv run pytest
bash scripts/validate.sh

4. Run from deterministic fixtures

uv run python -m ledger.cli \
  --config config/ledger.yml \
  --fixture-dir tests/fixtures

5. Run against the live SEC API

Use live mode only after configuring a valid, monitored user agent:

uv run python -m ledger.cli --config config/ledger.yml --live

6. Build the local warehouse

uv run dbt build --project-dir dbt --profiles-dir dbt

Validation

The delivery gate verifies:

• Ruff formatting and linting
• strict mypy checks
• Python unit and pipeline tests
• a deterministic fixture ingestion smoke test
• dbt model builds and data tests
• Terraform formatting and validation

A passing CI run establishes that the reference implementation is internally consistent. It does not by itself prove production scale, cloud deployment, or long-running operational reliability.

Documentation

• Case Study: origin, engineering objective, implemented scope, failure modes, evidence, limitations, and next steps
• Architecture: design goals, components, and production extension points
• Engineering Decisions: major choices, rationale, and trade-offs
• Operations Guide: operating modes, monitoring, failure handling, replay, security, and deployment checks
• Customization Guide: adapting the template to another organization or source system

Template boundaries

This repository deliberately excludes:

• personal sprint journals and career-development notes
• generated dashboards, runtime logs, and private operational evidence
• real cloud project IDs, bucket names, service accounts, personal emails, and secrets
• issuer selections tied to a specific person or organization
• claims that the template itself is a currently hosted production service

These exclusions make the repository safe to reuse and keep its claims precise. The project should be evaluated as a production-oriented reference architecture and portfolio case study, not as proof of an active public deployment.

Limitations and next steps

The current template does not include a complete managed warehouse loader, production alert routing, schema-migration automation, distributed backfill controls, or production cost benchmarks.

The highest-value extensions are:

1. implement GCS and BigQuery storage adapters
2. add explicit schema contracts and migration checks
3. add freshness, volume, error-rate, and cost observability
4. add ephemeral cloud integration tests
5. add bounded replay and backfill controls
6. publish synthetic-data dashboards over the governed mart

See Customizing the template before deploying it in a real environment.