Skip to content

feat(export): run the public data export unattended and publish to object storage #852

Description

@bencap

Problem

The export assumes an operator: an SSH tunnel to production, DB_* environment variables exported by hand, the script run from a workstation, a ZIP written to the working directory, and verification by reading unzip | grep output. There is no machine-checkable success condition, so a scheduled run would have no basis for deciding whether its output is publishable.

Proposed behavior

The export runs on a schedule on the existing worker, against production, with no operator present, and publishes to the dump bucket.

  • Destination bucket and key prefix come from configuration. Local filesystem output stays available for development.
  • The archive is written under a dated key.
  • The run emits a result record: asOf, experiment set count, score set count, archive byte size, archive SHA-256, and object key.
  • The run maintains a published-release registry — the set of archives eligible to be served, and which one is current.
  • The generated archive is verified after generation, see feat(export): verify a generated archive before it enters the registry #853

Acceptance criteria

  • The export completes with no interactive input and no tunnel from a workstation, given only configuration and credentials.
  • The archive lands in the configured bucket under a dated key, byte-identical to what the manual procedure produces for the same database state.
  • The result record is emitted and includes the archive SHA-256 and object key.
  • A run that fails partway leaves nothing in the registry and no object reachable as current.
  • Failures reach the alerting path in Slack.
  • The registry names exactly one current archive at any time.
  • The registry is not uploaded if feat(export): verify a generated archive before it enters the registry #853 does not pass.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backend

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions