Design incremental Overture ingestion and measure planet-scale operation #301

What needs to be done, and why?

OpenMapX can ingest regional Overture snapshots, but refreshing a large region still requires snapshot-scale work and the storage layout assumes a bounded Geofabrik-style region. A planet-scale deployment needs measured partitioning and release-delta semantics without ever exposing a mixed release.

Scope and open questions

Reported starting point

Rechecked on 30 October 2026 against main at fe796a6405f215fda3b18032e8181ed8647ac108.

• The data manager discovers STAC releases, pulls regional partitions with DuckDB, validates them, and atomically promotes staging tables.
• Country-scale OSM/GERS conflation is resumable and has capacity, retention, and admin controls.
• Scheduled refresh currently treats releases as immutable snapshots. No ingestion path reads the Overture changelog, and the Overture schema records each row's release but not a predecessor release.
• docs/docs/features/overture-places.md no longer describes the regional database as country-scale only. It now describes snapshot refreshes and links to the planet runbook in docs/docs/developer/ambient-places-publication.md, which covers snapshot-based planet preparation, resource sizing and deployment scenarios (feat(places): add global OSM/Overture ambient map coverage #436). That runbook states that container defaults are not a planet capacity profile.

Design and prove a release-aware ingestion mode using Overture changelog data (added, removed, data_changed, unchanged) plus a partitioned planet-scale storage contract. Keep full snapshot ingestion as the recovery/fallback path.

Alternatives considered

Always rebuilding snapshots is simpler and remains the fallback, but scales poorly with global data. In-place updates without release staging risk partial/mixed data.

Area

Overture ingestion, data manager, PostgreSQL/PostGIS, operations.

References

• docs/docs/features/overture-places.md
• services/data-manager/src/jobs/overture/
• https://docs.overturemaps.org/gers/changelog/
• https://docs.overturemaps.org/getting-data/cloud-sources/

What would show that this is complete?

• An ADR compares regional snapshot, regional delta, and planet-scale layouts across object storage, DuckDB, and Postgres/PostGIS.
• Benchmarks record release churn, download/compute time, peak disk/memory, index cost, query latency, and recovery time for representative country and planet slices.
• A release contract records source release, schema version, partitions, checksums, and predecessor.
• Delta ingestion handles added, changed, and removed records idempotently and can resume from durable checkpoints.
• No query can observe tables or indexes from mixed Overture releases.
• Missing predecessors, schema changes, corrupt partitions, or excessive churn trigger a safe full-snapshot fallback.
• OSM/GERS links, search documents, and embeddings are invalidated only for affected records where correctness permits.
• Reconciliation samples or counts prove the delta result matches the equivalent full snapshot.
• Capacity preflight, retention, rollback, and failed-staging cleanup work for the new layout.
• Operator UI/metrics expose current release, predecessor, lag, mode, changed counts, and fallback reason.
• Planet mode has an explicit supported hardware/storage profile or a documented no-go result.

Dependencies

Snapshot-based planet preparation and global publication were subsequently delivered by #436, which closed #399. This task still covers release-aware incremental ingestion and its benchmarks; the original storage/layout questions under "Scope and open questions" above should be evaluated against that shipped snapshot path before scheduling new work.