Skip to content

Add sync and async export page iteration - #35

Open
ram4-dev wants to merge 1 commit into
prooflinedev:mainfrom
ram4-dev:export-page-iteration
Open

ram4-dev wants to merge 1 commit into
prooflinedev:mainfrom
ram4-dev:export-page-iteration

Conversation

@ram4-dev

@ram4-dev ram4-dev commented Nov 4, 2026

Copy link
Copy Markdown

What

Adds iter_export_pages to both ProoflineClient and AsyncProoflineClient. It reads one export window end to end by following the signed X-Proofline-Export-Next-Cursor continuations and yields every ExportPage:

for export in client.iter_export_pages(org_id, "findings", from_time=start, to_time=end):
    ...

Who it helps

Today every export user hand-rolls the loop the README describes: "Repeat with the returned cursor and the same dataset, window, format and grouping." That loop is easy to get wrong — a continuation sent with a different group_by, a window that drifts between pages, or a dropped cursor. The helper pins dataset, from/to, format and group_by across all requests and advances only cursor, so pulling reviews, findings, spend, finding_outcomes or finding_validations becomes a single for / async for over CSV or NDJSON pages. It targets the same audience as the existing iter_reviews: scripts and pipelines that want the whole window instead of one page.

Decisions

  • Yields ExportPage, not bare rows. Each page keeps its raw text, decoded rows, settled_until horizon and next_cursor, so callers can stream per-page CSV (the contract repeats the header on every page), append NDJSON, or flatten rows themselves while still keeping the checkpoint horizon.
  • Bounded like iter_reviews. max_pages defaults to 100 (1..1000; a page holds at most 5000 rows per the contract), raises ValueError outside that range, and raises ProoflineError (urn:proofline:sdk:pagination-limit, detail Export pagination reached the configured bound.) when a continuation is still offered at the bound. _pagination_limit gained a subject argument so the export message does not read as a review failure; existing review callers keep the old text.
  • Cycle refusal. A continuation that repeats — including one the caller passed as cursor= — raises urn:proofline:sdk:invalid-pagination with the same detail style as _next_cursor.
  • Absent or empty continuation ends the window, matching "Absent on the last page."
  • No cross-page settled_until enforcement. The contract documents the horizon as frozen for the export but does not promise identical values across separate continuations, so the helper reports each page's horizon instead of failing on a mismatch.
  • scripts/contracts.py now lists iter_export_pages in the helpers whitelist next to iter_reviews and wait_for_review. The check still requires both facades to expose exactly the 21 generated operations plus export_reviews, so the operation surface is unchanged; openapi.public.json and proofline/_generated/ are untouched.
  • Bounds validate lazily at the first next() / __anext__(), the same way iter_reviews validates page_size, max_pages and max_items.

Verification

Run from a clean checkout:

python3 scripts/check.py

That is the entry point of the Checks workflow in .github/workflows/ci.yml. It bootstraps the pinned tools from requirements-dev.txt and runs scripts/generate.py --check, scripts/contracts.py, ruff check ., ruff format --check ., mypy proofline scripts, pytest -q, python -m build (one wheel plus one sdist) and the install smoke test against a local HTTP peer.

Result on this branch: every step passed — All checks passed! (ruff check and format), Success: no issues found on 225 source files (mypy), Python operation and outcome-vocabulary contracts passed., 91 passed (pytest), Installed Python SDK sync/async HTTP smoke passed., and Python SDK generation, HTTP contracts, wheel and source installs passed. (exit 0, two runs: once after the code changes and once after the README addition).

New test: tests/test_http.py::test_export_iteration_keeps_the_window_and_refuses_cycles_or_bounds, parametrized over sync and async, driving a real loopback HTTP peer. It asserts that:

  • the dataset path, from, to, format and group_by stay identical on every request while only cursor advances;
  • three CSV pages are yielded with their rows, raw text, cursors and settled horizon, and the last page has no continuation;
  • a repeated continuation raises urn:proofline:sdk:invalid-pagination after two requests;
  • a caller-supplied cursor that comes back raises the same error after one request;
  • max_pages=2 raises ProoflineError with the export bound detail after two requests;
  • max_pages=0 and max_pages=1001 raise ValueError before any request is sent.

Focused run: python -m pytest -q -k export → 4 passed. Full suite before the change also passed (baseline python3 scripts/check.py on a clean checkout, exit 0).

Files changed: proofline/client.py (feature), tests/test_http.py (tests), scripts/contracts.py (helper whitelist), README.md (usage next to the existing export example).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant