Skip to content

Iterate export pages with bounded continuation cursors - #31

Open
darrikonn wants to merge 1 commit into
prooflinedev:mainfrom
darrikonn:feat/iter-export-pages
Open

darrikonn wants to merge 1 commit into
prooflinedev:mainfrom
darrikonn:feat/iter-export-pages

Conversation

@darrikonn

@darrikonn darrikonn commented Nov 4, 2026 •

Copy link
Copy Markdown

Who this helps

Anyone pulling the reviews, findings, finding_outcomes, finding_validations
or spend datasets into a warehouse or a report. export_reviews reads one page and
the README asked callers to loop on next_cursor by hand with the same dataset,
window, format and grouping. The TypeScript SDK and the CLI leave that loop to the
user as well, and the CLI README warns that partial output must never be treated as
complete. The Python SDK already owns this shape for reviews through iter_reviews,
so exports now get the same treatment.

What changed

  • iter_export_pages on ProoflineClient and AsyncProoflineClient yields
    ExportPage objects until the server omits X-Proofline-Export-Next-Cursor.
  • Bounded by max_pages (1..1000, default 100). Exhausting the bound with a cursor
    left raises urn:proofline:sdk:pagination-limit instead of stopping silently.
  • A repeated or empty continuation cursor raises urn:proofline:sdk:invalid-pagination.
    A settled horizon that changes between pages raises
    urn:proofline:sdk:invalid-response before that page is yielded, so rows from two
    snapshots are never mixed.
  • cursor resumes an interrupted export. An empty cursor is a ValueError on both
    export_reviews and the iterator, before any request.
  • from_time and to_time accept timezone-aware datetime values as well as
    RFC3339 strings on both methods. Naive datetimes raise ValueError before any
    request. ExportTime is exported for annotations.
  • Both facades build the export request through one export_request helper in
    proofline/exports.py, so dataset, format, cursor and window validation live in
    one place.
  • The README keeps the single-page export_reviews example and documents the
    iterator's behaviour with its own example below it, without describing arguments,
    like the rest of the file. scripts/contracts.py allows the new helper name.

Decisions

  • Pages, not rows. ExportPage keeps the page text and settled horizon, which a
    row stream would drop; rows are page.rows away. One method keeps the public
    surface and the contracts allow-list small.
  • max_pages is the only bound. Server pages are capped at 5000 rows or 1000
    groups and the transport caps a response at 8 MiB, so a page count bounds bytes
    and rows; a row bound would have to raise mid-page.
  • No client-side 31-day window or group_by check. The server is the authority
    on both.
  • Existing error urns only. The three SDK urns already cover bounds, cycles and
    malformed responses, so callers need no new vocabulary. Horizon drift carries
    status 0 like the other SDK-detected malformed responses. The shared
    pagination-limit detail now reads "Pagination reached the configured bound." for
    both pagers; no test or document depended on the old wording.
  • Wire format in exports.py, pagination policy in client.py. Time
    serialization and request building sit next to decode_export. The bound, the
    seen cursors and the frozen horizon live in one _ExportPager state object, so
    the sync and async drivers are the same short loop and the policy is written
    once. It shares the review pager's error constructors. New code reads top-down:
    each function is followed by the functions it calls, so the iterator precedes
    the single-page method it drives, the request builder precedes the time
    serializer, and the pager closes the module after the classes that drive it.
  • No version bump. Releases are separate commits in this repository.

Verification

  • python3 scripts/check.py passes end to end: client regeneration check, operation
    and vocabulary contracts, ruff lint and format, mypy under the project
    configuration, pytest (95 tests), wheel and sdist build, and the install smoke
    test for both artifacts.
  • New loopback tests run under both sync and async clients and cover: two CSV
    finding_outcomes pages with the query repeated and the cursor URL-encoded on
    page two, a single page, a repeated cursor, an empty cursor from the server, a
    resume cursor that repeats, horizon drift, the max_pages bound, a 404 on page
    two after the first page was yielded, aware datetimes serialized to Z on
    export_reviews directly, and ValueError with zero requests for a bad bound,
    an empty cursor, a naive from_time or to_time, an unknown dataset and an
    unknown format on both methods.
  • A hand-run mutation pass over the pager and the request builder (dropped guards,
    inverted bounds, a yield moved ahead of the horizon check, a forgotten resume
    cursor, parameters no longer sent) is killed in full by these tests, with
    sync-only and async-only mutants failing only their own mode.

export_reviews read one page and left the continuation loop to the caller.
iter_export_pages follows X-Proofline-Export-Next-Cursor on both clients with
the same dataset, window, format and grouping, bounded by max_pages (1..1000,
default 100). A repeated or empty cursor is refused as invalid pagination, a
settled horizon that changes between pages as an invalid response, and an
exhausted bound raises pagination-limit instead of stopping silently. A cursor
resumes an interrupted export.

Export windows accept timezone-aware datetimes as well as RFC3339 strings;
naive datetimes raise ValueError before any request, as does an empty cursor
on either method. Both facades build the export request through one helper,
so dataset, format, cursor and window validation live in one place, and
ExportTime joins the public exports. The README keeps the single-page example
and documents the iterator below it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant