Skip to content

Add lazy paginated export iterators for sync and async clients - #27

Open
AI1120 wants to merge 1 commit into
prooflinedev:mainfrom
AI1120:feat/export-page-iteration
Open

AI1120 wants to merge 1 commit into
prooflinedev:mainfrom
AI1120:feat/export-page-iteration

Conversation

@AI1120

@AI1120 AI1120 commented Nov 3, 2026

Copy link
Copy Markdown

Engineers exporting analytics currently have to maintain their own continuation loop around export_reviews. Add iter_export_pages to both client facades so they can consume CSV or NDJSON exports one page at a time while preserving the organization, dataset, time window, format, grouping, and optional starting cursor.

The iterator reuses the existing bounded transport and export decoder. It yields ExportPage objects so callers retain raw text, decoded rows, dataset version, and settled horizon. It follows cursors even when an intermediate page is empty, rejects empty or repeated continuation cursors before yielding that page, and raises pagination-limit if max_pages is exhausted while continuation remains. The default limit is 100 pages, configurable from 1 to 1000. Requests happen only when the consumer advances the iterator.

The existing single-page method and API contract remain unchanged. The contract checker recognizes the new facade helper. The README includes a streaming example and error behavior.

Verification:

  • Full pytest suite: 115 passed (26 new parameterized cases); one Windows pytest-cache file-lock warning.
  • Ruff lint and formatting, mypy (224 source files), and the API contract checker pass.
  • Wheel and source distribution build and install successfully; isolated HTTP smoke checks pass for each.
  • On this Windows archive checkout, the byte-exact generation check differs because of native line endings and the default encoding. With UTF-8 generation and newline normalization, all 210 generated files match. Generated code is unchanged; the Linux CI gate remains to be checked.
    HTTP tests cover sync/async behavior, CSV/NDJSON, lazy fetching, resumed cursors, exact filter preservation, empty intermediate pages, normal completion at the configured bound, invalid page limits, repeated and cyclic cursors, and an API refusal without pagination retries.

Implementation and verification performed with AI coding assistance.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant