DataFrame builders, conversions, and the cookbook recipe (issue #6, PR 2)

[craigmcchesney]

Phases 2–4 of issue #6, completing the ticket. Plan: plan/tickets/6/plan.md.

Stacked on #44 — based on that branch, so review it first; this PR retargets to main once #44 merges.

Phase 2 — data_frame.py

Builders for common.DataFrame, the shared time-series payload. It is also ingestion's ingestionDataFrame, so #17 extends this module rather than forking it.

• sampling_clock() / timestamp_list() / timestamp_count() relocated here from sample_status_client now that calculations frames are a second caller, and re-exported from it so existing imports keep working.
• Typed scalar column builders; data_column(), the escape hatch whose None entries become unset oneofs — the only way to express a gap on a shared axis; and the provenance helpers.
• data_frame() routes columns by type and enforces the server's shape rules client-side (non-blank names, non-empty values, count match, name uniqueness across all types) so an error names the offending column. Size caps stay server-side: they are deployment policy, and duplicating numbers that can change is how clients drift.
• Array/image/struct/serialized builders are deliberately Add ingestion API client (full surface: ingestData + streaming) #17's; hand-built columns of those kinds pass through today.

Phase 3 — data_frame_conversions.py

Pure Python (no extras): data_frame_timestamps(), column_values() — written standalone so the bucket query (#16) can reuse it — data_frame_columns(), column_metadata_dict().

Behind [analysis]: data_frame_to_pandas(), data_frame_from_pandas(), and the calculations_* bridges.

Two decisions worth a look:

• The pandas direction always emits a TimestampList, never an inferred SamplingClock. A clock is only correct if the spacing is exactly uniform at nanosecond precision, and inferring that from an index that merely looks regular would quietly move timestamps.
• NaN is fail-loud, with a message pointing at the two ways to express a gap. A dense typed column cannot represent one, so the alternatives are inventing a value or dropping a row.

Phase 4 — cookbook and docs

doc/cookbook/datasets-and-annotations.md continues the shared worked example (a dataset over the CXI_3443 shift's first hour, an orbit-drift annotation, a 1 Hz RMS calculation with provenance, the export). Plus the cookbook README entry, the README move of the three Annotation Service bullets from TODO to Current state, and the CLAUDE.md usage section promised in PR 1.

Two things found while building this

• A naming collision I introduced. Exporting the data_frame() function from dp_python_lib.client shadowed the data_frame module, so both import forms returned the function — breaking the from dp_python_lib.client import data_frame as dfb form the plan's own reference snippet uses. The function is no longer re-exported at package level; reach it as dfb.data_frame(...).
• An array column with no dims was treated as an empty product of 1, silently making every element its own sample. Caught by a test expecting a rejection; fixed in both the write and read paths.

Verification

• 699 tests with the [analysis] extra; 601 pass and 56 skip cleanly without it, verified in a real pandas-free venv rather than assumed.
• 25 integration tests + 12 subtests against dp-service fddf692, now including builder-made calculations read back on a sub-second axis — so the round trip exercises nanosecond arithmetic rather than whole seconds a float could also represent — provenance survival, and four export cases.
• All 101 cookbook snippets type-check. The CLAUDE.md snippet was extracted and run against the live server, not just type-checked: save, query, batch fetch, calculations read-back in both plain Python and pandas, export, and teardown all succeed end to end.
• ruff lint and format clean.

Not in this PR

Array/image/struct column builders (#17), bucket-query conversions (#16), and the attributes() / optional-criteria back-port to the older helpers (#40 / #41) — all out of scope per the plan.

🤖 Generated with Claude Code

https://claude.ai/code/session_019he3UCsAnqTDE2VQ73Djwn