DataFrame builders, conversions, and the cookbook recipe (issue #6, PR 2)

.dev/tools/check-cookbook-snippets.py

@@ -104,10 +104,26 @@
     QuerySampleStatusesRequestParams,
     sampling_clock,
     timestamp_list,
+    DataSetClient,
+    DataSetQuery,
+    DataSetQuery as DS,
+    SaveDataSetRequestParams,
+    data_block,
+    AnnotationsClient,
+    AnnotationQuery,
+    AnnotationQuery as AQ,
+    SaveAnnotationRequestParams,
+    calculations,
+    ExportClient,
+    ExportFormat,
+    ExportDataRequestParams,
+    calculations_spec,
 )
 
 from dp_python_lib.client import query_conversions as qc
 from dp_python_lib.client import sample_status_conversions as ssc
+from dp_python_lib.client import data_frame as dfb
+from dp_python_lib.client import data_frame_conversions as dfc
 
 client: MldpClient = MldpClient()
 begin: datetime = datetime(2024, 1, 1, tzinfo=timezone.utc)
@@ -117,6 +133,27 @@
 # building the query.  Snippets that demonstrate query *construction* build their own.
 params: QueryParams = QueryParams(
     begin_time=begin, end_time=end, pv_selector=PV.name_list(["BPMS:GUNB:314:X"]))
+
+# The datasets-and-annotations recipe is one continuous worked example: it saves a dataset, then an
+# annotation on it, then calculations, then exports and deletes them.  These are the handles the
+# recipe establishes in its own earlier snippets and legitimately carries forward, the way `client`
+# and `params` are carried above.  Server-assigned ids are strings.
+t0: datetime = datetime(2026, 2, 2, 18, 0, tzinfo=timezone.utc)
+t1: datetime = datetime(2026, 2, 2, 19, 0, tzinfo=timezone.utc)
+# Declared without an annotation so mypy infers `str` for the standalone snippets that consume these,
+# while the recipe's own snippets can still rebind them from an Optional accessor and narrow with an
+# assert, as conventions.md teaches.  An explicit `str` would conflict with those real assignments; an
+# explicit `str | None` would force every later snippet to re-narrow a handle the recipe already did.
+#
+# Seeding a handle here cannot prove the recipe actually binds that name -- a snippet binding `saved_id`
+# while later ones read `dataset_id` type-checked cleanly and was still broken end to end.  Names carried
+# across snippets are verified by reading the recipe as one continuous script, not by this preamble.
+# `str | None` is what the accessors return; the recipe narrows with an assert before use, and the
+# standalone snippets below do the same, so consumers see a plain `str`.
+dataset_id: str | None = "6aa1bb271a768e97db44d426"
+annotation_id: str | None = "6aa1bb271a768e97db44d427"
+calculations_id: str | None = "6aa1bb271a768e97db44d428"
+assert dataset_id is not None and annotation_id is not None and calculations_id is not None
 # --- end preamble ---
 """
 

README.md

@@ -69,6 +69,18 @@ for their service.
   transparent paging (`query_samples()` / `iter_query_samples()`) and server-streaming
   (`iter_query_samples_stream()`).  Results convert to pandas DataFrames, NumPy arrays, and Excel
   via the optional `[analysis]` extra.
+- **DataSets** — `client.annotation.datasets`.  Name a region of the archive (time ranges plus the
+  PVs covered over them) so it can be found, annotated, and exported later: `save_dataset()`,
+  `get_dataset()`, `query_datasets()`, `iter_datasets()`, `delete_dataset()`, and a
+  `get_datasets(ids)` batch fetch, with the `DataSetQuery` criterion helpers.
+- **Annotations and calculations** — `client.annotation.annotations`.  Attach conclusions and
+  derived values to those datasets, with column-level provenance recording what each column came
+  from: `save_annotation()`, `get_annotation()`, `query_annotations()`, `iter_annotations()`,
+  `delete_annotation()`, `get_calculations()`, and the `AnnotationQuery` helpers.  Calculations are
+  built with the `data_frame` builders and read back with `data_frame_conversions`, which also
+  bridges to pandas under the optional `[analysis]` extra.
+- **Export** — `client.annotation.export`.  Export a saved dataset, ad-hoc data blocks, and/or
+  calculations to HDF5, CSV, or XLSX.  The file is written on the server; there is no retrieval RPC.
 - **Provider registration** — `client.ingestion_client.register_provider()`.  The rest of the
   ingestion API is not yet implemented.
 
@@ -97,10 +109,6 @@ Note the v2 query API comes from unreleased dp-grpc work and will not work again
   - `queryTable()` — PV time-series data in tabular format
   - `queryPvStats()` — archive ingestion statistics for PVs
   - `queryProviders()` / `queryProviderStats()` — provider information and ingestion statistics
-- **Annotation Service**
-  - `saveDataSet()` / `queryDataSets()` — datasets over collections of PVs and time ranges
-  - `saveAnnotation()` / `queryAnnotations()` — annotations targeting a dataset
-  - `exportData()` — export datasets to common file formats
 - **Ingestion Stream Service**
   - `subscribeDataEvent()` — notification when a data condition in the ingestion stream triggers
 

doc/cookbook/README.md

@@ -26,6 +26,7 @@ client.
 | [Recording machine configuration](machine-configuration.md) | Defining configurations, recording when each was active, closing and opening intervals, and answering "what was the machine doing at 18:04?" |
 | [Querying time-series data](query.md) | Retrieving samples by PV name, by metadata, or by machine configuration, and converting results to pandas / NumPy / Excel |
 | [Labeling samples](sample-status.md) | Recording per-sample status codes, reading them back, and querying data with flagged samples excluded |
+| [DataSets and annotations](datasets-and-annotations.md) | Naming a region of the archive, attaching analysis results with column-level provenance, round-tripping calculations through pandas, and exporting |
 
 **Not yet covered: getting data in.**  `IngestionClient` currently exposes only
 `register_provider()`, so there is no ingestion recipe.
@@ -45,6 +46,8 @@ query recipes is the data the earlier recipes create:
   `MODE=09`), activated over a shift with `DEST=CXI` and `EXP=CXI_3443`.
 - Queries that retrieve those PVs by name, by *"every monitor in GUNB"*, and by *"whatever ran
   during the CXI shift"*.
+- A dataset naming the first hour of that shift, an annotation recording an orbit drift, and a 1 Hz
+  RMS calculation attached to it with provenance pointing back at the source PV.
 
 Attribute names and values are the facility's; tag values are illustrative placeholders.