Repository navigation
Add fastermodels.measurement: latency rows and quality rows, validated - #10
Merged
Merged
Conversation
A measurement row records one session of one model variant on one device and software stack, at one precision and one batch size, on one date: latency percentiles and their spread over repeats, device conditions (power mode, clock locking, temperature, throttling), the accuracy with its confidence interval measured through the deployed engine, and argmax parity against the reference model. Rows are stored as JSON Lines. - FIELDS: the schema as data (type, required, rules); SCHEMA_VERSION = 1. - validate(row): every problem, by field name. - status(row): whether the measurement is VALID and why not — parity below its floor, throttling, latency spread above 5 %, accuracy not measured through the deployed engine, latency not measured on the device, or any of these not recorded. It is a validity status, never a verdict against a latency or accuracy budget. - key(row): the identity a row is unique on; a software update on the device gives a new key. - read_rows / write_row: JSON Lines, append-only, refusing malformed and duplicate rows. to_parquet for publishing (optional `parquet` extra). - throughput(row) is derived, never stored. Reached as `fastermodels.measurement`; the generic names stay off the package top level.
How fast an engine runs depends on its graph (architecture, widths, numeric precision) and on the device and software stack, not on the values of its weights; how accurate it is depends on the weights and the evaluation set. A single row forced every speed measurement to carry an accuracy. Each row now has a `kind`: - latency: one engine timed on one device. It needs no accuracy, so an engine built from random weights (`weights='random'`) is measured like any other, which is what a latency predictor trains on. Graph fields (`graph_hash`, widths through macs/params, recipe), the engine's layer counts and the precisions it actually ran, device, conditions and latency percentiles. - quality: one artifact's accuracy on one evaluation set, through the engine it names (`engine_hash`, None only for the development model). The identity and provenance fields are defined once and shared. `FIELDS[kind]` is the schema; validate, status and key dispatch on the kind, a missing or unknown kind is one clear problem, and key uniqueness in a file is per kind. `join` pairs the VALID rows of both kinds that share an engine_hash, keeping identity per side (latency_run_id, quality_run_id, ...) and refusing two sides that disagree on the same engine. `to_parquet` writes one kind per file. `graph_hash(onnx_path)` hashes the ONNX graph with every initializer's values removed and its name, shape and type kept: two models of the same shape share it, another width or a renamed weight does not. It needs the new optional `onnx` extra.
nathanhubens
changed the title
to_parquet went through a pandas DataFrame, so the column types of the published file were whatever pandas inferred: a string column with a missing value read back as NaN under pandas 3 and as None under pandas 2, and a column nobody recorded had no type at all. A published file must not depend on the reader. The schema is now derived from FIELDS[kind]: str is a string column, int int64, float float64, bool bool, a dict a string holding canonical JSON, and a field that is not required is nullable. The table is built and written with pyarrow alone; pandas is no longer used, and the `parquet` extra is pyarrow only. The round-trip test reads the file back with pyarrow and compares it to the rows, None preserved, and pins the type and nullability of the columns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
fastermodels.measurement: the published definition of an on-device measurement, and its validator. Two row kinds, because speed and accuracy do not describe the same object.graph_hash), the numeric precision, the batch size, the device and its software stack, and whether the weights were trained or random: speed depends on the graph, not on the weights, so latency can be measured (and a latency model trained) on random-weight models. Carries p50/p90/p99, the spread over repeats, the device conditions (power mode, locked clocks, temperature, throttling) and what the engine ran (layer count, reformatting layers, per-layer precisions).API:
FIELDS(per kind),validate(row)(every problem, by field),status(row)(VALID / INVALID with reasons — measurement validity, never a verdict against a budget),key(row)(per kind; a software update on the device gives a new key),read_rows/write_row(JSON Lines, append-only, refusing malformed and duplicate rows),join(latency_rows, quality_rows)(VALID rows paired on the engine),throughput(row)(derived),to_parquet(one kind per file, optionalparquetextra),graph_hash(onnx_path)(optionalonnxextra). Reached asfastermodels.measurement; nothing is added to the package top level.Tests
nbdev_teston the full suite: Success, including a real Parquet round trip andgraph_hashon real exports (same shape with other weights → same hash; another width, a renamed initializer or another element type → a different one). Export and clean are idempotent;__version__is unchanged.