Add a device verification command - #37
Open
glopesdev wants to merge 40 commits into
Open
Conversation
The command, its folder, namespace and class are renamed from benchmark to verify, so the name describes conformance against the Harp specification rather than just the smaller part that measures timing. The measurement types keep the benchmark name, since NumericBenchmarkResult, BenchmarkSummary and BenchmarkRoundTrip do measure latency.
Replaces the literal register addresses and lengths across the suites with the constants declared by the client register classes. The four addresses the client cannot name are written in decimal to match the numbering used by the specification and by core.yml, and the UID address becomes internal so the serial number suite reads it instead of repeating the value. R_VERSION has no client constant either, so it gains Version, VersionPayload and SemanticVersion, mirroring the interface the code generator produces for a register carrying a payload specification. Each version field is read as three bytes, major, minor and patch, rather than as HarpVersion, which has no patch component. The Version suite gains a test reporting the parsed PROTOCOL, FIRMWARE, HARDWARE, CORE_ID and INTERFACE_HASH fields. Renames the PPS option to --pps-event, replacing a partial rename that left --pps-address in the option and --reg-clock in the skip message, so the skip message named a flag the command rejected. The same file also drops a conditional access on a non-nullable result property, which was the only build warning on the branch.
The progress line and the line clear that precedes each result are now written only when output is not redirected. Piping the command to a file or running it under CI would otherwise fail on Console.WindowWidth, which is unavailable in that case. Redirected output now carries only the result lines.
The read-only probes for R_HEARTBEAT, R_TAG and R_VERSION now send a payload matching the register type and length. The timestamp writability test compares the absolute difference, so a device that ignores the write no longer passes when its clock reads below the value written. The register dump requires all twenty allocated core addresses rather than the first eighteen, and its core boundary now excludes address 32, the first application register. The three operation control event tests now write distinct bit combinations: heartbeat only, alive only, and both. The heartbeat test previously also set ALIVE_EN and the indicator bits. Restores are wrapped so a failed restore no longer replaces the test result. On the command, --clock-samples is rejected when not positive, since zero yielded NaN statistics, and the unreachable report filename fallback is removed.
Tests now receive a shared connection instead of a port name, so a run opens the device once rather than once per test. A sweep goes from 49 port openings to one, or two when a reference clock device is configured, and completes in about half the time. Repeated runs of the same build now produce the same result for every test that does not depend on host timing. A denied open is retried against a ten second deadline rather than failing immediately.
The benchmark now times each command with a stopwatch instead of differencing consecutive device reply timestamps, removing the dependency on the internal timestamp resolution of the device, which the protocol quantizes to 32 microseconds. It reports the resulting statistics and always passes. Mean, median, standard deviation, minimum, maximum and percentiles still appear in the console table and the HTML report.
A failed read of the register would leave the restore writing zero, putting the device in Standby with every flag cleared. The original value is now nullable and the restore is skipped when it is absent.
The reset device suite now checks that a read reports exactly one boot provenance bit with every command bit cleared, so the only legal values are 0x40 and 0x80, and that writes setting BOOT_DEF or BOOT_EE are refused, since both are read-only state. Both write probes fail on current ATxmega firmware, which answers them with a write reply rather than an error.
The register dump test now resolves the expected addresses and payload types from the core register metadata embedded in Harp.Generators, rather than from a hardcoded range of twenty addresses. It requires the fifteen declared addresses, ignores any beyond them, and now also fails when a reply carries a payload type the schema does not declare. The generators reference moves to 0.7.0, which is what pins the schema the test reads.
Tests asserting specification text outside the stable baseline are marked with Prerelease on HarpTestAttribute and excluded during collection, so the progress counter and the report cover only what runs. The new --prerelease option includes them. Twenty-two of the fifty-one tests are marked, and the Behavior device now passes the whole default run.
Verification now reads the protocol version declared by the device and holds it to that revision. A device with no R_VERSION falls back to v1, and a version beyond coverage is reported before the run continues against v1. The --prerelease option also requires a device declaring v2, so on a v1 device it reports having no effect rather than silently including nothing. The console states the declared version and the revision applied, and the report header adds both plus the generator version supplying the register set.
The serial number test asserted a derivation from R_UID that was added after the v1.13.0 tag and removed again in protocol 225, so it failed on every conformant device. It is replaced by a readability test, which the deprecated register rules still require. The timestamp offset writability test asserted the opposite of every version of the specification, where the register is marked writable at the tag and on main, and is removed. The two operation control heartbeat tests are marked prerelease, since they write a bit and await events from a register the tagged specification does not define. A device without R_VERSION now passes the whole default run, which is 28 tests.
The verify option naming the device interface file was --device-yml, which named a serialization format rather than the content, and diverged from the word the toolkit already uses for that file in DeviceMetadata, ReadDeviceMetadata and MetadataPathArgument.
The shared connection now requests standby mode with register dumps disabled, so the board no longer streams application events during a run. The identity read gains a two-second deadline per attempt. Each phase now starts its own retry budget rather than sharing one, so the connection always gets its full retry window. Both phases also retry transport errors and timeouts. The run no longer hangs. Stale bytes on a reopened port still produce an occasional parse warning, but the parser resynchronizes and the run completes.
Writing the register sets the integer seconds and leaves the sub-second counter running, so the clock can cross a second boundary before it is read. The bound now allows one full crossing and is directional, so a clock reading below the written value fails instead of passing. The failure message reports the reply timestamp against the expected range.
The device identity is read once before the suite runs, each read bounded by its own 2000 ms deadline, so an unanswered read shows as not reported rather than stalling the run. The console prints one identity line and the report header carries WhoAmI, serial port, hardware version and firmware version. The report is now titled with the device name, with a Harp conformance report subtitle in place of the toolkit badge.
Requests sent through CommandAsync now fail with a TimeoutException when the device does not reply within 2000 ms, so an unanswered request fails only that test instead of stalling the whole run. The reply timeout, the identity reads and the connection probe share one constant. The protocol version read moves onto the connection as ReadProtocolVersionAsync, taking the payload length check and the all-zero normalization with it, which leaves ProtocolTarget a plain record struct over its declared version and the prerelease flag.
The report header now names the harp-tech/protocol commit behind the checks, linked to the repository tree at that commit.
Writing a past timestamp is now accepted only when the clock reads forward of the written value, where the previous absolute difference also accepted a clock behind it. The one second tolerance is unchanged, since this check compares integer seconds and a boundary crossing shows as exactly one. The round trip suite description no longer claims a set of tests that write registers, since it is one test that reads WhoAmI and reports latency statistics. The boot provenance failure message no longer cites the rule that a device without non-volatile memory must set BOOT_DEF. That rule postdates the stable baseline, where the check itself runs.
A new article covers running a verification, how the specification revision is determined, the report structure and how to act on a reported failure, the synchronization clock checks and the declared interface checks, with every option documented. The readme gains a section pointing at it, and the article joins the table of contents after code generation.
glopesdev
requested review from
a team,
MicBoucinha,
Poofjunior,
artursilva0,
banchan86,
bruno-f-cruz and
filcarv
September 9, 2026 22:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
harp.toolkit verifychecks a device against the Harp specification and reports where its behavior departs from what the standard requires, printing to the console as the run proceeds and optionally writing a shareable HTML report. Conformance in this ecosystem has rested on word of mouth and on each maintainer reading the specification text, so a core or device developer has had no way to check a device independently. A report is an artifact that can be attached to an issue or a release to help back a conformance claim with evidence.The command originates in #16, authored by @bruno-f-cruz, and those commits are preserved here.
What it checks
The suite declares 50 checks, 45 of them grouped one suite per core register, covering readability, payload length and type, read-only enforcement, operation control behavior and the register dump. Alongside those it measures round trip latency and synchronization clock alignment against a reference device, and validates a device against its own
device.ymlby generating an interface and parsing live replies with it. That set spans all three specification documents, so this is a conformance tool in substance rather than the benchmark the original name suggested.docs/articles/verify.mddocuments the command, its options and the report.The declared baseline
The original suite declared no target version while encoding several at once, so a failure could mean a device defect, an unreleased rule, or a rule that no longer exists.
A device is now held to the revision it declares in
R_VERSION, falling back to v1 when that register is absent, unreadable or reads all zeros, since a device predating the register predates the version field. Checks encoding text outside the stable baseline are hidden unless the--prereleaseoption is supplied, and the excluded count is reported. Of the 50 checks, 28 run by default and 22 require--prerelease, which applies only to a device declaring the matching major version.The report also names the harp-tech/protocol commit behind the checks, currently harp-tech/protocol@11b584b, linked to the tree at that commit. We can use this to decide disagreements with the specification text currently under ratification. A device matching the text at that commit while a check fails is a toolkit defect, and a check matching a text that is wrong is a specification issue. Bumping it is intended to be a deliberate act with its own commit, since it can change results.
The core register set comes from the
core.ymlasset embedded in Harp.Generators, so the package version fully determines the registers a run expects, and no copy of the schema is vendored here.What it found
Measured against a Behavior device, WhoAmI 1216, hardware 2.0, firmware 3.3. The default run reports 24 passed, 0 failed and 4 skipped, the skips being the checks that need
--clock-portor--metadata. With--prereleaseit reports 42 passed, 4 failed and 4 skipped. Both runs were taken on hardware after the final change, with the report rendered and its header checked, and the project and documentation site both build at zero warnings.All four failures are genuine device non-conformances against the v2 text the device itself declares, rather than test defects:
R_CORE_VERSION_HandR_CORE_VERSION_Lreport core 1.15 whileR_VERSIONreports PROTOCOL 2.0, where Device Registers and Operation requires the deprecated pair to equal the major and minor protocol version.BOOT_DEForBOOT_EEtoR_RESET_DEVis answered with a write acknowledgement, where both bits are read-only state that must be answered with an error reply. The ATxmega handler tests four exact equalities and then falls through to success, so the write matches nothing, changes nothing, and is reported as having succeeded.Reliability
A conformance tool has to be reproducible before anyone can act on its output, and two properties get it there.
The run opens one connection rather than one per check, taking 49 port transitions down to one, which halved the runtime and removed an intermittent disagreement between back-to-back runs of an identical build.
Requests carry a 2000 ms reply timeout and fail with a
TimeoutException. Without it a device that never answers a register stalls the whole run rather than failing one check, which is reachable on real firmware:core.picodrops any request above its core register set with a bare return and no reply.New dependencies
Four package references are added to a tool shipped through the dotnet tool channel, and Harp.Generators moves from 0.6.0 to 0.7.0 for the embedded core schema. RazorLight renders the HTML report, Spectre.Console renders the console output, and Microsoft.CodeAnalysis.CSharp with Microsoft.Extensions.DependencyModel compile the generated interface at runtime for the
--metadatacheck.PreserveCompilationContextis required by RazorLight.Known limitations
The synchronization clock measurement is usable but still developing, and the open question of acceptable alignment tolerances requires further tests and discussion.
Verification always mutates the device. Conformance cannot be established without exercising writes, read-only enforcement and event streams, so there is no read-only mode, and the run assumes a freshly powered device.