Add a device verification command

docs/README.md

@@ -58,6 +58,18 @@ dotnet harp.toolkit generate interface python
 
 See [Code Generation](https://harp-tech.org/toolkit/articles/generate.html) for authoring device metadata, generating firmware, and the available options.
 
+## Device Verification
+
+`harp.toolkit` can also check a device against the Harp specification, reporting where its behavior departs from the standard and writing the result as a shareable HTML report:
+
+```cmd
+dotnet harp.toolkit verify --port COM4 --report report.html
+```
+
+Verification writes to device registers and assumes a freshly powered device, so avoid running it against a device that is part of a running experiment.
+
+See [Device Verification](https://harp-tech.org/toolkit/articles/verify.html) for the specification used to check the device, the report structure, and the available options.
+
 ## Contributing
 
 Bug reports and contributions are welcome at [the GitHub repository](https://github.com/harp-tech/toolkit).

docs/articles/toc.yml

@@ -1,3 +1,4 @@
 - name: Introduction
   href: ../index.md
-- href: generate.md
+- href: generate.md
+- href: verify.md

docs/articles/verify.md

@@ -0,0 +1,124 @@
+# Device Verification
+
+`harp.toolkit` can check a device against the Harp specification and report where its behavior departs from what the standard requires. The checks span all three specification documents, covering the core register set and its access rules, the reply behavior required by the binary protocol, and alignment on the synchronization clock. Results print to the console as the run proceeds, and can be written to a shareable HTML report.
+
+A verification result records how a device behaved against a stated revision of the specification, and it confers no compliance status.
+
+> [!Warning]
+> Verification writes to device registers. Conformance cannot be established without exercising writes, read-only enforcement and event streams, so there is no read-only mode. Some checks leave the device clock and the operation control register in a changed state, and the run assumes a freshly powered device. Avoid verifying a device that is part of a running experiment.
+
+## Running a verification
+
+A verification needs only the serial port of the device.
+
+```ps1
+dotnet harp.toolkit verify --port COM3
+```
+
+Every check reports as passed, failed or skipped. A check is skipped when it needs an option that was not supplied, and the message names the option. A device that stops answering fails the check that was waiting on it, after a fixed 2000 ms, so a silent register costs one result rather than stalling the rest of the run.
+
+#### Serial port
+```ps1
+--port <port>
+```
+
+Specifies the name of the serial port used to communicate with the device. This option is required.
+
+#### Detailed results
+```ps1
+--verbose
+```
+
+Prints a detailed result for every check once the run finishes, including the statistics gathered by the measurements. Per-check progress is printed either way.
+
+## Specification version
+
+Harp devices do not all implement the same revision of the standard, so no single set of checks applies to every device.
+
+A device declares the revision it implements in `R_VERSION`. Where that register is absent, unreadable or reads all zeros, the device is held to v1, since a device that predates the register also predates the version field. Checks belonging to a revision outside that scope are neither run nor listed. A skipped result means a check that was in scope and did not run. The console and the report state how many checks were excluded.
+
+#### Include prerelease checks
+```ps1
+--prerelease
+```
+
+Also runs the checks that encode specification text outside the stable baseline. Those checks apply only to a device declaring the matching major version, so supplying the option for a device that declares v1 changes nothing. A failure reported under this option may reflect text that is still being ratified, which makes it worth checking intent carefully against the specification.
+
+## Sharing a report
+
+Console output is not an artifact. A report captures one run as a single HTML file that can be attached to an issue or a release.
+
+```ps1
+dotnet harp.toolkit verify --port COM3 --report report.html
+```
+
+The report is titled with the device name and opens with a header describing the run.
+
+- **WhoAmI** is the device identity class, read from `R_WHO_AM_I`.
+- **Serial port** is the port used to reach the device.
+- **Hardware version** and **Firmware version** are read from the device at startup, and read as not reported for a device that does not answer them.
+- **Protocol version declared** is what the device reports in `R_VERSION`, or that no version was declared.
+- **Checked against** is the revision of the specification used to verify the device, together with the reason when that is narrower than what the device declared.
+- **Specification** links to the specification documents as they stood at the commit behind the checks.
+- **Register set** names the generator package supplying the core register metadata, which fully determines the register set the run expects.
+
+#### Report path
+```ps1
+--report <report>
+```
+
+Path of the HTML report written after the run. Without it the results are printed and not saved.
+
+### Acting on a reported failure
+
+The **Specification** link is what makes a disagreement decidable, so it is worth checking carefully before filing anything. Specification text moves between releases, and a check is written against one state of it.
+
+If the device matches the text at that commit and a check still fails, the check is wrong, and that belongs in the toolkit repository. If a check matches the text and the text itself is wrong, that belongs in the protocol repository.
+
+## Verifying the synchronization clock
+
+Supplying a second device as a clock reference enables the alignment checks. Both devices must be connected to the same synchronization clock bus.
+
+```ps1
+dotnet harp.toolkit verify --port COM3 --clock-port COM4 --pps-event 32
+```
+
+#### Clock reference port
+```ps1
+--clock-port <clock-port>
+```
+
+Serial port of the reference clock device. Supplying it enables the clock alignment checks.
+
+#### Tested device event register
+```ps1
+--pps-event <pps-event>
+```
+
+Address of the register on the tested device that reports the incoming pulse from the reference clock device. Supplying it enables the pulse alignment check, which also requires a clock reference port.
+
+Note that the pulse is a physical output that only some devices produce, and it is distinct both from the synchronization signal on the clock bus and from the software heartbeat. A device reports the pulse through an application register of its own, which is why the address has to be supplied explicitly.
+
+#### Sample count
+```ps1
+--clock-samples <clock-samples>
+```
+
+Number of pulse event pairs to collect for the alignment check. The default is 5, and the value must be greater than zero.
+
+## Verifying the declared interface
+
+A device can also be checked against its own declared interface rather than only against the standard. Supplying the device metadata generates an interface from it, reads every declared register from the live device, and parses each reply with the generated parsers. The identity, firmware and hardware versions declared in the metadata are cross-checked against what the device reports.
+
+```ps1
+dotnet harp.toolkit verify --port COM3 --metadata device.yml
+```
+
+#### Device metadata
+```ps1
+--metadata <metadata>
+```
+
+Path of the file describing the device registers. The file must exist.
+
+Unlike code generation, this option has no default, so a `device.yml` located in the current directory does not automatically enable these checks.

src/Harp.Toolkit/Generate/GenerateRegisterMetadataCommand.cs

@@ -3,8 +3,6 @@
 using Bonsai.Harp;
 using ExcelDataReader;
 using Harp.Generators;
-using YamlDotNet.Serialization;
-using YamlDotNet.Serialization.NamingConventions;
 
 namespace Harp.Toolkit.Generate;
 
@@ -16,10 +14,10 @@ public GenerateRegisterMetadataCommand()
         OutputPathOption outputPathOption = new();
         Argument<FileInfo> registerWorksheetPathArgument = ArgumentValidation.AcceptExistingOnly(
             new Argument<FileInfo>("registers.xls")
-        {
-            Description = "The path to the file describing the device registers.",
-            Arity = ArgumentArity.ExactlyOne
-        });
+            {
+                Description = "The path to the file describing the device registers.",
+                Arity = ArgumentArity.ExactlyOne
+            });
 
         Arguments.Add(registerWorksheetPathArgument);
         Options.Add(outputPathOption);

src/Harp.Toolkit/Generate/GeneratorHelper.cs

@@ -41,7 +41,7 @@ public static bool AssertNoGeneratorErrors(CompilerErrorCollection errors)
             Console.Error.WriteLine(errorLog.ToString());
             return !errors.HasErrors;
         }
-        
+
         return true;
     }
 }

src/Harp.Toolkit/Harp.Toolkit.csproj

@@ -6,14 +6,25 @@
     <Description>A tool for inspecting, updating and interfacing with Harp devices from the command-line.</Description>
     <TargetFramework>net8.0</TargetFramework>
     <ImplicitUsings>enable</ImplicitUsings>
+    <PreserveCompilationContext>true</PreserveCompilationContext> <!-- Necessary for RazorLight templates -->
   </PropertyGroup>
 
   <ItemGroup>
     <PackageReference Include="Bonsai.Harp" Version="3.6.1" />
     <PackageReference Include="ExcelDataReader.DataSet" Version="3.8.0" />
-    <PackageReference Include="Harp.Generators" Version="0.6.0" />
+    <PackageReference Include="Harp.Generators" Version="0.7.0" />
+    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.13.0" />
+    <PackageReference Include="Microsoft.Extensions.DependencyModel" Version="6.0.0" />
     <PackageReference Include="System.CommandLine" Version="2.0.1" />
     <PackageReference Include="System.Text.Encoding.CodePages" Version="10.0.2" />
+    <PackageReference Include="RazorLight" Version="2.3.1" />
+    <PackageReference Include="Spectre.Console" Version="0.54.0" />
   </ItemGroup>
 
-</Project>
+  <ItemGroup>
+    <None Update="Verify\ReportTemplate.cshtml">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </None>
+  </ItemGroup>
+
+</Project>

src/Harp.Toolkit/Program.cs

@@ -1,6 +1,7 @@
 using System.CommandLine;
 using Bonsai.Harp;
 using Harp.Toolkit.Generate;
+using Harp.Toolkit.Verify;
 
 namespace Harp.Toolkit;
 
@@ -16,6 +17,7 @@ static async Task Main(string[] args)
         rootCommand.Subcommands.Add(new ListCommand());
         rootCommand.Subcommands.Add(new UpdateFirmwareCommand());
         rootCommand.Subcommands.Add(new GenerateCommand());
+        rootCommand.Subcommands.Add(new VerifyCommand());
         rootCommand.SetAction(async parseResult =>
         {
             var portName = parseResult.GetRequiredValue(portNameOption);

src/Harp.Toolkit/TaskExtensions.cs

@@ -1,4 +1,4 @@
-namespace Harp.Toolkit;
+namespace Harp.Toolkit;
 
 static class TaskExtensions
 {
@@ -13,4 +13,4 @@ internal static async Task<T> WithTimeout<T>(this Task<T> task, int? millisecond
         }
         else throw new TimeoutException("There was a timeout while awaiting the device response.");
     }
-}
+}

src/Harp.Toolkit/Verify/ClockTestOptions.cs

@@ -0,0 +1,18 @@
+namespace Harp.Toolkit.Verify;
+
+/// <summary>
+/// Options for clock alignment and PPS synchronization tests run against a reference clock device.
+/// </summary>
+/// <param name="ClockPort">
+/// Serial port of the reference clock device (WhiteRabbit). Enabling this option runs the
+/// simultaneous WhoAmI timestamp comparison test.
+/// </param>
+/// <param name="PpsEvent">
+/// Address of the register on the tested device that reports the incoming PPS pulse from the
+/// reference clock device. When provided, also runs the PPS alignment test.
+/// </param>
+/// <param name="ClockSamples">Number of PPS event pairs to collect for the PPS alignment test.</param>
+internal record ClockTestOptions(
+    string ClockPort,
+    int? PpsEvent = null,
+    int ClockSamples = 5);

src/Harp.Toolkit/Verify/CoreSchema.cs

@@ -0,0 +1,49 @@
+using System.Reflection;
+using Harp.Generators;
+
+namespace Harp.Toolkit.Verify;
+
+/// <summary>
+/// Provides the core register metadata embedded in the Harp.Generators assembly, which declares
+/// the register set and payload types a conformant device must implement.
+/// </summary>
+internal static class CoreSchema
+{
+    const string ResourceName = "Harp.Generators.core.yml";
+
+    static readonly Lazy<DeviceMetadata> metadata = new(ReadMetadata);
+    static readonly Lazy<string> version = new(ReadVersion);
+
+    /// <summary>
+    /// Gets the core register metadata declared by the pinned generator version.
+    /// </summary>
+    public static DeviceMetadata Metadata => metadata.Value;
+
+    /// <summary>
+    /// Gets the version of the generator package supplying the core register metadata. This
+    /// version fully determines the register set, since the metadata ships inside the package.
+    /// </summary>
+    public static string Version => version.Value;
+
+    static string ReadVersion()
+    {
+        var assembly = typeof(InterfaceGenerator).Assembly;
+        var informational = assembly
+            .GetCustomAttribute<AssemblyInformationalVersionAttribute>()?.InformationalVersion;
+        var text = informational ?? assembly.GetName().Version?.ToString();
+        if (string.IsNullOrEmpty(text))
+            return "unknown";
+
+        var metadataSeparator = text.IndexOf('+');
+        return metadataSeparator < 0 ? text : text[..metadataSeparator];
+    }
+
+    static DeviceMetadata ReadMetadata()
+    {
+        using var stream = typeof(InterfaceGenerator).Assembly.GetManifestResourceStream(ResourceName)
+            ?? throw new InvalidOperationException(
+                $"The core register metadata resource '{ResourceName}' was not found in the Harp.Generators assembly.");
+        using var reader = new StreamReader(stream);
+        return MetadataDeserializer.Instance.Deserialize<DeviceMetadata>(reader);
+    }
+}

src/Harp.Toolkit/Verify/DeviceIdentity.cs

@@ -0,0 +1,9 @@
+using Bonsai.Harp;
+
+namespace Harp.Toolkit.Verify;
+
+internal readonly record struct DeviceIdentity(
+    int WhoAmI,
+    string? Name,
+    HarpVersion? HardwareVersion,
+    HarpVersion? FirmwareVersion);

src/Harp.Toolkit/Verify/GeneratedInterfaceCompiler.cs

@@ -0,0 +1,77 @@
+using System.Reflection;
+using System.Text;
+using Harp.Generators;
+using Harp.Toolkit.Generate;
+using Microsoft.CodeAnalysis;
+using Microsoft.CodeAnalysis.CSharp;
+using Microsoft.Extensions.DependencyModel;
+
+namespace Harp.Toolkit.Verify;
+
+/// <summary>
+/// Generates the C# interface for a device.yml (via <see cref="InterfaceGenerator"/>),
+/// compiles it in-memory, and returns its register address-to-type map so callers can
+/// invoke each register's own generated parser reflectively.
+/// </summary>
+internal static class GeneratedInterfaceCompiler
+{
+    public static IReadOnlyDictionary<int, Type> Compile(DeviceMetadata deviceOnlyMetadata, string rawDeviceYaml, string @namespace)
+    {
+        var generator = new InterfaceGenerator(deviceOnlyMetadata, @namespace);
+        var implementation = generator.GenerateImplementation();
+        if (!GeneratorHelper.AssertNoGeneratorErrors(generator.Errors))
+            throw new InvalidOperationException("Interface generation from device.yml completed with errors.");
+
+        var syntaxTree = CSharpSyntaxTree.ParseText(implementation.Device);
+        var compilation = CSharpCompilation.Create(
+            $"HarpGeneratedInterface_{@namespace}",
+            new[] { syntaxTree },
+            GetMetadataReferences(),
+            new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary, allowUnsafe: true));
+
+        // The generated Device class's static constructor reads device.yml back from
+        // an embedded "{Namespace}.device.yml" manifest resource (e.g. to expose it via
+        // the Metadata property) - without it, merely accessing RegisterMap throws.
+        var rawYamlBytes = Encoding.UTF8.GetBytes(rawDeviceYaml);
+        var deviceYamlResource = new ResourceDescription(
+            $"{@namespace}.device.yml",
+            () => new MemoryStream(rawYamlBytes),
+            isPublic: true);
+
+        using var peStream = new MemoryStream();
+        var result = compilation.Emit(peStream, manifestResources: new[] { deviceYamlResource });
+        if (!result.Success)
+        {
+            var errors = string.Join(Environment.NewLine, result.Diagnostics.Where(d => d.Severity == DiagnosticSeverity.Error));
+            throw new InvalidOperationException($"Failed to compile the interface generated from device.yml:{Environment.NewLine}{errors}");
+        }
+
+        var assembly = Assembly.Load(peStream.ToArray());
+        var deviceType = assembly.GetType($"{@namespace}.Device")
+            ?? throw new InvalidOperationException($"Compiled assembly does not contain type '{@namespace}.Device'.");
+        var registerMapProperty = deviceType.GetProperty("RegisterMap", BindingFlags.Public | BindingFlags.Static)
+            ?? throw new InvalidOperationException($"'{@namespace}.Device' does not expose a static RegisterMap property.");
+
+        return (IReadOnlyDictionary<int, Type>)registerMapProperty.GetValue(null)!;
+    }
+
+    // Reuses this project's existing PreserveCompilationContext setup (already required
+    // for RazorLight's own runtime compilation) to resolve the full reference-assembly
+    // closure, including Bonsai.Harp/Bonsai.Core, which the generated code depends on.
+    private static IReadOnlyList<MetadataReference> GetMetadataReferences()
+    {
+        var context = DependencyContext.Default
+            ?? throw new InvalidOperationException("No DependencyContext available for compiling the generated interface.");
+
+        var paths = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+        foreach (var library in context.CompileLibraries)
+        {
+            foreach (var path in library.ResolveReferencePaths())
+            {
+                paths.Add(path);
+            }
+        }
+
+        return paths.Select(path => (MetadataReference)MetadataReference.CreateFromFile(path)).ToList();
+    }
+}