diff --git a/docs/README.md b/docs/README.md index d60d975..82e97fd 100644 --- a/docs/README.md +++ b/docs/README.md @@ -58,6 +58,18 @@ dotnet harp.toolkit generate interface python See [Code Generation](https://harp-tech.org/toolkit/articles/generate.html) for authoring device metadata, generating firmware, and the available options. +## Device Verification + +`harp.toolkit` can also check a device against the Harp specification, reporting where its behavior departs from the standard and writing the result as a shareable HTML report: + +```cmd +dotnet harp.toolkit verify --port COM4 --report report.html +``` + +Verification writes to device registers and assumes a freshly powered device, so avoid running it against a device that is part of a running experiment. + +See [Device Verification](https://harp-tech.org/toolkit/articles/verify.html) for the specification used to check the device, the report structure, and the available options. + ## Contributing Bug reports and contributions are welcome at [the GitHub repository](https://github.com/harp-tech/toolkit). diff --git a/docs/articles/toc.yml b/docs/articles/toc.yml index 1ce9c53..2571da1 100644 --- a/docs/articles/toc.yml +++ b/docs/articles/toc.yml @@ -1,3 +1,4 @@ - name: Introduction href: ../index.md -- href: generate.md \ No newline at end of file +- href: generate.md +- href: verify.md \ No newline at end of file diff --git a/docs/articles/verify.md b/docs/articles/verify.md new file mode 100644 index 0000000..3dbb867 --- /dev/null +++ b/docs/articles/verify.md @@ -0,0 +1,124 @@ +# Device Verification + +`harp.toolkit` can check a device against the Harp specification and report where its behavior departs from what the standard requires. The checks span all three specification documents, covering the core register set and its access rules, the reply behavior required by the binary protocol, and alignment on the synchronization clock. Results print to the console as the run proceeds, and can be written to a shareable HTML report. + +A verification result records how a device behaved against a stated revision of the specification, and it confers no compliance status. + +> [!Warning] +> Verification writes to device registers. Conformance cannot be established without exercising writes, read-only enforcement and event streams, so there is no read-only mode. Some checks leave the device clock and the operation control register in a changed state, and the run assumes a freshly powered device. Avoid verifying a device that is part of a running experiment. + +## Running a verification + +A verification needs only the serial port of the device. + +```ps1 +dotnet harp.toolkit verify --port COM3 +``` + +Every check reports as passed, failed or skipped. A check is skipped when it needs an option that was not supplied, and the message names the option. A device that stops answering fails the check that was waiting on it, after a fixed 2000 ms, so a silent register costs one result rather than stalling the rest of the run. + +#### Serial port +```ps1 +--port +``` + +Specifies the name of the serial port used to communicate with the device. This option is required. + +#### Detailed results +```ps1 +--verbose +``` + +Prints a detailed result for every check once the run finishes, including the statistics gathered by the measurements. Per-check progress is printed either way. + +## Specification version + +Harp devices do not all implement the same revision of the standard, so no single set of checks applies to every device. + +A device declares the revision it implements in `R_VERSION`. Where that register is absent, unreadable or reads all zeros, the device is held to v1, since a device that predates the register also predates the version field. Checks belonging to a revision outside that scope are neither run nor listed. A skipped result means a check that was in scope and did not run. The console and the report state how many checks were excluded. + +#### Include prerelease checks +```ps1 +--prerelease +``` + +Also runs the checks that encode specification text outside the stable baseline. Those checks apply only to a device declaring the matching major version, so supplying the option for a device that declares v1 changes nothing. A failure reported under this option may reflect text that is still being ratified, which makes it worth checking intent carefully against the specification. + +## Sharing a report + +Console output is not an artifact. A report captures one run as a single HTML file that can be attached to an issue or a release. + +```ps1 +dotnet harp.toolkit verify --port COM3 --report report.html +``` + +The report is titled with the device name and opens with a header describing the run. + +- **WhoAmI** is the device identity class, read from `R_WHO_AM_I`. +- **Serial port** is the port used to reach the device. +- **Hardware version** and **Firmware version** are read from the device at startup, and read as not reported for a device that does not answer them. +- **Protocol version declared** is what the device reports in `R_VERSION`, or that no version was declared. +- **Checked against** is the revision of the specification used to verify the device, together with the reason when that is narrower than what the device declared. +- **Specification** links to the specification documents as they stood at the commit behind the checks. +- **Register set** names the generator package supplying the core register metadata, which fully determines the register set the run expects. + +#### Report path +```ps1 +--report +``` + +Path of the HTML report written after the run. Without it the results are printed and not saved. + +### Acting on a reported failure + +The **Specification** link is what makes a disagreement decidable, so it is worth checking carefully before filing anything. Specification text moves between releases, and a check is written against one state of it. + +If the device matches the text at that commit and a check still fails, the check is wrong, and that belongs in the toolkit repository. If a check matches the text and the text itself is wrong, that belongs in the protocol repository. + +## Verifying the synchronization clock + +Supplying a second device as a clock reference enables the alignment checks. Both devices must be connected to the same synchronization clock bus. + +```ps1 +dotnet harp.toolkit verify --port COM3 --clock-port COM4 --pps-event 32 +``` + +#### Clock reference port +```ps1 +--clock-port +``` + +Serial port of the reference clock device. Supplying it enables the clock alignment checks. + +#### Tested device event register +```ps1 +--pps-event +``` + +Address of the register on the tested device that reports the incoming pulse from the reference clock device. Supplying it enables the pulse alignment check, which also requires a clock reference port. + +Note that the pulse is a physical output that only some devices produce, and it is distinct both from the synchronization signal on the clock bus and from the software heartbeat. A device reports the pulse through an application register of its own, which is why the address has to be supplied explicitly. + +#### Sample count +```ps1 +--clock-samples +``` + +Number of pulse event pairs to collect for the alignment check. The default is 5, and the value must be greater than zero. + +## Verifying the declared interface + +A device can also be checked against its own declared interface rather than only against the standard. Supplying the device metadata generates an interface from it, reads every declared register from the live device, and parses each reply with the generated parsers. The identity, firmware and hardware versions declared in the metadata are cross-checked against what the device reports. + +```ps1 +dotnet harp.toolkit verify --port COM3 --metadata device.yml +``` + +#### Device metadata +```ps1 +--metadata +``` + +Path of the file describing the device registers. The file must exist. + +Unlike code generation, this option has no default, so a `device.yml` located in the current directory does not automatically enable these checks. diff --git a/src/Harp.Toolkit/Generate/GenerateRegisterMetadataCommand.cs b/src/Harp.Toolkit/Generate/GenerateRegisterMetadataCommand.cs index 6c9a0c6..07e734f 100644 --- a/src/Harp.Toolkit/Generate/GenerateRegisterMetadataCommand.cs +++ b/src/Harp.Toolkit/Generate/GenerateRegisterMetadataCommand.cs @@ -3,8 +3,6 @@ using Bonsai.Harp; using ExcelDataReader; using Harp.Generators; -using YamlDotNet.Serialization; -using YamlDotNet.Serialization.NamingConventions; namespace Harp.Toolkit.Generate; @@ -16,10 +14,10 @@ public GenerateRegisterMetadataCommand() OutputPathOption outputPathOption = new(); Argument registerWorksheetPathArgument = ArgumentValidation.AcceptExistingOnly( new Argument("registers.xls") - { - Description = "The path to the file describing the device registers.", - Arity = ArgumentArity.ExactlyOne - }); + { + Description = "The path to the file describing the device registers.", + Arity = ArgumentArity.ExactlyOne + }); Arguments.Add(registerWorksheetPathArgument); Options.Add(outputPathOption); diff --git a/src/Harp.Toolkit/Generate/GeneratorHelper.cs b/src/Harp.Toolkit/Generate/GeneratorHelper.cs index eeb0a05..a4db8b6 100644 --- a/src/Harp.Toolkit/Generate/GeneratorHelper.cs +++ b/src/Harp.Toolkit/Generate/GeneratorHelper.cs @@ -41,7 +41,7 @@ public static bool AssertNoGeneratorErrors(CompilerErrorCollection errors) Console.Error.WriteLine(errorLog.ToString()); return !errors.HasErrors; } - + return true; } } diff --git a/src/Harp.Toolkit/Harp.Toolkit.csproj b/src/Harp.Toolkit/Harp.Toolkit.csproj index 0fadfae..bbfa6ee 100644 --- a/src/Harp.Toolkit/Harp.Toolkit.csproj +++ b/src/Harp.Toolkit/Harp.Toolkit.csproj @@ -6,14 +6,25 @@ A tool for inspecting, updating and interfacing with Harp devices from the command-line. net8.0 enable + true - + + + + + - + + + PreserveNewest + + + + \ No newline at end of file diff --git a/src/Harp.Toolkit/Program.cs b/src/Harp.Toolkit/Program.cs index f269381..7fc160e 100644 --- a/src/Harp.Toolkit/Program.cs +++ b/src/Harp.Toolkit/Program.cs @@ -1,6 +1,7 @@ using System.CommandLine; using Bonsai.Harp; using Harp.Toolkit.Generate; +using Harp.Toolkit.Verify; namespace Harp.Toolkit; @@ -16,6 +17,7 @@ static async Task Main(string[] args) rootCommand.Subcommands.Add(new ListCommand()); rootCommand.Subcommands.Add(new UpdateFirmwareCommand()); rootCommand.Subcommands.Add(new GenerateCommand()); + rootCommand.Subcommands.Add(new VerifyCommand()); rootCommand.SetAction(async parseResult => { var portName = parseResult.GetRequiredValue(portNameOption); diff --git a/src/Harp.Toolkit/TaskExtensions.cs b/src/Harp.Toolkit/TaskExtensions.cs index 60668ad..19f2309 100644 --- a/src/Harp.Toolkit/TaskExtensions.cs +++ b/src/Harp.Toolkit/TaskExtensions.cs @@ -1,4 +1,4 @@ -namespace Harp.Toolkit; +namespace Harp.Toolkit; static class TaskExtensions { @@ -13,4 +13,4 @@ internal static async Task WithTimeout(this Task task, int? millisecond } else throw new TimeoutException("There was a timeout while awaiting the device response."); } -} \ No newline at end of file +} diff --git a/src/Harp.Toolkit/Verify/ClockTestOptions.cs b/src/Harp.Toolkit/Verify/ClockTestOptions.cs new file mode 100644 index 0000000..0f1fd19 --- /dev/null +++ b/src/Harp.Toolkit/Verify/ClockTestOptions.cs @@ -0,0 +1,18 @@ +namespace Harp.Toolkit.Verify; + +/// +/// Options for clock alignment and PPS synchronization tests run against a reference clock device. +/// +/// +/// Serial port of the reference clock device (WhiteRabbit). Enabling this option runs the +/// simultaneous WhoAmI timestamp comparison test. +/// +/// +/// Address of the register on the tested device that reports the incoming PPS pulse from the +/// reference clock device. When provided, also runs the PPS alignment test. +/// +/// Number of PPS event pairs to collect for the PPS alignment test. +internal record ClockTestOptions( + string ClockPort, + int? PpsEvent = null, + int ClockSamples = 5); diff --git a/src/Harp.Toolkit/Verify/CoreSchema.cs b/src/Harp.Toolkit/Verify/CoreSchema.cs new file mode 100644 index 0000000..76629b5 --- /dev/null +++ b/src/Harp.Toolkit/Verify/CoreSchema.cs @@ -0,0 +1,49 @@ +using System.Reflection; +using Harp.Generators; + +namespace Harp.Toolkit.Verify; + +/// +/// Provides the core register metadata embedded in the Harp.Generators assembly, which declares +/// the register set and payload types a conformant device must implement. +/// +internal static class CoreSchema +{ + const string ResourceName = "Harp.Generators.core.yml"; + + static readonly Lazy metadata = new(ReadMetadata); + static readonly Lazy version = new(ReadVersion); + + /// + /// Gets the core register metadata declared by the pinned generator version. + /// + public static DeviceMetadata Metadata => metadata.Value; + + /// + /// Gets the version of the generator package supplying the core register metadata. This + /// version fully determines the register set, since the metadata ships inside the package. + /// + public static string Version => version.Value; + + static string ReadVersion() + { + var assembly = typeof(InterfaceGenerator).Assembly; + var informational = assembly + .GetCustomAttribute()?.InformationalVersion; + var text = informational ?? assembly.GetName().Version?.ToString(); + if (string.IsNullOrEmpty(text)) + return "unknown"; + + var metadataSeparator = text.IndexOf('+'); + return metadataSeparator < 0 ? text : text[..metadataSeparator]; + } + + static DeviceMetadata ReadMetadata() + { + using var stream = typeof(InterfaceGenerator).Assembly.GetManifestResourceStream(ResourceName) + ?? throw new InvalidOperationException( + $"The core register metadata resource '{ResourceName}' was not found in the Harp.Generators assembly."); + using var reader = new StreamReader(stream); + return MetadataDeserializer.Instance.Deserialize(reader); + } +} diff --git a/src/Harp.Toolkit/Verify/DeviceIdentity.cs b/src/Harp.Toolkit/Verify/DeviceIdentity.cs new file mode 100644 index 0000000..083d3c1 --- /dev/null +++ b/src/Harp.Toolkit/Verify/DeviceIdentity.cs @@ -0,0 +1,9 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify; + +internal readonly record struct DeviceIdentity( + int WhoAmI, + string? Name, + HarpVersion? HardwareVersion, + HarpVersion? FirmwareVersion); diff --git a/src/Harp.Toolkit/Verify/GeneratedInterfaceCompiler.cs b/src/Harp.Toolkit/Verify/GeneratedInterfaceCompiler.cs new file mode 100644 index 0000000..a2239b0 --- /dev/null +++ b/src/Harp.Toolkit/Verify/GeneratedInterfaceCompiler.cs @@ -0,0 +1,77 @@ +using System.Reflection; +using System.Text; +using Harp.Generators; +using Harp.Toolkit.Generate; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using Microsoft.Extensions.DependencyModel; + +namespace Harp.Toolkit.Verify; + +/// +/// Generates the C# interface for a device.yml (via ), +/// compiles it in-memory, and returns its register address-to-type map so callers can +/// invoke each register's own generated parser reflectively. +/// +internal static class GeneratedInterfaceCompiler +{ + public static IReadOnlyDictionary Compile(DeviceMetadata deviceOnlyMetadata, string rawDeviceYaml, string @namespace) + { + var generator = new InterfaceGenerator(deviceOnlyMetadata, @namespace); + var implementation = generator.GenerateImplementation(); + if (!GeneratorHelper.AssertNoGeneratorErrors(generator.Errors)) + throw new InvalidOperationException("Interface generation from device.yml completed with errors."); + + var syntaxTree = CSharpSyntaxTree.ParseText(implementation.Device); + var compilation = CSharpCompilation.Create( + $"HarpGeneratedInterface_{@namespace}", + new[] { syntaxTree }, + GetMetadataReferences(), + new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary, allowUnsafe: true)); + + // The generated Device class's static constructor reads device.yml back from + // an embedded "{Namespace}.device.yml" manifest resource (e.g. to expose it via + // the Metadata property) - without it, merely accessing RegisterMap throws. + var rawYamlBytes = Encoding.UTF8.GetBytes(rawDeviceYaml); + var deviceYamlResource = new ResourceDescription( + $"{@namespace}.device.yml", + () => new MemoryStream(rawYamlBytes), + isPublic: true); + + using var peStream = new MemoryStream(); + var result = compilation.Emit(peStream, manifestResources: new[] { deviceYamlResource }); + if (!result.Success) + { + var errors = string.Join(Environment.NewLine, result.Diagnostics.Where(d => d.Severity == DiagnosticSeverity.Error)); + throw new InvalidOperationException($"Failed to compile the interface generated from device.yml:{Environment.NewLine}{errors}"); + } + + var assembly = Assembly.Load(peStream.ToArray()); + var deviceType = assembly.GetType($"{@namespace}.Device") + ?? throw new InvalidOperationException($"Compiled assembly does not contain type '{@namespace}.Device'."); + var registerMapProperty = deviceType.GetProperty("RegisterMap", BindingFlags.Public | BindingFlags.Static) + ?? throw new InvalidOperationException($"'{@namespace}.Device' does not expose a static RegisterMap property."); + + return (IReadOnlyDictionary)registerMapProperty.GetValue(null)!; + } + + // Reuses this project's existing PreserveCompilationContext setup (already required + // for RazorLight's own runtime compilation) to resolve the full reference-assembly + // closure, including Bonsai.Harp/Bonsai.Core, which the generated code depends on. + private static IReadOnlyList GetMetadataReferences() + { + var context = DependencyContext.Default + ?? throw new InvalidOperationException("No DependencyContext available for compiling the generated interface."); + + var paths = new HashSet(StringComparer.OrdinalIgnoreCase); + foreach (var library in context.CompileLibraries) + { + foreach (var path in library.ResolveReferencePaths()) + { + paths.Add(path); + } + } + + return paths.Select(path => (MetadataReference)MetadataReference.CreateFromFile(path)).ToList(); + } +} diff --git a/src/Harp.Toolkit/Verify/HarpTestAttribute.cs b/src/Harp.Toolkit/Verify/HarpTestAttribute.cs new file mode 100644 index 0000000..42b091d --- /dev/null +++ b/src/Harp.Toolkit/Verify/HarpTestAttribute.cs @@ -0,0 +1,9 @@ +namespace Harp.Toolkit.Verify; + +[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)] +public class HarpTestAttribute : Attribute +{ + public string? Description { get; set; } + + public bool Prerelease { get; set; } +} diff --git a/src/Harp.Toolkit/Verify/HtmlReportGenerator.cs b/src/Harp.Toolkit/Verify/HtmlReportGenerator.cs new file mode 100644 index 0000000..6b72e72 --- /dev/null +++ b/src/Harp.Toolkit/Verify/HtmlReportGenerator.cs @@ -0,0 +1,21 @@ +using System.Reflection; +using RazorLight; + +namespace Harp.Toolkit.Verify; + +public static class HtmlReportGenerator +{ + public static async Task GenerateAsync(Report report) + { + var engine = new RazorLightEngineBuilder() + .UseFileSystemProject(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location)) + .UseMemoryCachingProvider() + .Build(); + + // The template is copied to the output directory under Verify/ReportTemplate.cshtml + // RazorLight expects the path relative to the project root (which we set to the assembly location) + string templatePath = Path.Combine("Verify", "ReportTemplate.cshtml"); + + return await engine.CompileRenderAsync(templatePath, report); + } +} diff --git a/src/Harp.Toolkit/Verify/ProtocolReference.cs b/src/Harp.Toolkit/Verify/ProtocolReference.cs new file mode 100644 index 0000000..43af721 --- /dev/null +++ b/src/Harp.Toolkit/Verify/ProtocolReference.cs @@ -0,0 +1,27 @@ +namespace Harp.Toolkit.Verify; + +/// +/// Identifies the specification text encoded by the conformance checks. +/// +internal static class ProtocolReference +{ + /// + /// The harp-tech/protocol commit carrying that specification text. + /// + public const string Commit = "11b584bdd2eb45a8b55ebc540512864ff0667326"; + + /// + /// The major protocol version that specification text targets. + /// + public const int PrereleaseMajorVersion = 2; + + /// + /// Gets the abbreviated commit, for display where a link carries the full reference. + /// + public static string ShortCommit => Commit[..7]; + + /// + /// Gets the address of the specification documents as they stood at that commit. + /// + public static string TreeUrl => $"https://github.com/harp-tech/protocol/tree/{Commit}"; +} diff --git a/src/Harp.Toolkit/Verify/ProtocolTarget.cs b/src/Harp.Toolkit/Verify/ProtocolTarget.cs new file mode 100644 index 0000000..58a3ed1 --- /dev/null +++ b/src/Harp.Toolkit/Verify/ProtocolTarget.cs @@ -0,0 +1,27 @@ +using Harp.Toolkit.Verify.Suites; + +namespace Harp.Toolkit.Verify; + +internal enum ProtocolScope +{ + V1, + V2, + Unsupported, +} + +internal readonly record struct ProtocolTarget(SemanticVersion? DeclaredVersion, bool PrereleaseRequested) +{ + public ProtocolScope Scope + { + get + { + var major = DeclaredVersion.HasValue ? DeclaredVersion.GetValueOrDefault().Major : 0; + if (major > ProtocolReference.PrereleaseMajorVersion) + return ProtocolScope.Unsupported; + + return major == ProtocolReference.PrereleaseMajorVersion ? ProtocolScope.V2 : ProtocolScope.V1; + } + } + + public bool IncludePrerelease => PrereleaseRequested; +} diff --git a/src/Harp.Toolkit/Verify/Report.cs b/src/Harp.Toolkit/Verify/Report.cs new file mode 100644 index 0000000..608d991 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Report.cs @@ -0,0 +1,19 @@ +namespace Harp.Toolkit.Verify; + +public class Report +{ + public string DeviceName { get; set; } = "Unknown Device"; + public string PortName { get; set; } = string.Empty; + public string WhoAmI { get; set; } = string.Empty; + public string HardwareVersion { get; set; } = string.Empty; + public string FirmwareVersion { get; set; } = string.Empty; + public DateTime RunDate { get; set; } = DateTime.Now; + public bool IncludePrerelease { get; set; } + public string ProtocolNotice { get; set; } = string.Empty; + public string DeclaredProtocolVersion { get; set; } = string.Empty; + public string CheckedProtocolVersion { get; set; } = string.Empty; + public string ProtocolCommit { get; set; } = string.Empty; + public string ProtocolCommitUrl { get; set; } = string.Empty; + public string RegisterSetVersion { get; set; } = string.Empty; + public List Suites { get; set; } = new(); +} diff --git a/src/Harp.Toolkit/Verify/ReportTemplate.cshtml b/src/Harp.Toolkit/Verify/ReportTemplate.cshtml new file mode 100644 index 0000000..29943a1 --- /dev/null +++ b/src/Harp.Toolkit/Verify/ReportTemplate.cshtml @@ -0,0 +1,141 @@ +@using Harp.Toolkit.Verify +@model Harp.Toolkit.Verify.Report + + + + + + + Test Report - @Model.DeviceName + + + + +
+
+

@Model.DeviceName

+

Harp conformance report • @Model.RunDate.ToString("MMMM dd, yyyy HH:mm:ss")

+
+ +
+
+
+
WhoAmI
+
@Model.WhoAmI
+
Serial port
+
@Model.PortName
+
Hardware version
+
@Model.HardwareVersion
+
Firmware version
+
@Model.FirmwareVersion
+
Protocol version declared
+
@Model.DeclaredProtocolVersion
+
Checked against
+
@Model.CheckedProtocolVersion
+
Specification
+
@Model.ProtocolCommit
+
Register set
+
Harp.Generators @Model.RegisterSetVersion
+
+
+
+ + @if (Model.ProtocolNotice.Length > 0) + { + + } + + @foreach (var suite in Model.Suites) + { +
+
+

@suite.Name

+

@suite.Description

+
+
+
+ + + + + + + + + + + @foreach (var test in suite.Results) + { + + + + + + + } + +
Test CaseStatusResult DetailsMessage
+
@test.Name
+
@test.Description
+ 		+ + @(test.Result?.Status.ToString().ToUpper() ?? "SKIPPED") + + + @if (test.Result is NumericBenchmarkResult bsr) + { +
+
+
Mean: @bsr.Summary.Mean.ToString("F4")
+
Median: @bsr.Summary.Median.ToString("F4")
+
StdDev: @bsr.Summary.StdDev.ToString("F4")
+
+
+
Min: @bsr.Summary.Min.ToString("F4")
+
Max: @bsr.Summary.Max.ToString("F4")
+
+
+ } + else + { + var valProp = test.Result?.GetType().GetProperty("Value"); + object? val = null; + if (valProp != null) + { + val = valProp.GetValue(test.Result); + } + + if (val != null) + { + @val + } + } + + @if (test.Result is ErrorResult er && er.Exception != null) + { +
+ 
@er.Exception.ToString()
+
+ } + 		+ @test.Result?.Message +
+
+
+
+ } +
+ + diff --git a/src/Harp.Toolkit/Verify/Result.cs b/src/Harp.Toolkit/Verify/Result.cs new file mode 100644 index 0000000..a5cb605 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Result.cs @@ -0,0 +1,161 @@ + +namespace Harp.Toolkit.Verify; + + +public enum Status +{ + Passed, + Failed, + Skipped, + Error +} + +public interface IResult +{ + string? Message { get; } + + Status Status { get; } +} + +public class ErrorResult(Exception exception) : IResult +{ + public string? Message { get; } = exception.Message; + public Status Status { get; } = Status.Error; + public Exception Exception { get; } = exception; +} + +public class Result : IResult +{ + public Result(T value, Status status, string message = "") + { + Status = status; + Value = value; + Message = message; + } + + public Result(T value, Func predicate, Func? messageFactory = null) + { + bool evaluation = predicate(value); + Status = evaluation ? Status.Passed : Status.Failed; + Value = value; + Message = messageFactory?.Invoke(value, evaluation) ?? string.Empty; + } + + + public string Message { get; } + public Status Status { get; } + public T Value { get; } + + public override string? ToString() + { + return $"Result(Status={Status}, Value={Value}, Message={Message})"; + } +} + + +public class AssertionResult : Result +{ + public AssertionResult(bool value, string message = "") + : base(value, value ? Status.Passed : Status.Failed, message) + { + } + + public AssertionResult(bool value, Func? messageFactory = null) + : base( + value, + v => v, + messageFactory is null ? null : ((value, evaluation) => messageFactory(value))) + { + + } +} + + +public class NumericBenchmarkResult : Result +{ + + public NumericBenchmarkResult(double[] values, Status status, string message = "") + : base(values, status, message) + { + Summary = new BenchmarkSummary(values); + } + + public NumericBenchmarkResult(BenchmarkSummary summary, Status status, string message = "") + : base(summary.Values, status, message) + { + Summary = summary; + } + + public NumericBenchmarkResult(double[] values, Func predicate, Func? messageFactory = null) + : base(values, predicate, messageFactory) + { + Summary = new BenchmarkSummary(values); + } + + public BenchmarkSummary Summary { get; } +} + + +public class BenchmarkSummary +{ + public readonly double[] Values; + + + public BenchmarkSummary(double[] values) + { + Values = values ?? Array.Empty(); + // TODO consider copying here since we are mutating + Array.Sort(Values); + } + + public double Mean => Values.Length == 0 ? double.NaN : Values.Average(); + + public double StdDev + { + get + { + if (Values.Length == 0) return double.NaN; + var mean = Mean; + var sumOfSquares = Values.Sum(v => (v - mean) * (v - mean)); + return Math.Sqrt(sumOfSquares / Values.Length); + } + } + + public double Median + { + get + { + if (Values.Length == 0) return double.NaN; + int mid = Values.Length / 2; + if (Values.Length % 2 == 0) + return (Values[mid - 1] + Values[mid]) / 2.0; + else + return Values[mid]; + } + } + + public double Max => Values.Length == 0 ? double.NaN : Values[Values.Length - 1]; + + public double Min => Values.Length == 0 ? double.NaN : Values[0]; + + public double Percentile99 => Percentile(0.99); + public double Percentile01 => Percentile(0.01); + + public double Percentile(double percentile) + { + if (Values.Length == 0) return double.NaN; + if (percentile < 0f || percentile > 1.0f) + { + throw new ArgumentOutOfRangeException(nameof(percentile), "Percentile must be between 0 and 1."); + } + + double rank = percentile * (Values.Length - 1); + int lower = (int)Math.Floor(rank); + int upper = (int)Math.Ceiling(rank); + if (lower == upper) return Values[lower]; + // Apparently this is how you solve rounding with percentiles + // https://en.wikipedia.org/wiki/Percentile + double weight = rank - lower; + return Values[lower] * (1 - weight) + Values[upper] * weight; + } +} diff --git a/src/Harp.Toolkit/Verify/Runner.cs b/src/Harp.Toolkit/Verify/Runner.cs new file mode 100644 index 0000000..ed5ca46 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Runner.cs @@ -0,0 +1,57 @@ +using System.Runtime.CompilerServices; + +namespace Harp.Toolkit.Verify; + +public class Runner +{ + private readonly List suites = new(); + private readonly bool includePrerelease; + + public Runner(bool includePrerelease) + { + this.includePrerelease = includePrerelease; + } + + public int TestCount => suites.Sum(s => s.GetTestCount(includePrerelease)); + + public int PrereleaseTestCount => suites.Sum(s => s.GetPrereleaseTestCount()); + + public IEnumerable CollectSuites() + { + return suites.AsReadOnly(); + } + + public async IAsyncEnumerable<(Suite Suite, MethodResult Result)> RunAllAsync(VerifyConnection connection, [EnumeratorCancellation] CancellationToken cancellationToken = default, Action? onTestStart = null) + { + foreach (var suite in suites) + { + await foreach (var result in suite.RunAllAsync(connection, includePrerelease, cancellationToken, (testName, testDesc) => onTestStart?.Invoke(suite, testName, testDesc))) + { + yield return (suite, result); + } + } + } + + public void AddSuite(Suite suite) + { + if (suite == null) + { + throw new ArgumentNullException(nameof(suite)); + } + suites.Add(suite); + } + + public void ClearSuites() + { + suites.Clear(); + } + + public bool RemoveSuite(Suite suite) + { + if (suite == null) + { + throw new ArgumentNullException(nameof(suite)); + } + return suites.Remove(suite); + } +} diff --git a/src/Harp.Toolkit/Verify/Suite.cs b/src/Harp.Toolkit/Verify/Suite.cs new file mode 100644 index 0000000..3a8b871 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suite.cs @@ -0,0 +1,117 @@ +using System.Reflection; +using System.Runtime.CompilerServices; +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify; + + +public abstract class Suite +{ + public abstract string Description { get; } + + /// + /// Tests whose number and identity is only known at runtime. + /// During test collection, these will be enumerated and run + /// after the fixed methods. + /// + protected virtual IReadOnlyList DynamicTests { get; } = new List(); + + public int GetTestCount(bool includePrerelease) => CollectTests(includePrerelease).Count() + DynamicTests.Count; + + public int GetPrereleaseTestCount() => CollectAllTests().Count(x => x.Attribute.Prerelease); + + private IEnumerable<(MethodInfo Method, HarpTestAttribute Attribute)> CollectAllTests() + { + return GetType() + .GetMethods(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance) + .Select(m => (Method: m, Attribute: m.GetCustomAttribute()!)) + .Where(x => x.Attribute != null); + } + + private IEnumerable<(MethodInfo Method, HarpTestAttribute Attribute)> CollectTests(bool includePrerelease) + { + return CollectAllTests().Where(x => includePrerelease || !x.Attribute.Prerelease); + } + + public async IAsyncEnumerable RunAllAsync(VerifyConnection connection, bool includePrerelease, [EnumeratorCancellation] CancellationToken cancellationToken = default, Action? onTestStart = null) + { + foreach (var (method, attr) in CollectTests(includePrerelease)) + { + cancellationToken.ThrowIfCancellationRequested(); + + // Notify that test is starting + onTestStart?.Invoke(method.Name, attr.Description ?? string.Empty); + + IResult testResult; + try + { + object? resultObj = method.Invoke(this, new object[] { connection }); + if (resultObj is Task task) + { + testResult = await task; + } + else if (resultObj is IResult syncResult) + { + testResult = syncResult; + } + else + { + throw new InvalidOperationException($"Test method '{method.Name}' must return IResult or Task."); + } + } + catch (Exception ex) + { + testResult = new ErrorResult(ex.InnerException ?? ex); + } + yield return new MethodResult + { + Result = testResult, + Name = method.Name, + Description = attr.Description ?? string.Empty + }; + } + + foreach (var test in DynamicTests) + { + cancellationToken.ThrowIfCancellationRequested(); + + onTestStart?.Invoke(test.Name, test.Description); + + IResult testResult; + try + { + testResult = await test.Run(connection, cancellationToken); + } + catch (Exception ex) + { + testResult = new ErrorResult(ex); + } + yield return new MethodResult + { + Result = testResult, + Name = test.Name, + Description = test.Description + }; + } + } +} + +/// +/// A test whose name and behavior is determined at runtime rather than declared +/// with on a fixed method. +/// +public record DynamicTest(string Name, string Description, Func> Run); + +public class SuiteResult +{ + public required string Name { get; set; } + public string Description { get; set; } = string.Empty; + public List Results { get; set; } = new(); +} + +public class MethodResult +{ + public required string Name { get; set; } + public string Description { get; set; } = string.Empty; + public required IResult Result { get; set; } +} diff --git a/src/Harp.Toolkit/Verify/Suites/ClockTestSuite.cs b/src/Harp.Toolkit/Verify/Suites/ClockTestSuite.cs new file mode 100644 index 0000000..dd63e30 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/ClockTestSuite.cs @@ -0,0 +1,92 @@ + +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class ClockTestSuite : Suite +{ + private readonly ClockTestOptions? options; + + public ClockTestSuite(ClockTestOptions? options) + { + this.options = options; + } + + public override string Description => "Tests clock alignment and PPS synchronization accuracy against a reference clock device."; + + [HarpTest(Description = "Sends 100 simultaneous WhoAmI reads to both devices and compares embedded timestamps to measure clock offset.")] + public async Task SimultaneousWhoAmI(VerifyConnection device) + { + if (options is null) + return new Result(false, Status.Skipped, "No clock port provided (--clock-port)."); + + const int n = 100; + double[] deltas = new double[n]; + var probe = WhoAmI.FromPayload(MessageType.Read, default); + + using var clockDevice = await VerifyConnection.OpenAsync(options.ClockPort); + + for (int i = 0; i < n; i++) + { + var results = await Task.WhenAll(device.CommandAsync(probe), clockDevice.CommandAsync(probe)); + deltas[i] = results[0].GetTimestamp() - results[1].GetTimestamp(); + await Task.Delay(Random.Shared.Next(20, 70)); + } + + var summary = new BenchmarkSummary(deltas); + return new NumericBenchmarkResult( + summary, + Status.Passed, + $"Clock offset: mean={summary.Mean:F6}s ({summary.Mean * 1e3:F3}ms), " + + $"stddev={summary.StdDev:F6}s ({summary.StdDev * 1e3:F3}ms)"); + } + + [HarpTest(Description = "Subscribes to PPS events on both devices and compares timestamps to measure hardware clock synchronization accuracy.")] + public async Task PpsEventAlignment(VerifyConnection device) + { + if (options is null) + return new Result(false, Status.Skipped, "No clock port provided (--clock-port)."); + if (!options.PpsEvent.HasValue) + return new Result(false, Status.Skipped, "No tested device register provided (--pps-event)."); + + // The clock device (WhiteRabbit) emits a TimestampSecond event on every PPS tick. + // ALIVE_EN (0x80) | OP_MODE (0x01) enables those events. + // TODO: consider also supporting Heartbeat via HEARTBEAT_EN (0x04) | OP_MODE (0x01). + const int clockDeviceReg = TimestampSeconds.Address; + const byte clockDeviceOpCtrl = 0x81; + + var listenDuration = TimeSpan.FromSeconds(options.ClockSamples + 5); + using var clockDevice = await VerifyConnection.OpenAsync(options.ClockPort); + + var allMessages = await Task.WhenAll( + clockDevice.WriteAndCollectAsync( + [HarpMessage.FromByte(OperationControl.Address, MessageType.Write, clockDeviceOpCtrl)], + listenDuration), + device.WriteAndCollectAsync( + [HarpMessage.FromByte(OperationControl.Address, MessageType.Write, 0x01)], + listenDuration)); + + var clockEvents = allMessages[0] + .Where(m => m.MessageType == MessageType.Event && m.Address == clockDeviceReg) + .Take(options.ClockSamples).ToList(); + var testedEvents = allMessages[1] + .Where(m => m.MessageType == MessageType.Event && m.Address == options.PpsEvent!.Value) + .Take(options.ClockSamples).ToList(); + + int pairCount = Math.Min(clockEvents.Count, testedEvents.Count); + if (pairCount == 0) + return new AssertionResult(false, + $"No event pairs received within {listenDuration.TotalSeconds}s " + + $"(clock register {clockDeviceReg}, tested register {options.PpsEvent!.Value})."); + + var deltas = clockEvents + .Zip(testedEvents, (c, t) => c.GetTimestamp() - t.GetTimestamp()) + .ToArray(); + var summary = new BenchmarkSummary(deltas); + return new NumericBenchmarkResult( + summary, + Status.Passed, + $"PPS delta ({pairCount} pairs): mean={summary.Mean:F6}s ({summary.Mean * 1e3:F3}ms), " + + $"stddev={summary.StdDev:F6}s ({summary.StdDev * 1e3:F3}ms)"); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_ASSEMBLY_VERSION.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_ASSEMBLY_VERSION.cs new file mode 100644 index 0000000..205ebae --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_ASSEMBLY_VERSION.cs @@ -0,0 +1,17 @@ +namespace Harp.Toolkit.Verify.Suites; + +internal class R_ASSEMBLY_VERSION : Suite +{ + public override string Description => "AssemblyVersion Register Tests"; + + [HarpTest(Description = "Validates the deprecated register AssemblyVersion returns 0x00.")] + public async Task AssertReturnsZero(VerifyConnection device) + { + var value = await device.ReadAssemblyVersionAsync(); + return new AssertionResult( + value == 0x00, + x => x ? + "AssemblyVersion register correctly returned 0x00." : + $"AssemblyVersion register returned a non-zero value (0x{value:X2})."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CLOCK_CONFIG.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CLOCK_CONFIG.cs new file mode 100644 index 0000000..ccbd64d --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CLOCK_CONFIG.cs @@ -0,0 +1,32 @@ +using System.Text; +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_CLOCK_CONFIG : Suite +{ + public override string Description => "Clock Configuration Register Tests"; + + [HarpTest(Description = "Validates that ClockConfig register is readable.")] + public async Task IsReadable(VerifyConnection device) + { + return await RegisterHelpers.AssertReadableAsync(a => device.ReadByteAsync(a), ClockConfiguration.Address, "ClockConfig"); + } + + [HarpTest(Description = "Reports clock synchronization capability: REP_ABLE (bit 3) and GEN_ABLE (bit 4).")] + public async Task ReportSyncCapability(VerifyConnection device) + { + var value = await device.ReadByteAsync(ClockConfiguration.Address); + bool repAble = (value & (1 << 3)) != 0; + bool genAble = (value & (1 << 4)) != 0; + StringBuilder sb = new StringBuilder("ClockConfig sync capability:"); + sb.Append("\n"); + sb.Append(repAble ? "Device can repeat clock signal" : "Device cannot repeat clock signal"); + sb.Append("\n"); + sb.Append(genAble ? "Device can generate clock signal" : "Device cannot generate clock signal"); + sb.Append("\n"); + return new AssertionResult( + true, + sb.ToString()); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CORE_VERSION_H.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CORE_VERSION_H.cs new file mode 100644 index 0000000..8126972 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CORE_VERSION_H.cs @@ -0,0 +1,20 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_CORE_VERSION_H : Suite +{ + public override string Description => "Core Version High Register Tests"; + + [HarpTest(Description = "Validates that CoreVersionHigh matches byte 0 of R_VERSION.", Prerelease = true)] + public async Task AssertConsistentWithVersion(VerifyConnection device) + { + var versionArray = await device.ReadByteArrayAsync(Version.Address); + var registerValue = await device.ReadByteAsync(CoreVersionHigh.Address); + return new AssertionResult( + registerValue == versionArray[0], + x => x + ? $"CoreVersionHigh (0x{registerValue:X2}) matches R_VERSION byte 0." + : $"CoreVersionHigh (0x{registerValue:X2}) does not match R_VERSION byte 0 (0x{versionArray[0]:X2})."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CORE_VERSION_L.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CORE_VERSION_L.cs new file mode 100644 index 0000000..cb20f61 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_CORE_VERSION_L.cs @@ -0,0 +1,20 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_CORE_VERSION_L : Suite +{ + public override string Description => "Core Version Low Register Tests"; + + [HarpTest(Description = "Validates that CoreVersionLow matches byte 1 of R_VERSION.", Prerelease = true)] + public async Task AssertConsistentWithVersion(VerifyConnection device) + { + var versionArray = await device.ReadByteArrayAsync(Version.Address); + var registerValue = await device.ReadByteAsync(CoreVersionLow.Address); + return new AssertionResult( + registerValue == versionArray[1], + x => x + ? $"CoreVersionLow (0x{registerValue:X2}) matches R_VERSION byte 1." + : $"CoreVersionLow (0x{registerValue:X2}) does not match R_VERSION byte 1 (0x{versionArray[1]:X2})."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_DEVICE_NAME.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_DEVICE_NAME.cs new file mode 100644 index 0000000..851942c --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_DEVICE_NAME.cs @@ -0,0 +1,28 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_DEVICE_NAME : Suite +{ + public override string Description => "Device Name Register Tests"; + + [HarpTest(Description = "Validates that DeviceName register is readable.")] + public async Task IsReadable(VerifyConnection device) + { + try + { + await device.ReadByteArrayAsync(DeviceName.Address); + return new AssertionResult(true, "DeviceName is readable."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } + + [HarpTest(Description = "Validates that DeviceName register has exactly 25 bytes.")] + public async Task AssertLength(VerifyConnection device) + { + return await RegisterHelpers.AssertReadableArrayAsync(device, DeviceName.Address, DeviceName.RegisterLength, "DeviceName"); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_FW_VERSION_H.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_FW_VERSION_H.cs new file mode 100644 index 0000000..b6ac7ff --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_FW_VERSION_H.cs @@ -0,0 +1,20 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_FW_VERSION_H : Suite +{ + public override string Description => "Firmware Version High Register Tests"; + + [HarpTest(Description = "Validates that FwVersionHigh matches byte 3 of R_VERSION.", Prerelease = true)] + public async Task AssertConsistentWithVersion(VerifyConnection device) + { + var versionArray = await device.ReadByteArrayAsync(Version.Address); + var registerValue = await device.ReadByteAsync(FirmwareVersionHigh.Address); + return new AssertionResult( + registerValue == versionArray[3], + x => x + ? $"FwVersionHigh (0x{registerValue:X2}) matches R_VERSION byte 3." + : $"FwVersionHigh (0x{registerValue:X2}) does not match R_VERSION byte 3 (0x{versionArray[3]:X2})."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_FW_VERSION_L.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_FW_VERSION_L.cs new file mode 100644 index 0000000..1990d5b --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_FW_VERSION_L.cs @@ -0,0 +1,20 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_FW_VERSION_L : Suite +{ + public override string Description => "Firmware Version Low Register Tests"; + + [HarpTest(Description = "Validates that FwVersionLow matches byte 4 of R_VERSION.", Prerelease = true)] + public async Task AssertConsistentWithVersion(VerifyConnection device) + { + var versionArray = await device.ReadByteArrayAsync(Version.Address); + var registerValue = await device.ReadByteAsync(FirmwareVersionLow.Address); + return new AssertionResult( + registerValue == versionArray[4], + x => x + ? $"FwVersionLow (0x{registerValue:X2}) matches R_VERSION byte 4." + : $"FwVersionLow (0x{registerValue:X2}) does not match R_VERSION byte 4 (0x{versionArray[4]:X2})."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HEARTBEAT.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HEARTBEAT.cs new file mode 100644 index 0000000..13c224c --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HEARTBEAT.cs @@ -0,0 +1,27 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_HEARTBEAT : Suite +{ + internal const byte Address = 18; + public override string Description => "Heartbeat Register Tests"; + + [HarpTest(Description = "Validates that Heartbeat register is readable.", Prerelease = true)] + public async Task IsReadable(VerifyConnection device) + { + return await RegisterHelpers.AssertReadableAsync(a => device.ReadUInt16Async(a), Address, "Heartbeat"); + } + + [HarpTest(Description = "Validates that Heartbeat register is NOT writable.", Prerelease = true)] + public async Task IsNotWritable(VerifyConnection device) + { + var req = HarpMessage.FromUInt16(Address, MessageType.Write, 0); + var rejected = await RegisterHelpers.IsWriteRejectedAsync(device, req); + return new AssertionResult( + rejected, + x => x + ? "Heartbeat register correctly rejected write." + : "Heartbeat register should NOT be writable."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HW_VERSION_H.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HW_VERSION_H.cs new file mode 100644 index 0000000..9792179 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HW_VERSION_H.cs @@ -0,0 +1,20 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_HW_VERSION_H : Suite +{ + public override string Description => "Hardware Version High Register Tests"; + + [HarpTest(Description = "Validates that HwVersionHigh matches byte 6 of R_VERSION.", Prerelease = true)] + public async Task AssertConsistentWithVersion(VerifyConnection device) + { + var versionArray = await device.ReadByteArrayAsync(Version.Address); + var registerValue = await device.ReadByteAsync(HardwareVersionHigh.Address); + return new AssertionResult( + registerValue == versionArray[6], + x => x + ? $"HwVersionHigh (0x{registerValue:X2}) matches R_VERSION byte 6." + : $"HwVersionHigh (0x{registerValue:X2}) does not match R_VERSION byte 6 (0x{versionArray[6]:X2})."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HW_VERSION_L.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HW_VERSION_L.cs new file mode 100644 index 0000000..1468531 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_HW_VERSION_L.cs @@ -0,0 +1,20 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_HW_VERSION_L : Suite +{ + public override string Description => "Hardware Version Low Register Tests"; + + [HarpTest(Description = "Validates that HwVersionLow matches byte 7 of R_VERSION.", Prerelease = true)] + public async Task AssertConsistentWithVersion(VerifyConnection device) + { + var versionArray = await device.ReadByteArrayAsync(Version.Address); + var registerValue = await device.ReadByteAsync(HardwareVersionLow.Address); + return new AssertionResult( + registerValue == versionArray[7], + x => x + ? $"HwVersionLow (0x{registerValue:X2}) matches R_VERSION byte 7." + : $"HwVersionLow (0x{registerValue:X2}) does not match R_VERSION byte 7 (0x{versionArray[7]:X2})."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_OPERATION_CTRL.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_OPERATION_CTRL.cs new file mode 100644 index 0000000..87ce1e2 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_OPERATION_CTRL.cs @@ -0,0 +1,245 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_OPERATION_CTRL : Suite +{ + public override string Description => "Operation Control Register Tests"; + + [HarpTest(Description = "Validates that OP_MODE bits can be round-tripped between Standby (0) and Active (1).")] + public async Task OpModeRoundTrip(VerifyConnection device) + { + var original = await device.ReadByteAsync(OperationControl.Address); + byte currentMode = (byte)(original & 0x03); + byte newMode = currentMode == 0x01 ? (byte)0x00 : (byte)0x01; + byte newValue = (byte)((original & ~0x03) | newMode); + + try + { + await device.CommandAsync(HarpMessage.FromByte(OperationControl.Address, MessageType.Write, newValue)); + var readBack = await device.ReadByteAsync(OperationControl.Address); + byte readMode = (byte)(readBack & 0x03); + + return new AssertionResult( + readMode == newMode, + x => x + ? $"OpModeRoundTrip: OP_MODE correctly round-tripped to {newMode}." + : $"OpModeRoundTrip: wrote OP_MODE={newMode}, read back OP_MODE={readMode}."); + } + finally + { + await RestoreOperationControlAsync(device, original); + } + } + + [HarpTest(Description = "Validates that ALIVE_EN (deprecated, bit 7) can be toggled, or reports as unsupported.")] + public async Task AliveEnWritable(VerifyConnection device) + { + return await TestOptionalBitAsync(device, "AliveEn", 0x80); + } + + [HarpTest(Description = "Validates that OPLED_EN (optional, bit 6) can be toggled, or reports as unsupported.")] + public async Task OpLedEnWritable(VerifyConnection device) + { + return await TestOptionalBitAsync(device, "OpLedEn", 0x40); + } + + [HarpTest(Description = "Validates that VISUAL_EN (optional, bit 5) can be toggled, or reports as unsupported.")] + public async Task VisualEnWritable(VerifyConnection device) + { + return await TestOptionalBitAsync(device, "VisualEn", 0x20); + } + + [HarpTest(Description = "Validates that enabling HEARTBEAT_EN causes the device to emit R_HEARTBEAT events.", Prerelease = true)] + public async Task HeartbeatEnEmitsEvents(VerifyConnection device) + { + byte? originalOpCtrl = null; + + try + { + originalOpCtrl = await device.ReadByteAsync(OperationControl.Address); + var messages = await device.WriteAndCollectAsync( + new[] { HarpMessage.FromByte(OperationControl.Address, MessageType.Write, 0x05) }, + TimeSpan.FromSeconds(2.0)); + + bool received = messages.Any(m => m.Address == R_HEARTBEAT.Address && m.MessageType == MessageType.Event); + + return new AssertionResult( + received, + x => x + ? "HeartbeatEnEmitsEvents: heartbeat event received within 2s." + : "HeartbeatEnEmitsEvents: no heartbeat event received within 2s."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + finally + { + await RestoreOperationControlAsync(device, originalOpCtrl); + } + } + + [HarpTest(Description = "Validates that HEARTBEAT_EN (bit 2) takes precedence over ALIVE_EN (bit 7): when both are set, R_HEARTBEAT events are emitted and R_TIMESTAMP_SECOND events are not.", Prerelease = true)] + public async Task HeartbeatEnPrecedenceOverAliveEn(VerifyConnection device) + { + byte? originalOpCtrl = null; + + try + { + originalOpCtrl = await device.ReadByteAsync(OperationControl.Address); + // Set both ALIVE_EN (bit 7) and HEARTBEAT_EN (bit 2) with Active mode (bit 0) + var messages = await device.WriteAndCollectAsync( + new[] { HarpMessage.FromByte(OperationControl.Address, MessageType.Write, 0x85) }, + TimeSpan.FromSeconds(2.0)); + + bool receivedHeartbeat = messages.Any(m => m.Address == R_HEARTBEAT.Address && m.MessageType == MessageType.Event); + bool receivedTimestamp = messages.Any(m => m.Address == TimestampSeconds.Address && m.MessageType == MessageType.Event); + + if (!receivedHeartbeat) + return new AssertionResult(false, "HeartbeatEnPrecedenceOverAliveEn: no R_HEARTBEAT event received within 2s (expected HEARTBEAT_EN to take precedence)."); + if (receivedTimestamp) + return new AssertionResult(false, "HeartbeatEnPrecedenceOverAliveEn: R_TIMESTAMP_SECOND event received when HEARTBEAT_EN should suppress it."); + + return new AssertionResult(true, "HeartbeatEnPrecedenceOverAliveEn: R_HEARTBEAT events received and R_TIMESTAMP_SECOND correctly suppressed."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + finally + { + await RestoreOperationControlAsync(device, originalOpCtrl); + } + } + + [HarpTest(Description = "Validates that ALIVE_EN (deprecated, bit 7) causes R_TIMESTAMP_SECOND events to be emitted when HEARTBEAT_EN is not set.")] + public async Task AliveEnEmitsTimestampEvents(VerifyConnection device) + { + byte? originalOpCtrl = null; + + try + { + originalOpCtrl = await device.ReadByteAsync(OperationControl.Address); + // Set only ALIVE_EN (bit 7) with Active mode (bit 0); HEARTBEAT_EN (bit 2) is cleared + var messages = await device.WriteAndCollectAsync( + new[] { HarpMessage.FromByte(OperationControl.Address, MessageType.Write, 0x81) }, + TimeSpan.FromSeconds(2.0)); + + bool receivedTimestamp = messages.Any(m => m.Address == TimestampSeconds.Address && m.MessageType == MessageType.Event); + + if (!receivedTimestamp) + return new Result(false, Status.Skipped, "AliveEnEmitsTimestampEvents: ALIVE_EN is deprecated and R_TIMESTAMP_SECOND events were not emitted."); + + return new AssertionResult(true, "AliveEnEmitsTimestampEvents: R_TIMESTAMP_SECOND event received within 2s."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + finally + { + await RestoreOperationControlAsync(device, originalOpCtrl); + } + } + + [HarpTest(Description = "Validates that the DUMP bit triggers a burst of all core register reads after an OpCtrl write.")] + public async Task RegisterDump(VerifyConnection device) + { + byte? originalOpCtrl = null; + + try + { + originalOpCtrl = await device.ReadByteAsync(OperationControl.Address); + var messages = await device.WriteAndCollectAsync( + new[] { HarpMessage.FromByte(OperationControl.Address, MessageType.Write, (byte)(originalOpCtrl.GetValueOrDefault() | 0x08)) }, + TimeSpan.FromSeconds(1)); + + var opRegWriteResponse = messages.FirstOrDefault(m => m.Address == OperationControl.Address && m.MessageType == MessageType.Write); + if (opRegWriteResponse == null) + { + return new AssertionResult(false, "No response received for OpCtrl write."); + } + var replies = messages + .Where(m => m.MessageType == MessageType.Read) + .GroupBy(m => m.Address) + .ToDictionary(g => g.Key, g => g.First()); + + var declared = CoreSchema.Metadata.Registers.Values; + var missing = declared + .Where(r => !replies.ContainsKey(r.Address)) + .Select(r => r.Address) + .OrderBy(a => a) + .ToList(); + if (missing.Count > 0) + return new AssertionResult(false, + $"Missing Read replies for {missing.Count} declared core address(es): {string.Join(", ", missing)}."); + + var mismatched = declared + .Select(r => (Register: r, Replied: replies[r.Address].PayloadType & ~PayloadType.Timestamp)) + .Where(x => x.Replied != x.Register.Type) + .Select(x => $"address {x.Register.Address} declares {x.Register.Type} but replied {x.Replied}") + .ToList(); + if (mismatched.Count > 0) + return new AssertionResult(false, + $"Payload type mismatch on {mismatched.Count} core register(s): {string.Join("; ", mismatched)}."); + + return new AssertionResult(true, + $"All {declared.Count} declared core registers were dumped with the expected payload type."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + finally + { + await RestoreOperationControlAsync(device, originalOpCtrl); + } + } + + private static async Task RestoreOperationControlAsync(VerifyConnection device, byte? value) + { + if (!value.HasValue) + return; + + try + { + await device.CommandAsync(HarpMessage.FromByte(OperationControl.Address, MessageType.Write, value.GetValueOrDefault())); + } + catch + { + } + } + + private static async Task TestOptionalBitAsync(VerifyConnection device, string bitName, byte bitMask) + { + var original = await device.ReadByteAsync(OperationControl.Address); + byte toggled = (byte)(original ^ bitMask); + + try + { + try + { + await device.CommandAsync(HarpMessage.FromByte(OperationControl.Address, MessageType.Write, toggled)); + } + catch (HarpException) + { + return new Result(false, Status.Skipped, + $"{bitName} is optional/deprecated and not supported by this device."); + } + + var readBack = await device.ReadByteAsync(OperationControl.Address); + bool bitChanged = (readBack & bitMask) == (toggled & bitMask); + + return new AssertionResult( + bitChanged, + x => x + ? $"{bitName}: bit correctly toggled." + : $"{bitName}: bit did not change after write (expected {(toggled & bitMask) != 0}, got {(readBack & bitMask) != 0})."); + } + finally + { + await RestoreOperationControlAsync(device, original); + } + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_RESET_DEV.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_RESET_DEV.cs new file mode 100644 index 0000000..d7c1e96 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_RESET_DEV.cs @@ -0,0 +1,84 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_RESET_DEV : Suite +{ + const ResetFlags BootProvenance = ResetFlags.BootFromDefault | ResetFlags.BootFromEeprom; + + public override string Description => "Reset Device Register Tests"; + + [HarpTest(Description = "Validates that ResetDev register is readable.")] + public async Task IsReadable(VerifyConnection device) + { + return await RegisterHelpers.AssertReadableAsync(a => device.ReadByteAsync(a), ResetDevice.Address, "ResetDev"); + } + + [HarpTest(Description = "Validates that a read of ResetDev clears every command bit and reports exactly one boot provenance bit.")] + public async Task AssertBootProvenanceReported(VerifyConnection device) + { + var value = await device.ReadByteAsync(ResetDevice.Address); + var flags = (ResetFlags)value; + return new AssertionResult( + flags == ResetFlags.BootFromDefault || flags == ResetFlags.BootFromEeprom, + x => x + ? $"ResetDev read 0x{value:X2}, reporting {DescribeBootSource(flags)}." + : $"ResetDev read 0x{value:X2}. {DescribeReadViolation(flags)}"); + } + + [HarpTest(Description = "Validates that ResetDev rejects a write setting BOOT_DEF, which is read-only state.", Prerelease = true)] + public async Task BootFromDefaultIsNotWritable(VerifyConnection device) + { + return await AssertReadOnlyBitRejectedAsync(device, ResetFlags.BootFromDefault, "BOOT_DEF"); + } + + [HarpTest(Description = "Validates that ResetDev rejects a write setting BOOT_EE, which is read-only state.", Prerelease = true)] + public async Task BootFromEepromIsNotWritable(VerifyConnection device) + { + return await AssertReadOnlyBitRejectedAsync(device, ResetFlags.BootFromEeprom, "BOOT_EE"); + } + + static async Task AssertReadOnlyBitRejectedAsync(VerifyConnection device, ResetFlags bit, string bitName) + { + var request = HarpMessage.FromByte(ResetDevice.Address, MessageType.Write, (byte)bit); + var rejected = await RegisterHelpers.IsWriteRejectedAsync(device, request); + return new AssertionResult( + rejected, + x => x + ? $"ResetDev correctly rejected a write setting {bitName}." + : $"ResetDev accepted a write setting {bitName}, which is read-only state and must be answered with an error reply."); + } + + static string DescribeBootSource(ResetFlags flags) + { + return flags == ResetFlags.BootFromEeprom + ? "a boot from register values stored in non-volatile memory" + : "a boot from default register values"; + } + + static string DescribeReadViolation(ResetFlags flags) + { + var provenance = flags & BootProvenance; + if (provenance == BootProvenance) + return "Both BOOT_DEF and BOOT_EE are set, so the reported boot provenance is contradictory."; + if (provenance == 0) + return "Neither BOOT_DEF nor BOOT_EE is set, so no boot provenance is reported."; + + var commandBits = DescribeCommandBits(flags); + if (commandBits.Length > 0) + return $"A read reply must clear every command bit, but {commandBits} remained set."; + + var remaining = (byte)(flags & ~BootProvenance); + return $"A read reply must clear every bit outside the boot provenance field, but 0x{remaining:X2} remained set."; + } + + static string DescribeCommandBits(ResetFlags flags) + { + var names = new List(); + if (flags.HasFlag(ResetFlags.RestoreDefault)) names.Add("RST_DEF"); + if (flags.HasFlag(ResetFlags.RestoreEeprom)) names.Add("RST_EE"); + if (flags.HasFlag(ResetFlags.Save)) names.Add("SAVE"); + if (flags.HasFlag(ResetFlags.RestoreName)) names.Add("NAME_TO_DEFAULT"); + return string.Join(", ", names); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_SERIAL_NUMBER.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_SERIAL_NUMBER.cs new file mode 100644 index 0000000..dab5c5c --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_SERIAL_NUMBER.cs @@ -0,0 +1,14 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_SERIAL_NUMBER : Suite +{ + public override string Description => "Serial Number Register Tests"; + + [HarpTest(Description = "Validates that SerialNumber register is readable.")] + public async Task IsReadable(VerifyConnection device) + { + return await RegisterHelpers.AssertReadableAsync(a => device.ReadUInt16Async(a), SerialNumber.Address, "SerialNumber"); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TAG.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TAG.cs new file mode 100644 index 0000000..fc776a4 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TAG.cs @@ -0,0 +1,42 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_TAG : Suite +{ + private const byte Address = 17; + private const int ExpectedLength = 8; + public override string Description => "Tag Register Tests"; + + [HarpTest(Description = "Validates that Tag register is readable.", Prerelease = true)] + public async Task IsReadable(VerifyConnection device) + { + try + { + await device.ReadByteArrayAsync(Address); + return new AssertionResult(true, "Tag is readable."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } + + [HarpTest(Description = "Validates that Tag register has exactly 8 bytes.", Prerelease = true)] + public async Task AssertLength(VerifyConnection device) + { + return await RegisterHelpers.AssertReadableArrayAsync(device, Address, ExpectedLength, "Tag"); + } + + [HarpTest(Description = "Validates that Tag register is NOT writable.", Prerelease = true)] + public async Task IsNotWritable(VerifyConnection device) + { + var req = HarpMessage.FromByte(Address, MessageType.Write, new byte[ExpectedLength]); + var rejected = await RegisterHelpers.IsWriteRejectedAsync(device, req); + return new AssertionResult( + rejected, + x => x + ? "Tag register correctly rejected write." + : "Tag register should NOT be writable."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_MICRO.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_MICRO.cs new file mode 100644 index 0000000..42736b3 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_MICRO.cs @@ -0,0 +1,45 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_TIMESTAMP_MICRO : Suite +{ + public override string Description => "Timestamp Microseconds Register Tests"; + + [HarpTest(Description = "Validates that TimestampMicro register is readable.")] + public async Task IsReadable(VerifyConnection device) + { + try + { + await device.ReadUInt16Async(TimestampMicroseconds.Address); + return new AssertionResult(true, "TimestampMicro is readable."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } + + [HarpTest(Description = "Validates that TimestampMicro register is NOT writable.")] + public async Task IsNotWritable(VerifyConnection device) + { + var req = HarpMessage.FromUInt16(TimestampMicroseconds.Address, MessageType.Write, 0); + var rejected = await RegisterHelpers.IsWriteRejectedAsync(device, req); + return new AssertionResult( + rejected, + x => x + ? "TimestampMicro register correctly rejected write." + : "TimestampMicro register should NOT be writable."); + } + + [HarpTest(Description = "Validates that TimestampMicro value is within bounds (0 to 31249).")] + public async Task ValueWithinBounds(VerifyConnection device) + { + var microValue = await device.ReadUInt16Async(TimestampMicroseconds.Address); + return new AssertionResult( + microValue < 31250, + x => x + ? $"TimestampMicro value ({microValue}) is within expected bounds (< 31250)." + : $"TimestampMicro value ({microValue}) exceeds expected maximum (31249)."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_OFFSET.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_OFFSET.cs new file mode 100644 index 0000000..af2213d --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_OFFSET.cs @@ -0,0 +1,18 @@ +namespace Harp.Toolkit.Verify.Suites; + +internal class R_TIMESTAMP_OFFSET : Suite +{ + private const byte Address = 15; + public override string Description => "Timestamp Offset Register Tests"; + + [HarpTest(Description = "Validates the deprecated register TimestampOffset returns 0x00.", Prerelease = true)] + public async Task AssertReturnsZero(VerifyConnection device) + { + var value = await device.ReadByteAsync(Address); + return new AssertionResult( + value == 0x00, + x => x ? + "TimestampOffset register correctly returned 0x00." : + $"TimestampOffset register returned a non-zero value (0x{value:X2})."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_SECOND.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_SECOND.cs new file mode 100644 index 0000000..4a6472f --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_TIMESTAMP_SECOND.cs @@ -0,0 +1,74 @@ + +using Bonsai.Harp; +namespace Harp.Toolkit.Verify.Suites; + +internal class R_TIMESTAMP_SECOND : Suite +{ + public override string Description => "Timestamp Seconds Register Tests"; + + [HarpTest(Description = "Validates that the Timestamp Seconds register is writable.")] + public async Task IsWritable(VerifyConnection device) + { + const uint setSeconds = 42; + const double maximumElapsedSeconds = 2.0; + await device.WriteTimestampSecondsAsync(setSeconds); + await Task.Delay(1); + HarpMessage response = await device.CommandAsync(TimestampSeconds.FromPayload(MessageType.Read, default)); + double readSeconds = response.GetTimestamp(); + double elapsedSeconds = readSeconds - setSeconds; + return new AssertionResult( + elapsedSeconds >= 0 && elapsedSeconds < maximumElapsedSeconds, + (success) => success + ? "TimestampSeconds register is writable and updates as expected." + : $"Wrote {setSeconds} to TimestampSeconds and the reply timestamp was {readSeconds:F6}, " + + $"outside the expected range of {setSeconds} to {setSeconds + maximumElapsedSeconds}."); + } + + [HarpTest(Description = "Validates that TimestampSeconds register is readable.")] + public async Task IsReadable(VerifyConnection device) + { + try + { + await device.ReadTimestampSecondsAsync(); + return new AssertionResult(true, "TimestampSeconds is readable."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } + + [HarpTest(Description = "Validates that TimestampSeconds register is monotonically non-decreasing.")] + public async Task IsMonotonic(VerifyConnection device) + { + var first = await device.ReadTimestampSecondsAsync(); + await Task.Delay(100); + var second = await device.ReadTimestampSecondsAsync(); + return new AssertionResult( + second >= first, + x => x + ? $"TimestampSeconds is monotonic: {first} -> {second}." + : $"TimestampSeconds decreased from {first} to {second}."); + } + + [HarpTest(Description = "Validates that writing a past timestamp value takes effect and can be read back.")] + public async Task WritePastValueRoundTrip(VerifyConnection device) + { + const long maximumElapsedSeconds = 1; + var current = await device.ReadTimestampSecondsAsync(); + var tPast = current >= 10 ? current - 10 : 0u; + + await device.WriteTimestampSecondsAsync(tPast); + await Task.Delay(50); + + var readBack = await device.ReadTimestampSecondsAsync(); + var elapsedSeconds = (long)readBack - tPast; + + return new AssertionResult( + elapsedSeconds >= 0 && elapsedSeconds <= maximumElapsedSeconds, + x => x + ? $"Wrote {tPast} to TimestampSeconds and read it back as {readBack}." + : $"Wrote {tPast} to TimestampSeconds and read it back as {readBack}, " + + $"outside the expected range of {tPast} to {tPast + maximumElapsedSeconds}."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_UID.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_UID.cs new file mode 100644 index 0000000..bbaa994 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_UID.cs @@ -0,0 +1,30 @@ +namespace Harp.Toolkit.Verify.Suites; + +internal class R_UID : Suite +{ + internal const byte Address = 16; + private const byte ExpectedLength = 16; + public override string Description => "UID Register Tests"; + + [HarpTest(Description = "Validates that UID register has exactly 16 bytes.", Prerelease = true)] + public async Task AssertLength(VerifyConnection device) + { + var value = await device.ReadByteArrayAsync(Address); + return new AssertionResult( + value.Length == ExpectedLength, + x => x ? + $"Length is {ExpectedLength} as expected." : + $"Expected length of register to be {ExpectedLength}, got {value.Length} instead."); + } + + [HarpTest(Description = "Checks if the register value is 0, indicating it is likely not used.", Prerelease = true)] + public async Task AssertReturnsZero(VerifyConnection device) + { + var value = await device.ReadByteArrayAsync(Address); + string msg = value.All(x => x == 0) ? "Value of all bytes is 0. Register likely not being used." : $"Register returned a non-zero value: {BitConverter.ToString(value)}."; + return new Result( + value, + Status.Passed, + msg); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_VERSION.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_VERSION.cs new file mode 100644 index 0000000..b4e5e38 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_VERSION.cs @@ -0,0 +1,96 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal class R_VERSION : Suite +{ + public override string Description => "Version Register Tests"; + + [HarpTest(Description = "Validates that Version register is readable.", Prerelease = true)] + public async Task IsReadable(VerifyConnection device) + { + try + { + await device.ReadByteArrayAsync(Version.Address); + return new AssertionResult(true, "Version is readable."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } + + [HarpTest(Description = "Validates that Version register has exactly 32 bytes.", Prerelease = true)] + public async Task AssertLength(VerifyConnection device) + { + return await RegisterHelpers.AssertReadableArrayAsync(device, Version.Address, Version.RegisterLength, "Version"); + } + + [HarpTest(Description = "Validates that Version register is NOT writable.", Prerelease = true)] + public async Task IsNotWritable(VerifyConnection device) + { + var req = HarpMessage.FromByte(Version.Address, MessageType.Write, new byte[Version.RegisterLength]); + var rejected = await RegisterHelpers.IsWriteRejectedAsync(device, req); + return new AssertionResult( + rejected, + x => x + ? "Version register correctly rejected write." + : "Version register should NOT be writable."); + } + + [HarpTest(Description = "Validates that Version register declares the protocol major version being checked.", Prerelease = true)] + public async Task AssertDeclaresCheckedMajorVersion(VerifyConnection device) + { + try + { + var reply = await device.CommandAsync(HarpCommand.ReadByte(Version.Address)); + var payload = reply.GetPayloadArray(); + if (payload.Length != Version.RegisterLength) + { + return new AssertionResult( + false, + $"Version returned {payload.Length} bytes, expected {Version.RegisterLength}."); + } + + var declared = Version.GetPayload(reply).ProtocolVersion; + return new AssertionResult( + declared.Major == ProtocolReference.PrereleaseMajorVersion, + x => x + ? $"Version declares protocol {declared}, matching the major version being checked." + : $"Version declares protocol {declared}, but these checks are against major " + + $"version {ProtocolReference.PrereleaseMajorVersion}."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } + + [HarpTest(Description = "Reports the version information declared by the device.", Prerelease = true)] + public async Task ReportVersionInformation(VerifyConnection device) + { + try + { + var reply = await device.CommandAsync(HarpCommand.ReadByte(Version.Address)); + var payload = reply.GetPayloadArray(); + if (payload.Length != Version.RegisterLength) + { + return new AssertionResult( + false, + $"Version returned {payload.Length} bytes, expected {Version.RegisterLength}."); + } + + var version = Version.GetPayload(reply); + return new Result( + version, + Status.Passed, + $"PROTOCOL {version.ProtocolVersion}, FIRMWARE {version.FirmwareVersion}, " + + $"HARDWARE {version.HardwareVersion}, CORE_ID {version.CoreId}, " + + $"INTERFACE_HASH {Convert.ToHexString(version.InterfaceHash)}."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_WHO_AM_I.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_WHO_AM_I.cs new file mode 100644 index 0000000..ffb216e --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/R_WHO_AM_I.cs @@ -0,0 +1,30 @@ + +using Bonsai.Harp; +namespace Harp.Toolkit.Verify.Suites; + +internal class R_WHO_AM_I : Suite +{ + public override string Description => "WhoAmI Register Tests"; + + [HarpTest(Description = "Validates that the WhoAmI register exists and contains a value.")] + public async Task CheckWhoAmI(VerifyConnection device) + { + int value = await device.ReadWhoAmIAsync(); + return new Result( + value, + (v) => v > 0 && v < 9999, + (v, success) => success ? $"WhoAmI register contains valid value: {v}." : $"WhoAmI register contains invalid value: {v}."); + } + + [HarpTest(Description = "Validates that the WhoAmI register is NOT writable.")] + public async Task IsNotWritable(VerifyConnection device) + { + var req = HarpMessage.FromUInt16(WhoAmI.Address, MessageType.Write, 0); + var rejected = await RegisterHelpers.IsWriteRejectedAsync(device, req); + return new AssertionResult( + rejected, + x => x ? + "WhoAmI register correctly rejected write." : + "WhoAmI register should NOT be writable."); + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/RegisterHelpers.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/RegisterHelpers.cs new file mode 100644 index 0000000..183c644 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/RegisterHelpers.cs @@ -0,0 +1,49 @@ +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +internal static class RegisterHelpers +{ + public static async Task IsWriteRejectedAsync(VerifyConnection device, HarpMessage write) + { + try + { + await device.CommandAsync(write); + return false; + } + catch (HarpException) + { + return true; + } + } + + public static async Task AssertReadableArrayAsync(VerifyConnection device, int address, int expectedLength, string registerName) + { + try + { + var value = await device.ReadByteArrayAsync(address); + return new AssertionResult( + value.Length == expectedLength, + x => x + ? $"{registerName} is readable and has expected length ({expectedLength})." + : $"{registerName} returned {value.Length} bytes, expected {expectedLength}."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } + + public static async Task AssertReadableAsync(Func> readFunc, int address, string registerName) + { + try + { + await readFunc(address); + return new AssertionResult(true, $"{registerName} is readable."); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/CoreRegisters/Version.cs b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/Version.cs new file mode 100644 index 0000000..3681d03 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/CoreRegisters/Version.cs @@ -0,0 +1,164 @@ +using System.Text; +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify.Suites; + +/// +/// Provides methods for manipulating messages from the Version register. +/// +internal partial class Version +{ + /// + /// Represents the address of the register. This field is constant. + /// + public const int Address = 19; + + /// + /// Represents the payload type of the register. This field is constant. + /// + public const PayloadType RegisterType = PayloadType.U8; + + /// + /// Represents the length of the register. This field is constant. + /// + public const int RegisterLength = 32; + + static VersionPayload ParsePayload(byte[] payload) + { + VersionPayload result; + result.ProtocolVersion = ReadSemanticVersion(new ArraySegment(payload, 0, 3)); + result.FirmwareVersion = ReadSemanticVersion(new ArraySegment(payload, 3, 3)); + result.HardwareVersion = ReadSemanticVersion(new ArraySegment(payload, 6, 3)); + result.CoreId = ReadUtf8String(new ArraySegment(payload, 9, 3)); + result.InterfaceHash = GetSubArray(payload, 12, 20); + return result; + } + + /// + /// Returns the payload data for register messages. + /// + /// A object representing the register message. + /// A value representing the message payload. + public static VersionPayload GetPayload(HarpMessage message) + { + return ParsePayload(message.GetPayloadArray()); + } + + static SemanticVersion ReadSemanticVersion(ArraySegment segment) + { + var array = segment.Array!; + return new SemanticVersion( + array[segment.Offset], + array[segment.Offset + 1], + array[segment.Offset + 2]); + } + + static string ReadUtf8String(ArraySegment segment) + { + var array = segment.Array!; + var count = Array.IndexOf(array, (byte)0, segment.Offset, segment.Count) - segment.Offset; + return Encoding.UTF8.GetString(array, segment.Offset, count < 0 ? segment.Count : count); + } + + static byte[] GetSubArray(byte[] array, int offset, int count) + { + var result = new byte[count]; + Array.Copy(array, offset, result, 0, count); + return result; + } +} + +/// +/// Represents the payload of the Version register. +/// +internal struct VersionPayload +{ + /// + /// The semantic version of the Harp protocol implemented by the device. + /// + public SemanticVersion ProtocolVersion; + + /// + /// The semantic version of the device firmware application. + /// + public SemanticVersion FirmwareVersion; + + /// + /// The semantic version of the device hardware. + /// + public SemanticVersion HardwareVersion; + + /// + /// The three-character code of the Harp microcontroller core targeted by the device + /// firmware. + /// + public string CoreId; + + /// + /// The SHA-1 hash value of the device interface schema file, all zeros when the device + /// declares no schema for the controller to validate against. + /// + public byte[] InterfaceHash; + + /// + /// Returns a that represents the payload of the Version register. + /// + /// + /// A that represents the payload of the Version register. + /// + public override readonly string ToString() + { + return "VersionPayload { " + + "ProtocolVersion = " + ProtocolVersion + ", " + + "FirmwareVersion = " + FirmwareVersion + ", " + + "HardwareVersion = " + HardwareVersion + ", " + + "CoreId = " + CoreId + ", " + + "InterfaceHash = " + (InterfaceHash is null ? string.Empty : Convert.ToHexString(InterfaceHash)) + " " + + "}"; + } +} + +/// +/// Represents the semantic version of a device component reported by the Version register. +/// +internal readonly struct SemanticVersion +{ + /// + /// Initializes a new instance of the structure. + /// + /// The major version number. + /// The minor version number. + /// The patch version number. + public SemanticVersion(byte major, byte minor, byte patch) + { + Major = major; + Minor = minor; + Patch = patch; + } + + /// + /// The major version number. + /// + public byte Major { get; } + + /// + /// The minor version number. + /// + public byte Minor { get; } + + /// + /// The patch version number. + /// + public byte Patch { get; } + + /// + /// Returns a that represents the semantic version. + /// + /// + /// A containing the major, minor and patch numbers separated by dots. + /// + public override string ToString() + { + return $"{Major}.{Minor}.{Patch}"; + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/DeviceInterfaceSuite.cs b/src/Harp.Toolkit/Verify/Suites/DeviceInterfaceSuite.cs new file mode 100644 index 0000000..99fcced --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/DeviceInterfaceSuite.cs @@ -0,0 +1,148 @@ +using System.Reflection; +using Bonsai.Harp; +using Harp.Generators; + +namespace Harp.Toolkit.Verify.Suites; + +/// +/// Validates a live device against the C# interface actually generated from device.yml, +/// not just the schema: for every register, it reads the live reply and parses it with +/// that register's own generated GetPayload(HarpMessage) (found via the generated +/// , address -> register type, rather than +/// by guessing method names), catching real codegen/parser bugs (bad bit offsets, wrong +/// enum casts, mismatched payload struct fields) that a schema-only structural check +/// can't. It also cross-checks the WhoAmI/firmware/hardware version reported by the +/// device against device.yml. Core registers (WhoAmI, Heartbeat, version registers, etc.) +/// are covered automatically via the base +/// chain that generated code links into, so this only needs to generate/compile +/// device-specific registers. +/// +/// +/// Generating/compiling the interface and the identity check are always exactly one +/// test each, so they're plain methods like every other +/// suite. Only the per-register checks are s, since the +/// register set is only known once device.yml has been parsed. The compile itself +/// happens eagerly in the constructor (not inside ) +/// so the resulting register map is available up front to build those dynamic tests. +/// +internal class DeviceInterfaceSuite : Suite +{ + private readonly DeviceMetadata? metadata; + private readonly IReadOnlyDictionary? registerMap; + private readonly Exception? compileError; + private readonly IReadOnlyList dynamicTests; + + public DeviceInterfaceSuite(DeviceMetadata? metadata, string? rawYaml) + { + this.metadata = metadata; + + if (metadata is not null && rawYaml is not null) + { + try + { + registerMap = GeneratedInterfaceCompiler.Compile(metadata, rawYaml, $"Harp.{metadata.Device}"); + } + catch (Exception ex) + { + while (ex is TargetInvocationException or TypeInitializationException && ex.InnerException is not null) + ex = ex.InnerException; + compileError = ex; + } + } + + dynamicTests = registerMap is null + ? new List() + : BuildRegisterTests(registerMap); + } + + protected override IReadOnlyList DynamicTests => dynamicTests; + + public override string Description => + "Validates the C# interface generated from device.yml by parsing live register replies with its own generated parsers, and cross-checks WhoAmI/firmware/hardware versions."; + + [HarpTest(Description = "Generates and compiles the C# interface from device.yml.")] + public Task GenerateAndCompileInterface(VerifyConnection device) + { + IResult result = metadata is null + ? new Result(false, Status.Skipped, "No device metadata provided (--metadata).") + : registerMap is not null + ? new AssertionResult(true, $"Generated and compiled the interface with {registerMap.Count} registers.") + : new ErrorResult(compileError!); + return Task.FromResult(result); + } + + [HarpTest(Description = "Compares the WhoAmI/firmware/hardware version reported by the device against device.yml.")] + public async Task DeviceIdentity(VerifyConnection device) + { + if (metadata is null) + return new Result(false, Status.Skipped, "No device metadata provided (--metadata)."); + + var mismatches = new List(); + + int whoAmI = await device.ReadWhoAmIAsync(); + if (whoAmI != metadata.WhoAmI) + mismatches.Add($"WhoAmI: device={whoAmI}, device.yml={metadata.WhoAmI}"); + + // HarpVersion.Satisfies treats a null Major/Minor on the argument as a wildcard, + // so a device.yml that only pins a major version (minor left unset) is honored + // automatically - no need to hand-roll that comparison. + var firmware = await device.ReadFirmwareVersionAsync(); + if (metadata.FirmwareVersion is not null && !firmware.Satisfies(metadata.FirmwareVersion)) + mismatches.Add($"Firmware version: device={firmware}, device.yml={metadata.FirmwareVersion}"); + + var hardware = await device.ReadHardwareVersionAsync(); + if (metadata.HardwareTargets is not null && !hardware.Satisfies(metadata.HardwareTargets)) + mismatches.Add($"Hardware version: device={hardware}, device.yml={metadata.HardwareTargets}"); + + return new AssertionResult( + mismatches.Count == 0, + _ => mismatches.Count == 0 + ? $"WhoAmI={whoAmI}, Firmware={firmware}, Hardware={hardware} match device.yml." + : string.Join("; ", mismatches)); + } + + private static IReadOnlyList BuildRegisterTests(IReadOnlyDictionary registerMap) + { + return registerMap + .Where(entry => entry.Value.IsPublic) + .OrderBy(entry => entry.Key) + .Select(entry => new DynamicTest( + entry.Value.Name, + $"Reads register '{entry.Value.Name}' (address {entry.Key}) and parses the reply with its generated GetPayload parser.", + (device, cancellationToken) => CheckRegisterAsync(entry.Key, entry.Value, device, cancellationToken))) + .ToList(); + } + + private static async Task CheckRegisterAsync(int address, Type registerType, VerifyConnection device, CancellationToken cancellationToken) + { + const BindingFlags staticMembers = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static; + + var getPayload = registerType.GetMethod("GetPayload", staticMembers, null, new[] { typeof(HarpMessage) }, null); + if (getPayload is null) + return new Result(false, Status.Skipped, $"Generated register '{registerType.Name}' has no GetPayload(HarpMessage) parser."); + + var payloadType = (PayloadType)registerType.GetField("RegisterType", staticMembers)!.GetValue(null)!; + + HarpMessage reply; + try + { + reply = await device.CommandAsync(HarpCommand.Read(address, payloadType), cancellationToken); + } + catch (Exception ex) + { + return new ErrorResult(ex); + } + + try + { + var value = getPayload.Invoke(null, new object?[] { reply }); + return new AssertionResult(true, $"Register '{registerType.Name}' parsed successfully: {value}."); + } + catch (TargetInvocationException ex) + { + // Unwrap so a real generated-parser bug (bad bit offset, wrong enum cast, ...) + // is reported distinctly from the reflection-invocation exception itself. + return new ErrorResult(ex.InnerException ?? ex); + } + } +} diff --git a/src/Harp.Toolkit/Verify/Suites/RoundTripTestSuite.cs b/src/Harp.Toolkit/Verify/Suites/RoundTripTestSuite.cs new file mode 100644 index 0000000..45ca893 --- /dev/null +++ b/src/Harp.Toolkit/Verify/Suites/RoundTripTestSuite.cs @@ -0,0 +1,27 @@ + +using System.Diagnostics; +using Bonsai.Harp; +namespace Harp.Toolkit.Verify.Suites; + +internal class RoundTripTestSuite : Suite +{ + public override string Description => "Measures round trip latency statistics for a register read."; + + [HarpTest(Description = "Benchmarks the round trip time for a WhoAmI read command.")] + public async Task BenchmarkRoundTrip(VerifyConnection device) + { + const int n = 1000; + double[] elapsed = new double[n]; + HarpMessage probe = Bonsai.Harp.WhoAmI.FromPayload(MessageType.Read, default); + var clock = new Stopwatch(); + for (int i = 0; i < n; i++) + { + clock.Restart(); + await device.CommandAsync(probe); + elapsed[i] = clock.Elapsed.TotalMilliseconds; + } + var benchmark = new BenchmarkSummary(elapsed); + return new NumericBenchmarkResult(benchmark, Status.Passed, + $"Round trip WhoAmI read latency over {n} samples, in milliseconds."); + } +} diff --git a/src/Harp.Toolkit/Verify/VerifyCommand.cs b/src/Harp.Toolkit/Verify/VerifyCommand.cs new file mode 100644 index 0000000..ab5bebf --- /dev/null +++ b/src/Harp.Toolkit/Verify/VerifyCommand.cs @@ -0,0 +1,340 @@ +using System.CommandLine; +using Spectre.Console; +using Harp.Generators; +using Harp.Toolkit.Verify.Suites; +using Harp.Toolkit.Generate; + +namespace Harp.Toolkit.Verify; +public class VerifyCommand : Command +{ + public VerifyCommand() + : base("verify", "Verify device conformance against the Harp specification.") + { + PortNameOption portNameOption = new(); + Option fileOption = new("--report") + { + Description = "Path to the HTML report generated after running tests.", + Required = false, + }; + + Option verboseOption = new("--verbose") + { + Description = "Show detailed results for each test.", + Required = false, + }; + + Option prereleaseOption = new("--prerelease") + { + Description = "Include checks against specification text outside the stable baseline.", + Required = false, + }; + + Option clockPortOption = new("--clock-port") + { + Description = "Serial port of the reference clock device. Enables clock alignment tests.", + Required = false, + }; + + Option ppsEventOption = new("--pps-event") + { + Description = "Address of the register on the tested device (--port) that reports the incoming PPS pulse from the reference clock device. Enables the PPS alignment test, which also requires --clock-port.", + Required = false, + }; + + Option clockSamplesOption = new("--clock-samples") + { + Description = "Number of PPS event pairs to collect for the PPS alignment test.", + Required = false, + }; + clockSamplesOption.DefaultValueFactory = _ => 5; + clockSamplesOption.Validators.Add(result => + { + if (result.GetValueOrDefault() < 1) + result.AddError("The number of clock samples must be greater than zero."); + }); + + Option metadataOption = new("--metadata") + { + Description = "The path to the file describing the device registers. Enables validation of the generated interface against a live read of every declared register, and cross-checks the WhoAmI, firmware and hardware versions.", + Required = false, + }; + OptionValidation.AcceptExistingOnly(metadataOption); + + Options.Add(portNameOption); + Options.Add(fileOption); + Options.Add(verboseOption); + Options.Add(prereleaseOption); + Options.Add(clockPortOption); + Options.Add(ppsEventOption); + Options.Add(clockSamplesOption); + Options.Add(metadataOption); + SetAction(parsedResult => + { + string portName = parsedResult.GetRequiredValue(portNameOption); + FileInfo? reportFile = parsedResult.GetValue(fileOption); + bool verbose = parsedResult.GetValue(verboseOption); + bool prerelease = parsedResult.GetValue(prereleaseOption); + string? clockPort = parsedResult.GetValue(clockPortOption); + ClockTestOptions? clockOptions = clockPort is null ? null : new ClockTestOptions( + ClockPort: clockPort, + PpsEvent: parsedResult.GetValue(ppsEventOption), + ClockSamples: parsedResult.GetValue(clockSamplesOption)); + FileInfo? metadataPath = parsedResult.GetValue(metadataOption); + return RunVerification(portName, reportFile, verbose, prerelease, clockOptions, metadataPath, CancellationToken.None); + }); + } + + static async Task RunVerification(string portName, FileInfo? reportFile, bool verbose, bool prerelease, ClockTestOptions? clockOptions, FileInfo? metadataPath, CancellationToken cancellationToken) + { + AnsiConsole.MarkupLine($"Running tests on [bold]{portName}[/]..."); + if (clockOptions is not null) + AnsiConsole.MarkupLine($"Clock reference device: [bold]{clockOptions.ClockPort}[/]"); + + DeviceMetadata? deviceMetadata = null; + string? deviceRawYaml = null; + if (metadataPath is not null) + { + AnsiConsole.Markup($"Loading device metadata from [bold]{metadataPath.FullName}[/]..."); + deviceMetadata = GeneratorHelper.ReadDeviceMetadata(metadataPath.FullName); + deviceRawYaml = await File.ReadAllTextAsync(metadataPath.FullName, cancellationToken); + AnsiConsole.MarkupLine($" [green]Done![/] ({deviceMetadata.Registers.Count} registers)"); + } + + using var connection = await VerifyConnection.OpenAsync(portName, cancellationToken); + var identity = await connection.ReadDeviceIdentityAsync(cancellationToken); + var declaredVersion = await connection.ReadProtocolVersionAsync(cancellationToken); + var target = new ProtocolTarget(declaredVersion, prerelease); + var runner = new CoreRunner(target.IncludePrerelease, clockOptions, deviceMetadata, deviceRawYaml); + var notice = GetProtocolNotice(target, runner.PrereleaseTestCount); + + AnsiConsole.MarkupLine(DescribeDeviceIdentity(identity, portName)); + AnsiConsole.MarkupLine(DescribeProtocolSelection(target)); + if (notice.Length > 0) + { + var style = target.IncludePrerelease ? "grey" : "yellow"; + AnsiConsole.MarkupLine($"[{style}]{Markup.Escape(notice)}[/]"); + } + + var report = new Report + { + DeviceName = identity.Name is { Length: > 0 } name ? name : "Harp Device", + PortName = portName, + WhoAmI = identity.WhoAmI.ToString(), + HardwareVersion = identity.HardwareVersion?.ToString() ?? "not reported", + FirmwareVersion = identity.FirmwareVersion?.ToString() ?? "not reported", + RunDate = DateTime.Now, + IncludePrerelease = target.IncludePrerelease, + ProtocolNotice = notice, + DeclaredProtocolVersion = GetDeclaredVersion(target), + CheckedProtocolVersion = GetCheckedVersion(target), + ProtocolCommit = ProtocolReference.ShortCommit, + ProtocolCommitUrl = ProtocolReference.TreeUrl, + RegisterSetVersion = CoreSchema.Version + }; + + int currentTest = 0; + await foreach (var (suite, result) in runner.RunAllAsync(connection, cancellationToken, (suite, testName, testDesc) => + { + // Print "Running" status before test execution (without newline) + currentTest++; + if (!Console.IsOutputRedirected) + Console.Write($"({currentTest}/{runner.TestCount}) {suite.GetType().Name}::{testName} .... Running..."); + })) + { + // Clear the line by moving cursor to start and overwriting with spaces, then print result + if (!Console.IsOutputRedirected) + Console.Write($"\r{new string(' ', Console.WindowWidth - 1)}\r"); + AnsiConsole.MarkupLine($"[grey]({currentTest}/{runner.TestCount}) {suite.GetType().Name}::{result.Name}[/] .... {GetResultMarkup(result.Result)}"); + + var suiteResult = report.Suites.FirstOrDefault(s => s.Name == suite.GetType().Name); + if (suiteResult == null) + { + suiteResult = new SuiteResult + { + Name = suite.GetType().Name, + Description = suite.Description + }; + report.Suites.Add(suiteResult); + } + suiteResult.Results.Add(result); + } + + if (verbose) + { + AnsiConsole.WriteLine(); + AnsiConsole.Write(new Rule("[yellow]Detailed Results[/]")); + foreach (var suite in report.Suites) + { + AnsiConsole.MarkupLine($"[bold underline]{suite.Name}[/]"); + AnsiConsole.MarkupLine($"[dim]{suite.Description}[/]"); + + var table = new Table(); + table.AddColumn("Test Case"); + table.AddColumn("Status"); + table.AddColumn("Details"); + table.AddColumn("Message"); + + foreach (var test in suite.Results) + { + string details = ""; + string message = test.Result.Message ?? ""; + + if (test.Result is NumericBenchmarkResult bsr) + { + details = $"Mean: {bsr.Summary.Mean:F4}\nMedian: {bsr.Summary.Median:F4}\nStdDev: {bsr.Summary.StdDev:F4}\nMin: {bsr.Summary.Min:F4}\nMax: {bsr.Summary.Max:F4}\nPercentiles: 99th={bsr.Summary.Percentile99:F4}, 1st={bsr.Summary.Percentile01:F4}"; + } + else if (test.Result is ErrorResult er) + { + details = $"{er.Exception.GetType().Name}"; + } + else + { + var valProp = test.Result.GetType().GetProperty("Value"); + if (valProp != null) + { + var val = valProp.GetValue(test.Result); + details = val?.ToString() ?? ""; + } + } + + table.AddRow( + new Markup($"[bold]{Markup.Escape(test.Name)}[/]\n[dim]{Markup.Escape(test.Description)}[/]"), + new Markup(GetResultMarkup(test.Result)), + new Markup(Markup.Escape(details)), + new Markup(Markup.Escape(message)) + ); + } + AnsiConsole.Write(table); + AnsiConsole.WriteLine(); + } + } + + if (reportFile != null) + { + AnsiConsole.Markup("Generating HTML report..."); + string html = await HtmlReportGenerator.GenerateAsync(report); + string fileName = reportFile.FullName; + await File.WriteAllTextAsync(fileName, html, cancellationToken); + AnsiConsole.MarkupLine($"[green]Done![/] Report generated: [link]{fileName}[/]"); + } + } + + static string GetDeclaredVersion(ProtocolTarget target) + { + return target.DeclaredVersion.HasValue + ? target.DeclaredVersion.GetValueOrDefault().ToString() + : "not declared"; + } + + static string DescribeDeviceIdentity(DeviceIdentity identity, string portName) + { + var name = identity.Name is { Length: > 0 } deviceName ? deviceName : "unnamed device"; + return $"Device [bold]{Markup.Escape(name)}[/] on {portName}, WhoAmI [bold]{identity.WhoAmI}[/], " + + $"hardware {identity.HardwareVersion?.ToString() ?? "not reported"}, " + + $"firmware {identity.FirmwareVersion?.ToString() ?? "not reported"}."; + } + + static string DescribeProtocolSelection(ProtocolTarget target) + { + if (target.IncludePrerelease && target.Scope == ProtocolScope.V2) + { + return $"Checking against protocol version [bold]{GetDeclaredVersion(target)}[/], " + + "which is not yet ratified."; + } + + return $"Protocol version [bold]{GetDeclaredVersion(target)}[/], " + + $"checking against [bold]{GetCheckedVersion(target)}[/]."; + } + + static string GetCheckedVersion(ProtocolTarget target) + { + if (target.IncludePrerelease) + return $"v{ProtocolReference.PrereleaseMajorVersion}, which is not yet ratified"; + + return target.Scope == ProtocolScope.V1 + ? "v1" + : $"v1, since v{ProtocolReference.PrereleaseMajorVersion} is not yet ratified"; + } + + static string GetProtocolNotice(ProtocolTarget target, int count) + { + if (target.IncludePrerelease) + { + if (target.Scope == ProtocolScope.V2) + return $"Including {count} prerelease checks, which this device declares support for."; + + return $"Including {count} prerelease checks against protocol " + + $"v{ProtocolReference.PrereleaseMajorVersion}, which this device does not declare."; + } + + if (target.Scope == ProtocolScope.Unsupported) + { + return $"This device declares protocol {GetDeclaredVersion(target)}, which this toolkit " + + $"does not cover, so only the v1 baseline applies. {GetRerunHint(count)}".TrimEnd(); + } + + if (!target.DeclaredVersion.HasValue) + { + return "This device declares no protocol version, so only the v1 baseline applies. " + + $"Implementing R_VERSION is the first step of a v{ProtocolReference.PrereleaseMajorVersion} " + + $"migration. {GetRerunHint(count)}".TrimEnd(); + } + + return GetRerunHint(count); + } + + static string GetRerunHint(int count) + { + return count > 0 + ? $"{count} prerelease checks were not run. Rerun with --prerelease to include them." + : string.Empty; + } + + static string GetResultMarkup(IResult result) + { + return result.Status switch + { + Status.Passed => "[green]Passed[/]", + Status.Failed => "[red]Failed[/]", + Status.Error => "[red]Error[/]", + Status.Skipped => "[yellow]Skipped[/]", + _ => $"[white]{result.Status}[/]" + }; + } + + class CoreRunner : Runner + { + public CoreRunner( + bool includePrerelease, + ClockTestOptions? clockOptions = null, + DeviceMetadata? deviceMetadata = null, + string? deviceRawYaml = null) : base(includePrerelease) + { + AddSuite(new R_WHO_AM_I()); + AddSuite(new R_HW_VERSION_H()); + AddSuite(new R_HW_VERSION_L()); + AddSuite(new R_ASSEMBLY_VERSION()); + AddSuite(new R_CORE_VERSION_H()); + AddSuite(new R_CORE_VERSION_L()); + AddSuite(new R_FW_VERSION_H()); + AddSuite(new R_FW_VERSION_L()); + AddSuite(new R_TIMESTAMP_SECOND()); + AddSuite(new R_TIMESTAMP_MICRO()); + AddSuite(new R_OPERATION_CTRL()); + AddSuite(new R_RESET_DEV()); + AddSuite(new R_DEVICE_NAME()); + AddSuite(new R_SERIAL_NUMBER()); + AddSuite(new R_CLOCK_CONFIG()); + AddSuite(new R_TIMESTAMP_OFFSET()); + AddSuite(new R_UID()); + AddSuite(new R_TAG()); + AddSuite(new R_HEARTBEAT()); + AddSuite(new R_VERSION()); + AddSuite(new RoundTripTestSuite()); + AddSuite(new ClockTestSuite(clockOptions)); + AddSuite(new DeviceInterfaceSuite(deviceMetadata, deviceRawYaml)); + } + } +} + + diff --git a/src/Harp.Toolkit/Verify/VerifyConnection.cs b/src/Harp.Toolkit/Verify/VerifyConnection.cs new file mode 100644 index 0000000..0f4d75f --- /dev/null +++ b/src/Harp.Toolkit/Verify/VerifyConnection.cs @@ -0,0 +1,320 @@ +using System.Diagnostics; +using System.Reactive.Linq; +using System.Reactive.Subjects; +using System.Text; +using Bonsai.Harp; + +namespace Harp.Toolkit.Verify; + +/// +/// A single Harp connection shared by every test in a verification run. +/// +public sealed class VerifyConnection : IDisposable +{ + const int ConnectDelayMilliseconds = 200; + const int PortReleaseDelayMilliseconds = 300; + const int OpenTimeoutMilliseconds = 10000; + const int ReadTimeoutMilliseconds = 2000; + const int ReadyAttempts = 5; + const int ReadyTimeoutMilliseconds = 1000; + + readonly Subject requests = new(); + readonly IConnectableObservable messages; + readonly IDisposable subscription; + + VerifyConnection(string portName, int whoAmI) + { + WhoAmI = whoAmI; + var device = new Bonsai.Harp.Device(whoAmI) + { + PortName = portName, + IgnoreErrors = true, + OperationMode = OperationMode.Standby, + DumpRegisters = false, + }; + messages = device.Generate(requests).Publish(); + subscription = messages.Connect(); + } + + /// + /// Opens the shared connection, reading the device identity first so the connection can be + /// constructed with a known WhoAmI, which suppresses the device name probe that would + /// otherwise open a second port in the background. + /// + public static async Task OpenAsync(string portName, CancellationToken cancellationToken = default) + { + var whoAmI = await ReadIdentityAsync(portName, cancellationToken); + await Task.Delay(PortReleaseDelayMilliseconds, cancellationToken); + + var retryStart = Stopwatch.GetTimestamp(); + while (true) + { + var connection = new VerifyConnection(portName, whoAmI); + try + { + await connection.WaitUntilReadyAsync(cancellationToken); + return connection; + } + catch (Exception ex) when ( + IsRetryableOpenFailure(ex) && + IsWithinRetryBudget(retryStart) && + !cancellationToken.IsCancellationRequested) + { + connection.Dispose(); + await Task.Delay(PortReleaseDelayMilliseconds, cancellationToken); + } + catch + { + connection.Dispose(); + throw; + } + } + } + + static async Task ReadIdentityAsync(string portName, CancellationToken cancellationToken) + { + var retryStart = Stopwatch.GetTimestamp(); + while (true) + { + using var readTimeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + readTimeout.CancelAfter(ReadTimeoutMilliseconds); + try + { + using var probe = new AsyncDevice(portName); + return await probe.ReadWhoAmIAsync(readTimeout.Token); + } + catch (Exception ex) when ( + (IsRetryableOpenFailure(ex) || ex is OperationCanceledException) && + IsWithinRetryBudget(retryStart) && + !cancellationToken.IsCancellationRequested) + { + await Task.Delay(PortReleaseDelayMilliseconds, cancellationToken); + } + } + } + + static bool IsRetryableOpenFailure(Exception ex) + { + return ex is UnauthorizedAccessException || ex is IOException || ex is TimeoutException; + } + + static bool IsWithinRetryBudget(long retryStart) + { + return Stopwatch.GetElapsedTime(retryStart).TotalMilliseconds < OpenTimeoutMilliseconds; + } + + /// + /// The device identifier validated when the connection was opened. + /// + public int WhoAmI { get; } + + /// + /// Every message received from the device, before any reply correlation. + /// + public IObservable Messages => messages; + + /// + /// Sends a message without awaiting a reply. + /// + public void Write(HarpMessage message) => requests.OnNext(message); + + /// + /// Sends the specified request and awaits the matching reply, failing with a + /// if the device does not answer in time. + /// + public async Task CommandAsync(HarpMessage command, CancellationToken cancellationToken = default) + { + using var replyTimeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + replyTimeout.CancelAfter(ReadTimeoutMilliseconds); + var reply = messages.FirstAsync(message => + { + var match = message.IsMatch(command.Address, command.MessageType); + if (match && message.Error) + { + throw new HarpException(message); + } + + return match; + }).RunAsync(replyTimeout.Token); + + Write(command); + try + { + return await reply; + } + catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested) + { + throw new TimeoutException( + $"The device did not reply to a {command.MessageType} request at address " + + $"{command.Address} within {ReadTimeoutMilliseconds} ms."); + } + } + + /// + /// Sends the specified messages and collects everything received for the given duration. + /// + public async Task> WriteAndCollectAsync( + IEnumerable messagesToWrite, + TimeSpan listenDuration, + CancellationToken cancellationToken = default) + { + var collected = new List(); + using (messages.Subscribe(message => + { + lock (collected) + { + collected.Add(message); + } + })) + { + foreach (var message in messagesToWrite) + { + Write(message); + } + + await Task.Delay(listenDuration, cancellationToken); + } + + lock (collected) + { + return collected.ToList(); + } + } + + public async Task ReadByteAsync(int address, CancellationToken cancellationToken = default) + { + var reply = await CommandAsync(HarpCommand.ReadByte(address), cancellationToken); + return reply.GetPayloadByte(); + } + + public async Task ReadByteArrayAsync(int address, CancellationToken cancellationToken = default) + { + var reply = await CommandAsync(HarpCommand.ReadByte(address), cancellationToken); + return reply.GetPayloadArray(); + } + + public async Task ReadUInt16Async(int address, CancellationToken cancellationToken = default) + { + var reply = await CommandAsync(HarpCommand.ReadUInt16(address), cancellationToken); + return reply.GetPayloadUInt16(); + } + + public async Task ReadUInt32Async(int address, CancellationToken cancellationToken = default) + { + var reply = await CommandAsync(HarpCommand.ReadUInt32(address), cancellationToken); + return reply.GetPayloadUInt32(); + } + + public async Task ReadWhoAmIAsync(CancellationToken cancellationToken = default) + { + return await ReadUInt16Async(Bonsai.Harp.WhoAmI.Address, cancellationToken); + } + + public async Task ReadDeviceNameAsync(CancellationToken cancellationToken = default) + { + var payload = await ReadByteArrayAsync(DeviceName.Address, cancellationToken); + var terminator = Array.IndexOf(payload, (byte)0); + return Encoding.ASCII.GetString(payload, 0, terminator < 0 ? payload.Length : terminator); + } + + public async Task ReadAssemblyVersionAsync(CancellationToken cancellationToken = default) + { + return await ReadByteAsync(AssemblyVersion.Address, cancellationToken); + } + + public async Task ReadSerialNumberAsync(CancellationToken cancellationToken = default) + { + return await ReadUInt16Async(SerialNumber.Address, cancellationToken); + } + + public async Task ReadHardwareVersionAsync(CancellationToken cancellationToken = default) + { + var major = await ReadByteAsync(HardwareVersionHigh.Address, cancellationToken); + var minor = await ReadByteAsync(HardwareVersionLow.Address, cancellationToken); + return new HarpVersion(major, minor); + } + + public async Task ReadFirmwareVersionAsync(CancellationToken cancellationToken = default) + { + var major = await ReadByteAsync(FirmwareVersionHigh.Address, cancellationToken); + var minor = await ReadByteAsync(FirmwareVersionLow.Address, cancellationToken); + return new HarpVersion(major, minor); + } + + public async Task ReadTimestampSecondsAsync(CancellationToken cancellationToken = default) + { + return await ReadUInt32Async(TimestampSeconds.Address, cancellationToken); + } + + public async Task WriteTimestampSecondsAsync(uint seconds, CancellationToken cancellationToken = default) + { + await CommandAsync(HarpCommand.WriteUInt32(TimestampSeconds.Address, seconds), cancellationToken); + } + + /// + /// Reads the identity registers reported in the run header, leaving any register the device + /// does not answer within the read timeout unreported rather than failing the run. + /// + internal async Task ReadDeviceIdentityAsync(CancellationToken cancellationToken = default) + { + var name = await TryReadAsync(ReadDeviceNameAsync, cancellationToken); + var hardwareVersion = await TryReadAsync(ReadHardwareVersionAsync, cancellationToken); + var firmwareVersion = await TryReadAsync(ReadFirmwareVersionAsync, cancellationToken); + return new DeviceIdentity(WhoAmI, name, hardwareVersion, firmwareVersion); + } + + /// + /// Reads the protocol version the device declares, leaving it undeclared when the version + /// register is unreadable, carries an unexpected length, or reports all zeros. + /// + internal async Task ReadProtocolVersionAsync(CancellationToken cancellationToken = default) + { + var reply = await TryReadAsync( + token => CommandAsync(HarpCommand.ReadByte(Suites.Version.Address), token), + cancellationToken); + if (reply is null || reply.GetPayloadArray().Length != Suites.Version.RegisterLength) + return null; + + var protocolVersion = Suites.Version.GetPayload(reply).ProtocolVersion; + return protocolVersion.Major == 0 ? null : protocolVersion; + } + + static async Task TryReadAsync( + Func> read, + CancellationToken cancellationToken) + where T : class + { + try + { + return await read(cancellationToken); + } + catch (Exception) when (!cancellationToken.IsCancellationRequested) + { + return null; + } + } + + async Task WaitUntilReadyAsync(CancellationToken cancellationToken) + { + await Task.Delay(ConnectDelayMilliseconds, cancellationToken); + for (int attempt = 1; ; attempt++) + { + using var timeout = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + timeout.CancelAfter(ReadyTimeoutMilliseconds); + try + { + await ReadWhoAmIAsync(timeout.Token); + return; + } + catch (OperationCanceledException) when (attempt < ReadyAttempts && !cancellationToken.IsCancellationRequested) + { + } + } + } + + public void Dispose() + { + subscription.Dispose(); + requests.Dispose(); + } +}