Add benchmarking tool - #16
Conversation
|
@glopesdev was this the integration you had in mind for new tools?
|
@bruno-f-cruz Happy to create a namespace for particularly complex commands requiring many helper classes. An alternative is to create a separate library for the command backend and then add it as a dependency (e.g. I am going to do this with the reflex generator).
In principle not opposed to any dependencies, since this will be a self-contained dotnet tool, so it will be deployed and run completely independent from any bonsai project dependency, etc. But let's discuss the details on the benchmarks and reflex generator PRs. |
|
Sounds good. I guess the best way to go about this is to expose a single method (e.g.: https://github.com/harp-tech/Harp.Benchmark/blob/bb0529326aee7a12ab9b33d2dfeec573f8bbd2ac/src/Harp.Benchmark/Program.cs#L43) as the entry point of the tool and create a single command subclass in this repository that wraps it? |
|
@glopesdev took a first stab at this that would benefit from initial comments. Feel free to run the tool against a device; it should output something sensible. We can discuss some edge cases during one of the upcoming harp meetings |
de4629c to
327dcb4
Compare
327dcb4 to
b7df2d0
Compare
|
Decided to give this a try and I get random unauthorized port access errors on successive runs with just the core benchmark tests. It seems that each test opens its own connection, and the handle is not being released in time before the next test opens. Once it hits a snag a whole bunch of tests fail before it recovers. This might only surface on slower PCs, but curiously it does not happen when I run it the full device tests with the
Also, if the device isn't connected and the com port doesn't exist, the report says: "System.IO.FileNotFoundException: Could not find file 'COM9'." when something like the port could not be found might be a more obvious error message (and ideally it should throw that error without even running the tests). The HTML reports attached! |
This PR adds a new tool to run hardware tests of Harp devices against the Harp protocol specification.
The main features:
benchmarkcommand:dotnet run --project Harp.Toolkit benchmark --port COM3 --report here.html--verboseflag)