Skip to content

README.md

Latest commit

 

History

History
128 lines (99 loc) · 6.5 KB

README.md

File metadata and controls

128 lines (99 loc) · 6.5 KB
Raw

PreciseNumber Benchmarks

A BenchmarkDotNet suite covering the operations that dominate real use of PreciseNumber: building values, comparing them, arithmetic, rounding, text conversion, and conversion to and from the primitive numeric types.

Running

From the repository root:

# Pick benchmarks from an interactive list
dotnet run -c Release --project PreciseNumber.Benchmarks

# Run everything (slow: a full run is tens of minutes)
dotnet run -c Release --project PreciseNumber.Benchmarks -- --filter '*'

# Run one class, or one method
dotnet run -c Release --project PreciseNumber.Benchmarks -- --filter '*ArithmeticBenchmarks*'
dotnet run -c Release --project PreciseNumber.Benchmarks -- --filter '*.Multiply'

# Fewer iterations, for a quick read while iterating on a change
dotnet run -c Release --project PreciseNumber.Benchmarks -- --filter '*Comparison*' --job short

Release configuration is required — BenchmarkDotNet refuses to measure a debug build.

Results land in BenchmarkDotNet.Artifacts/results/ as GitHub-flavoured Markdown, CSV, HTML and JSON. That directory is gitignored; copy a table into a pull request when a change moves the numbers.

What is measured

Class Covers
ConstructionBenchmarks The constructor: counting significant digits and stripping trailing zeros
ComparisonBenchmarks Equality, ordering, CompareTo, Max, GetHashCode
ArithmeticBenchmarks +, -, *, /, %, negation, squaring
PowBenchmarks Raising to an integer power
RoundingBenchmarks Round, ReduceSignificance, Clamp
TextBenchmarks ToString, TryFormat, Parse
ConversionBenchmarks To and from int, long, double, float, decimal

Most classes are parameterised by Digits — 8, 30 and 200 significant digits. This is the axis that matters: a PreciseNumber holds its digits in a BigInteger, so an operation that touches each digit separately looks fine at 8 digits and falls apart at 200. Reading a table across the Digits column, rather than down a single value of it, is what catches that.

ComparisonBenchmarks and ArithmeticBenchmarks also separate operands whose exponents are far apart from operands in the same decade, because aligning two exponents is its own cost, distinct from the size of the operands.

What this type costs against a bare double

AbstractionCostBenchmarks is the one benchmark here whose answer is a ratio rather than a duration. Every other class says how long an operation takes, which is only readable beside something; this supplies the something — the primitive a caller would otherwise have used.

The same class, with the same loops and the same methodology, is in ktsu.SignificantNumber and ktsu.Semantics, so the three libraries answer one question the same way and their answers are comparable with each other as well as with double.

release Add Multiply
1.8.0 99.7× 293.9×
1.9.0 99.7× 288.2×
2.0.0 82.8× 248.8×
2.0.5 83.6× 249.8×

Becoming a value type in 2.0 took about 15% off the price of arbitrary precision, and six releases have held it there. The ratio is not expected to be 1 and is not a defect for being large — a double cannot do this at all. What the chart's third section is for is noticing the day it moves.

Three things decide how the number should be read:

  • These are loops. A single operation over operands that do not change is loop-invariant and the JIT hoists it out, which would leave the double side indistinguishable from an empty method and the ratio meaningless. Each iteration feeds the next, so there is nothing to hoist.
  • The loop's own cost biases toward 1, being paid by both sides, so a ratio is a floor on the real cost rather than the whole of it.
  • Both loops accumulate rather than compound, because this type carries as many digits as the arithmetic produces and a compounding chain would measure that growth instead of the operation. How the cost grows with digits is a different question, and ArithmeticBenchmarks answers it across the Digits axis.

Reading the results

Allocation is reported next to time. Both matter here, and they trade against each other: every operation returns a new instance, so a change that avoids an intermediate value shows up in the Allocated column before it shows up in Mean. A comparison that allocates at all is a regression — none of them should.

Divide is the one row whose cost is not driven by the operands alone. It produces a terminating quotient exactly and a repeating one to a chosen precision, so the Digits column moves it twice over: wider operands are more work to divide, and they also raise the precision the quotient is taken to. A number that looks expensive at 200 digits is doing proportionally more work, not doing the same work badly.

Benchmark operands come from a fixed digit pattern rather than a random source, so two runs on the same machine measure the same work. Numbers are still only comparable within a single run on a single machine; a cloud CI runner in particular is too noisy to compare against a previous run there.

Comparing two refs

That last constraint is why the Benchmarks workflow takes a baseline input. Given a ref — a tag, a branch or a commit — it measures that ref through a worktree and then this checkout, both in one job, and heads the two result tables separately. Two separate dispatches cannot be compared: they can land on different CPU generations, and that difference is larger than most changes worth catching.

Locally the same shape is a worktree and one --artifacts directory per ref, run one after the other rather than at the same time — two benchmark processes on one machine measure contention rather than the code:

git worktree add --detach /tmp/bench-baseline v1.9.0
(cd /tmp/bench-baseline && dotnet run -c Release --project PreciseNumber.Benchmarks -- \
  --filter '*ComparisonBenchmarks.Max' --job short --artifacts /tmp/bench/baseline)
dotnet run -c Release --project PreciseNumber.Benchmarks -- \
  --filter '*ComparisonBenchmarks.Max' --job short --artifacts /tmp/bench/head
git worktree remove --force /tmp/bench-baseline

--filter takes several patterns as separate arguments (--filter '*A.One' '*B.Two'). It does not accept them |-separated in one argument: that matches nothing, and BenchmarkDotNet then prints the full benchmark list and exits 0 rather than failing, so a comparison script written that way looks like it ran.