Skip to content

test: refuse drivers that emit numbers they never read - #168

Merged
davmoz merged 1 commit into
mainfrom
test/no-invented-numbers
Oct 7, 2026
Merged

davmoz merged 1 commit into
mainfrom
test/no-invented-numbers

Conversation

@davmoz

@davmoz davmoz commented Oct 7, 2026

Copy link
Copy Markdown
Member

Summary

A failed read must leave the value nil, so the host sends null. local w = 0 before a read, or decode(...) or 0 after one, turns a dead device into a reading: 0 W, 0 A, 0 % SoC, 50 Hz. Nothing downstream can tell it from a real one. A meter at 0 W looks like a balanced site, and a battery at 0 % looks empty and ready to charge.

  • drivers/tests/lua_harness/no_invented_numbers.lua runs a driver twice, A and B, with different register values, numeric config and clock. Then it fails every read for three polls and shortens every Modbus reply to half for three more. A number still emitted that is equal in both runs and was not read came from none of the driver's inputs, so it was invented. The data model's default SoC window (0.05 / 1.0) is the one allowed constant. Sourceful's driver registry runs this same file, byte for byte, on every publish and refuses a driver that fails it.
  • 44 of 91 drivers fail it today (592 fields). drivers/tests/test_no_invented_numbers.py ratchets against invented-number-baseline.json, the same way test_absent_register_settles.py does: a count that rises fails, a driver not listed must be clean, and a count that falls fails until the file is updated.
  • make invented-number-report ID=<id> prints the fields for one driver.

No driver changes. The baseline only records the current debt.

Driver evidence

  • Driver id: not applicable (test only)
  • Device make/model: not applicable
  • Firmware tested: not applicable
  • Protocol/register source: not applicable
  • Hardware test performed: not applicable
  • Known limits: a driver whose driver_init fails with the probe config (zaptec_cloud: no credentials) is not measured. A configured constant emitted while the device is down, such as ctek ev.phases, counts as invented.

Safety

  • The driver starts read-only, or this is a separately reviewed control change. (no driver change)
  • Vendor signs are converted at the driver boundary. (no driver change)
  • Cached telemetry becomes stale instead of being re-emitted as fresh. (no driver change)
  • No credentials, serial numbers, private addresses or site data are included.

Control evidence

  • HIL evidence or required: not applicable
  • Control changes include default-mode, lease-expiry and HIL evidence. (not applicable)

Checks

  • Commits include Signed-off-by.
  • make test-driver ID=<id>: pytest drivers/tests/test_no_invented_numbers.py 91 passed. With hello removed from the baseline and abb_meter raised by one, the expected 2 tests fail.
  • make check: 4901 passed, 923 skipped

🤖 Generated with Claude Code

A failed read must leave the value nil so the host sends null. `local w = 0`
before a read, or `decode(...) or 0` after one, turns a dead device into a
reading of 0 W, 0 % SoC or 50 Hz that nothing downstream can tell from a
real one.

lua_harness/no_invented_numbers.lua runs a driver twice with different
registers, config and clock, then fails every read and shortens every
Modbus reply. A number equal in both runs that was not read is invented.
It is the same file Sourceful's driver registry runs on every publish.

44 of 91 drivers fail it today, so test_no_invented_numbers.py ratchets
against invented-number-baseline.json like the absent-register test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: David Mozart <dmozartandraws@gmail.com>
@davmoz
davmoz merged commit 224c438 into main Oct 7, 2026
5 checks passed
@davmoz
davmoz deleted the test/no-invented-numbers branch October 7, 2026 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant