From 63b1f23ec1f8222a7d450d5610f072f7b23c1815 Mon Sep 17 00:00:00 2001 From: Roger Luethi Date: Wed, 5 Aug 2026 09:33:00 +0200 Subject: [PATCH] tests/e2e: gate on full config_db table coverage tests/e2e/coverage.py reports which config_db tables the golden set reaches, but nothing consumes it: it is wired into neither sonic_golden_test.sh nor the Zuul job, so the number is only seen by someone who runs `make sonic-e2e-coverage` by hand. That is how the "38 of 38" claim came to exist as an unverifiable one-off in the first place, and leaving it human-run lets it decay the same way again. Add a unit test asserting the set of emitted-but-uncovered tables is empty. It closes the one coverage failure nothing else catches: a newly emitted table arriving with no golden. Coverage lost in the other direction -- a table that had a golden becoming empty -- already fails the golden comparison, because the golden file itself changes, and the regeneration path is covered separately by the coverage guard in compare.py. It belongs in the unit suite rather than the E2E job because it needs neither NetBox nor a generated config, only the generator source and the committed goldens. It therefore costs milliseconds and runs on every change, where the E2E job runs on a file matcher and a 2400s budget. This needs no .zuul.yaml change: coverage.py already exposes emitted_tables() and covered_tables(). The assertion message names the missing tables and points at `make sonic-e2e-regen`, because the cost of this gate is that adding a generator table now obliges the same change to add golden coverage, and that is a full regeneration cycle rather than a two-line edit. If a table genuinely cannot be reached by any fixture, the message says to exclude it here with a stated reason -- one visible exception, not a silently growing allowlist. This lands after the last scenario because it can only pass once the golden set is complete; the report itself lands with the first goldens, where it is still useful at 30 of 38. Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi --- tests/unit/e2e/test_coverage.py | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 tests/unit/e2e/test_coverage.py diff --git a/tests/unit/e2e/test_coverage.py b/tests/unit/e2e/test_coverage.py new file mode 100644 index 00000000..3471c48a --- /dev/null +++ b/tests/unit/e2e/test_coverage.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: Apache-2.0 + +"""Gate on the golden set covering every config_db table the generator emits. + +tests/e2e/coverage.py is the human-facing report; this is the check that +actually fails. It lives in the unit suite rather than in the E2E job because +it needs neither NetBox nor a generated config -- only the generator source +and the committed goldens -- so it costs milliseconds and runs on every +change, instead of only when the E2E job's file matcher fires. + +What it catches is the one coverage failure nothing else does: a newly +emitted table that arrives with no golden. Coverage going the other way (a +table that had a golden becoming empty) already fails the golden comparison, +because the golden file itself changes. +""" + +from tests.e2e.coverage import covered_tables, emitted_tables + + +def test_every_emitted_table_has_golden_coverage(): + missing = sorted(emitted_tables() - covered_tables()) + + assert not missing, ( + "the generator can emit these config_db tables, but they are empty in " + "every file under tests/e2e/golden/: " + ", ".join(missing) + ". Seed a " + "device that populates them under tests/e2e/scenario/resources/ and " + "rewrite the goldens with `make sonic-e2e-regen`, or -- if the table " + "cannot be reached by any fixture -- exclude it here with a comment " + "saying why." + )