Gate on full config_db table coverage - #2571
Draft
ideaship wants to merge 1 commit into
Draft
Conversation
ideaship
changed the title
This was referenced Aug 5, 2026
berendt
force-pushed
the
sonic-e2e-v2-coverage-gate
branch
from
August 5, 2026 15:10
369711b to
34a693a
Compare
ideaship
force-pushed
the
sonic-e2e-v2-coverage-gate
branch
from
August 5, 2026 19:51
34a693a to
bdf2899
Compare
berendt
force-pushed
the
sonic-e2e-v2-coverage-gate
branch
from
August 6, 2026 10:56
bdf2899 to
8793a02
Compare
ideaship
force-pushed
the
sonic-e2e-v2-coverage-gate
branch
from
August 6, 2026 12:10
8793a02 to
f0dc3b3
Compare
tests/e2e/coverage.py reports which config_db tables the golden set reaches, but nothing consumes it: it is wired into neither sonic_golden_test.sh nor the Zuul job, so the number is only seen by someone who runs `make sonic-e2e-coverage` by hand. That is how the "38 of 38" claim came to exist as an unverifiable one-off in the first place, and leaving it human-run lets it decay the same way again. Add a unit test asserting the set of emitted-but-uncovered tables is empty. It closes the one coverage failure nothing else catches: a newly emitted table arriving with no golden. Coverage lost in the other direction -- a table that had a golden becoming empty -- already fails the golden comparison, because the golden file itself changes, and the regeneration path is covered separately by the coverage guard in compare.py. It belongs in the unit suite rather than the E2E job because it needs neither NetBox nor a generated config, only the generator source and the committed goldens. It therefore costs milliseconds and runs on every change, where the E2E job runs on a file matcher and a 2400s budget. This needs no .zuul.yaml change: coverage.py already exposes emitted_tables() and covered_tables(). The assertion message names the missing tables and points at `make sonic-e2e-regen`, because the cost of this gate is that adding a generator table now obliges the same change to add golden coverage, and that is a full regeneration cycle rather than a two-line edit. If a table genuinely cannot be reached by any fixture, the message says to exclude it here with a stated reason -- one visible exception, not a silently growing allowlist. This lands after the last scenario because it can only pass once the golden set is complete; the report itself lands with the first goldens, where it is still useful at 30 of 38. Assisted-by: Claude:claude-opus-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
sonic-e2e-v2-coverage-gate
branch
from
August 6, 2026 12:19
f0dc3b3 to
63b1f23
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the series tracked in #2562, which explains the ordering and what each PR covers. Based on the preceding PR in the stack, so review only the top commits here.
Thirty lines, and the reason the coverage number cannot quietly rot: a unit test
asserting that no table the generator can emit is left without a golden.
It closes the one coverage failure nothing else catches — a newly emitted
table arriving with no golden. Coverage lost the other way (a table that had a
golden becoming empty) already fails the golden comparison, because the golden
file itself changes.
It lives in the unit suite rather than the E2E job because it needs neither
NetBox nor a generated config, only the generator source and the committed
goldens. So it costs milliseconds and runs on every change, where the E2E job
runs on a file matcher and a 2400s budget. No
.zuul.yamlchange is needed.It lands last because it can only pass once the golden set is complete — which
is what the PR below achieves.