re-run denylisted tests for test_progs-bpf_gcc - #233
Draft
vineetgarc wants to merge 1 commit into
Draft
Conversation
The GCC BPF denylist is large and static, so tests stay on it long after the compiler has learnt to build them. Nothing in CI notices, because denylisted tests are never run. After the main test_progs-bpf_gcc pass, run a second pass over the tests the denylist excluded, by passing DENYLIST.test_progs-bpf_gcc back in as an allowlist. Tests reported as passing there are candidates for removal from the denylist. Only the runner specific list is re-run, not the merged denylist the main pass uses. Entries in the generic lists are excluded because they are broken or unstable in the VM rather than because of the compiler, so re-running them would risk taking the VM down without saying anything about GCC. The list is consumed as-is: test_progs parses list files itself, skipping blank lines and '#' comments, so it does not need to be normalized first. The second pass is informational. Its exit code is deliberately not appended to the exit status file, so tests that are still failing, which is the expected outcome for most of them, cannot fail the job. No JSON summary is written either: run.sh feeds every test_*.json to print_test_summary.py, which truncates the step summary for each one, so a second file there would hide the results of the main pass. Signed-off-by: Vineet Gupta <vineet.gupta@linux.dev>
vineetgarc
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GCC denylist is static and tests listed there are just never run. This means it can potentially miss out improvements due to compiler/selftest updates/improvements.
So ran an additional second pass over just denylist as allowlist. This run is purely informational: its exit code is deliberately not appended to the exit status file, so tests that are still failing, which is the expected outcome for most of them, cannot fail the job.