Repository navigation
Discard invalid choices before yielding search samples - #2778
林SO (Linxiushen) wants to merge 2 commits into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
Discarding invalid points bypasses the global stop check, allowing searches to exceed max_iter and max_time; the discard path needs stop enforcement and a bounded-search regression test.
1 open issue
What changed in this PR
This PR corrects search-point validation so Olive discards invalid conditional choices before yielding optimization samples or counting them as suggestions.
Changes:
- Unpack stored selections before validating values and nested choices.
- Add controls for invalid, ignored, and falsy values.
- Test conditional rejection across sequential/random samplers and joint/pass-by-pass searches.
| File | Description |
|---|---|
test/search/test_search_strategy.py |
Tests invalid-choice rejection and sample counts across search modes. |
test/search/test_search_point.py |
Adds flat and nested validation controls, including accepted values. |
olive/search/search_point.py |
Validates selected values rather than index-value pairs. |
Review effort: Balanced
|
|
||
| def _is_valid(values: dict[str, tuple[int, Any]]) -> bool: | ||
| for v in values.values(): | ||
| for _, v in values.values(): |
There was a problem hiding this comment.
Reproduced on the submitted head: both sequential and random samplers consumed all 100 invalid points despite the global iteration/time limits. The discard path now checks the existing global stop criterion before continuing, preserving the current post-step boundary behavior.
Added six regressions for both samplers (iteration limits 0/1 and a deterministic elapsed-time limit). They fail against the unchanged submitted production code, and the fix passes all 37 native test/search tests. Ruff, native formatting and Pylint also pass. Local validation covers CPU search/configuration behavior; the unrelated root model-building fixture was excluded with --confcutdir=test/search.

Describe your changes
SearchPointstores each selection as an(index, value)pair, butis_valid()checks the pair itself forINVALIDand nested choices. As a result, invalid conditional choices pass the strategy's discard gate. A two-choice search currently yields[None, valid_config]throughsample.passes_configsand counts both as suggested samples. The engine already skips theNoneconfiguration; this fixes the earlier strategy boundary.Unpack the selection before applying the existing recursive validation. Add flat and nested search-point controls, preserve ignored/falsy choices, and exercise conditional rejection through sequential/random samplers in joint/pass-by-pass searches.
Validation on base
1647eb2b589ae8a82b2dbe4429e5fe4335c40092:test/searchpass after the fix, with zero skips. Command:python -m pytest --confcutdir=test/search -o addopts= test/search -q --basetemp=<fresh-local-directory>.lintrunner -a --take RUFF-FORMAT,PYLINTand direct repository-config Ruff pass on all three changed files, as doesgit diff --check. The native Ruff adapter's Windows canonical paths miss the existingtest/**namespace exemption; direct Ruff uses that exemption without source suppressions.Follow-up on the automated review: the newly effective discard path must also honor the global stop criteria. It now checks the same
should_stopcondition before continuing, preserving the existing post-step iteration boundary behavior. Six additional regressions cover sequential/random samplers with iteration limits 0/1 and a deterministic elapsed-time limit. All six fail on the unchanged submitted head; the final code passes all 37test/searchtests, zero skips, using the command and CPU-only fixture scope above. Selected native formatting/Pylint, direct repository-config Ruff andgit diff --checkpass for the follow-up files.Checklist before requesting a review
lintrunner -a(selected format/Pylint adapters; Ruff checked directly as described above).(Optional) Issue link
Found while inspecting the search admission path. The reporting-only
SearchPoint.to_dict()addition in #2591 covers a separate behavior.