Skip to content

Discard invalid choices before yielding search samples - #2778

Open
林SO (Linxiushen) wants to merge 2 commits into
microsoft:mainfrom
Linxiushen:codex/fix-invalid-search-points
Open

林SO (Linxiushen) wants to merge 2 commits into
microsoft:mainfrom
Linxiushen:codex/fix-invalid-search-points

Conversation

@Linxiushen

@Linxiushen 林SO (Linxiushen) commented Oct 31, 2026 •

Copy link
Copy Markdown

Describe your changes

SearchPoint stores each selection as an (index, value) pair, but is_valid() checks the pair itself for INVALID and nested choices. As a result, invalid conditional choices pass the strategy's discard gate. A two-choice search currently yields [None, valid_config] through sample.passes_configs and counts both as suggested samples. The engine already skips the None configuration; this fixes the earlier strategy boundary.

Unpack the selection before applying the existing recursive validation. Add flat and nested search-point controls, preserve ignored/falsy choices, and exercise conditional rejection through sequential/random samplers in joint/pass-by-pass searches.

Validation on base 1647eb2b589ae8a82b2dbe4429e5fe4335c40092:

  • The eight new controls give 7 failures and 1 pass before the fix.
  • All 31 tests in test/search pass after the fix, with zero skips. Command: python -m pytest --confcutdir=test/search -o addopts= test/search -q --basetemp=<fresh-local-directory>.
  • The root autouse fixture is excluded because it creates unrelated ONNX/model fixtures. These tests exercise the actual search implementation; no model, optimization run, service, or GPU was used. The full repository suite was not run.
  • Native lintrunner -a --take RUFF-FORMAT,PYLINT and direct repository-config Ruff pass on all three changed files, as does git diff --check. The native Ruff adapter's Windows canonical paths miss the existing test/** namespace exemption; direct Ruff uses that exemption without source suppressions.

Follow-up on the automated review: the newly effective discard path must also honor the global stop criteria. It now checks the same should_stop condition before continuing, preserving the existing post-step iteration boundary behavior. Six additional regressions cover sequential/random samplers with iteration limits 0/1 and a deterministic elapsed-time limit. All six fail on the unchanged submitted head; the final code passes all 37 test/search tests, zero skips, using the command and CPU-only fixture scope above. Selected native formatting/Pylint, direct repository-config Ruff and git diff --check pass for the follow-up files.

Checklist before requesting a review

  • Add unit tests for this change.
  • Make sure all tests can pass. (The final 37 selected search tests pass; full suite not run.)
  • Update documents if necessary. (No documentation change needed.)
  • Lint and apply fixes to your code by running lintrunner -a (selected format/Pylint adapters; Ruff checked directly as described above).
  • Is this a user-facing change? If yes, give a description of this change to be included in the release notes: Discard invalid conditional choices before yielding optimization search samples.

(Optional) Issue link

Found while inspecting the search admission path. The reporting-only SearchPoint.to_dict() addition in #2591 covers a separate behavior.

Copilot AI balanced review requested due to automatic review settings October 31, 2026 12:20
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Discarding invalid points bypasses the global stop check, allowing searches to exceed max_iter and max_time; the discard path needs stop enforcement and a bounded-search regression test.

1 open issue
What changed in this PR

This PR corrects search-point validation so Olive discards invalid conditional choices before yielding optimization samples or counting them as suggestions.

Changes:

  • Unpack stored selections before validating values and nested choices.
  • Add controls for invalid, ignored, and falsy values.
  • Test conditional rejection across sequential/random samplers and joint/pass-by-pass searches.
File Description
test/​search/​test_search_strategy.py Tests invalid-choice rejection and sample counts across search modes.
test/​search/​test_search_point.py Adds flat and nested validation controls, including accepted values.
olive/​search/​search_point.py Validates selected values rather than index-value pairs.

Review effort: Balanced


def _is_valid(values: dict[str, tuple[int, Any]]) -> bool:
for v in values.values():
for _, v in values.values():

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced on the submitted head: both sequential and random samplers consumed all 100 invalid points despite the global iteration/time limits. The discard path now checks the existing global stop criterion before continuing, preserving the current post-step boundary behavior.

Added six regressions for both samplers (iteration limits 0/1 and a deterministic elapsed-time limit). They fail against the unchanged submitted production code, and the fix passes all 37 native test/search tests. Ruff, native formatting and Pylint also pass. Local validation covers CPU search/configuration behavior; the unrelated root model-building fixture was excluded with --confcutdir=test/search.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants