[Preetha Veeramalai (preetha-intel)]

Describe your changes

This is the single PR for compiling ONNX models with the OpenVINO plugin EP and assembling the results into an ONNX Runtime model package. Thanks to Xiaoyu Z (Xiaoyu Z (@xiaoyu-work)) for #2732: its improvements are merged here, so #2732 can be closed.

New / extended passes

• EPContextBinaryGenerator compiles with the OpenVINO plugin EP (aot via ModelCompiler, or jit). With target_devices it produces one EPContext model per device as a composite model.
• ORTModelPackage (olive/passes/onnx/model_package.py) assembles the compiled variants into an .ortpackage.

Plugin registration

• OpenVINOExecutionProvider is registered automatically during workflow setup from onnxruntime-ep-openvino, the same way onnxruntime_qnn is handled. An explicit library path still wins.

Device selection

• OrtEpDevices are filtered by the accelerator's device type before ep_device_filters is applied.
• Device IDs match regardless of format: B03E, b03e, 0xB03E, 0x00B03E and integers are equivalent. Malformed IDs are rejected. target_devices is normalized and de-duplicated.
• When a physical and a virtual device share an ID, the one exposing npu_platform is selected.

Output checks and retries

• If compilation or validation fails, the files written by that call are removed, so the workflow can be rerun with the same output path.

EPContext options

• embed_context and disable_cpu_fallback map to ep.context_embed_mode / session.disable_cpu_ep_fallback (and embed_compiled_data_into_model for aot). A setting is on if any source turns it on; the pass flag wins over an explicit "off" and logs a warning. Olive always sets ep.context_enable=1. All other options are passed to ORT unchanged.
• weight_sharing, update_genai_config and genai_config_role are not supported for OpenVINO and raise a ValueError.

ORT versions

Feature	Minimum ONNX Runtime
Plugin-EP compilation	1.23
ep.enable_weightless (raises a ValueError on older ORT)	1.30
Per-device binary naming (set_ep_context_binary_information, skipped with an info log if the binding is missing) and ORTModelPackage	1.32

Docs and install

• The guide is now docs/source/features/ihv-integration/openvino-model-package.md, linked from the IHV index and openvino.md. ORTModelPackage is added to the pass reference.

Testing

• Unit tests cover each item above (test_context_binary.py, test_ort_model_package.py, test_ort_inference.py). They run with mocked plugin devices on ORT 1.32.

Release note: Olive can compile ONNX models with the OpenVINO plugin EP for one or more NPU/GPU devices and package the results as an ONNX Runtime model package. Install with pip install olive-ai[openvino-ep]; requires ONNX Runtime >= 1.23 (>= 1.32 for model packaging).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved path-safety, registration-lifecycle, and package-completeness issues can cause unsafe or unusable output.

Review effort: Balanced
Findings: 2 · 6

Open (8)

• Protect shared registrations from premature unregistration · New
• Prevent path traversal in external-data staging · New
• Use the correct weightless option for supported ORT versions · New
• Reject missing requested device IDs unless partial packaging is enabled · New
• Reject duplicate generated variant names · New
• Handle weights when shared-asset staging is disabled · New
• Copy EPContext binaries using recorded relative paths · New
• Traverse all subgraph and attribute tensors for external data · New

What changed in this PR

Adds OpenVINO model packaging for ONNX Runtime’s OrtModelPackageApi, with per-device variants and shared weights.

Changes:

• Supports compilation and packaging of precompiled models.
• Extends EPContext generation with OpenVINO plugin support.
• Adds pass registration, configuration examples, documentation, and tests.

File	Description
test/​passes/​openvino/​test_openvino_model_package.py	Tests packaging and registration behavior.
olive/​passes/​openvino/​model_package.py	Implements compilation, asset staging, and manifests.
olive/​passes/​openvino/​examples/​model_package_config_template.json	Provides an example workflow.
olive/​passes/​onnx/​context_binary.py	Adds OpenVINO compilation and plugin registration handling.
olive/​olive_config.json	Registers the pass and supported providers.
docs/​README_OpenVINO.md	Documents configuration, usage, and package format.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Mayuresh Varerkar (MayureshV1)]

Preetha Veeramalai (@preetha-intel) . Can you please fix the below suggestions.
Also, overall guidance was to hold off merging this PR till Model Packaging support becomes a stable ABI, which is expected to be soon.

[Xiaoyu Z (xiaoyu-work)]

i'm a little bit confused about this pass. we have a CLI generate-model-package to generate a model package. I'd like to split this into 2 steps: EPContextBinaryGenerator and generate model package. the generating model package logic should be generic and all other ep can use it as well.

[Xiaoyu Z (xiaoyu-work)]

does this pass only create npu package? if we want to add support for cpu, gpu as well to ort-package. will this pass work? that's why i'm thinking having a generic packaging logic will be better here.

[Preetha Veeramalai (preetha-intel)]

The CLI generate-model-package is a GenAI specific model packaging which manually creates the manifest.json and component.json. But the OpenVINOModelPackage builds the package through ORT's ort.ModelPackage API. ORT owns the manifest, component and variant metadata, and the shared-asset hashing and URI.

I agree the packaging should be generic, and it already is in the pass. The ort.ModelPackage calls don't depend on OpenVINO, and only discovery and compile are OpenVINO-specific. Splitting it by moving the ORT-API packaging out of this pass can be done as the follow up once other EP's also want to accomodate the Olive but the CLI's GenAI packaging and the pass's ORT packaging are two different formats, so it can't be one generic CLI path without making it a different mode.

[Xiaoyu Z (xiaoyu-work)]

I've moved the context binary logic into #2732. can you check this pr?

[Preetha Veeramalai (preetha-intel)]

I've moved the context binary logic into #2732. can you check this pr?

I agree on the design where we have the EpContextBinary seperate from model packaging. Initially when this pass was added only OpenVINO use case was there hence this got swayed towards OV support.

I have now modified to have a seperate generic EPContextBinary and ORTModelPackage which uses the ORT modelPackage authoring API's which is very different from the CLI generate-model-package (Made specifically for GenAI where genai_config.json is a neccessity).

Xiaoyu Z (@xiaoyu-work) Can you review the updated code as it handles the end-to-end flow

[Mayuresh Varerkar (MayureshV1)]

Thanks for the restructuring into a generic EPContextBinaryGenerator + ORTModelPackage on top of ORT's authoring API, and for addressing my earlier comments (NVL, load_config removal, failing on missing devices). A few small doc and cleanup items are inline. I'll sync with you offline on sequencing with #2732 and the ORT 1.32 dependency (#33640) before the next round.

Could you also refresh the PR description? It still mentions package_only mode, allow_partial_package and docs/README_OpenVINO.md, which no longer match the code.

[Mayuresh Varerkar (MayureshV1)]

Preetha Veeramalai (@preetha-intel) thanks for the quick turnaround on aaecfaa. Status below.

Fixed in this update ✅

• Docs prerequisites now use onnxruntime>=1.32 + onnxruntime-ep-openvino + openvino (no onnxruntime_openvino).
• Stale "1.30.0 dev/rc" comment in _check_ort_version.
• Docs typo ("thats already available").
• Shared-asset weights are now hard-linked into the staging folder (_link_or_copy), so they're no longer copied twice.
• The leftover-binary check now also covers the binary name set via ep_context_binary_information.
• Boolean compiler options: strings "true"/"false"/"1"/"0" are parsed explicitly, and anything else is rejected.
• EP registration now also checks ort.get_ep_devices() instead of relying only on the "already registered" message text.

Could you resolve the two doc/comment threads that are now fixed?

Still to address

1. PR description. It still mentions package_only mode, allow_partial_package and docs/README_OpenVINO.md. Please update it to describe the current EPContextBinaryGenerator + ORTModelPackage split.
2. Overlap with Add standalone OpenVINO EPContext compilation support #2732. Both PRs still change the OpenVINO compile path in context_binary.py. Let's agree which PR owns EPContextBinaryGenerator and rebase the other on top, so reviewers aren't comparing two versions.
3. Dependency on ORT 1.32 (Add pybindings for Model Packaging onnxruntime#33640). ORTModelPackage needs onnxruntime.ModelPackage, and the native-API tests are skipped in CI. Please mark the PR as draft or blocked until a build with the bindings is available. Then post the results of the native tests run locally against such a build.
4. Code-level items, posted as inline comments:
   • The OpenVINO path ignores embed_context, weight_sharing and disable_cpu_fallback.
   • Session options only needed for generating the model are written into the package's runtime session_options.
   • The EP reuse fallback in registered_ep_library can hide a real registration failure.

[Mayuresh Varerkar (MayureshV1)]

Preetha Veeramalai (@preetha-intel) thanks for the updates in aaecfaa/c1f3b46. The registration fallback warning and the runtime session_options cleanup look good, and the refreshed description matches the code now.

Still open:

1. embed_context / weight_sharing / disable_cpu_fallback on the OpenVINO path. See my follow-up in that thread.
2. Mocked ORTModelPackage tests. They will likely fail on ORT < 1.32 because of the version check (inline).
3. create_shared_asset=False weights check. It is skipped when the source model comes from source_model_path (inline).
4. Overlap with #2732. Both PRs still change the OpenVINO compile path in context_binary.py. Let's settle which PR owns it before the next round.
5. ORT 1.32 / microsoft/onnxruntime#33640. Thanks for moving the PR to draft. Please post local native-test results once a build with the bindings is available.

Xiaoyu Z (@xiaoyu-work) could you take a look at your two open threads? The split into EPContextBinaryGenerator + ORTModelPackage landed in 578fcbc, and the <=1.23.2 question has an answer.

[Mayuresh Varerkar (MayureshV1)]

Preetha Veeramalai (@preetha-intel) thanks for a9d4da1. It covers the version-patched fixture, the source_model_path weights check, the registration-fallback tests and the enum ValueError. Three items are still open, inline:

1. The new EPContext option mapping overrides values the user sets explicitly.
2. _read_compat now calls a function that onnxruntime doesn't export.
3. The doc and docstring still say options are passed verbatim.

[github-advanced-security]

lintrunner found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

[Mayuresh Varerkar (MayureshV1)]

Preetha Veeramalai (@preetha-intel) thanks for e5af08c/579e7e1. The embed/CPU-fallback precedence, the _pybind_state restore and the README mapping table all look good, and the tests cover them.

None of the items below depend on #2732, so they can go in now:

1. CI: Python format fails (job). Everything it reports is in the new tests, apart from one advice-level item in model_package.py:

   • test_context_binary.py:150: from pathlib import Path inside the test reimports and shadows the module-level import (W0404/W0621). Please drop it.
   • RUF043 (match= with unescaped metacharacters). Use raw strings or re.escape:
     • test_context_binary.py:593, 600
     • test_ort_model_package.py:204, 295
   • SIM117 (nested with): test_context_binary.py:637, 650. Combine them into a single with.
   • test_context_binary.py:487, 490: RUF012 (mutable class attribute __members__) and B008 (FakeEnum() as a default argument).
   • W0212 (protected access) throughout both files, and W0621 for the mock_model_package / native_model_package fixtures. Add a file-level # pylint: disable=protected-access,redefined-outer-name like other Olive tests, or use @pytest.mark.usefixtures where the fixture value isn't used.
   • Advice R0916 (too many boolean expressions) at model_package.py:399 in _safe_relative_location. Optional: split the check into named conditions.

   Please run lintrunner -a locally to confirm it's clean.

2. PR description. It still says "ORT options are passed through verbatim; Olive sets no EPContext keys." After a9d4da1, Olive sets ep.context_enable, ep.context_embed_mode and session.disable_cpu_ep_fallback (plus embed_compiled_data_into_model for aot) from embed_context / disable_cpu_fallback. Please update it to match the README.

3. Nit: test_read_compat_reads_from_ort_pybind_module patches with create=True, so it still passes if get_compatibility_info_from_model is missing from _pybind_state. Dropping create=True (and skipping when the attribute is absent) would catch a rename.

Still open, tracked separately: the overlap with #2732, and ORT 1.32 / microsoft/onnxruntime#33640. Please post the local native-test results once a build with the bindings is available.

[Mayuresh Varerkar (MayureshV1)]

Xiaoyu Z (@xiaoyu-work) and I agreed to close #2732 and make #2717 the single PR for OpenVINO plugin-EP compilation and ORT model packaging. Thanks Xiaoyu Z (@xiaoyu-work) for #2732! Its improvements will be merged into #2717, and the PR will then be made available for review.

High-level plan for the next update of #2717:

• Plugin registration: register the OpenVINO plugin EP automatically during workflow setup, so OpenVINOExecutionProvider works without an explicit library path.
• Device selection: select devices by the accelerator's device type, match device IDs regardless of format (B03E, 0xB03E, …), and handle a physical and a virtual device that share an ID.
• Output checks: check that the compiled output contains EPContext nodes and their binaries, and clean up after a failed attempt so the workflow can be rerun.
• ORT versions: compilation will need ONNX Runtime >= 1.23. Only weightless compilation (>= 1.30), per-device binary naming and ORTModelPackage (>= 1.32) need newer versions.
• Docs and install: move the docs under docs/source and add an install extra for the plugin-EP flow.
• Lint and description: fix the lint failures and update the PR description.

I'll work with Preetha Veeramalai (@preetha-intel) to get the PR ready for review soon. It stays in draft until the ORT model-packaging bindings (microsoft/onnxruntime#33640) are merged, since ORTModelPackage depends on them.

Xiaoyu Z (@xiaoyu-work) since #2717 will carry the only version of the _run_qnn refactor, please review that part once it's ready.

[Preetha Veeramalai (preetha-intel)]

The changes are updated to accomodate #2732

[Mayuresh Varerkar (MayureshV1)]

Preetha Veeramalai (@preetha-intel) thanks for 5f2ca41. It brings over everything from #2732:

• automatic plugin registration
• device selection by accelerator type
• device-ID matching in any format (now stricter, with the extra invalid-ID cases)
• EPContext output checks and cleanup after a failed attempt
• the 1.23 / 1.30 / 1.32 version split
• the docs move and the openvino-ep extra

The lint and create=True items from my last review are addressed too.

A few things before this is ready for maintainer review:

1. PR description. It still says "ORT options are passed through verbatim; Olive sets no EPContext keys." Please update it to match the docs:
   • embed_context / disable_cpu_fallback map to ep.context_embed_mode / session.disable_cpu_ep_fallback (and embed_compiled_data_into_model for aot), and Olive always sets ep.context_enable=1.
   • Mention that it supersedes Add standalone OpenVINO EPContext compilation support #2732, crediting Xiaoyu Z (@xiaoyu-work).
   • List the ORT requirements: 1.23 for compilation, 1.30 for ep.enable_weightless, 1.32 for per-device binary naming and ORTModelPackage.
2. Validation results. CI can't exercise the plugin EP or the native ModelPackage path, so please post results from real runs:
   • the docs example via olive run in a clean venv (pip install -e ".[openvino-ep]", no EP library path in the config)
   • virtual NPU target_devices ["643E", "B03E", "D71D"] with aot and with jit
   • single-target compiles on a physical NPU and a physical GPU without ep_device_filters
   • EPContextBinaryGenerator on an ORT release older than 1.32 (default binary names)
   • the ORTModelPackage native tests on ORT with Add pybindings for Model Packaging onnxruntime#33640
3. Docs nit (openvino-model-package.md, Prerequisites). pip install "olive-ai[openvino-ep]" after pip install -e . replaces the editable install with the PyPI release, which doesn't have the openvino-ep extra yet. A single pip install -e ".[openvino-ep]" avoids that.
4. CI. The workflows on 5f2ca41 are waiting for approval. Xiaoyu Z (@xiaoyu-work) could you approve the runs? Please also take a look at the _run_qnn refactor, since this PR now carries the only version of it.

Once CI is green and the results are posted, I'll re-review and clear my earlier change request.

[Preetha Veeramalai (preetha-intel)]

Preetha Veeramalai (Preetha Veeramalai (@preetha-intel)) thanks for 5f2ca41. It brings over everything from #2732:

• automatic plugin registration
• device selection by accelerator type
• device-ID matching in any format (now stricter, with the extra invalid-ID cases)
• EPContext output checks and cleanup after a failed attempt
• the 1.23 / 1.30 / 1.32 version split
• the docs move and the openvino-ep extra

The lint and create=True items from my last review are addressed too.

A few things before this is ready for maintainer review:

1. PR description. It still says "ORT options are passed through verbatim; Olive sets no EPContext keys." Please update it to match the docs:

   • embed_context / disable_cpu_fallback map to ep.context_embed_mode / session.disable_cpu_ep_fallback (and embed_compiled_data_into_model for aot), and Olive always sets ep.context_enable=1.
   • Mention that it supersedes Add standalone OpenVINO EPContext compilation support #2732, crediting Xiaoyu Z (Xiaoyu Z (@xiaoyu-work)).
   • List the ORT requirements: 1.23 for compilation, 1.30 for ep.enable_weightless, 1.32 for per-device binary naming and ORTModelPackage.

2. Validation results. CI can't exercise the plugin EP or the native ModelPackage path, so please post results from real runs:

   • the docs example via olive run in a clean venv (pip install -e ".[openvino-ep]", no EP library path in the config)
   • virtual NPU target_devices ["643E", "B03E", "D71D"] with aot and with jit
   • single-target compiles on a physical NPU and a physical GPU without ep_device_filters
   • EPContextBinaryGenerator on an ORT release older than 1.32 (default binary names)
   • the ORTModelPackage native tests on ORT with Add pybindings for Model Packaging onnxruntime#33640

3. Docs nit (openvino-model-package.md, Prerequisites). pip install "olive-ai[openvino-ep]" after pip install -e . replaces the editable install with the PyPI release, which doesn't have the openvino-ep extra yet. A single pip install -e ".[openvino-ep]" avoids that.

4. CI. The workflows on 5f2ca41 are waiting for approval. Xiaoyu Z (Xiaoyu Z (@xiaoyu-work)) could you approve the runs? Please also take a look at the _run_qnn refactor, since this PR now carries the only version of it.

Once CI is green and the results are posted, I'll re-review and clear my earlier change request.

The comments are addressed along with porting from #2732 completely to #2717.
Fixed the failing CI tests as well.

Xiaoyu Z (@xiaoyu-work) Can you kindly help retrigger the pipelines.

[Mayuresh Varerkar (MayureshV1)]

Preetha Veeramalai (@preetha-intel) thanks for d4b86de. The #2732 port, the updated description and the docs install fix cover all the items we raised, and I found nothing further in the new code. Everything looks good from our side.

Xiaoyu Z (@xiaoyu-work) this is ready for you: could you approve the CI runs on d4b86de and review the PR, including the _run_qnn refactor?

[Mayuresh Varerkar (MayureshV1)]

Changes look good. Waiting for CI to pass to approve !

[Xiaoyu Z (xiaoyu-work)]

There is a failed test:

_ test_maybe_register_ep_libraries_warns_only_when_deprecated_openvino_wheel_installed[True] _
test\common\test_ort_inference.py:116: in test_maybe_register_ep_libraries_warns_only_when_deprecated_openvino_wheel_installed
    assert ("onnxruntime-openvino is deprecated" in caplog.text) == builtin_library_exists
E   AssertionError: assert ('onnxruntime-openvino is deprecated' in '') == True
E    +  where '' = <_pytest.logging.LogCaptureFixture object at 0x000001BA5C95AF90>.text