Repository navigation
perf(triton): size ocr_pipeline instances per profile within a memory budget - #243
Merged
Merged
Conversation
… budget The Python BLS model runs one request per instance, so a single instance capped OCR at about 21 images/s (42 with two, 80 with four). The installer profiles now set ocr_pipeline_instances (1, 2, 4), the model ships 2, and a per-instance GPU memory figure plus a per-profile Python budget are checked by an offline test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #240
What
models/ocr_pipelineships 2 instances (was 1). The installer profiles set the count through a newocr_pipeline_instancesfield: minimal 1, standard 2, full 4.load_profilereads it andgenerate_all_configsnow applies it toocr_pipeline, which no profile touched before.GPU_MIB_PER_INSTANCE(512) and each profile haspython_backend_budget_mib.tests/test_triton_instance_budget.pyparses the committed configs and fails when a Python-backend count exceeds the budget. It also runs the real shell path per profile on a copy.docs/PERFORMANCE.md(tuning section),INSTALLATION.mdprofile list, CHANGELOG.Measured (live, isolated project
ocrinst, Triton only, GPU 1, 12 GB, 0% foreign utilisation, torn down)Whole-card memory with det and rec engines loaded and the Python instances warm (640x480 frames, 16 clients): 1 instance 11082 MiB, 2 instances 11337, 4 instances 11850, so about 256 MiB per instance. The 512 MiB budget figure is 2x that for larger frames. 16 MP frames were not measured (the throwaway container had the default 64 MB shm). Throughput on this card was client-bound and only confirms the scaling (2.7, 5.5, 11.4 images/s for 1, 2, 4 instances); the throughput numbers in the issue come from Wave 0 on the A6000.
det and rec counts and batching are unchanged: Wave 0 showed GPU time under 10 ms per image for both, so the Python instance is the limit.
Tests
New test failed first (10 failed before the change, 10 pass after).
pre-commit run --from-ref origin/main --to-ref HEADexit 0.