Repository navigation
perf(bench): Wave 0 full-pipeline throughput on one A6000 (binding resource per stage) - #242
Merged
Merged
Conversation
…tion photo set wave0_suite.py sweeps client concurrency and Triton instance, queue-delay and preferred-batch arms (applied and restored through the model store), measures per-endpoint stage cost, and attributes API CPU with a thread census and py-spy. os_write_ceiling.py replays ingest-shaped documents into fresh indexes with the real mapping. ingest_micro_profile.py times the per-image primitives, wave0_report.py renders the tables. fetch_hires_set.py pins 500 CC-licensed 12 to 24 MP photos (commons_hires_500.json). baseline_suite.py gains --skip-cluster and --skip-settle and a pluggable gauges class. Refs #45, #210, #230.
One A6000, FP16 PE engine, sets A (COCO) and B (public 12 to 24 MP photos): client sweep, Triton arms, faces and OCR stage cost, API CPU attribution, OpenSearch write ceiling, binding resource per stage, and what the serving trade study's ceiling table got right and wrong. Raw JSON under docs/benchmarks/. Refs #45, #210, #230.
…ghput # Conflicts: # CHANGELOG.md
…ghput # Conflicts: # CHANGELOG.md # docs/PERFORMANCE.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #45, #210, #230. Also opens #239, #240, #241.
Extends the v0.6.0 baseline from the 4-client COCO run to the whole pipeline on one RTX A6000 (GPU 2, idle, FP16 PE engine, v0.5.0 images, isolated project, loopback only), 3 repetitions, medians in
docs/PERFORMANCE.md("0.6.0 Wave 0: full-pipeline throughput"), raw JSON indocs/benchmarks/v060_wave0_*.json.Headlines
preferred_batch_sizechange nothing (within 3 %).fetch_hires_set.py,commons_hires_500.json): host-bound, 1.0 to 1.3 API CPU-s per image (0.18 on COCO), 3 to 11 images/s, GPU 5 to 15 % busy; 98.5 % of API CPU is on the event-loop threads; throughput falls past 32 clients (perf(ingest): on 12-24 MP photos the API binds, not the GPU (1.2 CPU-s per image, 98 percent on the event-loop thread, throughput falls past 32 clients) #241).ocr_pipelineinstance (42 with 2, 80 with 4, perf(triton): ocr_pipeline runs one instance, so OCR is capped at about 21 images/s (2 instances 42, 4 instances 80) #240).ef_construction128 cuts index CPU per document by 38 %.Code:
wave0_suite.py,wave0_probes.py,wave0_lib.py,wave0_report.py,os_write_ceiling.py,ingest_micro_profile.py;baseline_suite.pygains--skip-cluster,--skip-settleand a pluggable gauges class. Tests undertests/benchandtests/datasets.Checks:
pre-commit run --from-ref origin/main --to-ref HEADexits 0,scripts/docs/check_docs_vs_code.pyclean,pytest tests/bench tests/datasetspasses.