diff --git a/docs/reference-matrix.md b/docs/reference-matrix.md index b53e9f979..decaaa985 100644 --- a/docs/reference-matrix.md +++ b/docs/reference-matrix.md @@ -448,6 +448,7 @@ the candidate mechanisms. | [Issue #60 report](https://github.com/InferMatrixLab/InferRSI/issues/60#issuecomment-6310920198), 2026-10-21; eight Git source inventories in the [owner metadata](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json) | Task/quality/workload separation across text, agentic coding, work agents, TTS, full-duplex interaction and image/video; selected pinned AIPerf/realtime/FDB/video sources rechecked and the current InferRSI metric contract inspected | Extend [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md) with [task selection](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/task-benchmark-selection.md) and [command templates](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/benchmark-commands.md). Candidate guidance; source review and bounded smoke do not qualify all tasks or establish a performance gain. Replay scores, reference parity and task quality retain separate meanings. | | [NVIDIA/NeMo-Speech.cpp @ 20999cc9](https://github.com/NVIDIA/NeMo-Speech.cpp/tree/20999cc921d2b14aa58708f61756a8b2e787ba20), inspected 2026-11-01 | Selected cache-aware encoder/config, runner, SDK bounds, endpoint policy, geometry test/CMake and ASR benchmark; complete 14-file readset and Apache-2.0 / parakeet MIT attribution retained in the reference | Extend [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md) with [cache-aware ASR stream boundaries](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/cache-aware-asr-stream-boundaries.md): first/steady mel geometry and decoded clock, stream-slot reset/reuse, EOU/EOF/reset and model-test SKIP boundaries. Source-derived candidate / empirical not_run; benchmark figures remain author-reported with precision, FE and trial limits. No model/device/quality/long-audio/load/service qualification. | | [yuekaizhang/faster-cosyvoice @ f1049c59](https://github.com/yuekaizhang/faster-cosyvoice/tree/f1049c59196c2c1a890079abc56056b43f27cd1d), inspected 2026-11-03; overall source rights UNKNOWN, selected file headers retain Apache-2.0 and unresolved inherited attribution | Bounded PCM client/server-credit, chunk planner, session/worker, Token2Wav and timing/quality-report inspection; [exact readset and per-file rights](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/faster-cosyvoice-source.json) | Extend [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md) with [dependency-aware TTS playout](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/dependency-aware-tts-playout.md): route-origin credit versus client zero-buffer playback, same-session Mel dependencies, cumulative vocoding and partial-cost/token-derived denominators. Original analysis with immutable links only; candidate / empirical not_run / efficacy not_evaluated. Author reports remain separate; no copied implementation/media, new runtime or hardware/quality/service qualification. | +| NVIDIA NCU 2025.3, Nsight Systems 2025.3, CUDA 12.9.1/10.2 archives, hash-pinned SMI document; Triton v3.2.0 @ `9641643da6c52000c807b5eeed05edaec4402a67`; PyTorch 2.7; [robust-kbench paper v1](https://arxiv.org/html/2509.14279v1); [typed source inventory](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json) | Selected clock/cache/replay, timing/graph, metric-name, flush, extension-identity and benchmark-exploit contracts, inspected 2026-11-15 | Extend `inference-benchmark-design` with [GPU measurement validity and profiler attribution](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md). Candidate / empirical not_run / efficacy not_evaluated; normalization/adoption is an unverified author proposal, four task-supplied cases lack raw evidence, and source review does not establish a gain or current-device metric availability. | ## SGLang versioned entry selection diff --git a/docs/searchspace-matrix.md b/docs/searchspace-matrix.md index 41756351f..bc1601d84 100644 --- a/docs/searchspace-matrix.md +++ b/docs/searchspace-matrix.md @@ -447,6 +447,14 @@ separate from the public release's inspected code paths. | Accelerator / Host / service boundary | Observe / Verify: select task quality and reference parity; freeze workload, timing, SLOs, units and failure accounting; distinguish agent replay from live task success | [inference-benchmark-design task reference](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/task-benchmark-selection.md) and [commands](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/benchmark-commands.md) | Cross-task measurement guidance, not new measured hardware coverage. A text HTTP smoke cannot validate TTS, duplex, image/video quality, repeated A/B or production capacity. No runtime adapter or automatic evaluator is added. | | Accelerator / Host / decoded-audio boundary | Observe / Verify: derive cache-aware ASR startup/steady mel geometry and decoded frontier; distinguish slot reset/reuse, EOU, EOF and full stream reset; retain optional-model test coverage | [inference-benchmark-design ASR reference](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/cache-aware-asr-stream-boundaries.md) | NeMo-Speech.cpp `20999cc921d2b14aa58708f61756a8b2e787ba20`; selected source and test definitions, author ASR benchmark only. Candidate / empirical not_run; no executed model, device, long-stream/load, ASR quality or service gain and no new runtime adapter. | +[GPU measurement validity and profiler attribution](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md) +extends `inference-benchmark-design` with clock/cache controls, conditional +within-run attribution, work-count signatures and timing-trap checks. This is +evaluation guidance, not a hardware-resource optimization mechanism or added +numbered-layer coverage; the owner's N/A declaration remains applicable. +Candidate / empirical not_run / efficacy not_evaluated; task-supplied numerical +observations are unverified and establish no device or service gain. + ## SGLang scope selection Use the [versioned entry comparison](../src/inferrsi/knowledge/data/skill_library/repositories/sglang/SKILL.md#choose-among-source-pins) diff --git a/docs/skill-coverage.md b/docs/skill-coverage.md index ab2806100..76135a095 100644 --- a/docs/skill-coverage.md +++ b/docs/skill-coverage.md @@ -4,7 +4,7 @@ Generated from the registry, the skill sidecars and the two maintained matrices. **Do not edit by hand**; regenerate after any registry, skill, matrix or frozen-plan change. -- Input digest: `sha256:fbc4d29e40a03beefb6c204f5defd7f817f5d5f68fb102926b3e746bc8801eee` +- Input digest: `sha256:d076a0096bce36fa313827cd89396972001950a685f39a087ba5e4b5d0161664` - Regenerate: `python tools/build_skill_coverage.py` - Verify without writing: `python tools/build_skill_coverage.py --check` @@ -132,7 +132,7 @@ optimization is fully covered or that any procedure has measured benefit. | [choose-parallel-training-plan](../src/inferrsi/knowledge/data/skill_library/methods/choose-parallel-training-plan/SKILL.md) | common | Compare stage, device-mesh and operator-sharding candidates for static distributed training while accounting for ... | Comparing stage, device-mesh and operator-sharding candidates for static distributed ... | failure_modes, input_contract, produced_artifact, refusal_point
3 advice | 1 Accelerator · Diagnose / Configure / Optimize / Verify
4 Network · Diagnose / Configure / Optimize / Verify | 1 · Systems sources (Alpa — [2201.12023v3](https://arxiv.org/abs/2201.12023v3)) | candidate · not_run · 1 readset(s) | no-eval | | [dense-performance-evidence](../src/inferrsi/knowledge/data/skill_library/methods/dense-performance-evidence/SKILL.md) | common | Taxonomy of performance evidence an optimization agent needs, ordered by collection cost and diagnostic power. Maps ... | Deciding which performance evidence to collect next and which claim each evidence level ... | failure_modes, input_contract, produced_artifact, refusal_point
4 advice | 1 Accelerator · Observe / Profile / Verify
2 RAM/Host · Verify
4 Network · Verify | 1 · Systems sources (NVIDIA/TensorRT-LLM @ f33e0e5fd)
2 · Methodology and corpus references (JiusiServe/vllm-evolve @ 43cc82c)
2 · Methodology and corpus references (microsoft/SkillOpt @ 343db22) | candidate · not_run · 0 readset(s) | — | | [evaluate-elastic-serving-placement](../src/inferrsi/knowledge/data/skill_library/methods/evaluate-elastic-serving-placement/SKILL.md) | common | Compare migrating an LLM request between model replicas with changing its sequence-parallel execution group, including ... | Comparing migration of a running LLM request between compatible model replicas with ... | failure_modes, input_contract, produced_artifact, refusal_point
3 advice | 1 Accelerator · Diagnose / Configure / Optimize / Verify
2 RAM/Host · Diagnose / Configure / Optimize / Verify
4 Network · Diagnose / Configure / Optimize / Verify | 1 · Systems sources (Llumnix — [2406.03243v1](https://arxiv.org/abs/2406.03243v1)) | candidate · not_run · 1 readset(s) | no-eval | -| [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md) | common | Select task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents ... | Selecting task quality gates, reference parity checks and performance workloads for text ... | failure_modes, input_contract, produced_artifact, refusal_point
7 advice | N/A: SKILL.md §§ Select the task and its evaluation contract, Statistical requirements, Shared-machine measurement discipline, and Search integrity and final-state verification design comparable evaluation; hardware identity and allocation isolation are comparison controls, not a hardware-resource mechanism. | 2 · Methodology and corpus references (JiusiServe/vllm-evolve @ 43cc82c)
2 · Methodology and corpus references ([Neroued/ninfer](https://github.com/Neroued/ninfer/tree/1882c6adbc9804 ...)
2 · Methodology and corpus references ([Xe-Forge repo + paper v1](https://github.com/IntelLabs/Xe-Forge/tree/ ...)
Task-specific benchmark selection (issue #60) ([Issue #60 report](https://github.com/InferMatrixLab/InferRSI/issues/6 ...)
Task-specific benchmark selection (issue #60) ([NVIDIA/NeMo-Speech.cpp @ 20999cc9](https://github.com/NVIDIA/NeMo-Spe ...)
Task-specific benchmark selection (issue #60) ([yuekaizhang/faster-cosyvoice @ f1049c59](https://github.com/yuekaizha ...) | candidate · not_run · 13 readset(s) | — | +| [inference-benchmark-design](../src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md) | common | Select task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents ... | Selecting task quality gates, reference parity checks and performance workloads for text ... | failure_modes, input_contract, produced_artifact, refusal_point
8 advice | N/A: SKILL.md §§ Select the task and its evaluation contract, Statistical requirements, Shared-machine measurement discipline, and Search integrity and final-state verification design comparable evaluation; hardware identity and allocation isolation are comparison controls, not a hardware-resource mechanism. | 2 · Methodology and corpus references (JiusiServe/vllm-evolve @ 43cc82c)
2 · Methodology and corpus references ([Neroued/ninfer](https://github.com/Neroued/ninfer/tree/1882c6adbc9804 ...)
2 · Methodology and corpus references ([Xe-Forge repo + paper v1](https://github.com/IntelLabs/Xe-Forge/tree/ ...)
Task-specific benchmark selection (issue #60) (NVIDIA NCU 2025.3, Nsight Systems 2025.3, CUDA 12.9.1/10.2 archives ...)
Task-specific benchmark selection (issue #60) ([Issue #60 report](https://github.com/InferMatrixLab/InferRSI/issues/6 ...)
Task-specific benchmark selection (issue #60) ([NVIDIA/NeMo-Speech.cpp @ 20999cc9](https://github.com/NVIDIA/NeMo-Spe ...)
Task-specific benchmark selection (issue #60) ([yuekaizhang/faster-cosyvoice @ f1049c59](https://github.com/yuekaizha ...) | candidate · not_run · 14 readset(s) | — | | [inference-optimization-casebook](../src/inferrsi/knowledge/data/skill_library/methods/inference-optimization-casebook/SKILL.md) | common | How to structure and use a casebook of past optimization decisions as a decision-support tool. Extracts the casebook ... | Structuring and querying a casebook of past optimization decisions as decision support ... | failure_modes, input_contract, produced_artifact, refusal_point
4 advice | 1 Accelerator · Optimize | 1 · Systems sources (NVIDIA/TensorRT-LLM @ f33e0e5fd) | candidate · not_run · 0 readset(s) | no-eval | | [kernel-writing-guide](../src/inferrsi/knowledge/data/skill_library/methods/kernel-writing-guide/SKILL.md) | common | Decision guide for writing custom inference kernels: when to write one, which framework to choose (Triton, CuTe DSL ... | Deciding whether to write a custom inference kernel, routing a kernel pattern to Triton ... | failure_modes, input_contract, produced_artifact, refusal_point
6 advice | 1 Accelerator · Optimize / Diagnose / Configure / Verify | 1 · Systems sources (Dao-AILab/causal-conv1d @ `cd81f0413cad2fc1e6f17e785ac39f59aae690cd` ...)
1 · Systems sources (NVIDIA/TensorRT-LLM @ f33e0e5fd)
1 · Systems sources (huggingface/transformers @ `0dad7b822255a0ae261ec45ae937371e859ffd1a` ...)
1 · Systems sources (mindspore-ai/akg @ `5aa15f3fe4d856eb60cd3b27447e04f5f1fb4dff` ...)
1 · Systems sources (sgl-project/sglang @ `16176278d0180f1a1a578e53e6eef432577876c2` ...)
1 · Systems sources (vllm-project/vllm @ `537a79f0fc5ba564d2bef43d54915b2f80f9f77d` (root ...) | candidate · not_run · 7 readset(s) | no-eval | | [optimization-search-policy](../src/inferrsi/knowledge/data/skill_library/methods/optimization-search-policy/SKILL.md) | common | Policy for how an optimization agent should select, order, and stop optimization experiments. Covers candidate ... | Selecting, ordering and stopping optimization experiments: candidate generation ... | failure_modes, input_contract, produced_artifact, refusal_point
5 advice | 1 Accelerator · Optimize | 2 · Methodology and corpus references (JiusiServe/vllm-evolve @ 43cc82c)
2 · Methodology and corpus references (RRSI — arXiv 2609.24972v3)
2 · Methodology and corpus references (arXiv 2608.01867 (CRISP))
2 · Methodology and corpus references (microsoft/SkillOpt @ 343db22) | candidate · not_run · 9 readset(s) | no-eval | diff --git a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json index e0d8fa184..8de8c664e 100644 --- a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json +++ b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/_provenance.json @@ -1,7 +1,7 @@ { "schema_version": 1, "repository": "https://github.com/InferMatrixLab/InferRSI", - "source_revision": "1e9c140f489f83ca3ecfba0e6e8b416ddb5091cc", + "source_revision": "3671a98ad340fb19e6595d1c8d818c7e9ca2577a", "registry": { "path": "src/inferrsi/knowledge/data/skill_library/skill-index.json", "sha256": "7904275c3383380db179287171e1430617aaa9ac4f439291eff8a71426987fbb" diff --git a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json index 6fba4e837..ee934336c 100644 --- a/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json +++ b/src/inferrsi/knowledge/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json @@ -34,7 +34,10 @@ "candidate-selection", "accepted-prefix", "state-restoration", - "completion-accounting" + "completion-accounting", + "gpu-timing", + "profiler-attribution", + "measurement-validity" ], "origin": [ "external" @@ -45,14 +48,15 @@ }, "checkout_ref": "checkout:inference-benchmark-design", "source_files": { - "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md": "fa7d91bf2b1c38299a7b3743cc362cb0399c4874d0cc42618161b77238e83a7a", - "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json": "061e07be83be10623b0500ef067709d05dd15fdd2c9d9aeb1e4404ef09fd6ca2", + "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md": "48e9f3deeca2176fa1b16095b9dd70e81b75733444eeb6a9bb24b1f9049b8529", + "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json": "0b4d160d0e24b3d86ba84402bef6d534bc71c179eae32458ccb436b12978a8fc", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/autotunebench-source.json": "bc343ece2bf40c4e8aa4bd8d84f32465d00633255557feda5fe7799ce24e8688", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/autotunebench-trial-validity.md": "cb3a7a5d94d056f868af07f92a16a6fef31eafd3199ed07ae79e93798e320b1d", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/benchmark-commands.md": "0898424e0bff3a6246e1eeeda6abd6c5bd974c047fe32f3bbd5ab9d31ce17cd3", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/cache-aware-asr-stream-boundaries.md": "4e0ea9c1bf6b096a49a91ad5266e90b8a456f4f5aca15eba8ac270a53c2309b5", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/dependency-aware-tts-playout.md": "e94866e718adc3bd7e272a4856e9a90b6f27cbc0ff3a44101e68e6995ed79e2c", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/faster-cosyvoice-source.json": "9efe8918208adddfb9a31d2abed080923a5ae0dfb6332438179af0de66f54e13", + "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md": "78d750321e65b69f1cc2006997d2e66494284e65cc7eb55d9f2c88e17aedb87c", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/justgoal-evaluation-contracts.md": "d25df795fd301e8a1bb436e5f8122d0bac2441c43104e59a3b310fd5a64f2f3e", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/justgoal-source.json": "60f2784384ea1a3b08a2ecafb60207d8e3707247dc7bc177b95b463d81f1cc16", "src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/ninfer-source.json": "ccda0fc6781e3c4dceb41521ab952445840b4afcc9f5a7d968857f76e8db6272", @@ -89,7 +93,10 @@ "candidate-selection", "accepted-prefix", "state-restoration", - "completion-accounting" + "completion-accounting", + "gpu-timing", + "profiler-attribution", + "measurement-validity" ], "status": "candidate", "profile": "local-skill-v1", @@ -436,14 +443,101 @@ "readset": "references/faster-cosyvoice-source.json", "license": "UNKNOWN overall; selected file-level Apache-2.0 headers and unresolved inherited attribution retained", "scope": "Selected PCM/playout, routed-credit, chunk-dependency and timing-denominator source audit; original analysis and immutable links only, no code/media copying or native/model/quality/service execution" + }, + { + "type": "document", + "url": "https://docs.nvidia.com/nsight-compute/2025.3/ProfilingGuide/index.html", + "version": "Nsight Compute 2025.3", + "sha256": "c9f3b812ce50a46a751f2c92eff628177554f4f6678bb3381468b1dac91b607e", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Replay, Overhead, Hardware Model, Metrics Structure, Range and Precision, Reproducibility, Clock Control and Cache Control; source contracts only, no profiler execution." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/nsight-compute/2025.3/NsightComputeCli/index.html", + "version": "Nsight Compute 2025.3", + "sha256": "c5129a8afe2439c04906f19312393972ae123062d2ee0894b3ba2b11af43471e", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Metric Collection and Profiling option defaults for clock/cache control and metric queries; actual target support must be checked separately." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/nsight-systems/2025.3/UserGuide/index.html", + "version": "Nsight Systems 2025.3", + "sha256": "1d42e35005fa9a4cb21b2c65d0018e4b065c3756dbba308dc0c41f79746e751f", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected CLI Profile options, CUDA graph/node tracing and CUDA API/kernel/launch/MemOps statistics reports; absence of NCU replay controls is an author inference, not a launcher guarantee." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-best-practices-guide/index.html", + "version": "CUDA Toolkit 12.9.1 archive", + "sha256": "0fcaf0c361d00de56a9870ba96db871a0b880904121e1dfb215ab9882470120b", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Assess/Optimize, Application Profiling, Amdahl's Law, Timing and Instruction Throughput; normalization/adoption procedure is an unverified author synthesis." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-programming-guide/index.html", + "version": "CUDA Toolkit 12.9.1 archive", + "sha256": "06499a0c6bfc6925cf54c76f6ba86a4986a081c44c602e10d4ef3b4f33a33865", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Concurrent Kernel Execution, CUDA Graphs (structure, definition/instantiation/execution) and Events; no constant launch/replay overhead or device test claimed." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/cuda/archive/10.2/nsight-compute/docs/NsightComputeCli/index.html", + "version": "CUDA Toolkit 10.2 archive, Nsight Compute CLI", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected nvprof Metric Comparison for dram__bytes_read.sum, dram__bytes_write.sum and smsp__inst_executed.sum. Legacy names do not guarantee current chip/tool availability; no raw-byte digest retained for this browser read." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/deploy/nvidia-smi/index.html", + "version": "Unversioned NVIDIA System Management Interface page, retrieved 2026-11-15; content pinned by SHA-256", + "sha256": "94d1737cdff11b5ef89b121e97365706feeb8ccc18978fecec571bdbbe6ec9ed", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Performance State, Clocks Event Reasons, clocks, temperature and power reporting. Public page may change; digest identifies inspected HTML, which is not redistributed." + }, + { + "type": "git", + "url": "https://github.com/triton-lang/triton", + "commit": "9641643da6c52000c807b5eeed05edaec4402a67", + "paths": [ + "python/triton/testing.py" + ], + "readset": "references/gpu-measurement-validity.md", + "license": "MIT", + "scope": "v3.2.0 do_bench: write-based cache.zero_ flush, preliminary estimate and per-sample event boundaries; tag resolved with git ls-remote, source inspected in public browser. No flush comparison run." + }, + { + "type": "document", + "url": "https://docs.pytorch.org/docs/2.7/cpp_extension.html", + "version": "PyTorch 2.7", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected load/load_inline and the extension name, build-directory and pybind11 module contract. Distinct names for separately loaded variants are author advice, not a documented universal duplicate-name failure; no raw-byte digest retained for this browser read." + }, + { + "type": "paper", + "url": "https://arxiv.org/html/2509.14279v1", + "id": "arxiv:2509.14279", + "version": "v1", + "sections": [ + "3", + "Appendix A", + "Appendix C.3" + ], + "readset": "references/gpu-measurement-validity.md", + "license": "CC-BY-4.0", + "scope": "Lange et al., Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization. Inspected benchmark loopholes, compromised-task filtering/examples and eager/compiled baseline comparisons; reports remain author-reported, not locally reproduced." } ], "empirical_validation": "not_run", "applicability": { - "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract", + "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; assessing GPU timing validity and profiler-based attribution of kernel/inference gains", "prerequisites": "The primary objective, timing boundary, quality gates, sample/failure accounting, required regimes, repetitions, seeds and decision rule are frozen before optimization; baseline and candidate share hardware, software stack, input distribution, benchmark configuration, warmup and measurement protocol, and on a shared machine also one time window and dedicated devices; cache-aware ASR also requires the selected model/config and first/steady geometry, decoded clock and reset/EOU/EOF boundaries to be established", "input_contract": "The task family and deployment use case; candidate quality evaluators and reference parity; the declared SLO and the caller's policy on which regimes must improve or must not regress; for an audit, a supplied fixed-workload report with request-to-trial grouping, exclusion and failure counts and the declared aggregation rule; for a treatment audit, the frozen delivery/arm-validity rules, run/candidate identities, feedback and consumer-input evidence, and per-arm raw records; for a kernel trial audit, complete saved candidate bytes and IDs, baseline/spec/reference/variant identities, result-origin fields, raw host outcome and return code, named gate and persisted ratio, actual oracle and the selecting/delivering caller; for the NInfer slice, a complete checkpoint/frontier identity, physical verify-block/state oracle, final adopted extents and correlated execution/termination records", - "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators", + "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for GPU attribution, a source-grounded measurement plan or interpretation binding environment, paired timing, work counts, predicted signature and E2E decomposition with explicit assumptions and unresolved evidence", "failure_modes": [ "Raw throughput is optimized in place of goodput, so requests served outside the SLO count toward a win", "One concurrency level or one undisclosed aggregate score stands in for all required regimes", @@ -460,7 +554,9 @@ "A displayed/manual best, baseline copy or edited candidate is reported as the eligible measured delivery", "Ratio omission alone is assumed to clear an earlier ratio, or baseline agreement is treated as independent semantic correctness", "Zero accepted drafts is treated as zero adopted input, or a matching token prefix is used to splice unrelated State and KV checkpoints", - "ResultReady or a commit receipt is treated as DeviceRetired, or selected full-batch intervals are compared with complete-cohort rates under one unlabeled denominator" + "ResultReady or a commit receipt is treated as DeviceRetired, or selected full-batch intervals are compared with complete-cohort rates under one unlabeled denominator", + "NCU replay durations are substituted for normal E2E evidence, or an untouched region is treated as a universal clock without checking common drift and critical-path assumptions", + "Unexplained gains, changed baseline modes or same-named JIT loads are adopted without binding actual work, module identity and predicted profiler signatures" ], "refusal_point": "Records missing coverage and high-variance results as INCONCLUSIVE rather than a win; a run without a frozen RunSpec, or whose validator changed after the plan was frozen, is INCONCLUSIVE regardless of metric values; sets no universal numeric gate or noise threshold; withholds causal treatment benefit/null conclusions when required delivery or arm eligibility is unverified, and preserves conflicts between stated protocol, exported code and source reports; withholds a kernel gain for unresolved performance or an unbound comparison and identifies missing independent semantic evidence when only baseline agreement is available; withholds native state-correctness and cancellation/reuse conclusions when the required same-block oracle, checkpoint identity or matching receipt is missing" }, @@ -471,7 +567,8 @@ "State the concurrency, trace, model, SKU and microbenchmark scope boundaries in any report that cites benchmark results.", "Trace treatment to its eligible consumer and apply the frozen arm-validity precedence before attributing benefit, saturation or a null.", "Bind the complete saved candidate and comparator cohort; keep result origin, actual oracle, named gate/ratio and delivered eligibility separate.", - "For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result." + "For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result.", + "For GPU comparisons, retain timestamped clocks/temperature/power, same-run region shares and work counts; test the predicted mechanism signature and paired E2E change before adopting a gain." ], "limitations": [ "Source-derived candidate; no local efficacy validation.", @@ -483,7 +580,8 @@ "The AutoTuneBench reference is source-only: original operational checkpoint/prompt receipts remain unavailable, source contradictions remain explicit, and no tuning, device/performance or agent-efficacy validation was run.", "The Xe-Forge reference is source-only and limited to inspected host/manual/Claude trial paths; named-verdict update predicates and caller defaults remain explicit. Native tests, backend/model/quality/performance/service behavior and agent efficacy were not evaluated, and source/package checks do not promote whole-model support.", "The NInfer reference inspects selected Qwen3.5 GDN/MTP source contracts and unexecuted test definitions; its historical author measurements use a different engine revision. Native state/model/device/quality/performance and efficacy remain unvalidated.", - "The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated." + "The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated.", + "The GPU measurement-validity reference is source-only. The within-run normalization and explanation-first adoption procedure is an unverified author proposal; four task-supplied numerical observations have no supplied raw evidence. GPU/profiler, model/quality, L1–L4 and efficacy validation were not run; legacy metric mappings require target-specific queries." ], "provenance_readsets": [ "references/task-benchmark-selection.md", @@ -498,7 +596,8 @@ "references/ninfer-state-and-completion-contracts.md", "references/ninfer-source.json", "references/dependency-aware-tts-playout.md", - "references/faster-cosyvoice-source.json" + "references/faster-cosyvoice-source.json", + "references/gpu-measurement-validity.md" ], "references": [ "../inference-optimization-casebook/cases/admission-drain-occupancy.md", @@ -510,10 +609,10 @@ "searchspace_not_applicable": "SKILL.md §§ Select the task and its evaluation contract, Statistical requirements, Shared-machine measurement discipline, and Search integrity and final-state verification design comparable evaluation; hardware identity and allocation isolation are comparison controls, not a hardware-resource mechanism." }, "applicability": { - "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract", + "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; assessing GPU timing validity and profiler-based attribution of kernel/inference gains", "prerequisites": "The primary objective, timing boundary, quality gates, sample/failure accounting, required regimes, repetitions, seeds and decision rule are frozen before optimization; baseline and candidate share hardware, software stack, input distribution, benchmark configuration, warmup and measurement protocol, and on a shared machine also one time window and dedicated devices; cache-aware ASR also requires the selected model/config and first/steady geometry, decoded clock and reset/EOU/EOF boundaries to be established", "input_contract": "The task family and deployment use case; candidate quality evaluators and reference parity; the declared SLO and the caller's policy on which regimes must improve or must not regress; for an audit, a supplied fixed-workload report with request-to-trial grouping, exclusion and failure counts and the declared aggregation rule; for a treatment audit, the frozen delivery/arm-validity rules, run/candidate identities, feedback and consumer-input evidence, and per-arm raw records; for a kernel trial audit, complete saved candidate bytes and IDs, baseline/spec/reference/variant identities, result-origin fields, raw host outcome and return code, named gate and persisted ratio, actual oracle and the selecting/delivering caller; for the NInfer slice, a complete checkpoint/frontier identity, physical verify-block/state oracle, final adopted extents and correlated execution/termination records", - "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators", + "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for GPU attribution, a source-grounded measurement plan or interpretation binding environment, paired timing, work counts, predicted signature and E2E decomposition with explicit assumptions and unresolved evidence", "failure_modes": [ "Raw throughput is optimized in place of goodput, so requests served outside the SLO count toward a win", "One concurrency level or one undisclosed aggregate score stands in for all required regimes", @@ -530,7 +629,9 @@ "A displayed/manual best, baseline copy or edited candidate is reported as the eligible measured delivery", "Ratio omission alone is assumed to clear an earlier ratio, or baseline agreement is treated as independent semantic correctness", "Zero accepted drafts is treated as zero adopted input, or a matching token prefix is used to splice unrelated State and KV checkpoints", - "ResultReady or a commit receipt is treated as DeviceRetired, or selected full-batch intervals are compared with complete-cohort rates under one unlabeled denominator" + "ResultReady or a commit receipt is treated as DeviceRetired, or selected full-batch intervals are compared with complete-cohort rates under one unlabeled denominator", + "NCU replay durations are substituted for normal E2E evidence, or an untouched region is treated as a universal clock without checking common drift and critical-path assumptions", + "Unexplained gains, changed baseline modes or same-named JIT loads are adopted without binding actual work, module identity and predicted profiler signatures" ], "refusal_point": "Records missing coverage and high-variance results as INCONCLUSIVE rather than a win; a run without a frozen RunSpec, or whose validator changed after the plan was frozen, is INCONCLUSIVE regardless of metric values; sets no universal numeric gate or noise threshold; withholds causal treatment benefit/null conclusions when required delivery or arm eligibility is unverified, and preserves conflicts between stated protocol, exported code and source reports; withholds a kernel gain for unresolved performance or an unbound comparison and identifies missing independent semantic evidence when only baseline agreement is available; withholds native state-correctness and cancellation/reuse conclusions when the required same-block oracle, checkpoint identity or matching receipt is missing" }, @@ -541,7 +642,8 @@ "State the concurrency, trace, model, SKU and microbenchmark scope boundaries in any report that cites benchmark results.", "Trace treatment to its eligible consumer and apply the frozen arm-validity precedence before attributing benefit, saturation or a null.", "Bind the complete saved candidate and comparator cohort; keep result origin, actual oracle, named gate/ratio and delivered eligibility separate.", - "For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result." + "For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result.", + "For GPU comparisons, retain timestamped clocks/temperature/power, same-run region shares and work counts; test the predicted mechanism signature and paired E2E change before adopting a gain." ], "limitations": [ "This generated abstract points to the complete packaged skill; external evidence and upstream repositories remain source links.", @@ -557,7 +659,8 @@ "The AutoTuneBench reference is source-only: original operational checkpoint/prompt receipts remain unavailable, source contradictions remain explicit, and no tuning, device/performance or agent-efficacy validation was run.", "The Xe-Forge reference is source-only and limited to inspected host/manual/Claude trial paths; named-verdict update predicates and caller defaults remain explicit. Native tests, backend/model/quality/performance/service behavior and agent efficacy were not evaluated, and source/package checks do not promote whole-model support.", "The NInfer reference inspects selected Qwen3.5 GDN/MTP source contracts and unexecuted test definitions; its historical author measurements use a different engine revision. Native state/model/device/quality/performance and efficacy remain unvalidated.", - "The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated." + "The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated.", + "The GPU measurement-validity reference is source-only. The within-run normalization and explanation-first adoption procedure is an unverified author proposal; four task-supplied numerical observations have no supplied raw evidence. GPU/profiler, model/quality, L1–L4 and efficacy validation were not run; legacy metric mappings require target-specific queries." ], "verification": { "locally_verified": false, diff --git a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md index 03369757b..65920e4f6 100644 --- a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md +++ b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/SKILL.md @@ -35,6 +35,12 @@ For treatment-enabled tuning comparisons or claimed protocol integrity, read Trace feedback to the eligible consumer input and apply the frozen arm-validity precedence before interpreting benefit, saturation or a null. +For GPU timing comparisons, read +[measurement validity and profiler attribution](references/gpu-measurement-validity.md). +Separate normal E2E timing from profiler replay, test within-run attribution +assumptions and retain work counts plus the predicted mechanism signature. +This is candidate guidance; its task-supplied observations remain unverified. + For hybrid speculative state-restoration or completion evidence, read the [NInfer validation contract](references/ninfer-state-and-completion-contracts.md). Freeze a coupled checkpoint and same-physical-block state oracle; distinguish diff --git a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json index 1060dde98..cd8f0d8fc 100644 --- a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json +++ b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/metadata.json @@ -24,7 +24,10 @@ "candidate-selection", "accepted-prefix", "state-restoration", - "completion-accounting" + "completion-accounting", + "gpu-timing", + "profiler-attribution", + "measurement-validity" ], "status": "candidate", "profile": "local-skill-v1", @@ -371,14 +374,101 @@ "readset": "references/faster-cosyvoice-source.json", "license": "UNKNOWN overall; selected file-level Apache-2.0 headers and unresolved inherited attribution retained", "scope": "Selected PCM/playout, routed-credit, chunk-dependency and timing-denominator source audit; original analysis and immutable links only, no code/media copying or native/model/quality/service execution" + }, + { + "type": "document", + "url": "https://docs.nvidia.com/nsight-compute/2025.3/ProfilingGuide/index.html", + "version": "Nsight Compute 2025.3", + "sha256": "c9f3b812ce50a46a751f2c92eff628177554f4f6678bb3381468b1dac91b607e", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Replay, Overhead, Hardware Model, Metrics Structure, Range and Precision, Reproducibility, Clock Control and Cache Control; source contracts only, no profiler execution." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/nsight-compute/2025.3/NsightComputeCli/index.html", + "version": "Nsight Compute 2025.3", + "sha256": "c5129a8afe2439c04906f19312393972ae123062d2ee0894b3ba2b11af43471e", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Metric Collection and Profiling option defaults for clock/cache control and metric queries; actual target support must be checked separately." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/nsight-systems/2025.3/UserGuide/index.html", + "version": "Nsight Systems 2025.3", + "sha256": "1d42e35005fa9a4cb21b2c65d0018e4b065c3756dbba308dc0c41f79746e751f", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected CLI Profile options, CUDA graph/node tracing and CUDA API/kernel/launch/MemOps statistics reports; absence of NCU replay controls is an author inference, not a launcher guarantee." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-best-practices-guide/index.html", + "version": "CUDA Toolkit 12.9.1 archive", + "sha256": "0fcaf0c361d00de56a9870ba96db871a0b880904121e1dfb215ab9882470120b", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Assess/Optimize, Application Profiling, Amdahl's Law, Timing and Instruction Throughput; normalization/adoption procedure is an unverified author synthesis." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-programming-guide/index.html", + "version": "CUDA Toolkit 12.9.1 archive", + "sha256": "06499a0c6bfc6925cf54c76f6ba86a4986a081c44c602e10d4ef3b4f33a33865", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Concurrent Kernel Execution, CUDA Graphs (structure, definition/instantiation/execution) and Events; no constant launch/replay overhead or device test claimed." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/cuda/archive/10.2/nsight-compute/docs/NsightComputeCli/index.html", + "version": "CUDA Toolkit 10.2 archive, Nsight Compute CLI", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected nvprof Metric Comparison for dram__bytes_read.sum, dram__bytes_write.sum and smsp__inst_executed.sum. Legacy names do not guarantee current chip/tool availability; no raw-byte digest retained for this browser read." + }, + { + "type": "document", + "url": "https://docs.nvidia.com/deploy/nvidia-smi/index.html", + "version": "Unversioned NVIDIA System Management Interface page, retrieved 2026-11-15; content pinned by SHA-256", + "sha256": "94d1737cdff11b5ef89b121e97365706feeb8ccc18978fecec571bdbbe6ec9ed", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected Performance State, Clocks Event Reasons, clocks, temperature and power reporting. Public page may change; digest identifies inspected HTML, which is not redistributed." + }, + { + "type": "git", + "url": "https://github.com/triton-lang/triton", + "commit": "9641643da6c52000c807b5eeed05edaec4402a67", + "paths": [ + "python/triton/testing.py" + ], + "readset": "references/gpu-measurement-validity.md", + "license": "MIT", + "scope": "v3.2.0 do_bench: write-based cache.zero_ flush, preliminary estimate and per-sample event boundaries; tag resolved with git ls-remote, source inspected in public browser. No flush comparison run." + }, + { + "type": "document", + "url": "https://docs.pytorch.org/docs/2.7/cpp_extension.html", + "version": "PyTorch 2.7", + "readset": "references/gpu-measurement-validity.md", + "scope": "Inspected load/load_inline and the extension name, build-directory and pybind11 module contract. Distinct names for separately loaded variants are author advice, not a documented universal duplicate-name failure; no raw-byte digest retained for this browser read." + }, + { + "type": "paper", + "url": "https://arxiv.org/html/2509.14279v1", + "id": "arxiv:2509.14279", + "version": "v1", + "sections": [ + "3", + "Appendix A", + "Appendix C.3" + ], + "readset": "references/gpu-measurement-validity.md", + "license": "CC-BY-4.0", + "scope": "Lange et al., Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimization. Inspected benchmark loopholes, compromised-task filtering/examples and eager/compiled baseline comparisons; reports remain author-reported, not locally reproduced." } ], "empirical_validation": "not_run", "applicability": { - "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract", + "scope": "Selecting task quality gates, reference parity checks and performance workloads for text, agentic coding, work agents, cache-aware ASR, TTS, full-duplex audio/video, image and video inference, and designing goodput-based multi-regime comparisons that fit InferRSI's evidence trail; interpreting treatment delivery, arm validity and stated-versus-observed protocol enforcement in supplied tuning reports; interpreting source-bound kernel trial origin, host evaluator outcomes, unresolved performance and measured-only versus manual candidate selection; designing and interpreting the selected NInfer hybrid speculative state-restoration and completion validation contract; assessing GPU timing validity and profiler-based attribution of kernel/inference gains", "prerequisites": "The primary objective, timing boundary, quality gates, sample/failure accounting, required regimes, repetitions, seeds and decision rule are frozen before optimization; baseline and candidate share hardware, software stack, input distribution, benchmark configuration, warmup and measurement protocol, and on a shared machine also one time window and dedicated devices; cache-aware ASR also requires the selected model/config and first/steady geometry, decoded clock and reset/EOU/EOF boundaries to be established", "input_contract": "The task family and deployment use case; candidate quality evaluators and reference parity; the declared SLO and the caller's policy on which regimes must improve or must not regress; for an audit, a supplied fixed-workload report with request-to-trial grouping, exclusion and failure counts and the declared aggregation rule; for a treatment audit, the frozen delivery/arm-validity rules, run/candidate identities, feedback and consumer-input evidence, and per-arm raw records; for a kernel trial audit, complete saved candidate bytes and IDs, baseline/spec/reference/variant identities, result-origin fields, raw host outcome and return code, named gate and persisted ratio, actual oracle and the selecting/delivering caller; for the NInfer slice, a complete checkpoint/frontier identity, physical verify-block/state oracle, final adopted extents and correlated execution/termination records", - "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators", + "produced_artifact": "A frozen benchmark design with evaluators, reference parity, primary objective, timing boundary, quality gates, required regimes, repetition and seed rules and an explicit decision config, plus the evidence-tree layout a PASS claim must produce; for a supplied report, the computed threshold result and missing evidence. Candidate knowledge, not qualification of any model or backend; for a treatment audit, a contract-bound eligibility and interpretation report separating treatment exposure, observed checks, comparator identity, anomaly disposition and source conflicts; for a kernel trial audit, a source-bound interpretation separating artifact/cohort identity, origin, oracle, performance resolution and selected-versus-delivered eligibility, without promoting kernel parity into a gain or whole-model acceptance; for the NInfer slice, a source-bound validation plan or report interpretation that separates state-oracle evidence, final adoption, retirement and workload denominators; for GPU attribution, a source-grounded measurement plan or interpretation binding environment, paired timing, work counts, predicted signature and E2E decomposition with explicit assumptions and unresolved evidence", "failure_modes": [ "Raw throughput is optimized in place of goodput, so requests served outside the SLO count toward a win", "One concurrency level or one undisclosed aggregate score stands in for all required regimes", @@ -395,7 +485,9 @@ "A displayed/manual best, baseline copy or edited candidate is reported as the eligible measured delivery", "Ratio omission alone is assumed to clear an earlier ratio, or baseline agreement is treated as independent semantic correctness", "Zero accepted drafts is treated as zero adopted input, or a matching token prefix is used to splice unrelated State and KV checkpoints", - "ResultReady or a commit receipt is treated as DeviceRetired, or selected full-batch intervals are compared with complete-cohort rates under one unlabeled denominator" + "ResultReady or a commit receipt is treated as DeviceRetired, or selected full-batch intervals are compared with complete-cohort rates under one unlabeled denominator", + "NCU replay durations are substituted for normal E2E evidence, or an untouched region is treated as a universal clock without checking common drift and critical-path assumptions", + "Unexplained gains, changed baseline modes or same-named JIT loads are adopted without binding actual work, module identity and predicted profiler signatures" ], "refusal_point": "Records missing coverage and high-variance results as INCONCLUSIVE rather than a win; a run without a frozen RunSpec, or whose validator changed after the plan was frozen, is INCONCLUSIVE regardless of metric values; sets no universal numeric gate or noise threshold; withholds causal treatment benefit/null conclusions when required delivery or arm eligibility is unverified, and preserves conflicts between stated protocol, exported code and source reports; withholds a kernel gain for unresolved performance or an unbound comparison and identifies missing independent semantic evidence when only baseline agreement is available; withholds native state-correctness and cancellation/reuse conclusions when the required same-block oracle, checkpoint identity or matching receipt is missing" }, @@ -406,7 +498,8 @@ "State the concurrency, trace, model, SKU and microbenchmark scope boundaries in any report that cites benchmark results.", "Trace treatment to its eligible consumer and apply the frozen arm-validity precedence before attributing benefit, saturation or a null.", "Bind the complete saved candidate and comparator cohort; keep result origin, actual oracle, named gate/ratio and delivered eligibility separate.", - "For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result." + "For hybrid speculative comparisons, freeze the coupled restore point, same-physical-block oracle, zero-draft/zero-adoption controls and complete-cohort versus steady-phase denominators before interpreting a result.", + "For GPU comparisons, retain timestamped clocks/temperature/power, same-run region shares and work counts; test the predicted mechanism signature and paired E2E change before adopting a gain." ], "limitations": [ "Source-derived candidate; no local efficacy validation.", @@ -418,7 +511,8 @@ "The AutoTuneBench reference is source-only: original operational checkpoint/prompt receipts remain unavailable, source contradictions remain explicit, and no tuning, device/performance or agent-efficacy validation was run.", "The Xe-Forge reference is source-only and limited to inspected host/manual/Claude trial paths; named-verdict update predicates and caller defaults remain explicit. Native tests, backend/model/quality/performance/service behavior and agent efficacy were not evaluated, and source/package checks do not promote whole-model support.", "The NInfer reference inspects selected Qwen3.5 GDN/MTP source contracts and unexecuted test definitions; its historical author measurements use a different engine revision. Native state/model/device/quality/performance and efficacy remain unvalidated.", - "The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated." + "The Faster CosyVoice reference is source-only: server route credit and client simulated playback remain distinct, reported costs/quality retain their boundaries, overall source rights are UNKNOWN, and native/model/device/audio quality/service/performance are not_run with efficacy not_evaluated.", + "The GPU measurement-validity reference is source-only. The within-run normalization and explanation-first adoption procedure is an unverified author proposal; four task-supplied numerical observations have no supplied raw evidence. GPU/profiler, model/quality, L1–L4 and efficacy validation were not run; legacy metric mappings require target-specific queries." ], "provenance_readsets": [ "references/task-benchmark-selection.md", @@ -433,7 +527,8 @@ "references/ninfer-state-and-completion-contracts.md", "references/ninfer-source.json", "references/dependency-aware-tts-playout.md", - "references/faster-cosyvoice-source.json" + "references/faster-cosyvoice-source.json", + "references/gpu-measurement-validity.md" ], "references": [ "../inference-optimization-casebook/cases/admission-drain-occupancy.md", diff --git a/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md new file mode 100644 index 000000000..018c2e49b --- /dev/null +++ b/src/inferrsi/knowledge/data/skill_library/methods/inference-benchmark-design/references/gpu-measurement-validity.md @@ -0,0 +1,175 @@ +# GPU measurement validity and attribution of gains + +Status: source-derived candidate; empirical validation not_run; efficacy +not_evaluated. Public source observations are cited below. The attribution and +adoption procedure is an author proposal derived from those sources, unverified +as a complete method. The final section contains task-supplied author reports +with unavailable evidence, not locally reproduced cases or efficacy evidence. +Source versions, inspected sections and available document hashes are retained +in the owning `metadata.json`; the unversioned SMI page is hash-pinned there. + +## Freeze the measurement and environment + +1. Bind workload, quality oracle, baseline execution mode, warmup, timed boundary, + repetition/aggregation rule, cache policy, streams and graph policy before + comparison. Retain paired baseline/candidate samples, including failures; + service claims still need the campaign's E2E evidence. This is proposed + measurement guidance, not a new numeric gate; see the owning skill's contract. +2. Record SM/memory clocks, temperature, power draw/limit and throttle reasons + alongside timestamped samples; record co-tenancy and allocation changes. + Boost state and thermal/power limits can move GPU region times without a + code change; competing work can alter scheduling and memory traffic. + They need not affect every region equally. E2E time alone cannot attribute + a gain to the patch. Sources: [NCG] §§ Clock Control, Reproducibility; + [SMI] §§ Performance State, Clocks Event Reasons. Recording is author advice. +3. Record profiler version, replay mode and actual control settings. Nsight + Compute 2025.3 defaults to `--clock-control base` and `--cache-control all`: + it attempts base clocks and flushes GPU caches before kernel replay passes. + Thermal throttling can still override clock control; platform support varies. + Sources: [NCC] § Profiling; [NCG] §§ Clock Control, Cache Control. +4. Ordinary execution and Nsight Systems tracing do not automatically apply + those NCU replay controls. This is an inference from [NCS] § CLI Profile + options versus [NCC], not a guarantee about an external launcher. Record + external clock locks and explicit cache flushes. Use NCU for isolated kernel + diagnosis; its kernel durations are not E2E evidence. Retain separate normal + paired E2E measurements and assess profiler overhead. [NCG] §§ Replay, + Overhead; [NCS] § CUDA Graph Trace warns of node-tracing overhead. + +## Attribute changes within each paired run + +Author-proposed diagnostic, derived from [CBP] §§ Application Profiling, +Amdahl's Law and [NCG] § Clock Control; its reliability is unverified here. + +1. Instrument stable regions and retain their times and E2E time from the same + run. Use region shares, not a kernel time divided by a different run's total. + Keep baseline/candidate interleaved in one allocation. Select an untouched + reference region `U` with identical work, placement and dependency behavior. +2. Compare `region_time / U_time` within each run. `U` acts as an internal clock + only if it shares the relevant clock sensitivity and is unaffected by the + patch. Verify its counts and environment; use multiple controls if possible. + A CPU-bound or differently memory-bound region is not a universal GPU clock. + Common movement is a confounding signal, not proof that normalization works. +3. Predict the E2E change from measured changed-region speedups and baseline + shares using an Amdahl decomposition. For disjoint serial regions, with no + drift and an unchanged residual, define `f_i = B_i / T_B` and `s_i = B_i/C_i`: + + ```text + predicted candidate/baseline time = 1 - sum(f_i) + sum(f_i / s_i) + observed candidate/baseline time = T_C / T_B + ``` + + If explicitly assuming common multiplicative drift `d = U_C/U_B`, use + `s_i = (B_i/U_B)/(C_i/U_C)` and compare the prediction to `(T_C/T_B)/d`. + That correction assumes the residual shares the drift too; model unaffected + host time separately. Retain raw and normalized ratios and their uncertainty. +4. Check prediction against the observed paired E2E ratio under the frozen + campaign uncertainty rule. A mismatch leaves attribution unresolved: inspect + noise, overlap/critical-path effects and side effects in other regions. + Do not sum overlapping kernel durations as wall time. Use the dependency + timeline for concurrent work; a faster off-path kernel need not shorten E2E. + Sources for these limits: [CPG] §§ Concurrent Kernel Execution, CUDA Graphs; + the mismatch interpretation is an author inference, not a causal proof. + +## Retain work counts as well as time + +For identical work and comparable collection conditions, these author-selected +counts are less directly sensitive to clocks/temperature than durations or rates. +They explain small changes; they do not independently establish a speedup. + +| Work signature | Collection and interpretation | +| --- | --- | +| Kernel set and launch count | [NCS] CUDA GPU Kernel Summary / launch trace: retain kernel identity and instances per region, not just aggregate time. | +| DRAM read/write bytes | `dram__bytes_read.sum`, `dram__bytes_write.sum`; [NCM] § nvprof Metric Comparison names these counters. Retain both directions under the same cache/replay policy. | +| Executed instructions / FLOPs | `smsp__inst_executed.sum` counts warp instructions, not FLOPs ([NCM]). Derive FLOPs only with documented operation, datatype, active-thread and tensor-instruction semantics; count FMA as two operations ([CBP] § Instruction Throughput). | +| Host↔device copy bytes | [NCS] CUDA GPU MemOps Summary (by Size): separate HtoD/DtoH bytes, instances and transfer types. | +| Synchronization count | [NCS] CUDA API trace: count named host synchronization calls per region. API calls do not enumerate every device barrier or dependency. | +| CUDA graph node count | [CPG] § CUDA Graphs: retain graph topology/node types and replay count; [NCS] node tracing exposes activities. A graph launch count is not its node count. | + +The metric names above have an official legacy mapping in the CUDA 10.2 NCU +archive [NCM], not a promise of availability on every current GPU. Query the +installed NCU/device with `--query-metrics`; resolve suffixes with +`--query-metrics-mode suffix --metrics ` ([NCC] § Metric Collection). +Retain supported names, units, collection passes and per-region sums; mark absent +counters unavailable. Do not replace a count with percent-of-peak or per-second. + +Author caveats: these are conditional invariants, not constants. Cache residency, +co-tenancy and replay can change DRAM traffic; data-dependent branches, spin +loops and algorithms can change instructions. [NCG] §§ Range and Precision, +Cache Control describes asynchronous traffic and multi-pass variability. +For FLOPs, keep the derivation with the raw counters; instruction count alone +does not distinguish useful arithmetic, address work or redundant computation. + +## Require an explanation before adoption + +Author-proposed acceptance guidance, motivated by [CBP]'s Assess / Optimize +cycle and [RB] § 3 and Appendix A; no efficacy claim or repository-wide gate. + +Before measurement, state a mechanism and predicted signature: region, metric, +direction and rough size. For example, removing a materialized intermediate +predicts fewer DRAM bytes in the affected region; fusion predicts fewer launches +and perhaps less intermediate traffic. These are hypotheses to verify, not +automatic consequences or numeric thresholds. Retain counters, timelines, +paired E2E samples, quality results and the Amdahl check together. Adopt only +when retained data supports the explanation and frozen campaign gates hold. +Leave unexplained numeric gains unadopted / INCONCLUSIVE for attribution; +reject a contradicted mechanism/candidate rather than narrating it as a win. +Distinguish this adoption decision from the campaign's PASS/FAIL/INCONCLUSIVE. + +## Audit common timing traps + +- **L2 flushing:** Triton v3.2.0 `do_bench` writes a cache buffer with + `cache.zero_()` before each measured start event ([TR]). Its preliminary + runtime estimate includes the flush, whereas per-sample events surround `fn`. + Record the method, buffer size, stream/order and cold-versus-warm intent. + Author advice: compare flush methods as a separate control; an excluded flush + may still change the next kernel's cache/memory state. Do not infer that a + read-based flush is equivalent or superior without device-specific evidence. +- **Events versus host timers:** CUDA launches are asynchronous. Use synchronized + host boundaries for completed wall time; CUDA events timestamp stream progress + and require completion before reading elapsed time ([CBP] § Timing). + Capture/instantiation, host submission, graph replay and kernel execution have + different costs ([CPG] § CUDA Graphs). Author advice: expose fixed launch/replay + costs with matched repetition counts; amortize only what deployment amortizes. + Events are not pure instruction time when their interval contains GPU idle gaps. +- **Baseline identity:** state eager versus `torch.compile`, compiler settings, + warmup and graph mode. Speedup is `baseline_time/candidate_time`; changing the + denominator changes the claim (arithmetic / author advice). [RB] Appendix C.3 + explicitly distinguishes eager and compiled comparisons; neither is universal. +- **JIT extension identity:** give separately loaded variants distinct extension + names and retain the actual module/function and built-library identity. This + is author advice inferred from [PT] `load`: the build directory uses `name`, + which must match the pybind11 module. The docs do not establish that every + repeated name fails; do not assume two same-named loads are independent. +- **Benchmark exploits:** [RB] § 3 and Appendix A report inefficient baselines, + low-variation outputs and input-specific shortcuts passing exposed checks. + Author advice: inspect actual timed work and use independent varied-input + quality tests. A correctness flag or extreme ratio alone cannot clear a gain. + +## Author-reported observations: unverified + +Task-supplied reports only; raw samples, configurations and profiler artifacts +were not supplied. The numbers below are not public-source findings, reproduced +results, reusable thresholds or evidence that this procedure is effective. + +| Reported scope | Reported observation | Limit | +| --- | --- | --- | +| One H200 campaign | Write-based L2 flush inflated timings about 9% versus read-based flushing. | Cause and flush equivalence unverified. | +| Shared nodes | Same-run interleaved ratios were reported trustworthy while cross-run baselines drifted ±7–15%. | Reliability, uncertainty and common-drift assumptions unverified. | +| Same code, different weight placement | Timing differed about 4%. | Placement and causal mechanism unverified. | +| Same candidate, different baseline | 7.35x versus eager, 1.43x versus `torch.compile`. | Baseline configurations, quality and timings unverified. | + +Original synthesis; no upstream code, figures or raw reports redistributed. +NVIDIA/PyTorch document rights remain with their owners; Triton is MIT-licensed; +Lange et al.'s [RB] is CC BY 4.0. Source review is not GPU execution, model quality, +L1–L4 qualification, independent review or proof of agent efficacy. + +[NCG]: https://docs.nvidia.com/nsight-compute/2025.3/ProfilingGuide/index.html +[NCC]: https://docs.nvidia.com/nsight-compute/2025.3/NsightComputeCli/index.html +[NCS]: https://docs.nvidia.com/nsight-systems/2025.3/UserGuide/index.html +[SMI]: https://docs.nvidia.com/deploy/nvidia-smi/index.html +[CBP]: https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-best-practices-guide/index.html +[CPG]: https://docs.nvidia.com/cuda/archive/12.9.1/cuda-c-programming-guide/index.html +[NCM]: https://docs.nvidia.com/cuda/archive/10.2/nsight-compute/docs/NsightComputeCli/index.html +[TR]: https://github.com/triton-lang/triton/blob/9641643da6c52000c807b5eeed05edaec4402a67/python/triton/testing.py +[PT]: https://docs.pytorch.org/docs/2.7/cpp_extension.html +[RB]: https://arxiv.org/html/2509.14279v1 diff --git a/tests/check_installed.py b/tests/check_installed.py index 3a41b4d40..c11bf433e 100644 --- a/tests/check_installed.py +++ b/tests/check_installed.py @@ -533,7 +533,7 @@ def _inventory(root, prefix): **_inventory(files("inferrsi").joinpath("knowledge", "data"), "knowledge/data"), **_inventory(files("inferrsi").joinpath("harness", "data"), "harness/data"), } -assert len(inventory) == 923, f"installed resource count is {len(inventory)}, expected 923" +assert len(inventory) == 924, f"installed resource count is {len(inventory)}, expected 924" assert sum(name.startswith("harness/data/") for name in inventory) == len(REPO_TYPES) # Every pinned record path and every registered skill body resolves inside that # same enumeration, so the inventory and the records describe one resource set.