Add no_cache scenario and --history-len to FlexKV inference benchmark

[Clebrate]

Description

This PR extends the FlexKV HSTU inference benchmark with a no_cache path and a --history-len CLI flag, so no-cache, naive GPU-hit, and naive CPU-hit can be compared on the same request shape.

• --scenarios now accepts gpu_hit, cpu_hit, ssd_hit, and no_cache. no_cache must be run by itself; it uses forward_nokvcache and does not start FlexKV.
• --history-len controls the cached/recomputed prefix (default 1024). Timed length is still history_len + append_history_len unless --only-onboard.

Example (CPU-only FlexKV, no SSD config):

cd examples/hstu
export PYTHONPATH=${PYTHONPATH}:$(realpath ../)

#no_cache
python3 ./inference/benchmark/inference_benchmark_flexkv.py  --disable-cudagraph  --scenarios no_cache --batch-size 1 --history-len 1024 --append-history-len 0 --timed-iters 10 --only-onboard

#cpu_hit / gpu_hit
python3 ./inference/benchmark/inference_benchmark_flexkv.py  --disable-cudagraph  --scenarios gpu_hit,cpu_hit --batch-size 1 --history-len 1024 --append-history-len 0 --timed-iters 10 --only-onboard

Do not pass --flexkv-config-path unless SSD is intended (ssd_cache_gb: 0 and no ssd_cache_dir keeps SSD off).

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[greptile-apps]

Confidence Score: 5/5

The changes since the previous review appear safe to merge, although the two existing non-blocking CLI-validation findings remain outstanding.

Findings

1.  Empty latency samples crash ▶
2.  Negative history length crashes ▶

Summary

• Adds a standalone no_cache scenario using forward_nokvcache.
• Adds configurable history length and per-iteration latency reporting.
• Corrects request action-feature sizing to include candidate tokens.
• Makes offload queue access compatible with the backend-owned task collection.
• Prevents no_cache from being combined with cache-backed scenarios.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Parse benchmark arguments] --> B{Selected scenarios}
    B -->|no_cache only| C[Build model without KV cache]
    C --> D[Run forward_nokvcache]
    B -->|gpu_hit / cpu_hit / ssd_hit| E[Build model with FlexKV]
    E --> F{Scenario}
    F -->|gpu_hit| G[Reuse GPU-resident prefix]
    F -->|cpu_hit| H[Evict from GPU and onboard from CPU]
    F -->|ssd_hit| I[Apply pressure and onboard from SSD]
    D --> J[Report latency samples and summary]
    G --> J
    H --> J
    I --> J

Loading

_{Reviews (3) · Last reviewed commit: "Fix FlexKV benchmark action length and o..."}

[greptile-apps]

Empty latency samples crash

--timed-iters accepts zero or negative values, which produce no request batches. The new latency summary then divides by len(wall_ms) and calls min and max on the empty list, so any scenario run with --timed-iters 0 crashes instead of completing. Validate that the iteration count is positive before running a scenario. The same aggregation pattern appears in the GPU-hit, CPU-hit, and SSD-hit summaries.

[greptile-apps]

Negative history length crashes

The new --history-len option accepts negative integers, and the value is later used directly as a tensor dimension in build_request. For example, --history-len -1 aborts request construction before the benchmark can run. Reject negative history lengths during argument validation so users receive a clear error.