Repository navigation
Conversation
With num_postprocess_workers > 0, per-request metrics are built in the postprocess worker on a record that has no prompt_token_ids, so record_stats never set PROMPT_TOKENS and trtllm_prompt_tokens_total was never incremented. trtllm_prompt_cached_tokens_total was still counted, leaving the token-level prefix hit rate without a denominator. The main-process GenerationResult does hold the prompt, so it now fills PROMPT_TOKENS in when it merges a finished request's metrics from the worker, keyed on the cached-token count the worker reports once per finished first candidate. Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
With KVCacheManagerV2, a prefix served by a KV cache connector (e.g. Mooncake store) is allocated in the scheduling pass and booked as missed blocks, so the block-level hit rate in Prometheus, iteration stats and per-request kv_cache_metrics only reflected on-device radix-tree reuse, while cached_tokens (and aiperf) also included the connector prefix. When the connector prefix is delivered, rebook the pending allocations for the blocks it fills completely from missed to reused, in the global, per-request and per-iteration views. The connector's share is also tracked per life cycle, reported as iterConnectorReusedBlocks in kvCacheIterationStatsByLifecycle and exported as the kv_cache_iter_connector_reused_blocks_total Prometheus counter. Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
brb-nv
changed the title
Signed-off-by: Balaram Buddharaju <169953907+brb-nv@users.noreply.github.com>
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #77412 [ run ] triggered by Bot. Commit: |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
How it works
When the connector prefix is delivered (phase 3, in prepare_resources), the blocks phase 2's resize just booked as misses are still uncommitted.
_deliver_connector_prefixnow calls a new_KVCache.record_connector_reuse, which moves those blocks from missed to reused in the global, per-request and per-iteration views.Only blocks the connector fills completely are moved. A partly filled last block, for example the one cut off where the offer stops one token before the end of the prompt, stays a miss.
Everything runs in Python and only moves counts between existing fields, so the C++ stats structs don't change and no rebuild is needed.
One structural change: subtract_allocation_range used to stop at the first segment that ended before the range being removed. Splitting segments breaks the ordering that relied on across life cycles, so it now checks every segment. The lists are small because they're cleared at each commit.
The device vs. connector split is still visible
Each life cycle's entry in
kvCacheIterationStatsByLifecyclegets a newiterConnectorReusedBlocksfield.It's exported as a new Prometheus counter,
trtllm_kv_cache_iter_connector_reused_blocks_total. It's a subset ofkv_cache_iter_reused_blocks_total, so reused minus connector-reused is the on-device reuse.Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.