Repository navigation
fix(jq_link): drop vcon.uuid from metric attributes - #224
Merged
Merged
Conversation
The jq link tagged its counters (vcon_filtered_out, filter_errors, string_body_array_retries) with the vCon uuid. Every filtered vCon became its own time series, and with cumulative temporality each series is re-exported on every interval for the life of the worker. On a deployment that filters out most of its input, one counter grew to over 1M series and roughly 460M samples an hour before a restart reset it, filling the metrics store. Keep link.name only. The uuid is already in the log line beside each increment. Tests assert one series per link across many vCons; they fail on the old attributes (3 series for 3 vCons) and pass with this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
howethomas
added a commit
that referenced
this pull request
Nov 4, 2026
…vCon attributes (#226) * fix(metrics): drop vcon.uuid from link metrics and guard against per-vCon attributes #154 put vcon.uuid on the metrics of 17 links. Each distinct value is its own time series, and with cumulative temporality every series is re-exported on each interval for the life of the worker, so metric volume grows without bound until a restart. #224 fixed the jq link after it filled a metrics store; this removes the same label from the rest: analyze, analyze_and_label, analyze_vcon, check_and_tag, datatrails, deepgram_link, detect_engagement, diet, groq_whisper, hugging_face_whisper, hugging_llm_link, openai_transcribe, post_analysis_to_slack, sampler, scitt, tag_router and webhook. Span attributes keep the uuid; spans are per-event and do not have this problem. lib.metrics now drops per-vCon keys (vcon.uuid, vcon_uuid, vcon_id, vcon.id) from counter, up/down counter and histogram attributes and warns once per metric, so a new or out-of-tree link cannot reintroduce it. The CON-6 metric tests asserted vcon.uuid was present; they now assert no per-vCon key appears on any data point. With the guard disabled, 3 guard tests fail; with the guard disabled and the links reverted, 27 fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(links): stop expecting vcon.uuid on metric attributes The per-link test files still asserted vcon.uuid in the attributes passed to the metric helpers. Update them to match the links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The jq link labels its counters (
conserver.link.jq.vcon_filtered_out,filter_errors,string_body_array_retries) withvcon.uuid. Every vCon the filter rejects becomes its own time series. The counters are cumulative, so each worker re-exports every series it has created on every export interval until the process restarts, and sample volume grows for as long as the process runs.On a deployment where the filter rejects most of its input, this one counter reached over 1M series and about 460M samples an hour before a restart reset it. That was more than 95% of all metric samples, and it filled the metrics store.
Change
link.nameonly on the jq link's metric attributes. The uuid is already in the log line written next to each increment.TestJqLinkMetricstoconserver/tests/test_link_metrics.py. It runs three vCons through the link and asserts one series per link forvcon_filtered_outandfilter_errors.Testing
assert 3 == 1: three series for three vCons) and pass with this change.conserver/tests/test_link_metrics.pyandconserver/links/jq_link/test_jq_link.pypass: 40 tests.Not in this PR
#154 added
vcon.uuidto metrics in about 15 other links (webhook, diet, deepgram_link, groq_whisper, analyze*, check_and_tag, scitt and others). Their volume is lower, but they have the same unbounded cardinality. That needs its own change, and the CON-6 metric tests that assertvcon.uuidwould need updating with it.🤖 Generated with Claude Code