prometheus: purge deleted stream metric series in streams mode (CON-555)

[squiidz]

Ref: CON-555

Problem

In streams mode, deleting a stream via DELETE /streams/{id} stops the stream and removes it from the API, but its metric series (labeled stream="<id>") remain registered in the prometheus exporter forever. /metrics keeps exposing every deleted stream's counters/gauges/timers with frozen values until the process restarts, growing cardinality without bound for workloads that churn streams.

Fix

Implements the optional service.MetricsExporterSeriesDeleter interface (added in redpanda-data/benthos#487) on the prometheus exporter by calling DeletePartialMatch on every registered counter, gauge, summary and histogram vec. The benthos stream manager invokes it with {stream: <id>} when a stream is deleted (or replaced by an update — for prometheus that shows up as an ordinary counter reset, which scrapers handle).

Depends on redpanda-data/benthos#487: this implementation is inert until the benthos dependency is bumped to a release containing the stream manager hook. The unit tests exercise the exporter directly and pass against the currently pinned version, so this can merge ahead of the bump.

Testing

• Unit tests covering summary- and histogram-mode timers plus counters and gauges: series for the matched label value disappear from /metrics, series for other label values and unlabelled series are untouched.
• Verified end to end against a build with the benthos change (via a local go.mod replace, not part of this PR), using the reproduction from the ticket:

==> series with stream="foo" BEFORE delete: 21
==> deleted stream foo
==> series with stream="foo" AFTER delete: 0
PASS: all 21 series purged after DELETE

[claude]

DeletePartialMatch is applied unconditionally to every registered vec, including vecs that do not declare the label being matched on.

In prometheus/client_golang (pinned here at v1.23.2 per go.mod#L136), MetricVec.DeletePartialMatch → metricMap.deleteByLabels → matchPartialLabels, which looks each requested label key up in the vec's variable label names and skips the comparison entirely when the key is not one of them. If none of the requested keys are variable labels of that vec, the match is vacuously true and every series in the vec is deleted.

Failure scenario: DeleteSeriesPartialMatch({"stream": "foo"}) is called while a metric vec exists that has no stream label — e.g. the zero-label uptime counter created at metrics_prometheus_test.go#L189-L191, or any global/non-stream-scoped vec. All of its series are dropped from /metrics, so deleting one stream silently wipes unrelated counters and gauges. Note this also means the new test's own assert.Contains(t, body, "\nuptime 9") assertion at line 208 should be failing — worth confirming the test actually passes locally before merging.

Suggested fix: record the declared label names on promCounterVec/promGaugeVec/promTimingVec/promTimingHistVec (they already carry count) and skip any vec that does not declare every key present in labels before calling DeletePartialMatch.

[squiidz]

Checked against the pinned client_golang (v1.23.2): the premise is inverted. In matchPartialLabels (prometheus/vec.go:468-483), when a requested key is not one of the vec's variable labels, indexOf returns validLabel == false, the inner branch is skipped, and the function hits return false — an unknown label key is a non-match for that vec, not a wildcard. There is no vacuously-true path; the loop only continues on an actual value match.

Consistent with that, TestPrometheusDeleteSeriesPartialMatch passes as written, including the uptime 9 assertion this comment predicted would fail. Added a labeled vec without the stream label (batch_created{mechanism="count"}) to the test to pin the non-match behavior explicitly.

[claude]

The changelog entry ships a placeholder link: [#TBD](https://github.com/redpanda-data/connect/pull/TBD) resolves to a 404. Every other entry in this file uses the real PR number (see CHANGELOG.md#L16-L23). Replace TBD with this PR's number (4742) before merging.

[claude]

This entry announces user-visible behavior that will not actually happen with the benthos version this PR pins. The commit body itself states the exporter-side implementation "is inert until the benthos dependency is bumped to a release containing the stream manager hook", and this PR changes only CHANGELOG.md and the two prometheus files — there is no go.mod bump, and nothing in this repo calls DeleteSeriesPartialMatch outside the new tests (see

connect/internal/impl/prometheus/metrics_prometheus.go

Lines 526 to 545 in 99bca83

	// service.MetricsExporterSeriesDeleter interface.
	func (p *metrics) DeleteSeriesPartialMatch(labels map[string]string) {
	promLabels := prometheus.Labels(labels)
	p.mut.Lock()
	defer p.mut.Unlock()
	for _, pv := range p.counters {
	pv.ctr.DeletePartialMatch(promLabels)
	}
	for _, pv := range p.gauges {
	pv.ctr.DeletePartialMatch(promLabels)
	}
	for _, pv := range p.timers {
	pv.sum.DeletePartialMatch(promLabels)
	}
	for _, pv := range p.timersHist {
	pv.sum.DeletePartialMatch(promLabels)
	}
	}

).

As written, the next release would ship a "Fixed" note for a fix users cannot observe. Suggested fix: either include the benthos dependency bump in this PR so the stream manager actually invokes the deleter, or hold the changelog entry until that bump lands (documenting the current limitation instead, per CONTRIBUTING.md §1.2.3 "Known limitations and edge cases are documented").

[squiidz]

Valid — the exporter side is inert until the benthos dependency contains the stream-manager hook (redpanda-data/benthos#487), so a release cut before that bump would ship a "Fixed" note for unobservable behavior. Removed the entry from this PR; it moves to the bump-benthos PR that activates the fix end to end. (This also supersedes the #TBD link comment above.)