Skip to content

test: check that a non-leader unit sends trace data - #2922

Open
tonyandrewmeyer wants to merge 5 commits into
canonical:mainfrom
tonyandrewmeyer:test-tracing-follower-unit
Open

tonyandrewmeyer wants to merge 5 commits into
canonical:mainfrom
tonyandrewmeyer:test-tracing-follower-unit

Conversation

@tonyandrewmeyer

@tonyandrewmeyer tonyandrewmeyer commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Deploys the traced charm with two units in test_direct_connection and waits for spans from both, because every tracing integration test ran a single unit, so nothing exercised a follower against a real provider. Only the leader can read its own application databag, which is where #2918's bug came from.

This doesn't reproduce #2918 itself (Tempo always offers otlp_http once it sees our request), but it would catch a follower failing on the normal path. get_spans now keeps each span's resource so the test can tell which unit sent it.

I ran test_direct_connection in a VM with Juju 3.6.29 on K8s (set up with concierge prepare -p k8s) and it passed, with spans from both test-tracing/0 and test-tracing/1, so the resource attribute parsing works against a real Tempo.

tonyandrewmeyer and others added 5 commits November 19, 2026 08:49
Deploy the traced charm with two units in test_direct_connection and
wait for spans from both. Every tracing integration test previously ran
a single unit, so nothing exercised a follower against a real provider,
and only the leader can read its own application databag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tonyandrewmeyer
tonyandrewmeyer marked this pull request as ready for review November 20, 2026 23:28
@chapman-von-monty

Copy link
Copy Markdown

Code review: canonical/operator PR #2922

Reviewed revision: 733114ac14c3393cd388c2d928b649bbc866ce8a
Base: 9ddfe17174e7926b73458742ac6244114b433d8e
Scope: Immutable saved diff /research/PR.diff; it changes only test/integration/test_tracing.py and is test-only.

1. Verdict: APPROVE

No blocking or non-blocking code changes are requested. The new wait condition checks the unit identities carried by spans, rather than merely counting spans, so the test will continue polling until it sees both the leader and follower or the existing bounded wait expires. The PR notes that this exercises the follower's normal tracing path but does not directly reproduce #2918; that is an explicit coverage tradeoff, not a defect in the stated test purpose.

2. Blocking Issues (Must Fix)

None found.

3. Non-Blocking Suggestions & Nitpicks

No optional code changes suggested.

Coverage caveat: The existing integration workflow runs this test on its Kubernetes matrix, but .github/workflows/integration.yaml:3–6 triggers only on manual dispatch and a weekly schedule, not on pull requests. This is pre-existing workflow behavior and is not changed by this PR; it means the new assertion is not automatically PR-gating.

4. Downstream & Ecosystem Considerations

No production API, runtime behavior, packaging, or downstream charm contract changes. This adds integration coverage using Juju and Tempo. Its execution remains subject to the existing manual/scheduled integration-test coverage noted above.

Review evidence and caveats

  • test/integration/test_tracing.py:44–52 deploys two units and waits for both test-tracing/0 and test-tracing/1; the final assertion reports whichever unit identity is missing. This distinguishes the expected units and would fail if only the leader emitted spans.
  • test/integration/test_tracing.py:129–149 raises Tempo's search limit and retains each span's resource; :153–160 extracts the OTLP juju_unit resource attribute's stringValue. The backend emits this resource attribute as a string (tracing/ops_tracing/_backend.py:61–65), and the backend test checks it (tracing/test/test_backend.py:67–83). The PR description also reports a successful run against real Tempo with spans from both units; that is author-reported evidence, not independently reproduced here.
  • The polling remains bounded by the existing wait_spans timeout (60 seconds, one-second retries; test/integration/test_tracing.py:99–116 at base). The predicate itself requires both identities, so a partial result does not count as success. The existing temporary-model fixture teardown covers the extra unit, and the port-forward retains its context-managed cleanup.
  • No tests, builds, hooks, or dependency setup were run. Juju integration tests must run in a VM under project rules; this reviewer did not run them. Conclusions are based on static inspection of the saved diff and relevant base-revision helpers/workflow. The PR summary records the author's VM run with Juju 3.6.29 on Kubernetes; it was not independently verified.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants