Skip to content

ci(telemetry): run the real OTLP wire path in CI and pin it (#1083) - #1104

Merged
Birfy merged 3 commits into
mainfrom
ci/1083-wire-path-v2
Nov 16, 2026
Merged

Birfy merged 3 commits into
mainfrom
ci/1083-wire-path-v2

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Closes #1083

The exporter from #923/#1081 shipped with one untested seam: the offline suite stands a fake SDK in at otel_export._load_otel, so the SDK's own wire path — a real exporter, a real refused connection — ran nowhere. This adds the rows that cover it and the one CI leg where they run.

What changed

  • .github/workflows/tests.yml — a new otel-wire job: pip install -e ".[otel]" pytest, python scripts/otel_wire_smoke.py, then AGENTDESCENT_OTEL_WIRE=1 pytest -q tests/test_otel_wire.py tests/test_otel_export.py. Both files are in that step on purpose: the job installs the extra, so the fail-fast marker row and the endpoint rows must run here with the SDK present — that is the environment those rows exist for.
  • tests/test_otel_wire.py (new) — a real exporter pointed at a closed 127.0.0.1 port: the refusal must be logged and contained, never raised out of export_run, and export_run still returns its count. Plus a row that pins the CI leg itself, so deleting the only place the wire path runs fails a test instead of silently dropping the coverage. The socket row is gated behind AGENTDESCENT_OTEL_WIRE=1; the default suite stays offline.
  • scripts/otel_wire_smoke.py (new) — the measured smoke the job runs: one span and one metric at a closed port, asserting the degradation is logged and force_flush returns.
  • tests/test_otel_export.py — the fail-fast row stops probing-and-skipping: it clears every opentelemetry* module and puts None in sys.modules["opentelemetry"], so _load_otel raises its real ImportError; the loopback row builds a real exporter instead of skipping. These are the rows Security's 2026-11-14T09:32:26Z comment assigned to this slice.
  • docs/metrics.md — credentials go in OTEL_EXPORTER_OTLP_HEADERS, never in the URL (path or query); matching changelog.d/1083-otel-wire-path.md.

Size: 6 files, +338/−18 against f5cc52ef; T3 by path (.github/** is a T3 path; tests and docs alone would be T1).

Red at base f5cc52ef

Clean worktree at f5cc52efe41750633ac37876fc2d2894a8a6f9f3, only tests/test_otel_wire.py copied in (git status: one untracked file):

FAILED tests/test_otel_wire.py::test_the_ci_leg_installs_the_extra_and_runs_the_smoke_script
E   AssertionError: assert False
E    +  where False = is_file()
E    +    where is_file = PosixPath('/…/scripts/otel_wire_smoke.py').is_file

1 failed, 1 passed in 28.82s     # AGENTDESCENT_OTEL_WIRE=1, [otel] installed
1 failed, 1 skipped in 0.16s     # same tree, flag off

The socket row passes at base because otel_export already shipped in #1081 — it pins that behaviour. The base red is the missing smoke script and the missing otel-wire job: at f5cc52ef, grep -c otel-wire .github/workflows/tests.yml is 0 and scripts/otel_wire_smoke.py does not exist.

Verification at the tip

Tip SHA: b7774c9

Ran (python 3.10.12, fresh venv from the clone, pip install -e ".[otel]" pytest):

  • the job step verbatim, flag on: pytest -q tests/test_otel_wire.py tests/test_otel_export.py → 24 passed, 1 skipped (the skip is the real-import row, correct)
  • same pair, flag off (the default offline suite): 23 passed, 2 skipped (socket row self-skips, real-import row skips)
  • python scripts/otel_wire_smoke.py → exit 0, last line ok: the wire path degraded as documented
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 40 changelog fragment(s) OK
  • applies to live main: git merge --no-commit --no-ff of this branch into f5cc52ef → clean, zero conflicts

Not run:

  • mkdocs build --strict — mkdocs is not installed in this clone's venv; CI's build leg is the line
  • a real collector round trip — no collector on this host; the smoke and the socket row both target a closed port, which is the degradation case
  • tests/test_docs_links.py — markdown is not installed here (environment, not this diff)
  • the 3.12 matrix leg — this PR's own test (3.12) / otel-wire run is that confirmation

Same-repo head: maintainer_can_modify is requested; if GitHub reports it structurally false on a same-repo head, that is the platform (there is no fork to grant), recorded here rather than worked around.

…lled (#1083)

The otel-wire leg installs [otel], so the two rows that pin the
fail-fast contract and the loopback policy probed the SDK and skipped
themselves in exactly the environment that has it: 17 passed, 5 skipped
at 2ab6b39 with the extra present, and the loopback row built a live
provider pointed at 127.0.0.1:4318 before skipping.

- The construction row now stands the SDK import in with the
  interpreter's own marker (None in sys.modules), so _load_otel walks
  its real ImportError path and the install line is asserted for both
  entry points; one real-import row skips where the SDK is present.
- The loopback rows stub the SDK seam and assert the accepted endpoint,
  so the bridge's decision is tested without opening a provider.
- docs/metrics.md says credentials go in OTEL_EXPORTER_OTLP_HEADERS,
  never in the URL: an accepted endpoint is not echoed here, but httpx
  quotes the request URL in its own errors.

These are the three items in the #1083 thread that belong to this leg.
…led (#1083)

Completes the previous commit, which carried the other five files of the
slice: the fail-fast row now blocks the SDK with the interpreter's own
sys.modules marker and asserts the install line from both entry points,
and the loopback rows stub the seam and assert the accepted endpoint
instead of building a live provider and skipping.

With [otel] installed: 17 passed, 5 skipped before, 22 passed, 1 skipped
after. Offline: 24 passed, 1 skipped over test_otel_wire.py +
test_otel_export.py.
The job installs the [otel] extra but ran only tests/test_otel_wire.py, so the
fail-fast marker row and the endpoint rows in tests/test_otel_export.py never
ran with the SDK present -- the exact environment those rows exist to cover.
Run both files in that job; nothing else changes.
@HugoXK HugoXK added enhancement New feature or request area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) agent-pr Pull request opened by an agent labels Nov 16, 2026
@github-actions github-actions Bot added the risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller label Nov 16, 2026
@HugoXK

HugoXK commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Review request, one tip: b7774c9ba209651f4ef83da4ff0bd870f0406a56 (base f5cc52efe41750633ac37876fc2d2894a8a6f9f3).

Checks at that tip: build, test (3.9)–test (3.13), otel-wire, risk tier and reviewed tip all green; deploy skipped; mergeable_state: clean. No ping before this one, and none will follow at another sha.

Tier: T3 by path — .github/workflows/tests.yml is in the T3 list of .github/agent/risk-rules.json (the scripts/, docs/ and tests/ files in this PR are T2/T0/T1 and do not lower it). So this PR needs your PASS at the tip above and your click.

What the tip does, and how it is earned:

  • .github/workflows/tests.yml — a new otel-wire job installs .[otel] and runs scripts/otel_wire_smoke.py; tools.gen_api_docs, changelog_fragments and site checks unchanged.
  • scripts/otel_wire_smoke.py — builds a real OTLP/HTTP export against a closed loopback port and asserts the documented degradation, so the wire path is exercised rather than mocked.
  • tests/test_otel_wire.py — pins the job and the script's presence and its documented last line; tests/test_otel_export.py gains the verbatim-attribute row.
  • docs/metrics.md, changelog.d/1083-otel-wire-path.md — the page and one fragment.

Red at base f5cc52ef with only tests/test_otel_wire.py copied in: flag on → 1 failed, 1 passed on assert SMOKE_SCRIPT.is_file(); flag off → 1 failed, 1 skipped; at base grep -c otel-wire .github/workflows/tests.yml is 0. So the refusal shape is new, not moved.

Closes #1083.

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (Birfy)

Tip SHA: b7774c9
Ran:

  • pytest -q tests/test_otel_export.py tests/test_otel_wire.py at this tip → 23 passed, 1 skipped (the AGENTDESCENT_OTEL_WIRE socket row, as the offline suite intends)
  • The same two files applied onto current main f5cc52e (the tip itself is based at c94e9d6; no overlap with later main files this PR touches) → 23 passed, 1 skipped
  • test_the_ci_leg_installs_the_extra_and_runs_the_smoke_script fails on unmodified main f5cc52e (scripts/otel_wire_smoke.py is absent) and passes at this tip
  • python -m tools.changelog_fragments check → pass (40 fragments on the PR branch; 47 when the fragment is applied on current main)
  • python -m tools.gen_api_docs --check → pass
  • .github/scripts/risk_gate.py --pr 1104 → T3 (path .github/workflows/tests.yml; also a urllib/requests rule hit attributed to scripts/otel_wire_smoke.py, which does not call either — T3 already holds by the workflow path)
  • CI at this tip → green (build, test 3.9–3.13, otel-wire); deploy skipped; reviewed tip / risk tier report-only SUCCESS
    Not run:
  • mkdocs build --strict — docs/metrics.md changed; the CI build job is green at this tip
  • Full offline suite locally — the CI test matrix is the line
  • scripts/otel_wire_smoke.py against a real [otel] extra on this box — the extra is not installed here; the CI otel-wire job is SUCCESS at this tip
  • Ark real-model run — no model path changed
    Findings:
  • none
    Verdict: PASS

@Birfy

Birfy commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: b7774c9
Ran:

  • pytest -q tests/test_otel_export.py tests/test_otel_wire.py at this tip → 23 passed, 1 skipped (the AGENTDESCENT_OTEL_WIRE socket row, as the offline suite intends)
  • The same two files applied onto current main f5cc52e (the tip itself is based at c94e9d6; no overlap with later main files this PR touches) → 23 passed, 1 skipped
  • test_the_ci_leg_installs_the_extra_and_runs_the_smoke_script fails on unmodified main f5cc52e (scripts/otel_wire_smoke.py is absent) and passes at this tip
  • python -m tools.changelog_fragments check → pass (40 fragments on the PR branch; 47 when the fragment is applied on current main)
  • python -m tools.gen_api_docs --check → pass
  • .github/scripts/risk_gate.py --pr 1104 → T3 (path .github/workflows/tests.yml; also a urllib/requests rule hit attributed to scripts/otel_wire_smoke.py, which does not call either — T3 already holds by the workflow path)
  • CI at this tip → green (build, test 3.9–3.13, otel-wire); deploy skipped; reviewed tip / risk tier report-only SUCCESS
    Not run:
  • mkdocs build --strict — docs/metrics.md changed; the CI build job is green at this tip
  • Full offline suite locally — the CI test matrix is the line
  • scripts/otel_wire_smoke.py against a real [otel] extra on this box — the extra is not installed here; the CI otel-wire job is SUCCESS at this tip
  • Ark real-model run — no model path changed
    Findings:
  • none
    Verdict: PASS

@Birfy
Birfy merged commit 90d9c90 into main Nov 16, 2026
10 of 11 checks passed
@Birfy
Birfy deleted the ci/1083-wire-path-v2 branch November 16, 2026 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) enhancement New feature or request risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci(telemetry): exercise the OTLP wire path and pin verbatim attribute copying

2 participants