[alukach]

Closes #265. The decision is recorded in ADR-015, which amends ADR-008.

From 2026-12-01, Cloudflare bills Workers logs and traces by volume (announcement, pricing). Paid plans include 50 GB ingested and 10 GB-month stored per cycle, then charge $0.25/GB ingested and $0.10/GB-month stored. At our current sampling, that would cost about $990/month. This PR brings it to about $15/month and keeps the signals we actually debug with.

Research (prod, 2026-09-29 → 2026-10-28)

Measured from the GraphQL analytics API (workersInvocationsAdaptive) and the observability API (/workers/observability/usage for bytes, /telemetry/query for counts by log type):

Prod requests	986M per 30 days (~33M/day)
Logs stored (at 25% sampling)	~42 GB/day, ~5.2 KB per event. ~99.9% are invocation logs (~37.8M/day after scaling up from the sample)
Our own log lines	~29k WARN + ~60 ERROR per day across all traffic (incl. the per-5xx WARN line in src/lib.rs)
Traces stored (at 25% sampling)	~80 GB/day, ~1.6 KB per span, ~6 automatic spans per request
Total	~122 GB/day, ~3.7 TB/month
Staging	2M requests per 30 days, ~1.3 GB of telemetry
Axiom destinations	axiom-logs / axiom-traces have failed since 2026-10-05: "Destination delivery failed. Check the receiver configuration and credentials."

Byte sizes are only recorded from 2026-10-06, so byte figures average Oct 7–28.

Queries used

All against account 1335bb5b87a1c2e1de2ca03408c23b71, run 2026-10-29.

Requests per Worker, 30 days — POST /graphql (Cloudflare's analytics API):

query($a:String!,$s:Time!,$e:Time!){viewer{accounts(filter:{accountTag:$a}){
  workersInvocationsAdaptive(limit:100, filter:{datetime_geq:$s, datetime_lt:$e}, orderBy:[sum_requests_DESC]){
    sum{requests errors subrequests} dimensions{scriptName}}}}}
# s=2026-09-29T00:00:00Z  e=2026-10-29T00:00:00Z

Telemetry events and bytes per Worker, per day — GET /accounts/{id}/workers/observability/usage?from=<ms>&to=<ms> (breakdown[].count, breakdown[].eventSize; dataset cloudflare-workers = logs, otel = traces; sizes recorded from 2026-10-06).

Log events by type and level, 1 day — POST /accounts/{id}/workers/observability/telemetry/query:

{"queryId":"adhoc-breakdown","dry":true,"view":"calculations","ignoreSeries":true,
 "timeframe":{"from":1761436800000,"to":1761523200000},
 "parameters":{"datasets":["cloudflare-workers"],
  "filters":[{"key":"$metadata.service","operation":"eq","type":"string","value":"source-data-proxy"}],
  "calculations":[{"operator":"count"}],
  "groupBys":[{"type":"string","value":"$metadata.type"},{"type":"string","value":"$metadata.level"}]}}

cf-worker-event = invocation logs, cf-worker = our own lines; counts are scaled up for sampling.

Responses by status class, 7 days — POST /accounts/{id}/analytics_engine/sql:

SELECT intDiv(double2,100) AS cls, SUM(_sample_interval) AS n
FROM source_data_proxy_production
WHERE timestamp > NOW() - INTERVAL '7' DAY GROUP BY cls ORDER BY cls

Observability settings and destination status — GET /accounts/{id}/workers/scripts/{name}/settings, GET /accounts/{id}/workers/observability/destinations.

Estimated monthly cost from Dec 1

	Ingested	Stored	Cost
Current config	~3,660 GB	~855 GB-month	~$990, plus ~$360 export if Axiom worked and is billed at the blog's $0.10/GB export rate
This PR	~103 GB (traces 96, our logs ~2, staging ~5)	~24 GB-month	~$15

Changes

Prod

• Logs: sample 100% of our own lines and turn off invocation logs. Every WARN/ERROR line is now kept, where 25% sampling used to drop three-quarters of them. The invocation records we lose duplicate the ANALYTICS dataset, which already records every request (status, duration, bytes, path, product).
• Traces: 1% baseline, about 330k traced requests/day.
• redact_query_string = true: traces store full URLs with query strings, and presigned requests carry X-Amz-Signature there. The trade-off is losing list-type/prefix details in traces.
• Workers Issues enabled: free during the beta. It groups uncaught exceptions, failed invocations and 5xx responses, which covers the failed-invocation records we used to get from invocation logs.
• Axiom destinations removed: export has been broken for 24 days without anyone noticing, and Cloudflare now provides SQL queries, dashboards and alerts natively.

PR previews (wrangler.preview.toml): redact query strings and enable Issues; sampling stays at 100%.

Staging: capture 100% of logs, invocation logs and traces (~5 GB/month). This resolves the TODO about separate rates for prod and staging.

public-log-stream: left without observability on purpose. It isn't critical, and its free built-in metrics already show its error rate.

Checked that the TOML parses, and that Wrangler 4.155.0 (what CI's wrangler@4 installs) accepts redact_query_string and observability.issues, which need ≥ 4.134.

Follow-ups (outside wrangler.toml)

• Delete the axiom-logs / axiom-traces destinations in the dashboard. They are account-level objects, so this PR doesn't remove them.
• Add a zone Trace Rule that forces 100% sampling for a debug header (e.g. x-source-trace) so we can trace a single request on demand. coolnewgeo.com is on the Free plan, and I couldn't verify through the API that Trace Rules are available.
• Set up Alerts on source_data_proxy_production: 5xx rate and p99 duration_ms.
• Watch the latent Analytics Engine cost. It isn't billed today; if billing turns on, ~986M writes/month ≈ $244/month.

🤖 Generated with Claude Code

[claude]

Claude finished @alukach's task in 15s —— View job

---

✅ No blocking issues — safe to merge.

I read the wrangler.toml, wrangler.preview.toml and ADR-008 diffs. I did not read ADR-015 itself, and I did not run Wrangler or deploy anything.

• The prod and staging observability blocks are consistent. Staging is a separate [env.staging.observability] tree, so it doesn't inherit prod values, and every key it needs is set explicitly.
• redact_query_string and [observability.issues] are set at the top level, in wrangler.preview.toml and in staging. The PR description says Wrangler 4.155.0 accepts both keys.
• Removing the destinations entries is correct. The Axiom exports have been failing since 2026-10-05, and persist = true keeps logs and traces in Cloudflare Observability.
• The 5xx WARN line at src/lib.rs:437 and the refusal warnings at src/lib.rs:627, 639 and 715 log request IDs and reasons, not URLs, so the redaction doesn't leave a query string leaking through our own logs. I only grepped for warn! and URL/query mentions in src/lib.rs, so this isn't a full audit of every log call.

Two non-blocking notes:

• Observability is now 100% sampling for logs and 1% for traces, with invocation logs off. A request that fails without a 5xx WARN, such as an uncaught exception or a client abort, will only show up through Workers Issues, which is a beta feature. The PR already accepts this trade-off, so I'd just keep an eye on it.
• ADR-008's amendment says the old paragraph "describes the configuration before 2026-10-29". That date is correct only if this merges today. Consider linking to the ADR-015 date instead.

Simplify (ponytail)

• No findings. The change deletes config and adds no new code.

---

💰 Estimated review cost: $0.10 · 0m14s · 5 turns

[github-actions]

🚀 Latest commit deployed to https://source-data-proxy-pr-266.source-coop.workers.dev

• Date: 2026-10-29T18:32:01Z
• Commit: b414915