Skip to content

Give Traefik a GC target it can live under, and more room - #195

Merged
HarryCordewener merged 1 commit into
mainfrom
claude/traefik-hardening
Sep 28, 2026
Merged

HarryCordewener merged 1 commit into
mainfrom
claude/traefik-hardening

Conversation

@HarryCordewener

Copy link
Copy Markdown
Member

What happened (2026-09-28, ~22:55 UTC onward)

A distributed crawler behind Cloudflare requested random facet combinations on /games and /zh-Hans/games at about 100 req/s (64,240 requests in ten minutes, of which 60,246 got a 429). web stayed healthy throughout, answering /health in 1.5 ms and /games in 0.56 s from inside its network. Traefik failed:

  • Memory was at 511.9/512 MiB against GOMEMLIMIT=96MiB. A Go runtime over its soft limit collects continuously, and Traefik was at 62% CPU.
  • Its load-balancer health probe to web missed the 3 s timeout, so it marked the only backend down. Every path returned 503, then 522 once Cloudflare timed out on Traefik.
  • The host swapped about 870 MB between 22:55 and 23:01.

Change

Before After
GOMEMLIMIT 96MiB 640MiB
mem_limit / memswap_limit 512m 768m
websecure readTimeout / idleTimeout 60s / 180s (defaults) 15s / 30s
Access log every status everything except 429
  • No writeTimeout: it bounds the whole response, and MCP crawl_run_cycle or a slow reader of a large listing legitimately runs long.
  • Why 429s are dropped from the log: they were 94% of the lines, so the 100 MiB json-file rotation covered only ten minutes and the start of the flood had already rotated out.
  • Memory: the host was overcommitted before this change and still is (limits sum to about 4.1 GB on 3.73 GiB). The compose comment now says so.

Verification

  • Ran traefik:v3.7 with the new flags: it accepted them, and its parsed config shows readTimeout: 15s, idleTimeout: 30s and statusCodes: ["100-428","430-599"].
  • Parsed the overlay YAML.
  • Not run: docker compose config --quiet. The workstation has only podman-compose, which can't handle the existing !reset tag, so it needs to run on the box.

Deploy

Watchtower does not apply compose changes. In /opt/muindex:

git pull --ff-only origin main && docker compose config --quiet && docker compose up -d --no-deps traefik

Not in this PR: stopping the flood at the edge, which needs a Cloudflare WAF rule on stacked facet queries.

🤖 Generated with Claude Code

On 2026-09-28 a facet flood through Cloudflare took the site down while web
answered /health in 1.5 ms. Traefik sat at 511.9/512 MiB with GOMEMLIMIT at
96MiB, so the Go collector ran continuously; at 62% CPU its own health probe
to web missed the 3s timeout, the only backend was marked down, and every path
served 503 and then 522.

- GOMEMLIMIT 96MiB -> 640MiB, mem_limit 512m -> 768m: a soft target far below
  the working set turns memory pressure into CPU starvation.
- websecure readTimeout 15s, idleTimeout 30s (defaults 60s/180s): the flood
  held ~2,600 established and ~1,600 CLOSE_WAIT connections.
- Access log omits 429s: they were 94% of lines, so 100 MiB of rotation
  covered ten minutes and the start of the incident was already gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 28, 2026

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Essentials

Run ID: c7c23f70-56b9-4876-9e21-e28a30071619

📥 Commits

Reviewing files that changed from the base of the PR and between 630f655 and 0d5a909.

📒 Files selected for processing (2)
  • deploy/compose.production.yaml
  • docs/deploy.md

Comment @coderabbitai help to get the list of available commands.

@HarryCordewener
HarryCordewener merged commit 9032f3d into main Sep 28, 2026
2 of 3 checks passed
@HarryCordewener
HarryCordewener deleted the claude/traefik-hardening branch September 28, 2026 23:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant