feat(deploy): nginx configuration for local, dev, release and production (#203) by adamjohnwright · Pull Request #221 · reactome/WebsiteAngular

adamjohnwright · 2026-09-16T18:28:10Z

Part of #203. Nothing changes on any server. This puts the configuration in
the repository so it is reviewable and rebuildable, and so the site can
eventually be started locally. The cutover is a separate change with
prerequisites that are not met yet.

Four environments, one set of routes

	local	dev	release	production
TLS	no	yes	yes	yes
Cloudflare real IPs	no	yes	yes	yes
Named AI/SEO crawlers blocked	no	yes	yes	yes
Anything calling itself a bot blocked	no	yes	yes	NO
Tina admin	reachable	denied	denied	denied
Rate limit	none	100 r/s	100 r/s	600 r/s

The row that carries the risk is the fourth. That rule blocks Googlebot and
Bingbot along with everything else — safe where nothing should be indexed,
catastrophic on the host that must be. One configuration for both would have
deindexed reactome.org. Asserted by test, not by reading:

dev         Googlebot → 403      GPTBot → 403
production  Googlebot → allowed  GPTBot → 403

What a root-level audit of Apache contradicted

I could not read most of the Apache configuration until Adam ran a read-only
audit script. Three things in my first draft were wrong:

Real client IPs were missing. Apache runs mod_remoteip with
CF-Connecting-IP and sixteen trusted ranges — the comment in
beta-bot-blocks.conf saying it is not configured is stale, and is corrected
in this PR. Omitting it would have left the new rate limit covering the whole
internet with one bucket.
The TLS include named a file that does not exist.
options-ssl-nginx.conf is written by certbot's nginx plugin, and only the
apache and dns-cloudflare plugins are installed. nginx does not start with a
missing include.
location ^~ /admin is a prefix match and also denied /adminfoo, a URL
Apache serves. Apache's rule is LocationMatch "^/admin(/|$)"; it is a regex now.

What an adversarial review of my own work then found

The config claimed connection pooling it did not have.
upstream-proxy.conf said pooling was "configured deliberately in each
upstream block" and it was configured in none, while setting
proxy_http_version 1.1 and Connection "". That pairing without keepalive
tells the backend to hold sockets open with no pool to keep them in — a way of
causing the CLOSE-WAIT pile this migration is partly meant to cure. Measured
on the dev box: 76 CLOSE-WAIT against 19 ESTABLISHED to Tomcat. Every
upstream now sets keepalive 32 with a 10s timeout, shorter than Tomcat's 20s.
local.conf could not start. It named compose services that do not exist;
nginx refuses to start when an upstream will not resolve. DeltaSignal and the
chatbot are resolved per request now, so their absence is a 502 on those routes
rather than a site that will not boot.

Verified

All four pass nginx -t, and behaviour was exercised with curl per environment:
the admin denial and its /adminfoo exception, the bot rules and the CUBOT and
curl exceptions, the chat redirect, /api reaching the proxy, and unknown hosts
returning 503.

Known and deliberate

http2 on is new — Apache here has no http2 module, so beta is HTTP/1.1 today.
The rate limit matches mod_evasive's per-site number but has no per-page
equivalent to its DOSPageCount 12. Slightly more permissive; better tuned
against real traffic than guessed.
Local is not runnable yet and the README says so: only app exists in
docker-compose.yml.

Before any cutover

Certbot first. beta.reactome.org and reactome.org renew with
authenticator = apache; stop Apache without migrating them and renewal fails
silently. The safe path is already proven on that box — dev.reactome.org renews
via dns-cloudflare — so it is two certificates onto a working path, not a new
capability. The wikis and login.dev are not needed and can simply go.

🤖 Generated with Claude Code

beta.reactome.org is served by Apache configured by hand on the dev box. Nothing about that arrangement is in any repository, so it is unreviewed, unversioned, and reconstructable only from memory if the box is rebuilt. This is that configuration, as nginx, read off the box on 2026-09-16: the admin denial, both Tomcat services, the chat with its server-sent events, the site itself, and the twelve bot-blocking rules translated from the Apache include. Plus the one route Apache does not have -- `/api/` for DeltaSignal, which is why DeltaSignal cannot work on beta at all today: the Angular side calls it as a bare relative path, routed only by the dev server's proxy, which does not exist in a built artifact. On 8090, not the 8080 its own compose binds, because 8080 here is Tomcat and an unconfigured deployment would get 404s from a real server rather than a clear failure. **Not in use, and no compose service.** Adding one would invite `docker compose up` to take port 443 from Apache on a box where that is the live site. The compose entry belongs in the same change as the cutover, and the cutover is gated on certbot: it authenticates through Apache today, and swapping the proxy without migrating that stops renewal silently -- beta loses HTTPS in 90 days with nothing to warn anyone. Checked against a running nginx, not just read: the admin denial, the bot rules, the CUBOT and curl exceptions, and the chat redirect. That caught a real difference -- `location ^~ /admin` is a prefix match and also denied `/adminfoo`, which Apache serves. It is a regex now, matching `LocationMatch "^/admin(/|$)"` exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Reviewing the translation against what Apache is actually doing, rather than against the beta vhost alone, found four things missing. Each would have been a silent regression at cutover. **Rate limiting.** mod_evasive is active here -- 12 requests per page per second, 100 per site, blocked for 60. Nothing replaced it. On a box whose Tomcat heap fills within minutes when something walks the pathway URLs, that is the only thing between a crawler and the database. Added, with the note that it is keyed on the client address and therefore means nothing until the real-IP block is switched on: until then every request appears to come from Cloudflare and one limit covers the whole internet. **gzip.** mod_deflate is enabled on the Apache this replaces. **The retired dev site.** `001-reactome.conf` is dev.reactome.org: a Joomla 3.10.12 stack, end of life since 2023, which was the entry vector on production and release in September. The only thing making this host safe is that the vhost returns 503 for everything. A configuration that answered only for beta would let a request for dev.reactome.org fall to the default server and be served beta's content under a retired name. There is now an explicit default server returning 503. That vhost is also the strongest argument for this migration, which is worth writing down: it still carries `Options +Includes` and `XBitHack on` with mod_include loaded. Server-Side Includes are an output filter, so no PHP-shaped rule touches them -- a file dropped as .shtml, or as .html with the execute bit, runs as www-data. nginx has no mod_include. The class disappears rather than being held shut by one line. **Upstreams are named once**, because they are going to move: node in this repository is to take over most of what Tomcat serves, and the site should be startable locally. Both want one edit rather than six, and in compose these become service names. Verified against a running nginx: dev.reactome.org and an unknown host get 503, beta's admin denial, bot rules and :80 redirect all still behave. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A root-level audit of Apache, which I could not read before, contradicted two things in this config. **Real client IPs were left out, and they are configured today.** Apache runs `RemoteIPHeader CF-Connecting-IP` with sixteen trusted Cloudflare ranges. The bot-block file's comment saying mod_remoteip is "not configured" is simply out of date. Omitting it would have been a silent regression, and worse than cosmetic: the rate limit added in the last commit is keyed on the client address, so without this it would have covered the entire internet with a single bucket. The same ranges are now carried here, with a note that Cloudflare publishes changes and a stale list stops resolving clients. **The TLS include named a file that does not exist.** `/etc/letsencrypt/options-ssl-nginx.conf` is written by certbot's *nginx* plugin, and only python3-certbot-apache and python3-certbot-dns-cloudflare are installed here. nginx does not start when an include is missing, and a cutover is the worst moment to discover that. The settings are written out instead. Both found by asking the machine rather than by reading the config I had already written, which is the same lesson as `location ^~ /admin`. Shared pieces split into common/ because this config is meant for production too, where some of it must differ -- see the next commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Four environments -- local, dev, release, production -- sharing routes and differing only where they must. The differences are the point. **The blanket bot rule cannot go to production.** It blocks anything self-identifying as a bot, which is safe where nothing should be indexed and catastrophic on the host that must be: it takes Googlebot and Bingbot with it. Writing one configuration for both would have shipped that. It is now `common/block-all-automation.conf`, included by dev and release only, and the difference is asserted by test: dev Googlebot -> 403 GPTBot -> 403 production Googlebot -> allowed GPTBot -> 403 **The admin rule is repeated per environment rather than shared.** nginx cannot pick an include by variable, and the workaround would have hidden the one decision that has to be obvious at a glance: whether a CMS editor is reachable from the internet. Reachable locally, where editing content is the point; denied everywhere else. **Timeouts moved to the routes that need them.** A shared default plus a per-route override does not compile -- nginx rejects a duplicate directive in the same context -- and the routes genuinely differ: an export takes minutes, the chat holds a connection for an hour. Also corrected, in the Apache file rather than here: the note claiming mod_remoteip is not configured. It is, with the full Cloudflare range list. That stale sentence is why the first nginx draft shipped without real-IP handling, and the rate limit that depends on it would have covered the whole internet with one bucket. The correction says so rather than quietly editing the line. All four configurations pass `nginx -t` and were exercised with curl. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…have Adversarial review of my own nginx work, and the worst finding is the one it was most confident about. `common/upstream-proxy.conf` said pooling "is configured deliberately in each upstream block". It was configured in none of them, while the file set `proxy_http_version 1.1` and `proxy_set_header Connection ""`. That pairing without a `keepalive` directive tells the backend to hold the socket open while nginx has no pool to keep it in -- which is a way of *causing* the CLOSE-WAIT pile this configuration is partly meant to cure. 76 CLOSE-WAIT against 19 ESTABLISHED is the measurement it would have reproduced. Every upstream now sets `keepalive 32` with `keepalive_timeout 10s`, shorter than Tomcat's 20s default so the backend never closes a pooled socket first, and the comment describes what is there rather than what was intended. Second finding: `local.conf` named compose services that do not exist -- `content-service`, `deltasignal`, `chatbot` -- and nginx refuses to start when an upstream will not resolve. "Someone can spin up the website locally" would have failed at the first absent service. DeltaSignal and the chatbot are now resolved per request through a variable and a resolver, so their absence is a 502 on those two routes rather than a site that will not start. Someone who only wanted to look at a pathway should not need a Julia solver running. The content service is deliberately not treated that way: a site without it is not worth starting, and failing loudly is correct. The README says local is not runnable yet, because it is not. All four configurations still pass, and the behaviour that matters was re-checked after the refactor: Googlebot 403 on dev and through on production, admin denied, the retired host 503. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Corrections from Adam, who knows what these machines are for. **This box needs dev.reactome.org and beta.reactome.org, and nothing else.** The wikis are on the release machine, login.dev is gone, and production serves reactome.org. Eventually only dev.reactome.org remains, once the Angular site is the one going forward. So the retired host gets its own certificate rather than being handed beta's: it has one, it already renews via dns-cloudflare, and presenting the wrong name would make the retirement a TLS error instead of the clean 503 it is meant to be. The catch-all for names this box does not serve still answers 503 -- the point being that an unknown name is never quietly served beta's content. **Locally there are no certificates at all.** local.conf is plain HTTP on port 80 and mounts nothing from /etc/letsencrypt; verified, it starts with that path absent. Running the site on your own machine should not require obtaining a certificate for a hostname you do not own. Also written down, because it changes what the risk is rather than removing it: renewal is already automated here, by certbot.timer twice daily and a cron file besides, both running `certbot -q renew`. That is the problem. `renew` uses each certificate's stored authenticator, beta's is still `apache`, and `-q` means the failure after Apache stops is silent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

I wrote them, and I should not have. `deploy/nginx/dev.conf` -- merged and public since #221 -- named the retired site's software and exact version, said it was end of life, stated it had been the entry vector for a real compromise, said the only thing making the host safe was a single 503, and named the precise misconfiguration still present. That is a roadmap: what to attack, that it worked before, how thin the mitigation is, and where to look. I scrubbed the release SOP for far less this morning and then wrote this myself a few hours later. The operational reasoning is kept, because it is the actual argument for the migration: the retired site runs on an Apache loading modules this configuration does not load, so a class of problem that has to be held shut stops being reachable. The specifics belong in internal notes. Also removed: a product-and-version reference in RELEASE-TESTING.md's figures row. The lesson there -- an origin answering a missing path with index.html, a 200 that is not an image -- is the useful part and stays. Reviewed before merging, as asked. Behaviour re-checked afterwards: every retired name returns 503 over both HTTP and HTTPS, for every path including /admin, /ContentService/, /api/ and /chat, so the retired block cannot proxy anything; beta still redirects :80 and proxies :443. Both files the config needs, reactome-origin.crt and .key, exist on the box. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…issed #224 (#225) * fix(deploy): present the certificate Apache actually presents Corrects this branch, and corrects something I told Adam with more confidence than the evidence supported. I said dev.reactome.org's certificate expiring would turn the retirement into a browser warning, and built a server block around that. Asking the running server what it serves for each name says otherwise: beta.reactome.org presents beta.reactome.org (Let's Encrypt) dev.reactome.org presents CloudFlare Origin Certificate reactome.org presents CloudFlare Origin Certificate www.reactome.org presents CloudFlare Origin Certificate `001-reactome.conf` sets `SSLCertificateFile /etc/ssl/cloudflare/reactome-origin.crt`, with the Let's Encrypt line commented out beneath it. So **only one Let's Encrypt certificate on this box is used at all**, beta's, and the dev.reactome.org and reactome.org certificates are leftovers presenting nothing. The retired names therefore present the Origin certificate here too, which is what a faithful translation means. All traffic arrives through Cloudflare, so an Origin certificate is the right thing on this leg -- it is issued by Cloudflare, long-lived, and not certbot's business, which is why none of the certbot work touches it. The separate reactome.org block added in the previous commit is folded into the same server: same certificate, same 503, one block. Verified: dev.reactome.org, reactome.org, www.reactome.org and an unknown name all 503; beta still proxies. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(deploy): dev.reactome.org's certificate is kept on purpose Nothing presents it today, which is exactly why it would be deleted as cruft by someone tidying up -- I nearly recommended it. It is there because the Angular site is expected to move from beta.reactome.org to dev.reactome.org, and the certificate is pre-provisioned for that. Also records what the box actually presents for each name, established by asking the running server rather than by listing what is on disk. Only beta's Let's Encrypt certificate is in use; the retired names present a Cloudflare Origin certificate that certbot does not manage. reactome.org's certificate was deleted: nothing presented it, and it shared a name with dev.reactome.org's, which made combined renewals fight over the same challenge record. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(deploy): take the security specifics out of a public repository I wrote them, and I should not have. `deploy/nginx/dev.conf` -- merged and public since #221 -- named the retired site's software and exact version, said it was end of life, stated it had been the entry vector for a real compromise, said the only thing making the host safe was a single 503, and named the precise misconfiguration still present. That is a roadmap: what to attack, that it worked before, how thin the mitigation is, and where to look. I scrubbed the release SOP for far less this morning and then wrote this myself a few hours later. The operational reasoning is kept, because it is the actual argument for the migration: the retired site runs on an Apache loading modules this configuration does not load, so a class of problem that has to be held shut stops being reachable. The specifics belong in internal notes. Also removed: a product-and-version reference in RELEASE-TESTING.md's figures row. The lesson there -- an origin answering a missing path with index.html, a 200 that is not an image -- is the useful part and stays. Reviewed before merging, as asked. Behaviour re-checked afterwards: every retired name returns 503 over both HTTP and HTTPS, for every path including /admin, /ContentService/, /api/ and /chat, so the retired block cannot proxy anything; beta still redirects :80 and proxies :443. Both files the config needs, reactome-origin.crt and .key, exist on the box. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

adamjohnwright and others added 5 commits September 16, 2026 18:25

adamjohnwright changed the title ~~feat(deploy): write beta's configuration down, as nginx (#203)~~ feat(deploy): nginx configuration for local, dev, release and production (#203) Sep 16, 2026

adamjohnwright and others added 3 commits September 16, 2026 15:21

Merge branch 'main' into feat/nginx-config-in-repo

6626d72

Merge branch 'main' into feat/nginx-config-in-repo

369e3db

adamjohnwright mentioned this pull request Sep 16, 2026

DeltaSignal's API is published on all interfaces, unlike every other local service #222

Open

adamjohnwright merged commit a88bdcb into main Sep 16, 2026
6 checks passed

adamjohnwright deleted the feat/nginx-config-in-repo branch September 16, 2026 19:58

Navigation Menu

Uh oh!

feat(deploy): nginx configuration for local, dev, release and production (#203) - #221

feat(deploy): nginx configuration for local, dev, release and production (#203)#221
adamjohnwright merged 8 commits into
mainfrom
feat/nginx-config-in-repo

adamjohnwright commented Sep 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

adamjohnwright commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Four environments, one set of routes

What a root-level audit of Apache contradicted

What an adversarial review of my own work then found

Verified

Known and deliberate

Before any cutover

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

adamjohnwright commented Sep 16, 2026 •

edited

Loading