feat(deploy): nginx configuration for local, dev, release and production (#203)

deploy/apache/beta-bot-blocks.conf

@@ -13,10 +13,20 @@
 # correct than a list of names that grows every month.
 #
 # NOTE ON IP RULES: this host sits behind Cloudflare, so every request arrives
-# from a Cloudflare address. `Require not ip` and %{REMOTE_ADDR} therefore match
-# Cloudflare, not the client -- they block nothing or everything. Only header
-# matching (User-Agent below) is reliable until mod_remoteip is configured with
-# Cloudflare's ranges; see the bottom of this file.
+# from a Cloudflare address. Left alone, `Require not ip` and %{REMOTE_ADDR}
+# would match Cloudflare rather than the client, and block nothing or everything.
+#
+# mod_remoteip IS configured on the dev box -- `RemoteIPHeader CF-Connecting-IP`
+# with the trusted ranges, and the `cloudflare` LogFormat recording the restored
+# address. Verified 2026-09-16. This paragraph previously said it was not, which
+# is worth a correction rather than a quiet edit: it is the reason the nginx
+# translation in deploy/nginx first shipped without real-IP handling, and the
+# rate limit that depends on it would have covered the whole internet with one
+# bucket.
+#
+# The rules below still match on User-Agent rather than address. That is a
+# deliberate choice, not a limitation: a crawler's address changes and its
+# self-description mostly does not.
 
 <IfModule mod_rewrite.c>
   RewriteEngine On
@@ -73,6 +83,9 @@
 #
 #   a2enmod remoteip && systemctl reload apache2
 #
+# ALREADY APPLIED on the dev box; kept here as the record of what was done and
+# what to reapply if this host is rebuilt.
+#
 # <IfModule mod_remoteip.c>
 #   RemoteIPHeader CF-Connecting-IP
 #   RemoteIPTrustedProxy 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22

deploy/nginx/README.md

@@ -0,0 +1,165 @@
+# nginx for reactome.org
+
+Four environments, one set of shared routes. **None of this is in use yet** —
+beta is still served by hand-configured Apache on the dev box. This exists so
+the configuration is reviewable, diffable and rebuildable, and so the site can
+eventually be started locally.
+
+```
+common/     what every environment shares
+local.conf       your machine          compose service names, no TLS
+dev.conf         beta.reactome.org     the development box
+release.conf     release.reactome.org  the staged release
+production.conf  reactome.org          the public site
+```
+
+## What differs, and why
+
+|                                         | local         | dev     | release | production |
+| --------------------------------------- | ------------- | ------- | ------- | ---------- |
+| TLS                                     | no            | yes     | yes     | yes        |
+| Real client IPs via Cloudflare          | no            | yes     | yes     | yes        |
+| Named AI/SEO crawlers blocked           | no            | yes     | yes     | **yes**    |
+| _Anything_ calling itself a bot blocked | no            | yes     | yes     | **NO**     |
+| Tina admin                              | **reachable** | denied  | denied  | denied     |
+| Rate limit                              | none          | 100 r/s | 100 r/s | 600 r/s    |
+| Unknown hostnames                       | —             | 503     | 503     | 503        |
+
+Two rows carry the whole risk of this arrangement.
+
+**The blanket bot rule must never reach production.** It blocks Googlebot and
+Bingbot along with everything else. Safe on a host that must not be indexed,
+catastrophic on the one that must. It lives in `common/block-all-automation.conf`
+and is included by `dev` and `release` only. Verified by test, not by reading:
+
+```
+dev         Googlebot → 403      GPTBot → 403
+production  Googlebot → allowed  GPTBot → 403
+```
+
+**The admin rule is repeated in each environment rather than shared.** nginx
+cannot choose an include by variable, and the workaround would have hidden the
+one decision that must be obvious at a glance: whether a CMS editor is reachable
+from the internet. It is reachable locally — editing content is the point of
+running locally — and denied everywhere else.
+
+## Verified
+
+Against a running `nginx:alpine`, per environment:
+
+```
+/admin        → 403      /adminfoo    → passes through   (matches Apache)
+/admin/cms    → 403      /administer  → passes through   (matches Apache)
+GPTBot        → 403      CUBOT phone  → passes through
+curl          → passes through        dev.reactome.org → 503
+www.reactome.org → 301 to reactome.org
+```
+
+`/adminfoo` is in that list because the first draft blocked it: `location ^~
+/admin` is a prefix match, while Apache's `LocationMatch "^/admin(/|$)"` is not.
+Found by asking the running config rather than by reading it.
+
+## Local is not runnable yet, and says so
+
+`local.conf` expects compose services named `app`, `content-service`,
+`deltasignal` and `chatbot`. **Only `app` exists** in `docker-compose.yml` today.
+So this file is the shape of the answer, not the answer: starting the site
+locally still needs those services defined.
+
+DeltaSignal and the chatbot are resolved _per request_ rather than at startup,
+through a variable and a resolver, so their absence gives a 502 on those two
+routes instead of stopping nginx entirely. Named in an upstream block they would
+be resolved at startup, and one missing service would refuse to start the whole
+site — which is a poor welcome for someone who only wanted to look at a pathway.
+
+`content-service` is deliberately **not** treated that way: a site with no content
+service is not worth starting, and failing loudly is the right answer.
+
+## Upstreams move; that is expected
+
+`common/routes.conf` names `site`, `content`, `analysis`, `deltasignal` and
+`chatbot`. Each environment defines them. Two changes are coming and this shape
+absorbs both: node in this repository taking over most of what Tomcat serves, and
+the site being startable locally, where they become compose service names.
+
+`/api/` for DeltaSignal exists in none of the Apache configuration, which is why
+DeltaSignal cannot work on beta today — the Angular side calls it as a bare
+relative path, routed only by the dev server's proxy, which does not exist in a
+built artifact. Note the port on the dev box: **8090**, not the 8080 DeltaSignal's
+own compose binds, because 8080 there is Tomcat.
+
+## Connections, which is a reason this is worth doing
+
+Measured on the dev box, sockets to Tomcat's 8080:
+
+```
+76 CLOSE-WAIT      19 ESTABLISHED
+```
+
+Four in five connections leaked. Whether Apache or Tomcat is at fault is not
+settled and does not need to be — the pairing produces it, and it goes when
+Tomcat does.
+
+What this configuration changes is that connection handling is chosen rather than
+inherited: every upstream sets `keepalive`, with `keepalive_timeout 10s` — shorter
+than Tomcat's 20s default, so the backend never closes a pooled socket first.
+
+That pairing is not optional. `proxy_http_version 1.1` with `Connection ""` and
+**no** `keepalive` tells the backend to hold the socket open while nginx has no
+pool to keep it in, which is a way of causing this pile rather than curing it. An
+earlier draft of these files did exactly that — described the pooling and
+configured none of it — and it was caught by reviewing the configuration against
+its own comments.
+
+## Certificates: only the servers have them
+
+`local.conf` is plain HTTP on port 80 and touches no certificate at all —
+verified, it starts with nothing mounted at `/etc/letsencrypt`. Running the site
+on your own machine should not require obtaining a certificate for a hostname you
+do not own.
+
+Deployed environments present certificates the server already holds. This box
+needs exactly two:
+
+|                     |                                                 |
+| ------------------- | ----------------------------------------------- |
+| `beta.reactome.org` | the site                                        |
+| `dev.reactome.org`  | the retired host, so its 503 is not a TLS error |
+
+and eventually only one, when the Angular site is the one going forward. The
+wikis live on the release machine; `login.dev` is gone. The `reactome.org`
+certificate here is presented only because the retired vhost lists that name as
+an alias — production serves it.
+
+## Before any of this serves traffic
+
+**Certbot first, and proved.** Renewal is already automated — `certbot.timer`
+twice daily and a `/etc/cron.d/certbot` besides, both running `certbot -q renew`.
+That is the risk rather than the reassurance: `renew` uses each certificate's
+_stored_ authenticator, and `beta.reactome.org` still stores
+`authenticator = apache`. The day Apache stops, that renewal begins failing, and
+`-q` means it fails without saying anything. The certificate expires 60-odd days
+later.
+
+Only beta needs moving: `dev.reactome.org` already renews via `dns-cloudflare`,
+which is the same path, already working on this machine.
+
+The safe path is already proven on that box: `dev.reactome.org` renews via
+`dns-cloudflare`, and `python3-certbot-dns-cloudflare` is installed.
+`python3-certbot-nginx` is **not** — which is why `common/tls.conf` writes the TLS
+settings out rather than including `/etc/letsencrypt/options-ssl-nginx.conf`, a
+file that does not exist here. A missing include stops nginx from starting, and a
+cutover is the worst moment to learn that.
+
+```
+certbot renew --cert-name beta.reactome.org --dry-run    # must pass first
+```
+
+Then: ports 80 and 443 handed over atomically, with a one-command rollback, and
+**not during a curator review cycle** — beta is the QA gate.
+
+## What is deliberately absent
+
+No `docker-compose` service. Adding one would invite `docker compose up` to take
+port 443 from Apache on a box where that is the live site. It belongs in the same
+change as the cutover.

deploy/nginx/common/block-ai-crawlers.conf

@@ -0,0 +1,14 @@
+# Named AI and SEO crawlers, blocked on every host.
+#
+# Safe for production: this list contains no search engine anyone wants to be
+# indexed by. It is the same list the Apache configuration carries, kept in step
+# deliberately -- add here and there together.
+#
+# What is NOT here, and must not be added here, is the rule that blocks anything
+# self-identifying as a bot. That one is safe on beta precisely because beta must
+# not be indexed, and would deindex reactome.org. It lives in the beta config
+# alone, with the reason beside it.
+map $http_user_agent $blocked_crawler {
+    default 0;
+    "~*(Amzn-SearchBot|Amazonbot|PetalBot|AhrefsBot|SemrushBot|DataForSeo|MJ12bot|DotBot|CCBot|Barkrowler|Bytespider|GPTBot|ClaudeBot|Claude-Web|anthropic-ai|PerplexityBot|Bytedance|SeekportBot|serpstatbot|ZoominfoBot|Diffbot|ImagesiftBot|Timpibot|Omgilibot|meta-externalagent|Applebot-Extended)" 1;
+}

deploy/nginx/common/block-all-automation.conf

@@ -0,0 +1,24 @@
+# Block anything self-identifying as automation.
+#
+# ONLY for hosts that must not be indexed — the dev box and release. It blocks
+# Googlebot and Bingbot along with everything else, so including it in
+# production would deindex reactome.org. That is not a hypothetical caution: the
+# Apache configuration this came from carries the same warning in the same words.
+#
+# Safe on a non-indexed host precisely because the default there is "no
+# automation at all", which is a far easier rule to keep correct than a list of
+# names that grows every month.
+#
+# CUBOT is an Android phone brand whose user agent contains "bot", so it is
+# excluded first: a rule that blocks a real device is worse than one that misses
+# a crawler. Word boundaries are not an option — nearly every crawler is
+# "<something>bot", so \bbot\b would match almost none of them.
+#
+# curl and wget are deliberately absent: they are how these hosts are checked
+# from outside, and a blocked health check is worse than a scraped page.
+map $http_user_agent $blocked_automation {
+    default 0;
+    "~*(python-requests|python-urllib|aiohttp|httpx|Go-http-client|Java/|okhttp|libwww-perl|Scrapy|node-fetch|axios)" 1;
+    "~*CUBOT"                                                         0;
+    "~*(bot|crawler|crawling|spider|scraper|fetcher|archiver|indexer)" 1;
+}

deploy/nginx/common/cloudflare-real-ip.conf

@@ -0,0 +1,36 @@
+# Real client IPs behind Cloudflare.
+#
+# Every request arrives from a Cloudflare address, so without this $remote_addr
+# is Cloudflare's: logs describe Cloudflare, and any rate limit keyed on the
+# client address covers the entire internet with one bucket.
+#
+# These ranges are copied from the Apache `mod_remoteip` configuration already
+# running on the dev box, which uses the same header and the same list. They are
+# not invented, and they are not permanent -- Cloudflare publishes changes at
+# https://www.cloudflare.com/ips/ and a stale range silently stops resolving the
+# clients behind it.
+real_ip_header    CF-Connecting-IP;
+real_ip_recursive on;
+
+set_real_ip_from 173.245.48.0/20;
+set_real_ip_from 103.21.244.0/22;
+set_real_ip_from 103.22.200.0/22;
+set_real_ip_from 103.31.4.0/22;
+set_real_ip_from 141.101.64.0/18;
+set_real_ip_from 108.162.192.0/18;
+set_real_ip_from 190.93.240.0/20;
+set_real_ip_from 188.114.96.0/20;
+set_real_ip_from 197.234.240.0/22;
+set_real_ip_from 198.41.128.0/17;
+set_real_ip_from 162.158.0.0/15;
+set_real_ip_from 104.16.0.0/13;
+set_real_ip_from 104.24.0.0/14;
+set_real_ip_from 172.64.0.0/13;
+set_real_ip_from 131.0.72.0/22;
+set_real_ip_from 2400:cb00::/32;
+set_real_ip_from 2606:4700::/32;
+set_real_ip_from 2803:f800::/32;
+set_real_ip_from 2405:b500::/32;
+set_real_ip_from 2405:8100::/32;
+set_real_ip_from 2a06:98c0::/29;
+set_real_ip_from 2c0f:f248::/32;

deploy/nginx/common/routes-core.conf

@@ -0,0 +1,48 @@
+# The routes every environment shares EXCEPT the two a local machine may not be
+# running: DeltaSignal and the chatbot. Deployed environments include
+# routes.conf, which is this plus those two; local.conf includes this and
+# defines them itself, resolved per request so a missing service does not stop
+# nginx from starting.
+#
+# Upstreams are named, not addressed: each environment file defines `site`,
+# `content`, `analysis`, `deltasignal` and `chatbot` to suit itself — localhost
+# ports on a host, service names in compose, and something else again when node
+# in this repository takes over what Tomcat serves.
+#
+# What is NOT here is anything that differs by environment: bot blocking, rate
+# limiting, TLS, and the Tina admin all live in the environment files, because
+# getting one of them wrong in the wrong place is the expensive kind of mistake.
+#
+# The admin rule in particular is deliberately repeated in each environment
+# rather than shared. nginx cannot choose an include by variable, and the
+# workaround would have hidden the one decision that must be obvious at a glance:
+# whether a CMS editor is reachable from the internet.
+
+location /ContentService/ {
+    include /etc/nginx/common/upstream-proxy.conf;
+    proxy_pass http://content/ContentService/;
+    # A diagram or SBML export takes minutes. Apache's default is 300s and the
+    # site relies on it; nginx defaults to 60 and would cut them off.
+    proxy_read_timeout 300s;
+    proxy_send_timeout 300s;
+}
+
+location /AnalysisService/ {
+    include /etc/nginx/common/upstream-proxy.conf;
+    proxy_pass http://analysis/AnalysisService/;
+    # An analysis over a large identifier list is not quick either.
+    proxy_read_timeout 300s;
+    proxy_send_timeout 300s;
+}
+
+
+
+# The site itself.
+location / {
+    include /etc/nginx/common/upstream-proxy.conf;
+    proxy_pass http://site/;
+    proxy_set_header Upgrade    $http_upgrade;
+    proxy_set_header Connection $connection_upgrade;
+    proxy_read_timeout 300s;
+    proxy_send_timeout 300s;
+}

deploy/nginx/common/routes.conf

@@ -0,0 +1,67 @@
+# The routes every environment shares. Included inside a server block.
+#
+# Upstreams are named, not addressed: each environment file defines `site`,
+# `content`, `analysis`, `deltasignal` and `chatbot` to suit itself — localhost
+# ports on a host, service names in compose, and something else again when node
+# in this repository takes over what Tomcat serves.
+#
+# What is NOT here is anything that differs by environment: bot blocking, rate
+# limiting, TLS, and the Tina admin all live in the environment files, because
+# getting one of them wrong in the wrong place is the expensive kind of mistake.
+#
+# The admin rule in particular is deliberately repeated in each environment
+# rather than shared. nginx cannot choose an include by variable, and the
+# workaround would have hidden the one decision that must be obvious at a glance:
+# whether a CMS editor is reachable from the internet.
+
+location /ContentService/ {
+    include /etc/nginx/common/upstream-proxy.conf;
+    proxy_pass http://content/ContentService/;
+    # A diagram or SBML export takes minutes. Apache's default is 300s and the
+    # site relies on it; nginx defaults to 60 and would cut them off.
+    proxy_read_timeout 300s;
+    proxy_send_timeout 300s;
+}
+
+location /AnalysisService/ {
+    include /etc/nginx/common/upstream-proxy.conf;
+    proxy_pass http://analysis/AnalysisService/;
+    # An analysis over a large identifier list is not quick either.
+    proxy_read_timeout 300s;
+    proxy_send_timeout 300s;
+}
+
+# DeltaSignal. Apache has no such route, which is why DeltaSignal cannot work on
+# beta today: the Angular side calls /api/pathways, /api/parse and /api/solve as
+# bare relative paths, routed only by proxy.conf.js — the dev server's proxy,
+# which does not exist in a built artifact.
+location /api/ {
+    include /etc/nginx/common/upstream-proxy.conf;
+    proxy_pass http://deltasignal/api/;
+    # Solving a perturbation is not a fast request.
+    proxy_read_timeout 300s;
+    proxy_send_timeout 300s;
+}
+
+# The chatbot. A plain proxy renders its UI and then hangs with no replies: it is
+# server-sent events and needs buffering off as well as the upgrade headers. The
+# Apache configuration records having hit exactly this.
+location = /chat { return 302 /chat/; }
+location /chat/ {
+    include /etc/nginx/common/upstream-proxy.conf;
+    proxy_pass http://chatbot/chat/;
+    proxy_set_header Upgrade    $http_upgrade;
+    proxy_set_header Connection $connection_upgrade;
+    proxy_buffering off;
+    proxy_read_timeout 3600s;
+}
+
+# The site itself.
+location / {
+    include /etc/nginx/common/upstream-proxy.conf;
+    proxy_pass http://site/;
+    proxy_set_header Upgrade    $http_upgrade;
+    proxy_set_header Connection $connection_upgrade;
+    proxy_read_timeout 300s;
+    proxy_send_timeout 300s;
+}