Repository navigation
Force HTTP1 or enable HTTP2 #353
Description
We have deployed according to the steps in #322.
However, we have a problem that only seems to affect some clients:
$ curl -v 'https://stag.foo.com/'
* Trying <ip>...
* TCP_NODELAY set
* Connected to stag.foo.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=stag.foo.com
* start date: Feb 23 01:50:57 2021 GMT
* expire date: May 24 01:50:57 2021 GMT
* subjectAltName: host "stag.foo.com" matched cert's "stag.foo.com"
* issuer: <bar>
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd84900d600)
> GET / HTTP/2
> Host: stag.foo.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host stag.foo.com left intact
curl: (16) Error in the HTTP2 framing layer
* Closing connection 0
Notice the error:
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
Most clients do not experience this, however:
% curl -v 'https://stag.foo.com/'
* Trying <ip>...
* TCP_NODELAY set
* Connected to stag.foo.com (<ip>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: CN=stag.foo.com
* start date: Mar 15 02:07:24 2021 GMT
* expire date: Mar 29 02:07:24 2021 GMT
* subjectAltName: host "stag.foo.com" matched cert's "stag.foo.com"
* issuer: <bar>
* SSL certificate verify ok.
> GET / HTTP/1.1
> Host: stag.foo.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 3618
< Content-Type: text/html
< Date: Mon, 15 Mar 2021 17:48:39 GMT
< Etag: "604f7271-e22"
< Last-Modified: Mon, 15 Mar 2021 14:42:57 GMT
< Server: nginx/1.15.12
<
* Connection #0 to host stag.foo.com left intact
<!DOCTYPE html><html lang=en>...</html>* Closing connection 0
The difference from the failing output appears to start here:
> GET / HTTP/1.1
Therefore, when the request succeeds, it's because the client falls back to HTTP/1.1. When the request fails, it's because the client attempts to use HTTP/2, which the server claims to support but fails to do so.
The solution appears to be either:
- Force HTTP/1.1, or
- Enable HTTP/2
How can we configure this stack to force HTTP/1.1 or enable HTTP/2?
Activity
abrichr commentedon Mar 19, 2021
This appears to be a known issue:
traefik/traefik#6392
traefik/traefik#7776
traefik/traefik#7814
One workaround (described in https://community.traefik.io/t/traefik-advertises-http-2-when-backend-doesnt-support-it/1776) appears to involve creating a
traefik.tomlfile for "dynamic" configuration with the following:Here this would probably also require the addition of something like the following to
traefik.ymland/ordocker-compose.ymlunder thetraefikimage:Another workaround (described in https://community.traefik.io/t/traefikv2-http-2-0/1199) involves setting the following environment variables:
Here this would involve the addition of the following to
traefik.ymland/ordocker-compose.ymlunder thetraefikimage:Will test these out and report back.
abrichr commentedon Mar 19, 2021
Adding the variables in both
docker-compose.ymlandtraefik.ymlyielded the following change:abrichr commentedon Mar 19, 2021
Tried various combinations of specifying minVersion and cipherSuites in the .toml file and in the traefik.yml, with no luck.
Turning on debug logging in traefik.yml shows this whenever an attempt is made to connect from the affected machine:
Also:
None of the container IPs match the target in the Traefik log of 10.0.0.2...
🤔
pdbs commentedon Dec 20, 2022
Same for Traefik v2.9.6: traefik.tcp.routers.gf7tcp.tls.passthrough=true results in error "404 not found" in Microsoft Edge.
In Internet Explorer 11 (no HTTP/2 ?) TLS passthrough to Glassfish 7 (HTTP/2 enabled)
Log has no handshake error message - Traefik has the correct IP & port in dashboard.
Setting TLS options does not help, loglevel DEBUG does not print any error message.
But I only have this problem in a docker swarm environment. In docker standalone the same docker image (traefik & glassfish) works fine with
Very strange behaviour.
Edit: It might have something to do with HTTP/2 and wildcard certificates. I will change that and report.
pdbs commentedon Dec 22, 2022
It seems that adding
to dynamic configuration file solves the problem, but only if the certificate is exclusively used on the tcp router (and not on other https routers) since it contains wildcard DNS SANs and this can not be handled by traefik. Maybe adding a configuration value to [[tls.certificates]] which indicates all hostnames the certificate should be used for could help - or defining the certificate file on service level value.
I assume the problem is not (only) http2 in traefik since browsers acted differently: Sometimes they where showing the webserver's index.html, sometimes 404 from traefik although different certificates were used. But setting the order
"http/1.1", "h2"solved the problem. curl always worked, adding GODEBUG env variables worked for the tcp router but left the http routers unusable.🧱 Add failed status for docker builder when it fails (#353)