Sporadic random HTTP 504

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

The app-client schema looks like this:

app (nginx + fastcgi) <-> traefik <-> client (my own)

Client connects directly to the traefik, traefik connects directly to the backend app. There is no load balancers, proxies, or anything above TCP/IP between those components.

Amount of traffic: tens of thousands of requests a day

Around 20-50 requests a day end up with HTTP 504 from traefik.

{
    "ClientAddr": "10.50.1.12:45790",
    "ClientHost": "10.50.1.12",
    "ClientPort": "45790",
    "ClientUsername": "-",
    "DownstreamContentSize": 15,
    "DownstreamStatus": 504,
    "Duration": 15193763,
    "KubernetesIngressName": "<redacted>",
    "KubernetesIngressNamespace": "<redacted>",
    "KubernetesServiceName": "<redacted>",
    "KubernetesServicePort": "80",
    "OriginContentSize": 15,
    "OriginDuration": 14882260,
    "OriginStatus": 504,
    "Overhead": 311503,
    "RequestAddr": "<redacted>",
    "RequestContentSize": 343,
    "RequestCount": 34460,
    "RequestHost": "<redacted>",
    "RequestMethod": "POST",
    "RequestPath": "/api/session/app",
    "RequestPort": "-",
    "RequestProtocol": "HTTP/1.1",
    "RequestScheme": "https",
    "RetryAttempts": 0,
    "RouterName": "<redacted>@kubernetesingressnginx",
    "ServiceAddr": "10.51.14.246:8080",
    "ServiceName": "<redacted>@kubernetesingressnginx",
    "ServiceURL": "http://10.51.14.246:8080",
    "StartLocal": "2026-09-09T03:29:20.779600412Z",
    "StartUTC": "2026-09-09T03:29:20.779600412Z",
    "TLSCipher": "TLS_AES_128_GCM_SHA256",
    "TLSVersion": "1.3",
    "entryPointName": "websecure",
    "level": "info",
    "msg": "",
    "request_User-Agent": "GuzzleHttp/7",
    "time": "2026-09-09T03:29:20Z"
}

The request duration varies from single digit milliseconds (like 7.4ms), to hundreds of ms (530ms)

There is no deterministic pattern, it just happens occasionally, once. And then everything is fine.

I'm assessing traefik as a replacement for ingress-nginx kubernetes ingress operator. ingress-nginx serves fine, with no unexpected http/networking problems.

The client simply receives HTTP 504 with Gateway Timeout response.

The backend application just sees the connection closed while it still processes the request.

What did you see instead?

.

What version of Traefik are you using?

Version: 3.7.13 Codename: langres Go version: go1.26.8 Built: 2026-09-04T12:31:42Z OS/Arch: linux/amd64

What is your environment & configuration?

# (paste your configuration here)

Add more configuration information here.

    - --entryPoints.metrics.address=:9100/tcp
    - --entryPoints.traefik.address=:8080/tcp
    - --entryPoints.web.address=:8000/tcp
    - --entryPoints.websecure.address=:8443/tcp
    - --api.dashboard=true
    - --ping=true
    - --metrics.prometheus=true
    - --metrics.prometheus.entrypoint=metrics
    - --providers.kubernetescrd
    - --providers.kubernetescrd.allowEmptyServices=true
    - --providers.kubernetesingress
    - --providers.kubernetesingress.allowEmptyServices=true
    - --providers.kubernetesingress.ingressendpoint.publishedservice=adm-traefik-adm-prod/helm-traefik
    - --providers.kubernetesingressnginx
    - --providers.kubernetesingressnginx.controllerClass=k8s.io/adm-prod
    - --providers.kubernetesingressnginx.disableSvcExternalName=false
    - --providers.kubernetesingressnginx.httpEntryPoint=web
    - --providers.kubernetesingressnginx.httpsEntryPoint=websecure
    - --providers.kubernetesingressnginx.ingressClass=nginx-adm-prod
    - --providers.kubernetesingressnginx.ingressClassByName=false
    - --providers.kubernetesingressnginx.watchIngressWithoutClass=false
    - --entryPoints.websecure.http.tls=true
    - --log.level=INFO
    - --accesslog=true
    - --accesslog.format=json
    - --accesslog.fields.defaultmode=keep
    - --accesslog.fields.headers.defaultmode=drop
    - --accesslog.fields.headers.names.User-Agent=keep
    - --entryPoints.websecure.http.middlewares=adm-traefik-adm-prod-traefik-headers@kubernetescrd,adm-traefik-adm-prod-traefik-compress@kubernetescrd

If applicable, please paste the log output in DEBUG level

I was able to catch one of such requests in debug logging mode too:

2026-09-14T04:24:21Z DBG github.com/traefik/traefik/v3/pkg/proxy/httputil/proxy.go:205 > 504 Gateway Timeout error=""read tcp 10.35.74.121:56870->10.35.77.113:8080: i/o timeout"""

It was for a request that is Duration: 227725160, that is - 227.7ms.

The backend application was alive and well, and was ready to serve the response back at 04:24:23.572408 (so it took 2 seconds to generate the response), but by that moment traefik has already closed the connection.

So, the question is - where does such a short timeout come from in a trafik configuration? And why it's so random It almost looks like the reverse proxy transport is not concurrency-safe.