Diagnosing 504 Gateway Time-out: An nginx · ALB · gunicorn Timeout Alignment Runbook

502 and 504 are completely different incidents

A 502 Bad Gateway means "the connection or the response itself failed." Typical cases include connection refused, a protocol error, or the upstream closing the connection mid-response. A 504 Gateway Time-out, by contrast, means the connection was established and the request was forwarded, but no response headers came back within the allotted time. In other words, a 502 is "I knocked and got the door slammed in my face"; a 504 is "I knocked, nobody answered, so I left first." The diagnostic starting points differ, so applying a 502 checklist to a 504 will send you down the wrong path. If you suspect a 502, start with Diagnosing nginx 502 Bad Gateway: a cause table and copy-paste commands to fix it in 5 minutes. This post is 504-only.

The core question for a 504 is just one: "Who hung up first?" 504s are showing up again as more endpoints take tens of seconds to respond—LLM API proxies, report generation, large batch triggers. Add a service mesh or API gateway and you get yet another timeout layer, making the culprit even harder to find. That's why "just set proxy_read_timeout to 300 seconds" only hides the symptom and stretches the outage.

30-second triage: name the culprit from hang-up time and response headers

Step 1 — Measure hang-up time with curl

curl -o /dev/null -s -w \
'code=%{http_code} total=%{time_total} connect=%{time_connect} ttfb=%{time_starttransfer}\n' \
'https://api.example.com/reports/heavy'

A healthy result looks like code=200 total=1.2 .... When a 504 reproduces, you get something like:

code=504 total=60.043 connect=0.031 ttfb=60.041

Here's the decision logic.

• Hang-up time repeats at a round number (30/60/65/100/300 seconds) → A configured timeout fired. Tracing the setting is the right move.
• Scattered values like 12s, 41s, 88s → App/DB latency or queuing. A load problem, not a config problem.
• Large connect value → Connection-phase issue (backlog, security group, connection exhaustion). Not really a 504; it's a connection-layer problem.

The fastest check is to repeat the same request 5 times and look at the total distribution.

for i in $(seq 1 5); do
  curl -o /dev/null -s -w "%{http_code} %{time_total}\n" 'https://api.example.com/reports/heavy'
done

Step 2 — Use response headers to see who generated the response

Whoever produced the 504 response is whoever hung up.

curl -sS -D - -o /dev/null 'https://api.example.com/reports/heavy'

Step 3 — Bypass the edge and peel the layers

If CloudFront sits in front, call the origin (ALB or nginx) directly and see how the same request ends.

# DNS를 무시하고 특정 오리진 IP로 직접 호출 (Host 헤더는 유지)
curl -o /dev/null -s -w 'code=%{http_code} total=%{time_total}\n' \
  --resolve api.example.com:443:10.0.12.34 \
  'https://api.example.com/reports/heavy'

From inside the nginx box, hit the upstream directly.

curl -o /dev/null -s -w 'code=%{http_code} total=%{time_total}\n' \
  http://127.0.0.1:8000/reports/heavy

30-second triage decision table

The last two rows are the key. The "ghost symptom"—users see 504 while app logs show a completed 200—almost always happens because an outer-layer timeout is shorter than an inner one.

Per-layer timeout table and alignment principles

Defaults vary by product version and deployment. The "Default" column below is for reference only. Always read the live value with the check command.

Command cheat sheet

# nginx: 실제 로드된 전체 설정에서 타임아웃 관련 지시어만 추출
nginx -T 2>/dev/null | grep -i -E 'timeout|keepalive'

# ALB: idle timeout 속성 확인
aws elbv2 describe-load-balancer-attributes \
  --load-balancer-arn arn:aws:elasticloadbalancing:ap-northeast-2:1234:loadbalancer/app/my-alb/abcd \
  --query "Attributes[?Key=='idle_timeout.timeout_seconds']"

# gunicorn: 실행 중 프로세스의 인자에서 timeout/worker 확인
ps aux | grep '[g]unicorn'
grep -i -E 'timeout|workers' /etc/gunicorn/gunicorn.conf.py

Alignment principle: outer layers must be longer than inner ones

Timeouts must be ordered outer layer > inner layer. When they invert, this happens:

1. The app takes 70 seconds and finishes a valid response.
2. But ALB idle timeout is 60s, so it drops the connection at 60s.
3. The user sees a 504. The app access log shows 200 ... 70.1s.
4. The dev team says "we're fine on our side," and root-cause work stretches for hours.

A recommended alignment set:

With this, the innermost layer fails first, so the real cause lands in the logs. Invert it and only the LB screams—nobody knows why.

Prescriptions by log message, and catching a genuinely slow app

nginx timeout logs: the trailing phrase is the diagnosis. Check /var/log/nginx/error.log.

grep -i 'timed out' /var/log/nginx/error.log | tail -20

The important fork: If nginx logs and gunicorn WORKER TIMEOUT appear together, the app gave up on its own. If you only have nginx logs and the app is silent, the proxy hung up first. The latter is likely an alignment problem.

Pinning down a genuinely slow app

Find long-running PostgreSQL queries

SELECT pid,
       now() - query_start AS duration,
       state,
       wait_event_type,
       left(query, 120) AS query
FROM pg_stat_activity
WHERE state <> 'idle'
  AND now() - query_start > interval '5 seconds'
ORDER BY duration DESC;

On a healthy system the result is empty or a few short batch queries. If multi-tens-of-seconds queries keep appearing at the same time as the 504, the cause is the DB. wait_event_type of Lock means lock contention; IO means disk/index issues.

Check the MySQL slow query log

SHOW VARIABLES LIKE 'slow_query_log%';
SHOW VARIABLES LIKE 'long_query_time';
-- 세션 단위 임시 활성화 예시
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;

Queuing diagnostic formula

For sync workers, theoretical requests per second is roughly:

처리량(req/s) ≈ 워커 수 / 평균 응답시간(초)

8 workers at 0.8s average response → about 10 req/s ceiling. If 30 req/s arrives, the surplus queues, and even though each request is fast, wait time accumulates into 504s. Query tuning won't fix this; worker count, worker type, and autoscaling will.

Distinguishing N+1 from external API latency

• Hundreds of DB queries per request, each under 1ms → N+1. Fix with select_related/join/batch loading.
• Only a few DB queries, but most of the response time sits in one interval → external API call. Always set an explicit timeout on the caller and attach a retry policy. If the external API has no timeout, your 504s become hostage to someone else's outage.

Change the architecture before you raise timeouts

The right approach: async jobs + polling/SSE

Work that is legitimately long-running—report generation, LLM calls, large aggregations—is not something you should endure by raising timeouts. Accept the request, immediately return 202 Accepted plus a job ID, and have the client poll status or receive progress over SSE/WebSocket. Holding an HTTP connection for 60 seconds is fragile against LB restarts, deploys, and scale-in.

If you introduce streaming (SSE), also check proxy buffering. With buffering on, nginx holds tokens the app is streaming and flushes them in a batch—and you still hit the header-wait timeout.

If you still must raise it: scope it to a location

A global raise delays connection recycling for the whole service because of one slow endpoint. Always narrow the scope.

# 일반 트래픽: 짧고 엄격하게 유지
server {
    listen 80;
    server_name api.example.com;

    proxy_connect_timeout 5s;
    proxy_send_timeout    60s;
    proxy_read_timeout    60s;

    location / {
        proxy_pass http://app_upstream;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # 오래 걸리는 엔드포인트만 예외 처리
    location /api/reports/generate {
        proxy_pass http://app_upstream;
        proxy_connect_timeout 5s;
        proxy_send_timeout    180s;
        proxy_read_timeout    180s;
        proxy_set_header Host $host;
    }

    # SSE/스트리밍 엔드포인트: 버퍼링 해제가 핵심
    location /api/stream {
        proxy_pass http://app_upstream;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 300s;
    }
}

Always syntax-check before applying.

sudo nginx -t && sudo nginx -s reload

If nginx -t prints syntax is ok / test is successful, you're good. On error, do not reload—fix the file and line in the message first.

Safety-range guide: Even an exception location is pointless if it's longer than the front-end LB idle timeout. Raise nginx to 180s while ALB is 60s and users still get 504 at 60s. Always raise from the outside in, in order.

Kubernetes adjustments

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "180"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "180"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "5"
spec:
  ingressClassName: nginx
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /api/reports
            pathType: Prefix
            backend:
              service:
                name: api-svc
                port:
                  number: 8000

Common reasons annotations don't take effect:

The most reliable check is reading the config actually loaded by the controller.

kubectl get ingress api-ingress -o yaml

kubectl -n ingress-nginx exec deploy/ingress-nginx-controller -- \
  nginx -T 2>/dev/null | grep -i -E 'proxy_read_timeout|proxy_send_timeout'

If you see proxy_read_timeout 180s;, it took effect. If it's still 60s, the annotation was not applied. For layer-by-layer tracing of Kubernetes 5xx in general, see Analyzing K8s 5xx errors: a 7-step Ingress/Gateway API debugging guide.

Preventing recurrence: timeout inventory and monitoring

504s repeat when nobody knows who put what value where. Keep this table in a wiki or repo and refresh it every quarter.

Three metrics to watch:

• HTTPCode_ELB_5XX_Count — 5xx the LB generated itself. A spike means the LB layer, not the app.
• upstream_response_time percentiles (p95/p99) — Always include $upstream_response_time in the nginx access log and collect it.
• Headroom of p99 vs. timeout threshold — Alert when p99 exceeds 70% of the threshold. Act before 504s fire, not after.

Example access log format:

log_format timing '$remote_addr - $status $request_time '
                  'upstream=$upstream_response_time '
                  'addr=$upstream_addr "$request"';

access_log /var/log/nginx/access.log timing;

A large gap between $request_time (end-to-end from the client) and $upstream_response_time (upstream processing) means a slow client network or request-body transfer. If both are similarly large, the app is slow. That one-line difference alone cuts diagnosis time on the next 504.

FAQ

Q. Will raising proxy_read_timeout to 300 seconds make 504s go away? A. They may disappear from the user's screen, but connections and workers stay occupied that much longer, so concurrent throughput drops. A modest traffic bump then queues into a larger outage. If the front-end LB idle timeout is shorter, it has no effect at all. Convert long-running work to async jobs, and if you raise a timeout, limit it to that location.

Q. Users see 504 but app logs show 200. Why? A. Classic timeout inversion. The app finished a valid response, but an outer layer (ALB or CloudFront) already dropped the connection and returned 504 to the client. Align timeouts outer > inner so the innermost layer fails first and the cause lands in the logs.

Q. How should I treat while connecting to upstream vs. while reading response header from upstream? A. The first means TCP connect never completed—look at upstream process down, security groups, backlog exceeded. The second means connect and request forwarding succeeded but no response came, so app processing delay is the most likely cause. Raising the timeout on the latter is usually the wrong answer; profile slow queries and external API calls first.