certificate has expired: a five-branch diagnosis (nginx · openssl)

The cert was renewed — so why is it still showing as expired?

There's a scene that plays out all too often at 2 a.m. The certbot renew log says Congratulations, all renewals succeeded, and ls -l /etc/letsencrypt/live/example.com/ shows file timestamps from moments ago. Yet the browser still throws NET::ERR_CERT_DATE_INVALID.

There's one premise you have to lock in before going any further.

The certificate file on disk and the certificate the server process is holding in memory are completely different objects.

nginx and haproxy read the certificate file into memory at start/reload time. After that, the process has no idea if the file changes. So "I checked the file" is not a diagnosis. Diagnosis means checking what the server actually serves on the socket.

The scope of this post is explicit. It covers only expiry of the certificate the server presents, chain construction, and protocol negotiation failures. Client-side trust store or CA bundle problems (missing internal root CA, JDK cacerts, Python certifi, etc.) live in a different cause layer, so this post only links out at those points.

Scope of applicability:

Error text → cause decision table (finish this in 30 seconds)

Don't scroll. Find the error string you're looking at in the left column.

The confirming evidence for each branch comes out of the diagnostic command set below in one pass. Blindly re-running certbot renew on a guess does nothing for (b)–(e) and only burns your rate limit.

30-second diagnostic command set — which line of the output to look at

1) Confirm the actual chain from the socket

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | head -40

Healthy (complete chain) output example:

Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R11
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Verify return code: 0 (ok)

Missing depth 1 output example:

Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R11
---
Verify return code: 21 (unable to verify the first certificate)

How to read it is simple.

• depth 0 = server certificate (leaf)
• depth 1 = intermediate CA
• depth 2 = root (usually omitted; omission is normal)

If depth 1 is missing entirely, the overwhelming likelihood is that ssl_certificate points at cert.pem instead of fullchain.pem. That's branch (c).

Confusing the two Verify return code values will cost you 30 minutes.

2) Extract only the dates the server presented

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Expected healthy result:

notBefore=Aug 20 03:11:02 2026 GMT
notAfter=Nov 18 03:11:01 2026 GMT
subject=CN = example.com
issuer=C = US, O = Let's Encrypt, CN = R11

The key point is that these values are from the socket, not the file. If notAfter is in the past, that's (a) or (b). If notBefore is in the future, that's (e).

3) Cross-check with curl

curl -vI https://example.com 2>&1 | grep -Ei 'expire date|start date|SSL certificate|issuer'

If you get SSL certificate problem: unable to get local issuer certificate but the browser is fine, that is almost certainly (c). Browsers fetch the missing intermediate via AIA and self-correct; curl, Java, and older mobile stacks do not.

4) Pin down the certificate path actually being served

nginx -T 2>/dev/null | grep -nE 'server_name|ssl_certificate' | head -40

nginx -T dumps every included config fully expanded. A leftover vhost in sites-enabled still pointing at the wrong path is a frequently reported case.

5) Confirm a missed reload — file vs. socket fingerprint

These two lines are the most operationally useful technique in this post.

# 파일 쪽 지문
openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -fingerprint -sha256

# 소켓 쪽 지문
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

• The two values match → the server is serving what's on disk. If it's expired, renewal itself failed (a)
• The two values differ → the file is new, the process is old. (b) missed reload, confirmed. Nothing more to look at

Supporting evidence is the process start time.

ss -tlnp | grep :443
PID=$(pgrep -f 'nginx: master' | head -1)
ps -o pid,lstart,cmd -p "$PID"
sudo lsof -p "$PID" | grep -i pem
stat -c '%n %y' /etc/letsencrypt/live/example.com/fullchain.pem

If the ps -o lstart value is earlier than the certificate file's mtime, that process has never read the new certificate.

6) no shared cipher family — confirming (d)

If nginx error.log has the following line, you can forget about dates and chains.

SSL_do_handshake() failed (SSL: error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher)

Walk the supported protocols yourself and see where it breaks.

for p in tls1 tls1_1 tls1_2 tls1_3; do
  printf '%-8s ' "$p"
  echo | openssl s_client -connect example.com:443 -servername example.com -$p 2>&1 \
    | grep -qE 'Verify return code|Cipher is' && echo OK || echo FAIL
done

If only TLS 1.2/1.3 are OK and 1.0/1.1 FAIL, that is a correct security configuration. The failing clients are old stacks, so raise the client rather than lower the server. Conversely, if TLS 1.2 also FAILs, suspect a mismatch between the ssl_ciphers setting and the key type (RSA/ECDSA). Serving an RSA-key certificate while leaving only ECDSA-only cipher suites will blow up with no shared cipher as-is.

Recommended baseline:

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;

7) Clock skew — (e)

date -u
timedatectl status | grep -E 'System clock|NTP'

If System clock synchronized: no, fix container/VM clock drift first. A server whose clock is skewed into the future will judge a valid certificate as certificate has expired; skewed into the past, it will emit certificate is not yet valid.

SNI multi-domain: when only a specific domain fails

When multiple vhosts share one IP, results diverge depending on whether -servername is present.

With servername:

openssl s_client -connect 203.0.113.10:443 -servername shop.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject
# subject=CN = shop.example.com

Without it:

openssl s_client -connect 203.0.113.10:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -subject
# subject=CN = www.example.com     ← default_server의 인증서

What that difference means, in a table.

SAN check:

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

X509v3 Subject Alternative Name:
    DNS:example.com, DNS:www.example.com

If you connect as shop.example.com and it is not in that list, that domain was never a subject of this certificate to begin with.

If you've gotten this far, Verify return code: 0 (ok), and the SAN is correct, the server-side problem is done. If a specific client still fails, the cause moves to the client trust store. For Java/Go stacks see x509 certificate signed by unknown authority class of problems; for Python clients see Five-way diagnosis of SSLCertVerificationError CERTIFICATE_VERIFY_FAILED.

Recovery runbook

cert.pem vs fullchain.pem — most of cause (c)

Wrong config:

server {
    listen 443 ssl;
    server_name example.com;

    ssl_certificate     /etc/letsencrypt/live/example.com/cert.pem;      # ← 서버 인증서만
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
}

Corrected:

server {
    listen 443 ssl;
    http2 on;
    server_name example.com;

    ssl_certificate     /etc/letsencrypt/live/example.com/fullchain.pem; # ← 서버+중간 CA
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_session_cache shared:SSL:10m;
    ssl_stapling on;
    ssl_stapling_verify on;
}

cert.pem is the leaf only; fullchain.pem is leaf + intermediate CA. One line of difference and depth 1 disappears.

Key–certificate match and chain verification

# 키와 인증서가 같은 쌍인지 (두 해시가 같아야 정상)
openssl x509 -noout -modulus -in /etc/letsencrypt/live/example.com/cert.pem | openssl sha256
openssl rsa  -noout -modulus -in /etc/letsencrypt/live/example.com/privkey.pem | openssl sha256

# 체인 검증
openssl verify -untrusted /etc/letsencrypt/live/example.com/chain.pem \
  /etc/letsencrypt/live/example.com/cert.pem
# 기대 출력: cert.pem: OK

For an ECDSA key, compare public keys with openssl ec -noout -text | grep pub -A3 instead of openssl rsa.

Apply order

sudo nginx -t                      # syntax is ok / test is successful 확인 필수
sudo systemctl reload nginx
# 반영 확인 — 소켓 지문 재확인
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -fingerprint -sha256

If you reload while nginx -t is failing, the old config is kept and you reproduce the "I fixed it but nothing changed" situation. haproxy can keep existing connections briefly depending on how sockets are handed over on reload, so always confirm that the renewal took effect on a new connection.

Bind reload to renewal (the proper way)

# certbot
sudo certbot renew --deploy-hook "systemctl reload nginx"

# 또는 훅 파일로 고정
sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh >/dev/null <<'EOF'
#!/bin/sh
/usr/bin/nginx -t && /bin/systemctl reload nginx
EOF
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

acme.sh does the same job with --reloadcmd.

acme.sh --install-cert -d example.com \
  --key-file /etc/nginx/ssl/example.com.key \
  --fullchain-file /etc/nginx/ssl/example.com.crt \
  --reloadcmd "nginx -t && systemctl reload nginx"

As the industry discussion about stepwise shortening of certificate lifetimes (from 90 days toward even shorter cycles) continues, manual renewal is effectively ceasing to be an option. Paradoxically, the shorter the cycle, the higher the incidence of missed-reload outages. Renewal success and service rollout are separate events. For concrete timelines and policy, check CA/Browser Forum and each CA's official notices directly.

Rollback

ls -l /etc/letsencrypt/archive/example.com/
# 롤백 전 증적 보존 (필수)
sudo cp -a /etc/letsencrypt/live/example.com /root/incident-$(date +%Y%m%d%H%M)/
sudo journalctl -u nginx --since "1 hour ago" > /root/incident-nginx.log

# 이전 버전으로 심볼릭 링크 되돌리기 (예: 12 → 11)
cd /etc/letsencrypt/live/example.com
sudo ln -sf ../../archive/example.com/fullchain11.pem fullchain.pem
sudo ln -sf ../../archive/example.com/privkey11.pem   privkey.pem
sudo nginx -t && sudo systemctl reload nginx

Rollback only makes sense if the previous certificate is still within its validity period. Rolling back to an expired one makes the situation worse.

Preventing recurrence — monitor the socket, not the file

A script that watches file mtime will never catch (b) missed reload. The thing you monitor must be the socket.

#!/usr/bin/env bash
# /usr/local/bin/tls-expiry-check.sh
set -uo pipefail

DOMAINS=(
  "example.com:443"
  "shop.example.com:443"
  "api.example.com:443"
)
THRESHOLD_DAYS="${THRESHOLD_DAYS:-30}"
NOW_EPOCH=$(date +%s)
EXIT_CODE=0

for entry in "${DOMAINS[@]}"; do
  host="${entry%%:*}"
  port="${entry##*:}"

  end_date=$(echo | timeout 10 openssl s_client \
      -connect "${host}:${port}" -servername "${host}" 2>/dev/null \
    | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)

  if [ -z "${end_date}" ]; then
    echo "CRITICAL ${host} - 인증서 조회 실패 (접속/핸드셰이크 오류)"
    EXIT_CODE=2
    continue
  fi

  end_epoch=$(date -d "${end_date}" +%s 2>/dev/null) || {
    echo "CRITICAL ${host} - 날짜 파싱 실패: ${end_date}"; EXIT_CODE=2; continue; }

  days_left=$(( (end_epoch - NOW_EPOCH) / 86400 ))

  if   [ "${days_left}" -lt 0 ]; then
    echo "CRITICAL ${host} - 이미 만료됨 (${end_date})"; EXIT_CODE=2
  elif [ "${days_left}" -lt "${THRESHOLD_DAYS}" ]; then
    echo "WARNING  ${host} - ${days_left}일 남음 (${end_date})"
    [ "${EXIT_CODE}" -lt 1 ] && EXIT_CODE=1
  else
    echo "OK       ${host} - ${days_left}일 남음"
  fi
done

exit "${EXIT_CODE}"

sudo chmod +x /usr/local/bin/tls-expiry-check.sh
/usr/local/bin/tls-expiry-check.sh

Expected healthy output:

OK       example.com - 74일 남음
OK       shop.example.com - 74일 남음
WARNING  api.example.com - 12일 남음 (Sep 13 08:22:10 2026 GMT)

Exit codes are 0 (OK) / 1 (WARNING) / 2 (CRITICAL), so you can hook this straight into Nagios/Zabbix-class tools or an internal alert script.

cron registration:

# crontab -e
0 9 * * * /usr/local/bin/tls-expiry-check.sh >> /var/log/tls-expiry.log 2>&1 || \
  curl -s -X POST -H 'Content-Type: application/json' \
    -d "{\"text\":\"[TLS] 인증서 만료 경고 발생 - $(hostname)\"}" "$WEBHOOK_URL"

systemd service + timer (recommended):

# /etc/systemd/system/tls-expiry-check.service
[Unit]
Description=TLS certificate expiry check (socket-based)
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
Environment=THRESHOLD_DAYS=30
ExecStart=/usr/local/bin/tls-expiry-check.sh
StandardOutput=journal
StandardError=journal

# /etc/systemd/system/tls-expiry-check.timer
[Unit]
Description=Run TLS expiry check daily

[Timer]
OnCalendar=*-*-* 09:00:00
RandomizedDelaySec=600
Persistent=true
Unit=tls-expiry-check.service

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now tls-expiry-check.timer
systemctl list-timers tls-expiry-check.timer

Persistent=true runs the job once right after boot if the server was down and missed the scheduled time. Especially important on machines that are not always-on, like batch servers.

Remember just one sentence. You monitor the socket, not the file. Looking at the socket catches renewal failure, missed reload, and a missing chain in one shot.

FAQ

Q. certbot renew succeeded but the browser still shows expired. What do I look at first? A. Diff the file fingerprint against the socket fingerprint. If the value from openssl x509 -in fullchain.pem -noout -fingerprint -sha256 differs from the fingerprint you get via openssl s_client, a missed reload is confirmed. nginx -t && systemctl reload nginx clears it immediately; to prevent recurrence, bind reload to renewal with --deploy-hook.

Q. The browser is fine, but curl and the Java app get unable to get local issuer certificate. A. High likelihood of a missing intermediate CA chain (no depth 1). Browsers fetch the missing piece via AIA; other stacks do not. Check that nginx ssl_certificate is not pointing at cert.pem, switch it to fullchain.pem, and reload. If after that Verify return code: 0 but a specific client still fails, from that point it is a client trust-store problem.

Q. nginx error.log keeps logging no shared cipher. A. This is a negotiation failure, not a certificate date problem. Typical cases: the server turned off TLS 1.0/1.1 and an old client is connecting, or you left only ECDSA-only suites in ssl_ciphers while serving an RSA-key certificate. First confirm which version it breaks on with the -tls1_2 and -tls1_3 options, then align the cipher list with the key type.