certificate has expired: a five-branch diagnosis (nginx · openssl)

The cert was renewed — so why is it still showing as expired?

There's a scene that plays out all too often at 2 a.m. The certbot renew log says Congratulations, all renewals succeeded, and ls -l /etc/letsencrypt/live/example.com/ shows file timestamps from moments ago. Yet the browser still throws NET::ERR_CERT_DATE_INVALID.

There's one premise you have to lock in before going any further.

The certificate file on disk and the certificate the server process is holding in memory are completely different objects.

nginx and haproxy read the certificate file into memory at start/reload time. After that, the process has no idea if the file changes. So "I checked the file" is not a diagnosis. Diagnosis means checking what the server actually serves on the socket.

The scope of this post is explicit. It covers only expiry of the certificate the server presents, chain construction, and protocol negotiation failures. Client-side trust store or CA bundle problems (missing internal root CA, JDK cacerts, Python certifi, etc.) live in a different cause layer, so this post only links out at those points.

Scope of applicability:

Error text → cause decision table (finish this in 30 seconds)

Don't scroll. Find the error string you're looking at in the left column.

The confirming evidence for each branch comes out of the diagnostic command set below in one pass. Blindly re-running certbot renew on a guess does nothing for (b)–(e) and only burns your rate limit.

30-second diagnostic command set — which line of the output to look at

1) Confirm the actual chain from the socket

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null | head -40

Healthy (complete chain) output example:

Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R11
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Verify return code: 0 (ok)

Missing depth 1 output example:

Certificate chain
 0 s:CN = example.com
   i:C = US, O = Let's Encrypt, CN = R11
---
Verify return code: 21 (unable to verify the first certificate)

How to read it is simple.

• depth 0 = server certificate (leaf)
• depth 1 = intermediate CA
• depth 2 = root (usually omitted; omission is normal)

If depth 1 is missing entirely, the overwhelming likelihood is that ssl_certificate points at cert.pem instead of fullchain.pem. That's branch (c).

Confusing the two Verify return code values will cost you 30 minutes.

2) Extract only the dates the server presented

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer

Expected healthy result:

notBefore=Aug 20 03:11:02 2026 GMT
notAfter=Nov 18 03:11:01 2026 GMT
subject=CN = example.com
issuer=C = US, O = Let's Encrypt, CN = R11

The key point is that these values are from the socket, not the file. If notAfter is in the past, that's (a) or (b). If notBefore is in the future, that's (e).

3) Cross-check with curl

curl -vI https://example.com 2>&1 | grep -Ei 'expire date|start date|SSL certificate|issuer'

If you get SSL certificate problem: unable to get local issuer certificate but the browser is fine, that is almost certainly (c). Chrome, Edge, Safari and others download the missing intermediate from the AIA (Authority Information Access) URL, and Firefox fills it in from a preloaded intermediate list. Firefox intermediate preloading curl, Java, and older mobile stacks do neither, so the server must send the chain.

4) Pin down the certificate path actually being served

nginx -T 2>/dev/null | grep -nE 'server_name|ssl_certificate' | head -40

nginx -T dumps every included config fully expanded. A leftover vhost in sites-enabled still pointing at the wrong path is a frequently reported case.

5) Confirm a missed reload — file vs. socket fingerprint

These two lines are the most operationally useful technique in this post.

# fingerprint on the file
openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -fingerprint -sha256

# fingerprint on the socket
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -fingerprint -sha256

• The two values match → the server is serving what's on disk. If it's expired, renewal itself failed (a)
• The two values differ → the file is new, the process is old. (b) missed reload, confirmed. Nothing more to look at

Supporting evidence is the process start time.

ss -tlnp | grep :443
PID=$(pgrep -f 'nginx: master' | head -1)
ps -o pid,lstart,cmd -p "$PID"
stat -L -c '%n %y' /etc/letsencrypt/live/example.com/fullchain.pem   # -L: time of the file the symlink points to

If the ps -o lstart value is earlier than the certificate file's mtime, the process did not read the new certificate at start-up. But nginx -s reload and systemctl reload re-read config and certificates without changing the master PID or start time, so use this only as supporting evidence and decide with the fingerprint comparison above. nginx does not keep certificate files open after reading them, so lsof cannot show this (the lsof step from the previous version was removed).

6) no shared cipher family — confirming (d)

If nginx error.log has the following line, you can forget about dates and chains.

SSL_do_handshake() failed (SSL: error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher)   # OpenSSL 1.0.x
SSL_do_handshake() failed (SSL: error:0A0000C1:SSL routines::no shared cipher)                         # OpenSSL 3.x

The error code format depends on the server's OpenSSL version, so search for the string no shared cipher.

Walk the supported protocols yourself and see where it breaks.

for p in tls1_2 tls1_3; do
  printf '%-8s ' "$p"
  openssl s_client -brief -connect example.com:443 -servername example.com -$p </dev/null 2>&1 \
    | grep -q 'CONNECTION ESTABLISHED' && echo OK || echo FAIL
done
# To test TLS 1.0/1.1 as well (an OpenSSL 3.x client only tries them at security level 0)
for p in tls1 tls1_1; do
  printf '%-8s ' "$p"
  openssl s_client -brief -connect example.com:443 -servername example.com -$p \
    -cipher 'DEFAULT:@SECLEVEL=0' </dev/null 2>&1 \
    | grep -q 'CONNECTION ESTABLISHED' && echo OK || echo FAIL
done

The previous loop also counted the Cipher is (NONE) line printed on failure, so it always reported OK. Decide on CONNECTION ESTABLISHED from -brief instead. Also, an OpenSSL 3.x client does not use anything below TLS 1.2 at the default security level (1), so a -tls1 FAIL may not be the server's doing. OpenSSL security levels

If only TLS 1.2/1.3 are OK and 1.0/1.1 FAIL, that is a correct security configuration. The failing clients are old stacks, so raise the client rather than lower the server. Conversely, if TLS 1.2 also FAILs, suspect a mismatch between the ssl_ciphers setting and the key type (RSA/ECDSA). Serving an RSA-key certificate while leaving only ECDSA-only cipher suites will blow up with no shared cipher as-is.

Recommended baseline:

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;

7) Clock skew — (e)

date -u
timedatectl status | grep -E 'System clock|NTP'

If System clock synchronized: no, fix container/VM clock drift first. A server whose clock is skewed into the future will judge a valid certificate as certificate has expired; skewed into the past, it will emit certificate is not yet valid.

SNI multi-domain: when only a specific domain fails

When multiple vhosts share one IP, results diverge depending on whether -servername is present.

With servername:

openssl s_client -connect 203.0.113.10:443 -servername shop.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject
# subject=CN = shop.example.com

Without it:

openssl s_client -connect 203.0.113.10:443 </dev/null 2>/dev/null \
  | openssl x509 -noout -subject
# subject=CN = www.example.com ← default_server certificate

What that difference means, in a table.

SAN check:

openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -ext subjectAltName

X509v3 Subject Alternative Name:
    DNS:example.com, DNS:www.example.com

If you connect as shop.example.com and it is not in that list, that domain was never a subject of this certificate to begin with.

If you've gotten this far, Verify return code: 0 (ok), and the SAN is correct, the server-side problem is done. If a specific client still fails, the cause moves to the client trust store. For Java/Go stacks see x509 certificate signed by unknown authority class of problems; for Python clients see Five-way diagnosis of SSLCertVerificationError CERTIFICATE_VERIFY_FAILED.

Recovery runbook

cert.pem vs fullchain.pem — most of cause (c)

Wrong config:

server {
    listen 443 ssl;
    server_name example.com;

 ssl_certificate /etc/letsencrypt/live/example.com/cert.pem; # ← certificate
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
}

Corrected:

server {
    listen 443 ssl;
    http2 on;
    server_name example.com;

 ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # ← + CA
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_session_cache shared:SSL:10m;
}

cert.pem is the leaf only; fullchain.pem is leaf + intermediate CA. One line of difference and depth 1 disappears.

OCSP stapling depends on the CA. Let's Encrypt stopped including OCSP URLs in certificates on 2025-05-07 and turned off its OCSP responders on 2025-08-06. With a Let's Encrypt certificate issued since then, ssl_stapling on; makes nginx log a warning that there is no OCSP responder URL and skip stapling. That is why ssl_stapling on; ssl_stapling_verify on; was removed from the corrected config. Enable it only for certificates from a CA that still provides OCSP, together with ssl_trusted_certificate and resolver. Let's Encrypt ending OCSP · nginx ssl_stapling

Key–certificate match and chain verification

# Do the key and certificate match? Compare public-key hashes (works for RSA and ECDSA; both must match)
openssl x509 -in /etc/letsencrypt/live/example.com/cert.pem -noout -pubkey | openssl sha256
sudo openssl pkey -in /etc/letsencrypt/live/example.com/privkey.pem -pubout | openssl sha256

# Chain verification
openssl verify -untrusted /etc/letsencrypt/live/example.com/chain.pem \
  /etc/letsencrypt/live/example.com/cert.pem
# Expected output: cert.pem: OK

The previous version used openssl rsa -modulus, but since certbot 2.0 the default key for new certificates is ECDSA (P-256), and openssl rsa fails on it. The -pubkey / pkey -pubout comparison above works for either key type. certbot changelog

Apply order

sudo nginx -t # syntax is ok / test is successful must confirm
sudo systemctl reload nginx
# confirm it applied — recheck the socket fingerprint
openssl s_client -connect example.com:443 -servername example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -fingerprint -sha256

If you reload while nginx -t is failing, the old config is kept and you reproduce the "I fixed it but nothing changed" situation. haproxy can keep existing connections briefly depending on how sockets are handed over on reload, so always confirm that the renewal took effect on a new connection.

Bind reload to renewal (the proper way)

# Hook for a single run (treat it as applying to this renew run; use the hook file below for a permanent hook)
sudo certbot renew --deploy-hook "systemctl reload nginx"

# Permanent hook file: certbot runs it after every successful renewal
sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh >/dev/null <<'EOF'
#!/bin/sh
/usr/bin/nginx -t && /bin/systemctl reload nginx
EOF
sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

The hook file runs only when a certificate was actually renewed. After adding it, run sudo certbot renew --dry-run to confirm the hook runs, then check the socket fingerprint again. certbot user guide: renewal hooks

acme.sh does the same job with --reloadcmd.

acme.sh --install-cert -d example.com \
  --key-file /etc/nginx/ssl/example.com.key \
  --fullchain-file /etc/nginx/ssl/example.com.crt \
  --reloadcmd "nginx -t && systemctl reload nginx"

Under CA/Browser Forum Ballot SC-081, the maximum validity of public TLS certificates drops, by issuance date, to 200 days from 2026-03-15, 100 days from 2027-03-15, and 47 days from 2029-03-15. CA/B Forum Ballot SC-081v3 The more often you renew, the more chances a missed reload has to cause an outage. Renewal success and service rollout are separate events. Check your CA's notices for the lifetime it actually issues.

Rollback

ls -l /etc/letsencrypt/archive/example.com/
# keep evidence before rollback (required)
sudo cp -a /etc/letsencrypt/live/example.com /root/incident-$(date +%Y%m%d%H%M)/
sudo journalctl -u nginx --since "1 hour ago" > /root/incident-nginx.log

# Point the symlinks back to the previous version (e.g. 12 → 11): switch all four files to the same number
N=11
cd /etc/letsencrypt/live/example.com
for f in cert chain fullchain privkey; do
  sudo ln -sfn "../../archive/example.com/${f}${N}.pem" "${f}.pem"
done
sudo openssl x509 -in fullchain.pem -noout -enddate      # confirm it is still valid first
sudo nginx -t && sudo systemctl reload nginx

Rollback only makes sense if the previous certificate is still within its validity period. Rolling back to an expired one makes the situation worse. Reverting only some files leaves a mismatched key and certificate, and nginx will not start. The next certbot renew recreates the links with a new number, so a rollback is a stopgap until the cause is fixed.

Preventing recurrence — monitor the socket, not the file

A script that watches file mtime will never catch (b) missed reload. The thing you monitor must be the socket.

#!/usr/bin/env bash
# /usr/local/bin/tls-expiry-check.sh
set -uo pipefail

DOMAINS=(
  "example.com:443"
  "shop.example.com:443"
  "api.example.com:443"
)
THRESHOLD_DAYS="${THRESHOLD_DAYS:-30}"
NOW_EPOCH=$(date +%s)
EXIT_CODE=0

for entry in "${DOMAINS[@]}"; do
  host="${entry%%:*}"
  port="${entry##*:}"

  end_date=$(echo | timeout 10 openssl s_client \
      -connect "${host}:${port}" -servername "${host}" 2>/dev/null \
    | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)

  if [ -z "${end_date}" ]; then
 echo "CRITICAL ${host} - certificate lookup failed (connect/handshake error)"
    EXIT_CODE=2
    continue
  fi

  end_epoch=$(date -d "${end_date}" +%s 2>/dev/null) || {
 echo "CRITICAL ${host} - date parse failed: ${end_date}"; EXIT_CODE=2; continue; }

  days_left=$(( (end_epoch - NOW_EPOCH) / 86400 ))

  if   [ "${days_left}" -lt 0 ]; then
 echo "CRITICAL ${host} - already expired (${end_date})"; EXIT_CODE=2
  elif [ "${days_left}" -lt "${THRESHOLD_DAYS}" ]; then
 echo "WARNING ${host} - ${days_left}days left (${end_date})"
    [ "${EXIT_CODE}" -lt 1 ] && EXIT_CODE=1
  else
 echo "OK ${host} - ${days_left}days left"
  fi
done

exit "${EXIT_CODE}"

sudo chmod +x /usr/local/bin/tls-expiry-check.sh
/usr/local/bin/tls-expiry-check.sh

Expected healthy output:

OK example.com - 74days left
OK shop.example.com - 74days left
WARNING api.example.com - 12days left (Sep 13 08:22:10 2026 GMT)

Exit codes are 0 (OK) / 1 (WARNING) / 2 (CRITICAL), so you can hook this straight into Nagios/Zabbix-class tools or an internal alert script.

cron registration:

# crontab -e
# cron does not support shell line continuation (\): keep each command on one line.
# cron has none of your login shell's variables, so define the webhook URL in the crontab itself.
WEBHOOK_URL=https://hooks.example.com/REPLACE_ME
0 9 * * * /usr/local/bin/tls-expiry-check.sh >> /var/log/tls-expiry.log 2>&1 || curl -s -X POST -H 'Content-Type: application/json' -d "{\"text\":\"[TLS] certificate expiry warning - $(hostname)\"}" "$WEBHOOK_URL"

systemd service + timer (recommended):

# /etc/systemd/system/tls-expiry-check.service
[Unit]
Description=TLS certificate expiry check (socket-based)
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
Environment=THRESHOLD_DAYS=30
ExecStart=/usr/local/bin/tls-expiry-check.sh
StandardOutput=journal
StandardError=journal

# /etc/systemd/system/tls-expiry-check.timer
[Unit]
Description=Run TLS expiry check daily

[Timer]
OnCalendar=*-*-* 09:00:00
RandomizedDelaySec=600
Persistent=true
Unit=tls-expiry-check.service

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now tls-expiry-check.timer
systemctl list-timers tls-expiry-check.timer

Persistent=true runs the job once right after boot if the server was down and missed the scheduled time. Especially important on machines that are not always-on, like batch servers.

Remember just one sentence. You monitor the socket, not the file. Looking at the socket catches renewal failure, missed reload, and a missing chain in one shot.

FAQ

Q. certbot renew succeeded but the browser still shows expired. What do I look at first? A. Diff the file fingerprint against the socket fingerprint. If the value from openssl x509 -in fullchain.pem -noout -fingerprint -sha256 differs from the fingerprint you get via openssl s_client, a missed reload is confirmed. nginx -t && systemctl reload nginx clears it immediately; to prevent recurrence, bind reload to renewal with --deploy-hook.

Q. The browser is fine, but curl and the Java app get unable to get local issuer certificate. A. High likelihood of a missing intermediate CA chain (no depth 1). Browsers fill in the missing piece via AIA download or a preloaded intermediate list; other stacks do not. Check that nginx ssl_certificate is not pointing at cert.pem, switch it to fullchain.pem, and reload. If after that Verify return code: 0 but a specific client still fails, from that point it is a client trust-store problem.

Q. nginx error.log keeps logging no shared cipher. A. This is a negotiation failure, not a certificate date problem. Typical cases: the server turned off TLS 1.0/1.1 and an old client is connecting, or you left only ECDSA-only suites in ssl_ciphers while serving an RSA-key certificate. First confirm which version it breaks on with the -tls1_2 and -tls1_3 options, then align the cipher list with the key type.