## The cert was renewed — so why is it still showing as expired? There's a scene that plays out all too often at 2 a.m. The `certbot renew` log says `Congratulations, all renewals succeeded`, and `ls -l /etc/letsencrypt/live/example.com/` shows file timestamps from moments ago. Yet the browser still throws `NET::ERR_CERT_DATE_INVALID`. There's one premise you have to lock in before going any further. > **The certificate file on disk and the certificate the server process is holding in memory are completely different objects.** nginx and haproxy read the certificate file into memory at start/reload time. After that, the process has no idea if the file changes. So "I checked the file" is not a diagnosis. Diagnosis means checking **what the server actually serves on the socket**. The scope of this post is explicit. It covers only **expiry of the certificate the server presents, chain construction, and protocol negotiation failures**. Client-side trust store or CA bundle problems (missing internal root CA, JDK cacerts, Python certifi, etc.) live in a different cause layer, so this post only links out at those points. Scope of applicability: | Item | Scope | |---|---| | OS | Linux in general (RHEL/Rocky 8–9, Ubuntu 20.04–24.04) | | Server | nginx 1.18+, haproxy 2.4+ | | Tools | OpenSSL 1.1.1 / 3.x, curl 7.x+, certbot / acme.sh | | Out of scope | Client trust stores, internal CA distribution, mTLS client certificates | --- ## Error text → cause decision table (finish this in 30 seconds) Don't scroll. Find the error string you're looking at in the left column. | Error text | Likely cause | Corroborating symptoms (to block misdiagnosis) | |---|---|---| | `certificate has expired` / `ERR_CERT_DATE_INVALID` / `Verify return code: 10` | **(a)** notAfter has actually passed — renewal itself never happened | File fingerprint and socket fingerprint are **the same**. Failure traces in the certbot log | | Same error, but the file is current | **(b)** missed reload | File and socket fingerprints **differ**. `ps -o lstart` start time < certificate renewal time | | `unable to get local issuer certificate` / `Verify return code: 21` (certain clients only) | **(c)** missing intermediate CA chain | Browser is fine (AIA fetch fills the gap); curl, Java, mobile fail. No depth 1 in `Certificate chain` | | `sslv3 alert handshake failure` / `SSL_ERROR_SYSCALL` / `no shared cipher` in nginx error.log | **(d)** protocol / cipher suite mismatch | Certificate dates are fine. Fails only for specific clients / specific TLS versions | | `certificate is not yet valid` / `notBefore` is in the future | **(e)** clock skew or replacement higher in the chain | `date -u` disagrees with real time, or container clock drift | The confirming evidence for each branch comes out of the diagnostic command set below in one pass. Blindly re-running `certbot renew` on a guess does nothing for (b)–(e) and only burns your rate limit. --- ## 30-second diagnostic command set — which line of the output to look at ### 1) Confirm the actual chain from the socket ```bash openssl s_client -connect example.com:443 -servername example.com /dev/null | head -40 ``` **Healthy (complete chain) output example:** ```text Certificate chain 0 s:CN = example.com i:C = US, O = Let's Encrypt, CN = R11 1 s:C = US, O = Let's Encrypt, CN = R11 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 --- Verify return code: 0 (ok) ``` **Missing depth 1 output example:** ```text Certificate chain 0 s:CN = example.com i:C = US, O = Let's Encrypt, CN = R11 --- Verify return code: 21 (unable to verify the first certificate) ``` How to read it is simple. - **depth 0** = server certificate (leaf) - **depth 1** = intermediate CA - **depth 2** = root (usually omitted; omission is normal) If depth 1 is missing entirely, the overwhelming likelihood is that `ssl_certificate` points at `cert.pem` instead of `fullchain.pem`. That's branch **(c)**. Confusing the two `Verify return code` values will cost you 30 minutes. | Code | Meaning | Action | |---|---|---| | `10 (certificate has expired)` | Date problem — (a) or (b) | Check whether renewal happened + whether reload happened | | `21 (unable to verify the first certificate)` | Chain problem — (c) | Fix the fullchain path | | `0 (ok)` | Server side is healthy | Anything after this is the client trust-store layer | ### 2) Extract only the dates the server presented ```bash openssl s_client -connect example.com:443 -servername example.com /dev/null \ | openssl x509 -noout -dates -subject -issuer ``` Expected healthy result: ```text notBefore=Aug 20 03:11:02 2026 GMT notAfter=Nov 18 03:11:01 2026 GMT subject=CN = example.com issuer=C = US, O = Let's Encrypt, CN = R11 ``` **The key point is that these values are from the socket, not the file.** If notAfter is in the past, that's (a) or (b). If notBefore is in the future, that's (e). ### 3) Cross-check with curl ```bash curl -vI https://example.com 2>&1 | grep -Ei 'expire date|start date|SSL certificate|issuer' ``` If you get `SSL certificate problem: unable to get local issuer certificate` but the browser is fine, that is almost certainly (c). Chrome, Edge, Safari and others download the missing intermediate from the AIA (Authority Information Access) URL, and Firefox fills it in from a preloaded intermediate list. [Firefox intermediate preloading](https://wiki.mozilla.org/Security/CryptoEngineering/Intermediate_Preloading) curl, Java, and older mobile stacks do neither, so the server must send the chain. ### 4) Pin down the certificate path actually being served ```bash nginx -T 2>/dev/null | grep -nE 'server_name|ssl_certificate' | head -40 ``` `nginx -T` dumps every included config fully expanded. A leftover vhost in `sites-enabled` still pointing at the wrong path is a frequently reported case. ### 5) Confirm a missed reload — file vs. socket fingerprint These two lines are the most operationally useful technique in this post. ```bash # 파일 쪽 지문 openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -fingerprint -sha256 # 소켓 쪽 지문 openssl s_client -connect example.com:443 -servername example.com /dev/null \ | openssl x509 -noout -fingerprint -sha256 ``` - **The two values match** → the server is serving what's on disk. If it's expired, renewal itself failed **(a)** - **The two values differ** → the file is new, the process is old. **(b) missed reload, confirmed**. Nothing more to look at Supporting evidence is the process start time. ```bash ss -tlnp | grep :443 PID=$(pgrep -f 'nginx: master' | head -1) ps -o pid,lstart,cmd -p "$PID" stat -L -c '%n %y' /etc/letsencrypt/live/example.com/fullchain.pem # -L: time of the file the symlink points to ``` If the `ps -o lstart` value is **earlier** than the certificate file's mtime, the process did not read the new certificate at start-up. But `nginx -s reload` and `systemctl reload` re-read config and certificates without changing the master PID or start time, so use this only as supporting evidence and decide with the fingerprint comparison above. nginx does not keep certificate files open after reading them, so `lsof` cannot show this (the `lsof` step from the previous version was removed). ### 6) no shared cipher family — confirming (d) If nginx `error.log` has the following line, you can forget about dates and chains. ```text SSL_do_handshake() failed (SSL: error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher) # OpenSSL 1.0.x SSL_do_handshake() failed (SSL: error:0A0000C1:SSL routines::no shared cipher) # OpenSSL 3.x ``` The error code format depends on the server's OpenSSL version, so search for the string `no shared cipher`. Walk the supported protocols yourself and see where it breaks. ```bash for p in tls1_2 tls1_3; do printf '%-8s ' "$p" openssl s_client -brief -connect example.com:443 -servername example.com -$p &1 \ | grep -q 'CONNECTION ESTABLISHED' && echo OK || echo FAIL done # To test TLS 1.0/1.1 as well (an OpenSSL 3.x client only tries them at security level 0) for p in tls1 tls1_1; do printf '%-8s ' "$p" openssl s_client -brief -connect example.com:443 -servername example.com -$p \ -cipher 'DEFAULT:@SECLEVEL=0' &1 \ | grep -q 'CONNECTION ESTABLISHED' && echo OK || echo FAIL done ``` The previous loop also counted the `Cipher is (NONE)` line printed on failure, so it always reported OK. Decide on `CONNECTION ESTABLISHED` from `-brief` instead. Also, an OpenSSL 3.x client does not use anything below TLS 1.2 at the default security level (1), so a `-tls1` FAIL may not be the server's doing. [OpenSSL security levels](https://docs.openssl.org/3.0/man3/SSL_CTX_set_security_level/) If only TLS 1.2/1.3 are OK and 1.0/1.1 FAIL, that is a **correct security configuration**. The failing clients are old stacks, so raise the client rather than lower the server. Conversely, if TLS 1.2 also FAILs, suspect a mismatch between the `ssl_ciphers` setting and the key type (RSA/ECDSA). Serving an RSA-key certificate while leaving only ECDSA-only cipher suites will blow up with `no shared cipher` as-is. Recommended baseline: ```nginx ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers off; ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384; ``` ### 7) Clock skew — (e) ```bash date -u timedatectl status | grep -E 'System clock|NTP' ``` If `System clock synchronized: no`, fix container/VM clock drift first. A server whose clock is skewed into the future will judge a valid certificate as `certificate has expired`; skewed into the past, it will emit `certificate is not yet valid`. --- ## SNI multi-domain: when only a specific domain fails When multiple vhosts share one IP, results diverge depending on whether `-servername` is present. **With servername:** ```bash openssl s_client -connect 203.0.113.10:443 -servername shop.example.com /dev/null \ | openssl x509 -noout -subject # subject=CN = shop.example.com ``` **Without it:** ```bash openssl s_client -connect 203.0.113.10:443 /dev/null \ | openssl x509 -noout -subject # subject=CN = www.example.com ← default_server의 인증서 ``` What that difference means, in a table. | Observation | Interpretation | Action | |---|---|---| | Healthy only with servername | Normal behavior. Only non-SNI clients fail | Old Android 4.x, Java 6, etc. → upgrade the client or split onto a dedicated IP | | A different CN even with servername | Typo in `server_name` or vhost mismatch → falls through to default_server | Check server_name with `nginx -T`, including whether a wildcard is present | | CN is correct but the browser reports a name mismatch | Host is not in the SAN | Check SAN below, then reissue | | Fails only behind L4/CDN | Front end is not forwarding SNI to origin | Configure SNI passthrough on the proxy (`proxy_ssl_server_name on`, etc.) | SAN check: ```bash openssl s_client -connect example.com:443 -servername example.com /dev/null \ | openssl x509 -noout -ext subjectAltName ``` ```text X509v3 Subject Alternative Name: DNS:example.com, DNS:www.example.com ``` If you connect as `shop.example.com` and it is not in that list, that domain was never a subject of this certificate to begin with. If you've gotten this far, `Verify return code: 0 (ok)`, and the SAN is correct, the server-side problem is done. If a specific client still fails, the cause moves to the client trust store. For Java/Go stacks see [x509 certificate signed by unknown authority class of problems](/blog/pkix-path-building-failed-suncertpathbuilderexception-30분-해결-런북); for Python clients see [Five-way diagnosis of SSLCertVerificationError CERTIFICATE_VERIFY_FAILED](/blog/sslcertverificationerror-해결-certificateverifyfailed-원인-5종-판별법). --- ## Recovery runbook ### cert.pem vs fullchain.pem — most of cause (c) **Wrong config:** ```nginx server { listen 443 ssl; server_name example.com; ssl_certificate /etc/letsencrypt/live/example.com/cert.pem; # ← 서버 인증서만 ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; } ``` **Corrected:** ```nginx server { listen 443 ssl; http2 on; server_name example.com; ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem; # ← 서버+중간 CA ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem; ssl_protocols TLSv1.2 TLSv1.3; ssl_session_cache shared:SSL:10m; } ``` `cert.pem` is the leaf only; `fullchain.pem` is leaf + intermediate CA. One line of difference and depth 1 disappears. **OCSP stapling depends on the CA.** Let's Encrypt stopped including OCSP URLs in certificates on 2025-05-07 and turned off its OCSP responders on 2025-08-06. With a Let's Encrypt certificate issued since then, `ssl_stapling on;` makes nginx log a warning that there is no OCSP responder URL and skip stapling. That is why `ssl_stapling on; ssl_stapling_verify on;` was removed from the corrected config. Enable it only for certificates from a CA that still provides OCSP, together with `ssl_trusted_certificate` and `resolver`. [Let's Encrypt ending OCSP](https://letsencrypt.org/2024/12/05/ending-ocsp/) · [nginx ssl_stapling](https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_stapling) ### Key–certificate match and chain verification ```bash # Do the key and certificate match? Compare public-key hashes (works for RSA and ECDSA; both must match) openssl x509 -in /etc/letsencrypt/live/example.com/cert.pem -noout -pubkey | openssl sha256 sudo openssl pkey -in /etc/letsencrypt/live/example.com/privkey.pem -pubout | openssl sha256 # Chain verification openssl verify -untrusted /etc/letsencrypt/live/example.com/chain.pem \ /etc/letsencrypt/live/example.com/cert.pem # Expected output: cert.pem: OK ``` The previous version used `openssl rsa -modulus`, but since certbot 2.0 the default key for new certificates is ECDSA (P-256), and `openssl rsa` fails on it. The `-pubkey` / `pkey -pubout` comparison above works for either key type. [certbot changelog](https://github.com/certbot/certbot/blob/master/certbot/CHANGELOG.md) ### Apply order ```bash sudo nginx -t # syntax is ok / test is successful 확인 필수 sudo systemctl reload nginx # 반영 확인 — 소켓 지문 재확인 openssl s_client -connect example.com:443 -servername example.com /dev/null \ | openssl x509 -noout -dates -fingerprint -sha256 ``` If you reload while `nginx -t` is failing, the old config is kept and you reproduce the "I fixed it but nothing changed" situation. haproxy can keep existing connections briefly depending on how sockets are handed over on reload, so always confirm that the renewal took effect on a **new** connection. ### Bind reload to renewal (the proper way) ```bash # Hook for a single run (treat it as applying to this renew run; use the hook file below for a permanent hook) sudo certbot renew --deploy-hook "systemctl reload nginx" # Permanent hook file: certbot runs it after every successful renewal sudo tee /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh >/dev/null <<'EOF' #!/bin/sh /usr/bin/nginx -t && /bin/systemctl reload nginx EOF sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh ``` The hook file runs only when a certificate was actually renewed. After adding it, run `sudo certbot renew --dry-run` to confirm the hook runs, then check the socket fingerprint again. [certbot user guide: renewal hooks](https://eff-certbot.readthedocs.io/en/stable/using.html) acme.sh does the same job with `--reloadcmd`. ```bash acme.sh --install-cert -d example.com \ --key-file /etc/nginx/ssl/example.com.key \ --fullchain-file /etc/nginx/ssl/example.com.crt \ --reloadcmd "nginx -t && systemctl reload nginx" ``` Under CA/Browser Forum Ballot SC-081, the maximum validity of public TLS certificates drops, by issuance date, to 200 days from 2026-03-15, 100 days from 2027-03-15, and 47 days from 2029-03-15. [CA/B Forum Ballot SC-081v3](https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/) The more often you renew, the **more chances a missed reload has to cause an outage**. Renewal success and service rollout are separate events. Check your CA's notices for the lifetime it actually issues. ### Rollback ```bash ls -l /etc/letsencrypt/archive/example.com/ # 롤백 전 증적 보존 (필수) sudo cp -a /etc/letsencrypt/live/example.com /root/incident-$(date +%Y%m%d%H%M)/ sudo journalctl -u nginx --since "1 hour ago" > /root/incident-nginx.log # Point the symlinks back to the previous version (e.g. 12 → 11): switch all four files to the same number N=11 cd /etc/letsencrypt/live/example.com for f in cert chain fullchain privkey; do sudo ln -sfn "../../archive/example.com/${f}${N}.pem" "${f}.pem" done sudo openssl x509 -in fullchain.pem -noout -enddate # confirm it is still valid first sudo nginx -t && sudo systemctl reload nginx ``` Rollback only makes sense if the previous certificate is still within its validity period. Rolling back to an expired one makes the situation worse. Reverting only some files leaves a mismatched key and certificate, and nginx will not start. The next `certbot renew` recreates the links with a new number, so a rollback is a stopgap until the cause is fixed. --- ## Preventing recurrence — monitor the socket, not the file A script that watches file mtime will never catch (b) missed reload. The thing you monitor must be the **socket**. ```bash #!/usr/bin/env bash # /usr/local/bin/tls-expiry-check.sh set -uo pipefail DOMAINS=( "example.com:443" "shop.example.com:443" "api.example.com:443" ) THRESHOLD_DAYS="${THRESHOLD_DAYS:-30}" NOW_EPOCH=$(date +%s) EXIT_CODE=0 for entry in "${DOMAINS[@]}"; do host="${entry%%:*}" port="${entry##*:}" end_date=$(echo | timeout 10 openssl s_client \ -connect "${host}:${port}" -servername "${host}" 2>/dev/null \ | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2) if [ -z "${end_date}" ]; then echo "CRITICAL ${host} - 인증서 조회 실패 (접속/핸드셰이크 오류)" EXIT_CODE=2 continue fi end_epoch=$(date -d "${end_date}" +%s 2>/dev/null) || { echo "CRITICAL ${host} - 날짜 파싱 실패: ${end_date}"; EXIT_CODE=2; continue; } days_left=$(( (end_epoch - NOW_EPOCH) / 86400 )) if [ "${days_left}" -lt 0 ]; then echo "CRITICAL ${host} - 이미 만료됨 (${end_date})"; EXIT_CODE=2 elif [ "${days_left}" -lt "${THRESHOLD_DAYS}" ]; then echo "WARNING ${host} - ${days_left}일 남음 (${end_date})" [ "${EXIT_CODE}" -lt 1 ] && EXIT_CODE=1 else echo "OK ${host} - ${days_left}일 남음" fi done exit "${EXIT_CODE}" ``` ```bash sudo chmod +x /usr/local/bin/tls-expiry-check.sh /usr/local/bin/tls-expiry-check.sh ``` Expected healthy output: ```text OK example.com - 74일 남음 OK shop.example.com - 74일 남음 WARNING api.example.com - 12일 남음 (Sep 13 08:22:10 2026 GMT) ``` Exit codes are 0 (OK) / 1 (WARNING) / 2 (CRITICAL), so you can hook this straight into Nagios/Zabbix-class tools or an internal alert script. **cron registration:** ```bash # crontab -e # cron does not support shell line continuation (\): keep each command on one line. # cron has none of your login shell's variables, so define the webhook URL in the crontab itself. WEBHOOK_URL=https://hooks.example.com/REPLACE_ME 0 9 * * * /usr/local/bin/tls-expiry-check.sh >> /var/log/tls-expiry.log 2>&1 || curl -s -X POST -H 'Content-Type: application/json' -d "{\"text\":\"[TLS] certificate expiry warning - $(hostname)\"}" "$WEBHOOK_URL" ``` **systemd service + timer (recommended):** ```ini # /etc/systemd/system/tls-expiry-check.service [Unit] Description=TLS certificate expiry check (socket-based) After=network-online.target Wants=network-online.target [Service] Type=oneshot Environment=THRESHOLD_DAYS=30 ExecStart=/usr/local/bin/tls-expiry-check.sh StandardOutput=journal StandardError=journal ``` ```ini # /etc/systemd/system/tls-expiry-check.timer [Unit] Description=Run TLS expiry check daily [Timer] OnCalendar=*-*-* 09:00:00 RandomizedDelaySec=600 Persistent=true Unit=tls-expiry-check.service [Install] WantedBy=timers.target ``` ```bash sudo systemctl daemon-reload sudo systemctl enable --now tls-expiry-check.timer systemctl list-timers tls-expiry-check.timer ``` `Persistent=true` runs the job once right after boot if the server was down and missed the scheduled time. Especially important on machines that are not always-on, like batch servers. Remember just one sentence. **You monitor the socket, not the file.** Looking at the socket catches renewal failure, missed reload, and a missing chain in one shot. --- ## FAQ **Q. `certbot renew` succeeded but the browser still shows expired. What do I look at first?** A. Diff the file fingerprint against the socket fingerprint. If the value from `openssl x509 -in fullchain.pem -noout -fingerprint -sha256` differs from the fingerprint you get via `openssl s_client`, a missed reload is confirmed. `nginx -t && systemctl reload nginx` clears it immediately; to prevent recurrence, bind reload to renewal with `--deploy-hook`. **Q. The browser is fine, but curl and the Java app get `unable to get local issuer certificate`.** A. High likelihood of a missing intermediate CA chain (no depth 1). Browsers fill in the missing piece via AIA download or a preloaded intermediate list; other stacks do not. Check that nginx `ssl_certificate` is not pointing at `cert.pem`, switch it to `fullchain.pem`, and reload. If after that `Verify return code: 0` but a specific client still fails, from that point it is a client trust-store problem. **Q. nginx error.log keeps logging `no shared cipher`.** A. This is a negotiation failure, not a certificate date problem. Typical cases: the server turned off TLS 1.0/1.1 and an old client is connecting, or you left only ECDSA-only suites in `ssl_ciphers` while serving an RSA-key certificate. First confirm which version it breaks on with the `-tls1_2` and `-tls1_3` options, then align the cipher list with the key type.

#Certificate expiry troubleshooting

certificate has expired: a five-branch diagnosis (nginx · openssl)