Why certificates still lapse, twelve years after Let's Encrypt

An expired TLS certificate is the most embarrassing outage available. It is total — every browser refuses, loudly, with an interstitial that says your site is dangerous. It is entirely predictable, months in advance, to the second. And it is still happening, constantly, at organisations that automated renewal years ago.

The interesting question is not "why don't people automate renewal". They did. The interesting question is why that turned out not to be the same thing as "certificates don't expire".

The renewal is not the delivery

certbot running on a cron is a program that obtains a certificate and writes it to disk. That is the whole of its job, and it is very good at it.

What takes a site down is everything downstream of that write:

• The renewal ran, wrote a new certificate, and nothing reloaded. nginx is still holding the old one in memory. The file on disk is perfect. The socket serves an expired certificate for as long as that process lives.
• The renewal ran on one host of three. The load balancer serves the fresh one two times out of three, so the failure is intermittent, which is worse than total: monitoring that checks once and gets lucky reports green.
• The certificate was renewed where it is terminated, which is not where anyone was looking. A CDN, an ingress controller and an origin can each hold their own copy, and only one of them faces the browser.
• The renewal wrote a valid leaf and the chain got shorter. The site works in every browser the person testing it owns, and fails in the older client, the Java service, or the payment processor's callback that does not carry the intermediate in its trust store.
• The domain's DNS-01 challenge broke months ago, renewal has been failing silently since, and the email about it goes to an alias nobody reads.

Every one of these is a certificate that is, on the machine that generated it, completely fine. Checking your renewal automation tells you nothing about any of them.

Watch the socket, not the pipeline

The single most useful correction is to move where you observe from.

A certificate's expiry is a fact about the bytes a client receives on a TLS handshake to a particular hostname and port, from outside your network, with no privileged access and no knowledge of how it got there. That measurement is indifferent to which of the failures above happened, because all of them produce the same observable: the thing on the wire is not the thing you think you renewed.

This is why HarpyWatch runs ssl-cert-expiry as a check against the address a visitor uses, rather than reading a file. The check connects, completes a handshake, and reads the certificate that was actually presented. If nginx was never reloaded, the file on disk is irrelevant — the socket tells the truth.

Expiry is the symptom, not the check

Once you are looking at the handshake, expiry is only the most obvious of several things that will take a site down, and it is not the one that will surprise you.

cert-chain-trust verifies the chain the server actually sends, in order, up to a trusted root. A missing intermediate is the classic version of this: it works in Firefox, which caches intermediates it has seen, and fails in a freshly installed client, which does not. The bug reproduces only on machines nobody develops on.

tls-handshake looks at the negotiation itself — protocol versions and cipher suites. A server quietly still offering TLS 1.0 is a compliance finding waiting to happen; a server that has just dropped 1.2 in the name of hardening is an outage for whatever embedded client has not been updated since.

domain-expiry watches the registration underneath all of it, which is worth saying out loud because it fails in exactly the same way for exactly the same reason. It is a renewal, on a long clock, whose reminders go to a mailbox that belonged to somebody who has left. When it lapses, the certificate is irrelevant — there is no name left to present it for.

Two numbers, both of which matter

For expiry specifically, there are two useful thresholds and most setups only have one.

The first is the alert you can act on: far enough out that a human can schedule work. Thirty days is conventional and reasonable, and if your renewal is automated it should essentially never fire.

The second is the one people skip: the alert that says automation has already failed. If renewal is meant to happen at sixty days and you are inside fourteen, the thirty-day warning has not just fired, it has fired and been ignored or missed, and something is wrong with the process rather than the certificate. That should be a different, louder signal, because it means the mechanism you were relying on is not working and nobody noticed.

The gap between those two numbers is the actual subject. A certificate does not expire because nobody knew the date. It expires because the date was known, the renewal was automated, the automation reported success, and no one ever checked what a browser gets when it connects.