GitHub Let a Free Certificate Expire and Killed Every Self-Hosted Runner on Earth
- Patrick Duggan
- 17 hours ago
- 5 min read
At 23:05:54 UTC tonight, a Let's Encrypt wildcard certificate reached the end of its 90-day life and every self-hosted GitHub Actions runner on the planet stopped taking jobs at the same second.
Matt Lucas at RedEye Security root-caused it publicly eleven minutes before GitHub's status page said a word. His write-up is here and it is correct end to end: GitHub Actions Cert Expiry and the Self-Hosted Runner Outage. We are not claiming this one. We confirmed it independently, from a different network, and we are writing because there is a second story sitting underneath his that nobody has said out loud yet.
What we verified, independently
We read the certificate off the wire ourselves rather than take anyone's word for it, including Matt's.
The wildcard on *.actions.githubusercontent.com is issued by Let's Encrypt R12, valid from April 20 2026 at 23:05:55 UTC, expiring July 19 2026 at 23:05:54 UTC. The same certificate fronts both pipelines.actions.githubusercontent.com, which every runner long-polls for jobs, and vstoken.actions.githubusercontent.com, which hands out tokens. Both were still serving the expired certificate at 00:21 UTC on July 20 — seventy-five minutes past death, no rotation.
Meanwhile api.github.com sits on a Sectigo certificate valid until September 29 and never blinked. That split is why the outage feels local and partial when you hit it: the API works, the web UI works, and only the thing your runners talk to is dead.
GitHub opened "Incident with GitHub Actions" at 23:34:03 UTC, twenty-nine minutes after the expiry. By 00:07 UTC the incident had spread to API Requests, which the status page moved to partial outage. So it did not stay contained to Actions.
Matt's one-command check is the right one and we will repeat it because it is the single most useful thing in this post: pipe an empty echo into openssl s_client -connect pipelines.actions.githubusercontent.com:443 -servername pipelines.actions.githubusercontent.com, pipe that into openssl x509 -noout -enddate, and read the date. If it is in the past, stop touching your own infrastructure. It is not your CA bundle, your clock, or your proxy.
The part nobody has said yet
GitHub's Actions control plane sits behind Azure Front Door. Microsoft owns GitHub. Microsoft owns Front Door. Microsoft runs one of the largest certificate estates on earth and Azure Front Door ships a managed-certificate product whose entire job is to rotate certificates automatically so that a human forgetting cannot become an outage.
They did not use it. That wildcard is a bring-your-own Let's Encrypt certificate, and the bring-your-own path puts rotation back on the tenant.
So the honest indictment is not "Let's Encrypt is risky" and it is not "Azure certificates are unreliable." We checked that second claim against our own estate before repeating it, and it did not survive: our Key Vault certificate has 121 days left and our container registry has 172, both Microsoft-issued, both fine. There is no Azure certificate epidemic in our data.
The real indictment is narrower and considerably worse. Somebody chose the manual path for the one wildcard that fronts every self-hosted runner on Earth, and then did not do the manual part. The platform offered to handle this. They opted out, and then went home. The 90-day lifetime is not the flaw — it is the feature, the forcing function that is supposed to prove your automation is real. Tonight it found out the automation was not there.
This is also not the first time. In March 2023 the certificate on objects.githubusercontent.com expired and broke release downloads across the ecosystem. Same failure class, different subdomain, three years apart.
The shape of it
We have been calling this the subtraction attack when an adversary does it deliberately: the attack wins by removing an artifact the defense assumed would be there, so the defense never fires because it is looking for something to be wrong rather than something to be missing.
Tonight was not an attack. But it is the same shape, and that is why it is worth writing down. Nothing was compromised, nothing was malicious, nothing was even misconfigured in a way a scanner would flag. The entire global outage lives in the absence of one action nobody performed. Every runner in the world trusted a rotation that never happened, and there was no artifact anywhere for anyone to inspect, because the whole failure was a thing that did not exist.
You cannot detect a missing renewal by looking harder at what is present. You detect it by watching the clock.
We had the same blind spot
Here is the part that is ours to own.
We publish about other people's dependency blind spots for a living. When we went looking tonight for whatever we had written about certificate expiry monitoring, our own corpus returned zero results. Not thin — zero. And when we checked the repository, scripts/ contained no certificate monitor of any kind. Nothing watched a notAfter date, ours or anybody's.
That is worse than it sounds, and it is a specific failure mode we already have a name for. A check that emits nothing reads exactly like a check that passes. There was no red light, because there was no light. No denominator, no signal, no way for the absence to ever feel wrong. We would have diagnosed tonight the same way Matt initially did — by suspecting our own trust store first — except slower, because he was already awake and looking.
So we built the thing. scripts/cert-expiry-watch.js walks eighteen endpoints: our own public properties, and the third-party hosts our operations actually rest on — the GitHub Actions control plane, the GitHub API, the Wix publishing pipeline, Microsoft Graph, our container registry, Key Vault, Spamhaus submissions, ThreatFox, GreyNoise, Cloudflare, and the OSV bucket our malicious-package sync reads. It alerts at 14, 7, and 1 days. It is dependency-free pure Node, it never sends a byte to the endpoints it inspects, and it prints its denominator on every run so that "found nothing" can never again be mistaken for "never looked."
First run caught both expired GitHub endpoints at negative one day, correctly, with the trust error attached rather than swallowed.
It also caught something of ours we did not know was broken. status.dugganusa.com has no DNS record at all. It is decommissioned, the live status page has been at analytics.dugganusa.com/status.html for some time, and stale references to the dead subdomain are still sitting in two of our own deployment docs. Fifteen of our endpoints are healthy, the nearest expiry is 47 days out, and one advertised hostname does not resolve. We found that in the first ninety seconds of having a tool that looks.
What to actually do
If your runners are down right now: verify with the openssl check above, confirm the date is in the past, and then stop changing things. Runners under systemd with a restart policy will reconnect on their own when GitHub rotates. Runners you launched by hand are dead until you restart them, and you will want to check every one of them afterward, because partial recovery is easy to miss — Matt found exactly that in his own fleet, one runner self-healed and two did not.
If your runners are fine: that is luck, not architecture. Go make a list of the external hostnames your pipeline cannot function without, and start watching their expiry dates. It is one cron job. Tonight was predictable to the second, ninety days in advance, by anyone who was counting.
Your availability includes other people's certificates. You do not get to opt out of that, and tonight neither did anybody else.
Root cause and first public diagnosis: Matt Lucas, RedEye Security — the original write-up. Independent confirmation, Azure Front Door analysis, and the monitoring gap are ours. Certificate details read live from two networks via openssl s_client; incident timing from the githubstatus.com API. As of publication the certificate has not rotated.
Every indicator in this post is in the feed. Free.
1.58M+ IOCs, STIX 2.1 / TAXII, 88% novel vs ThreatFox, exploited-CVE leads ahead of CISA. No credit card — a free API key in 30 seconds, and you can audit every claim above against the live endpoints.