Fifty-Nine of Our Seventy-Five Cron Jobs Stopped Firing for 4.6 Days. The Only Thing They Had in Common Was a Minute Field Divisible by Five. Our Status Page Called Them Healthy.
On Thursday morning at 02:10 UTC, fifty-nine of our seventy-five scheduled jobs stopped running. They did not error. They did not log a failure. They did not fire again for one hundred and nine hours, and nothing alerted, because the health endpoint that exists to catch exactly this was reporting thirty-five of them as healthy with one hundred percent seven-day uptime.
We found it on Sunday afternoon, by looking. Not by an alarm, not by a customer complaint, and not because anything downstream broke loudly enough to notice. We found it because a scheduled self-audit cross-tabulated the job list by a field nobody had thought to cross-tabulate before, and the answer fell out in one table.
The discriminator was arithmetic
Every job whose cron minute field is a multiple of five died. Every job whose minute field is not a multiple of five kept running. There were no exceptions in either direction.
Fifty-nine jobs in the first bucket, zero survivors. Sixteen jobs in the second bucket, zero casualties. Daily jobs at 0 3 * * *, ten-minute jobs at */10, twice-daily jobs at 35 9,21 * * * — all dead. Jobs at 3 * * * *, 8,23,38,53 * * * *, 48 6 * * * — all alive, firing on time, through the entire window.
The receipt is a ten-minute job called bounce-revoke-fast, which revokes free API keys whose registration email hard-bounces. Its history runs 01:40, 01:50, 02:00, 02:10 on September 17, and then nothing at all until 16:00:16 on September 21. A gap of 109.8 hours in a job that is supposed to run one hundred and forty-four times a day. The registry documentation calls that job the strongest registration control we have.
It was not a deploy gap
The first thing you check is whether the container was simply down or redeploying. It was not. Revision 0000357 was created on September 14 at 16:24 UTC and revision 0000358 on September 21 at 15:54 UTC, with nothing in between. One container, running continuously for seven days, serving traffic the whole time, which stopped firing seventy-nine percent of its own scheduled work somewhere around hour sixty and carried on as if nothing had happened.
What ended the outage was an unrelated deploy. We shipped a fix to a feed endpoint on Sunday afternoon, the container restarted, and the jobs came back. Nobody intended that. The first recovery fires land within minutes of the new revision. A restart heals it; as far as we can tell, nothing else does.
The mechanism, which we have not proven
Our server has two raw cron.schedule handlers registered outside the scheduler manager, both on a five-minute cadence, and the first of them — a Cloudflare analytics poll — is not wrapped in a try/catch. The node-cron version we run re-arms its own one-second polling timer on the line immediately after it emits to its listeners. A synchronous throw out of a listener therefore escapes before the re-arm executes, and that task's polling loop dies silently and permanently.
That is a real defect and it is the first place to look. It is also not sufficient. It explains one task dying, not fifty-nine independent tasks dying together in a way that correlates perfectly with minute-modulo-five. Something is shared between those tasks that we have not identified yet, and we are not going to pretend otherwise in order to close the finding. The honest state is: the pattern is exact and reproducible in the data, the mechanism is a hypothesis, and it needs a live reproduction before anyone believes it, us included.
We have been here before in shape. An audit in May found eighty-three percent of our crons running at under a quarter of their expected fire rate. Today's figure is seventy-nine percent. Different cause, possibly. Same class of failure, certainly: scheduled work that stops happening, and a system with no independent check on whether the schedule is being honored.
The health surface was the second failure, and it is worse
A four-day outage is a bug. A four-day outage that your monitoring reports as healthy is a design flaw, and it is the part of this we should be most uncomfortable about.
Our status endpoint computes seven-day uptime as successes divided by recorded fires. Read that again: the denominator is the number of times the job ran. A job that stops running entirely keeps one hundred percent uptime forever, because zero failures out of zero fires is a perfect score. "Never looked" and "all green" render identically, which is the exact failure mode we have written about in other people's systems and had sitting in our own.
There is a second, sharper version of the same defect. The endpoint infers an expected interval from the cron expression so it can mark a job stale, and its parser handles */N forms, hourly, daily and weekly. For comma-list schedules it gives up and returns null, with a comment saying so. The staleness check is then guarded on having an interval, so a null interval means the check never runs and the job falls through to healthy.
Twelve of our enabled jobs use comma-list schedules. They can never be marked stale. Not "were not marked stale during this incident" — cannot be, by construction, ever.
As I write this, hours after the restart, five of them are still reporting green:
virustotal-hunter runs 10,40 * * * *, which is every thirty minutes. It has not fired in 110.9 hours. Status: ok. Seven-day uptime: 100%.
greynoise-fp-check-daily runs 35 9,21 * * *. 115.6 hours silent. Status: ok. Uptime: 100%.
tweetfeed-sync, phishing-feed-sync and hunt-poc-digest are all between 111 and 116 hours silent, all reporting ok.
A thirty-minute job that has not run in four and a half days is not healthy, and a health endpoint that says otherwise is not an instrument, it is decoration. The fix on the detection side is small and we know what it is: expand list-form minute and hour fields, take the smallest gap between consecutive slots as the expected interval, and report UNKNOWN rather than ok when no interval can be inferred at all. An honest unknown beats a confident green every time.
Two things the sweep found on the way past
While we were in there, two other instruments turned out to be lying, and both deserve naming.
The first is the GreyNoise false-positive gate. This is the precision control we describe on our IP blocklist: before an address ships to consumers, it is checked against GreyNoise so that benign internet scanners and known-good infrastructure get suppressed rather than blocked. Its last run printed, verbatim, candidates=149, checking=80, checked=0, then Summary: checked=0 | FP-suppressed=0, then Writes: 0 fp_suppress upserts | task FAILED | sample verified: false, and on standard error, 429 rate-limited — stopping. The job recorded itself as SUCCESS. Its error field is null.
The fp_suppress index holds zero documents. Not zero this week — zero, total, since the gate shipped. The precision control we advertise has never suppressed a single address, and the reason nobody noticed is that "rate-limited before we checked anything" and "checked everything, found no false positives" produce byte-identical output. A zero with no denominator beside it is not a result, and we shipped one into a customer-facing claim.
The second is our IP blocklist itself. Of the 2,840 rows we serve, 1,716 come from Spamhaus DROP and 497 from a Tor exit-node list — 77.9% of the file is two redistributed public lists that any defender can pull for free from the source. Every source except our own honeypot lane bottoms out within about forty-eight hours, because a fixed fetch budget is consumed by those two lists rewriting the same values with fresh timestamps on every refresh. Our own first-party observations survive in the file; most other lanes do not reach back far.
I want to be precise about one thing here, because it cuts against us in a way that matters. The internal audit that surfaced this reported the served window as roughly three hours. I could not reproduce that figure. Pulling the file myself this evening, the honest numbers are the ones above: 48 hours for most lanes, 277 days for the honeypot lane, 77.9% public-list concentration. The concentration finding holds and it is the important one. The three-hour figure does not, and publishing it because it came from our own tooling would have been the identical sin this entire post is about.
Which half of the control was unassigned
Every finding here sits on a boundary where two halves of a control meet and only one of them has an owner.
The scheduler owns firing jobs. It did that correctly for three days and then stopped. Nobody owns noticing that it stopped — and the data was there the whole time. Our cron_history index recorded the silence perfectly, with beautiful fidelity, 74,118 records of exactly when everything did and did not happen. No process reads it and asks whether the gaps make sense.
The status endpoint owns classification. It classifies correctly for schedules its parser understands. Nobody owns the case where the parser does not understand, so that case became healthy by default instead of unknown by default.
The GreyNoise gate owns checking addresses. It correctly reported that it had checked zero. Nobody owns the question of whether zero-checked should be allowed to record as success.
The blocklist route owns fetching IOCs within a budget. Nobody owns asserting that rows served bears any relationship to rows eligible.
This is what we mean when we say the failures live in shared responsibility. Not that anyone was negligent — every one of those components did its own job correctly. The bug lives in the seam, and seams are unowned by default unless somebody goes and assigns them.
What we are not claiming
The root cause is not solved. We know the pattern exactly, we have a hypothesis about the mechanism, and we do not have a reproduction. Until we do, this will recur, and the next time it recurs the only thing that will have changed is that we will find it faster, because we now know to bucket the job list by minute-modulo-five before anything else.
We also have not fixed the detection half yet. The parser change is understood, small and not yet shipped. If you pull our status page tonight you will still see five green jobs that have not run since Wednesday.
Ninety-five percent confidence cap, as always, and in this case the missing five percent has a specific address: we do not know why fifty-nine tasks died together, and anyone who tells you they know why a distributed timer failed without reproducing it is selling something.
The thing that worked was the practice. A leper checks his hands on Tuesday because it is Tuesday, not because they hurt — the nerves are dead, so pain is not available as a signal, and the only defense is scheduled, deliberate inspection of the extremities whether or not anything seems wrong. Every one of these instruments was reporting green. The corpus of evidence was complete and correct and sitting in an index. All it took was somebody scheduled to go and look.
Was this useful? Rate this post. The widget is at the bottom of the page, and we read every response.
How do AI models see YOUR brand?
AIPM has audited 250+ domains. 15 seconds. Free while still in beta.
Was this useful? Thirty seconds, no cookies, no tracking, no third parties, your address hashed and never stored. If the box below does not load, the same question lives at https://analytics.dugganusa.com/nps.html?post=fifty-nine-of-our-seventy-five-cron-jobs-stopped-firing-for-4-6-days-the-only-thing-they-had-in-com
Comments