A Forecast That Says 0.0123 Cannot Be Checked. One That Says 'This Is August 6' Can. We Taught the Threat Model Déjà Vu, and It Named Five Dates — Four of Which Never Got Clean.
Patrick said it in seven words: left of boom plus Markov should lead to déjà vu.
He is right, and the machinery to do it has been running on our infrastructure for weeks. We just throw the answer away at the last step and print a probability instead.
What déjà vu is, mathematically
Strip the mysticism and déjà vu is three things in a row. You are in some state. You search your memory for the times you were in that same state and things looked roughly like this. You recall what happened next.
A Markov chain gives you the first part. Ours models a threat level with four states — normal, watch, warning, critical — and a matrix of transition probabilities learned from history. Nearest-neighbor search over embedded observations gives you the second: find the windows whose shape most resembles the shape of right now. The third part, what happened next, is just reading forward in the record from each of those matches.
That is the whole trick. State, similarity, precedent. No part of it is exotic, and all three have been live here since July.
Why a probability cannot be checked and a precedent can
Here is what our forecast endpoint returns right now, live, while the system sits at critical:
Next step: normal 0.0163, watch 0.0489, warning 0.5279, critical 0.4070. Eight steps out it relaxes to normal 0.6854.
Now ask the only question that matters about a forecast: how would you know if it was wrong? You cannot. A single 0.407 is unfalsifiable by construction. If the next half hour is critical, the number was not wrong. If it is normal, the number was not wrong either — it said 0.0163, and unlikely things happen. You need hundreds of repetitions before a probability says anything checkable, and by then the incident is a year old.
A precedent has the opposite property. If the system tells you "this looks like 2026-08-06 at 08:00," you can go read what we published that morning, pull the block events from that window, and decide for yourself in ninety seconds whether it rhymes. The claim is exposed. It can be caught being wrong, immediately, by anyone.
That is the difference between a number that sounds like knowledge and a statement that behaves like knowledge.
What we actually built
The model doc lives in Meilisearch as threat-level-v1. Pulled live while writing this: 664 observations, trainer version markov-backfill-1.1.0, signals block clustering plus KEV velocity plus proof-of-concept velocity, retrained this morning at 10:05 UTC.
It carries a held-out skill gate, which is the only reason any of the rest deserves your attention. The model assigns a probability to the state that actually came next, averaged over transitions it never trained on, and that average has to beat the best state-blind guess available on the same test set. Today: mean probability 0.594 against a must-beat of 0.394, over 3,411 held-out transitions, flagged beats-base-rate. The dated archives go back eight days and the gate has held every day in the range 0.582 to 0.594, against must-beats between 0.376 and 0.394.
The gate matters more than the forecast. An ungated chain is a random number generator wearing a lab coat, and ours spent months as exactly that before anyone scored it.
The third tier is the déjà vu part. It vector-searches 9,055 embedded precursor observations for windows resembling the current one, keeps only the ones that started from the same state as now, and blends their observed outcomes into the transition matrix at thirty percent weight.
The one line that made it fire
For most of its life that tier did nothing. The forecast reported neighbors_0 and silently fell back to the plain chain.
The reason was a single misplaced filter. The code searched for fifteen similar windows across all four states, then filtered to same-state afterward in JavaScript. From a rare state like critical, fifteen nearest neighbors are almost all normal, because most of history is normal — so the post-filter threw away everything and left zero or one survivor, below the minimum to blend. The fix was to move the constraint into the query, so the database returns fifteen same-state neighbors instead of fifteen neighbors of which one happens to match. Neighbors went from zero to fifteen. One line, in a filter string.
Nothing announced the defect. The endpoint had always returned applied: true with a neighbor count nobody read.
What it should return, and what that looks like
Today the endpoint hands back this and nothing more: applied true, neighbors found 5, window query, blend weight 0.3. Five precedents were located, consulted, weighted into the math, and then discarded unnamed. The most useful thing the system computed is the thing it refuses to say.
So I ran the retrieval by hand against the live corpus — same query shape, same same-state constraint — to show what the endpoint is sitting on.
The five nearest precedents to the current shape, with similarity scores between 0.891 and 0.897, are 2026-08-06 08:00, 2026-08-16 02:30, 2026-09-11 06:01, 2026-08-17 02:01 and 2026-08-15 10:00.
Four of the five never got clean. The August 16 window ran twelve of its next sixteen half-hours at critical and never decayed at all. September 11 held solid critical for five and a half hours, dipped briefly at seven, and climbed straight back. August 17 went quiet from hour three to hour seven and returned to critical by hour eight. August 6 dipped to normal within an hour and then sat at warning for six consecutive hours. Exactly one, August 15, settled and stayed down — and it settled at watch, not at normal.
Compare that to "critical 0.407." The precedent version tells a tired human at three in the morning something they can act on: this shape historically does not resolve itself inside an hour, the one time it looked like it had, it came back, and here are five dates to go read before you decide to go back to bed. Left of boom is worth nothing if the warning arrives in a form nobody can use. A named precedent is the form.
What would prove this wrong
If the named precedents do not rhyme, this feature is worse than useless, because it converts a vague number into a confident and specific falsehood. So the test has to be stated before the feature ships, not after.
The test: take every forecast where the system names precedents, wait out the window, and score whether the actual trajectory matched the modal precedent trajectory more often than chance. If it does not beat the same state-blind baseline the chain's own gate has to beat, the precedents are decoration and we retire them. That is the identical bar the Markov chain already has to clear daily, applied to the retrieval layer, and there is no honest reason to hold the new part to a weaker standard than the old part.
Three things I found while writing this, all of them ours
I went to verify the numbers in this post rather than quote them, and the verification found defects. That keeps happening, and it is the entire argument for doing it.
First, the similarity query is currently generic. The endpoint builds its search string from the most recent precursor observation's signal scores, and two different producers write to that index — a live one that records real values every thirty minutes, and an aggregator that writes a constant 0.16 with every signal at zero, once an hour. When the aggregator's row happens to be newest, every signal reads zero, and the query degrades to the phrase it uses when nothing is elevated: low threat normal baseline steady state. The system was searching for windows that looked calm while sitting at critical. The retrieval still returns same-state neighbors, so it is not broken, but similarity is doing far less work than the response implies.
Second, that same interleaving corrupts the outcome lookup. My first pass at "what happened next" reported that four of five precedents returned to normal within six minutes, which is absurd on its face and is what made me look. The aggregator's hourly constant-normal row was landing between the real observations and reading as an instant recovery. Every trajectory in the figure above comes from the live producer only, and the difference between the two versions is the difference between a useful answer and a comforting lie.
Third, the corpus is bigger than the search API will admit. The index holds 9,055 observations; a paged search tops out at 5,000 and returns no warning that it stopped. My first trajectory pass silently used the oldest five thousand and found nothing forward of any August match. The figure uses the documents endpoint, which has no such cap. This is the fourth time this month that a ceiling in a query path has quietly truncated something here.
The honest limits
Six hundred sixty-four observations is a small training frame and the critical row of the matrix rests on thirty-nine of them, which the model itself reports as medium confidence while every other row reads high. The signal set is three signals, chosen by ablation last night: block plus KEV scored 0.593, adding proof-of-concept velocity moved it to 0.594 while raising the critical evidence from fifteen observations to thirty-eight, and adding Tor telemetry dropped it to 0.530, so Tor is built and benched. Patrick's frame for that afternoon was that ablation is sanding, and there is no such thing as smooth, only levels of ablating.
We have eight days of dated gate archives and no longer-run scoring history, because the metric store keeps one document per metric and overwrites it. Neighbor surfacing is designed and argued for here; it is not shipped. And the whole apparatus forecasts a threat level derived from our own telemetry, which is a view of the internet from one small vantage point in Minnesota, not the weather.
Ninety-five percent, as always. Something in here is wrong and we would rather you find it fast than slow.
Was this useful? Rate this post. The widget is at the bottom of the page, and we read every response.
Every indicator in this post is in the feed. Free.
1.58M+ IOCs, STIX 2.1 / TAXII, 88% novel vs ThreatFox, exploited-CVE leads ahead of CISA. No credit card — a free API key in 30 seconds, and you can audit every claim above against the live endpoints.
Was this useful? Thirty seconds, no cookies, no tracking, no third parties, your address hashed and never stored. If the box below does not load, the same question lives at https://analytics.dugganusa.com/nps.html?post=a-forecast-that-says-0-0123-cannot-be-checked-one-that-says-this-is-august-6-can-we-taught-the-t
Comments