Matt Lucas of RedEye Security Translated Our 17 Detection Rules for Caver and Found Five That Did Not Do What They Said. All Five Are Fixed in the Live Catalog.
If you pulled our detection rules before November 7, 2026, pull them again. Matt Lucas at RedEye Security translated all 17 of them into the rule format of Caver, his company's SIEM, and in doing so found five rules whose logic did not do what its title said. We checked each finding against our catalog, confirmed all five, and the fixes went live the same day.
What he did
Matt took our public rule catalog and translated each rule into Caver's native format, one file per rule, with each file linking back to its source. Caver runs on an open OCSF Parquet lakehouse, so a translation forces someone to read every line of a rule as the engine would, and that is where the defects showed. He validated the results with Caver's own rule checker, which parses the search and checks which fields each step can still see. He says plainly that nothing was run against event data and nothing was run against a live key of ours. He also held back from publishing his work, because our terms say our data is not passed on without our written okay, and sent it to us instead.
The five defects
Rule 007, JSON-based SQL injection. A line meant to require at least two SQL keywords added up counts that are each at least one, so the total could never fall below three and the filter removed nothing. The regular expression above it was doing all the selecting. The rule still caught what the regex caught, but the extra safeguard we advertised was not there.
Rule 011, vulnerable driver loads. The search read as the index and the first event code, or the second event code. In Splunk's precedence that means the second code matched events from every index, not only Windows. We added parentheses.
Rule 012, living-off-the-land tactics. A filter kept only events scoring 60 or more, but 60 was the lowest score the rule could assign, so it never removed anything. We removed the dead line.
Rule 016, hex-named webshells in Windchill. The search matched file creations under the Windchill login folder, and then the final filter only kept events that matched a request pattern, a header or one staging file name. A newly written hex-named JSP file, the thing the rule exists to find, was dropped. We added a filename match to the final filter.
Rule 017, WSDL reconnaissance. A response-size test used a single equals sign inside the condition, where a comparison needs two. We fixed it.
Matt also noted that the Sigma versions of several rules wrote thresholds like abuse score greater than 15 as quoted text, which Sigma tooling treats as a literal string, not a number. We changed six of those to proper numeric comparisons across rules 001, 002, 004, 005 and 006.
What we checked ourselves
We did not take the list on trust. We pulled the live catalog and read the source of each rule. All five Splunk defects were present as he described them. For the Sigma point, abuse score above 15 appears as text in rule 001, and the other rules use the same pattern with other numbers, so the details of his note on rule 006 differ slightly from what we found: 006 uses 20, not 15. The point holds.
A finding about our feed, not our rules
While reviewing his integration page we measured something he had already hedged about. Today, a pull of our IP list with the confidence floor at 80 and the window at seven days returned 3,097 lines, and 1,790 of them contained a slash, meaning they are network ranges, mostly from the Spamhaus DROP list. A lookup that matches exact addresses cannot use a range, so more than half of that list would sit idle in his setup. He had said ranges never match. The size of the share is the new part. Authenticated pulls of that list contained no comment lines, so the step he included to strip them is harmless but, for a keyed pull, unnecessary.
What we did about it
These rules had no test that ran them. A rule that parses but filters nothing is a syntax-valid defect that a reader catches and a linter does not. So every rule now also has a SQL form built on OCSF field names, available from the rules API as the sql export format, and for each of the 17 a test runs the detection logic against a row that should match and a near-miss row that should not. The same day, we published a Caver integration page on our docs site, written with Matt: docs.dugganusa.com/siem/caver. It covers the TAXII subscription, the CSV lookup tables and their limits, and the rules.
Limits
Our fixes were made by reading the rules and reasoning about them. We did not run them in Splunk or in a Sigma toolchain. We have not run Matt's translated rules or tested Caver, and the claims about how Caver handles lookup tables and feeds are his, not ours. Our SQL tests ran on SQLite with two small shims. The OCSF mapping in the first step of each query, and the SQL dialect, have not been run on a lakehouse engine or on Caver. His own translation into Caver's rule format is unpublished and remains his work. One KQL query, for rule 007, checks the wrong fields for a content type, and we have not changed it yet.
Five percent of this is wrong somewhere. It always is. Matt found five in ours, which is how that works.
Every indicator in this post is in the feed. Free.
1.58M+ IOCs, STIX 2.1 / TAXII, 88% novel vs ThreatFox, exploited-CVE leads ahead of CISA. No credit card — a free API key in 30 seconds, and you can audit every claim above against the live endpoints.
Comments