All Hat, No Cattle? A Decade of Receipts
- Patrick Duggan
- Jan 8
- 3 min read
The Coconut Headphones
The appendix slide was my favorite. During World War II, Pacific islanders watched American military planes land on their islands and drop supplies. After the war ended, some islanders built replica runways out of bamboo. They carved wooden headphones. They waved landing sticks at the empty sky.
They were mimicking the form without understanding the function.
Richard Feynman called this "cargo cult science" - following the superficial appearance of methodology without understanding why it works.
My warning to the 2013 audience: Before you implement Chaos Monkey because Netflix does it, ask yourself if you're actually Netflix. Do you have their scale? Their redundancy? Their engineering team?
Or are you wearing coconut headphones?
A Decade Later
This morning, at 12:15 PM Central Time, our Meilisearch instance panicked. MDB_PANIC: Update of meta page failed or environment had fatal error. The search index that powers our threat intelligence platform was corrupted.
Here's what we did:
Deleted the Azure File Share
Recreated it with fresh storage
Deployed a new container revision
Waited 45 seconds
Searched for "Lazarus Group" - returned in 3 milliseconds
The salmon swims again.
The Receipts
Am I all hat, no cattle? Let me check:
Treat infrastructure as disposable
Kill and replace, don't nurse back to health
Don't name your servers
Build for failure
Containers are numbered (meilisearch--0000006), not named
When the index died, we shot it and replaced it
Core operations continued during the outage - blocking still worked, STIX feed still published
Total infrastructure cost: $77/month
We didn't scale up to Netflix complexity. We scaled down to right-sized cattle.
The Architecture of Tolerance
The key insight isn't "be Netflix." It's "be resilient at your scale."
Our Meilisearch index is an optimization layer, not the source of truth. The real data lives in Azure Tables. The STIX feed generates from primary sources. The blocklist operates from core storage.
Threat intel still publishes
IP blocking still works
The platform still operates
You just grep slower until the index rebuilds.
Indexing as last layer = inherent fault tolerance. Less efficient, but the system survives.
The Right-Sized Herd
The coconut headphone people deployed Kubernetes for a blog. They implemented microservices for a CRUD app. They built chaos engineering for systems that couldn't survive a single failure.
Meanwhile, we're running enterprise-grade threat intelligence on the infrastructure equivalent of a pickup truck. No Kubernetes. No service mesh. No complexity that adds attack surface.
Express. Boring. Works.
The Sermon Changes, The Gospel Doesn't
I'm not preaching Netflix anymore. I'm preaching right-sized resilience.
Cattle, not pets
Disposable, not precious
Designed for failure, not designed around it
But the implementation looks different at $77/month than it does at $77 million.
The rancher with 50 head doesn't need the same infrastructure as the rancher with 50,000. But they both need to know which cattle are sick and be willing to replace them.
This morning, one of our cattle got sick. We replaced it. Life goes on.
That's not "all hat." That's the whole point.
The 2013 deck is lost to time. The coconut headphones live on in my memory. The cattle keep grazing.
About DugganUSA: We publish free threat intelligence for the 99% who can't afford enterprise security. Our STIX 2.1 feed tracks 2,200+ blocked IPs with MITRE ATT&CK attribution. Built on $77/month of Azure Container Apps, designed to fail gracefully.
Her name is Renee Nicole Good.
Comments