The Bug Hunt: How Two Temporal Dependencies Killed Our Threat Intel Feed (And How We Fixed It in 26 Minutes)

Category: Engineering War Stories Tags: debugging, temporal-dependencies, stix-threat-intelligence, production-incidents, javascript-edge-cases

---

TL;DR

Our STIX 2.1 threat intelligence feed started throwing 500 errors on every request. Root cause: two temporal dependency bugs where variables were used before they were declared. Time from first error to production fix: 26 minutes. This is why we measure deployment velocity.

---

The Scene

4:55 PM CT: Threat Intel Hub dashboard shows errors. STIX feed endpoint returning:

{
  "error": "Failed to generate STIX feed",
  "message": "Cannot access 'infrastructureType' before initialization"
}

Console logs from production:

STIX feed error: ReferenceError: Cannot access 'infrastructureType' before initialization
at transformToSTIXIndicator (/app/routes/api/v1/stix-feed.js:515:11)

Impact: 100% failure rate on `/api/v1/stix-feed` endpoint. Zero threat intelligence indicators served.

---

Bug #1: The Classic Temporal Dependency

The Problem

Line 515 in `stix-feed.js`:

labels: [
  'malicious-ip',
  asshole.country || 'unknown-country',
  asshole.usageType?.toLowerCase().replace(/\s+/g, '-') || 'unknown-usage',
  // Issue #212: Add residential-proxy label when detected
  ...(infrastructureType === 'RESIDENTIAL' ? ['residential-proxy'] : [])
]

Variable `infrastructureType` used here. But where is it declared?

Line 568:

// BOT CLASSIFICATION - Pattern #32
const threatCategory = classifyThreatCategory(asshole);
const infrastructureType = classifyInfrastructure(asshole);

53 lines later. JavaScript's temporal dead zone strikes again.

The Fix

Move classification calls to line 444, before the labels array:

// BOT CLASSIFICATION - Pattern #32: Classify BEFORE using in labels
// Must be done early since labels array references infrastructureType
const threatCategory = classifyThreatCategory(asshole);
const infrastructureType = classifyInfrastructure(asshole);

// Map abuse score to confidence (0-100) let confidence = asshole.abuseScore || 0; // ... confidence boosting logic ... ```

Time to identify: 2 minutes (read error logs + check line 515) Time to fix: 30 seconds (move 2 lines) Commit: `c9f0af7`

---

Bug #2: The Edge Case That Wasn't

Deployed Bug #1 fix. Same error:

STIX feed error: SyntaxError: Unexpected end of JSON input
at JSON.parse (<anonymous>)
at transformToSTIXIndicator (/app/routes/api/v1/stix-feed.js:637:33)

The Problem

Line 637:

suspicious_keywords: JSON.parse(ispReputation.suspiciousKeywords || '[]')

Looks safe, right? Default to empty array string if falsy.

Wrong. Empty string `''` is falsy in boolean context but not in `||` fallback context when the value exists.

const value = '';  // From Azure Table Storage
value || '[]'      // Returns '' (the original empty string!)
JSON.parse('')     // SyntaxError: Unexpected end of JSON input

Azure Table Storage was returning `suspiciousKeywords: ''` (empty string, not null). The `||` operator saw a truthy-ish value and kept it.

The Fix

Check if string exists and is non-empty:

let suspiciousKeywords = [];
try {
  const keywordStr = ispReputation.suspiciousKeywords;
  if (keywordStr && keywordStr.trim()) {
    suspiciousKeywords = JSON.parse(keywordStr);
  }
} catch (e) {
  console.error(`Failed to parse suspicious keywords for ${ispReputation.ispName}:`, e.message);
}

indicator.x_dugganusa_isp_reputation = { // ... suspicious_keywords: suspiciousKeywords, // ... }; ```

Time to identify: 5 minutes (checked logs, tested endpoint, found line 637) Time to fix: 2 minutes (add trim() check + error handling) Commit: `19e349e`

---

The Deployment

Build & Deploy Pipeline

# Build Docker image (linux/amd64)
./build-and-push.sh

Build time: 3m 15s (React SPA + Node.js backend) Push time: 1m 45s (232MB image, only 3 layers changed) Deployment time: 2m 30s (Azure Container Apps revision rollout)

Total deployment time: 7m 30s

Verification

curl -s "https://analytics.dugganusa.com/api/v1/stix-feed?days=7&min_confidence=80" \
  | jq '.x_dugganusa_metadata.total_indicators'

Status: ✅ Serving 142 threat intelligence indicators (last 7 days, 80%+ confidence)

---

Why This Matters

1. Temporal Dependencies Are Sneaky

JavaScript's temporal dead zone catches a lot of developers. Variables exist in their scope but accessing them before declaration throws `ReferenceError`. This is intentional (prevents accidental hoisting bugs) but easy to miss during refactoring.

Lesson: When moving code around, check variable dependencies both up and down the file.

2. Empty String ≠ Falsy (In Some Contexts)

// These all behave differently:
null || 'default'       // 'default'
undefined || 'default'  // 'default'
'' || 'default'         // 'default'
' ' || 'default'        // ' ' (single space is truthy!)

// But in if statements: if ('') { } // false - empty string IS falsy if (' ') { } // true - whitespace is truthy ```

Lesson: When validating strings, use `.trim()` or explicit length checks. Don't rely on falsy coercion.

3. Fast Debugging = Fast Recovery

Total time from error to production fix: 26 minutes

• Error identified: 0m 0s (user reported)

• Root cause #1 found: 2m 0s

• Fix #1 applied: 2m 30s

• Build + deploy #1: 7m 30s

• Root cause #2 found: 5m 0s

• Fix #2 applied: 2m 0s

• Build + deploy #2: 7m 30s

• Logs in production: `az containerapp logs show` (Azure CLI)

• Error line numbers: Stack traces pointed directly to the problem

• Fast deployment pipeline: 7.5 minute turnaround from code to production

• No red tape: No change approval process for critical bug fixes

• Git commits during deployment: Don't wait for deploy to finish before committing

---

The Technical Details

STIX 2.1 Threat Intelligence Feed

Our endpoint serves indicators in STIX 2.1 format:

{
  "type": "bundle",
  "spec_version": "2.1",
  "objects": [
    {
      "type": "indicator",
      "id": "indicator--...",
      "pattern": "[ipv4-addr:value = '1.2.3.4']",
      "confidence": 85,
      "labels": ["malicious-ip", "CN", "datacenter"],
      "x_dugganusa_threat_intel": {
        "abuse_score": 80,
        "vt_detections": 3,
        "isp": "Cloudflare Inc."
      },
      "x_dugganusa_bot_classification": {
        "threat_category": "SCANNER",
        "infrastructure_type": "CLOUD",
        "residential_proxy": false
      }
    }
  ]
}

Classification Logic

Bot classification happens in `classifyInfrastructure()`:

function classifyInfrastructure(asshole) {
  const usageType = (asshole.usageType || '').toLowerCase();
  const isp = (asshole.isp || '').toLowerCase();

// Cloud providers if (isp.includes('amazon') || isp.includes('aws') || isp.includes('google cloud') || isp.includes('microsoft')) { return 'CLOUD'; }

// Residential if (usageType.includes('isp') || usageType.includes('residential')) { return 'RESIDENTIAL'; }

return 'UNKNOWN'; } ```

This function returns `'RESIDENTIAL'` when ISP usage type matches residential patterns. That value gets used in the `labels` array to tag residential proxies.

The bug: Function was called 53 lines after the labels array tried to use its return value.

---

Lessons Learned

1. Test Edge Cases in Data Validation

Empty strings from databases are common. Always validate:

// ❌ BAD
const value = data.field || 'default';

// ✅ GOOD const value = (data.field && data.field.trim()) || 'default'; ```

2. Variable Declaration Order Matters

• Forward dependencies: Variables used before they're declared

• Backward dependencies: Variables declared but never used

Tools like ESLint can catch some of these, but not all (especially in dynamic contexts).

3. Production Logs Are Your Friend

• Stack traces included line numbers

• Error messages were descriptive

• Logs were easily accessible (`az containerapp logs show`)

No logs = no debugging.

4. Fast Deployments Enable Fast Recovery

• Fix bug #1

• Deploy it

• Discover bug #2

• Fix bug #2

• Deploy it

All in 26 minutes. Compare to traditional enterprise deployments (hours to days).

---

The Broader Context

Why We Publish Free Threat Intelligence

Our STIX feed is free and public (CC0-1.0 license). No API keys, no rate limits, no paywalls.

• 99.5% of our codebase is public (4,780 files)

• Threat intelligence should be accessible to everyone

• Standing on the shoulders of giants (not hoarding on top of them)

• AbuseIPDB

• VirusTotal

• ThreatFox

• Team Cymru

• GreyNoise

• Our own multi-source correlation

Value add: We find threats that billion-dollar vendors miss through correlation analysis.

The Numbers

• 142 indicators (last 7 days, 80%+ confidence)

• 596 total blocked IPs across 31 countries

• Free, updated in real-time

• STIX 2.1 format (industry standard)

---

Conclusion

Two bugs, 26 minutes, zero downtime (except for the initial error window). This is why we measure:

• Time to detection: Instant (user-reported dashboard error)

• Time to diagnosis: 7 minutes total

• Time to fix: 4.5 minutes total

• Time to production: 15 minutes total (2x deployments)

Deployment velocity matters. Fast pipelines enable fast recovery. Fast recovery enables confident experimentation.

---

Try It Yourself

# Get our threat intel feed
curl https://analytics.dugganusa.com/api/v1/stix-feed | jq .

Free. Public. Real-time. No excuses.

---

Patrick Duggan DugganUSA LLC Minnesota

*Co-Authored-By: Claude (Sonnet 4.5) - Because this debugging session happened live during our deployment pipeline execution.*