9 min read Updated May 9, 2026

Detection Troubleshooting

This guide covers issues with Tamandua's detection capabilities, including YARA rules, Sigma rules, ML-based detection, and the detection engine itself.

Missing Detections

Symptoms

  • Known malware not detected
  • Sigma rules not triggering
  • ML model not scoring samples
  • Expected alerts not generated

Diagnostic Steps

  1. Check detection engine status

   # In IEx console
   TamanduaServer.Detection.Engine.status()
   TamanduaServer.Detection.list_rules()

  1. Verify rule is loaded

   TamanduaServer.Detection.get_rule("rule_name")

  1. Check telemetry is arriving

   TamanduaServer.Telemetry.recent_events(agent_id, limit: 10)

Common Causes and Solutions

Detection Engine Not Running

Error: 
Detection engine not responding
GenServer :detection_engine not found
Solution:
  1. Check engine status:

   Process.whereis(TamanduaServer.Detection.Engine)

  1. Restart if needed:

   Supervisor.restart_child(TamanduaServer.Supervisor, TamanduaServer.Detection.Engine)

Rules Not Loaded

Error: 
No rules loaded in detection engine
Solution:
  1. Check rules directory:

   ls -la /opt/tamandua/priv/yara_rules/
   ls -la /opt/tamandua/priv/sigma_rules/

  1. Reload rules:

   TamanduaServer.Detection.reload_rules()

  1. Check for load errors in logs

Telemetry Not Reaching Detection

Error: 
Events in queue but not processed
Solution:
  1. Check Broadway pipeline:

   TamanduaServer.Telemetry.Ingestor.status()

  1. Verify detection is subscribed to events
  2. Check for processing errors in logs

Event Type Not Monitored

Symptom: Specific event types not generating detections Solution:
  1. Enable event type in detection config:

   config :tamandua_server, TamanduaServer.Detection,
     enabled_event_types: [:process, :file, :network, :registry, :dns]

  1. Ensure agent is collecting that event type

ML Service Not Responding

Error: 
ML prediction failed: connection refused
ML service timeout
Solution:
  1. Check ML service:

   curl http://localhost:8000/health
   curl http://localhost:8000/predict -X POST -H "Content-Type: application/json" \
     -d '{"features": [...]}'

  1. Restart ML service:

   systemctl restart tamandua-ml

  1. Check ML service logs

False Positives

Symptoms

  • Legitimate software flagged as malicious
  • High volume of benign alerts
  • Alert fatigue for analysts

Common Causes and Solutions

Overly Broad Rules

Problem: Rule matches too many legitimate files/processes Solution:
  1. Review rule logic:

   rule = TamanduaServer.Detection.get_rule("broad_rule")
   IO.inspect(rule.conditions)

  1. Add exclusions:

   # Sigma rule with filter
   detection:
     selection:
       Image|endswith: '\powershell.exe'
     filter:
       User: 'SYSTEM'
       ParentImage|endswith: '\svchost.exe'
     condition: selection and not filter

Missing Allowlist

Solution:
  1. Create allowlist for known-good items:

   TamanduaServer.Detection.add_allowlist(%{
     type: :hash,
     value: "abc123...",
     reason: "Known good - Microsoft signed",
     expires_at: ~U[2025-12-31 23:59:59Z]
   })

  1. Allowlist by path:

   TamanduaServer.Detection.add_allowlist(%{
     type: :path,
     value: "C:\\Windows\\System32\\*",
     reason: "System directory"
   })

Environment-Specific Issues

Problem: Rules designed for different environment Solution:
  1. Create organization-specific rule variants
  2. Use rule tags to enable/disable by environment:

   tags:
     - attack.execution
     - environment.production

  1. Configure detection profiles per agent group

ML Model Miscalibrated

Symptom: ML scores too aggressive Solution:
  1. Adjust threshold:

   config :tamandua_server, TamanduaServer.Detection.ML,
     threshold: 0.85  # Increase from default 0.75

  1. Retrain model with environment-specific samples
  2. Use ensemble approach with multiple thresholds

Tuning Process

  1. Collect false positive data

   TamanduaServer.Alerts.false_positives(days: 30)
   |> Enum.group_by(& &1.rule_name)
   |> Enum.sort_by(fn {_, fps} -> length(fps) end, :desc)

  1. Analyze patterns

   TamanduaServer.Detection.analyze_false_positives("rule_name")

  1. Create suppression rules

   TamanduaServer.Detection.add_suppression(%{
     rule_name: "Suspicious PowerShell",
     conditions: %{
       parent_process: "sccm-agent.exe",
       user: "SYSTEM"
     },
     expires_at: ~U[2025-06-01 00:00:00Z]
   })

Rule Not Matching

Symptoms

  • Rule should fire but doesn't
  • Test events don't trigger alerts
  • Regex not matching expected strings

Diagnostic Steps

  1. Test rule in isolation

   event = %{type: :process, name: "powershell.exe", ...}
   TamanduaServer.Detection.test_rule("rule_name", event)

  1. Check rule syntax

   TamanduaServer.Detection.validate_rule(rule_content)

Common Causes and Solutions

Case Sensitivity

Problem: Rule expects lowercase but event has mixed case Solution: 
# Sigma - use modifiers
detection:
  selection:
    Image|endswith|nocase: 'powershell.exe'

// YARA - use nocase modifier
rule example {
  strings:
    $s1 = "powershell" nocase
  condition:
    $s1
}

Field Name Mismatch

Problem: Rule references field that doesn't exist in event Solution:
  1. Check event schema:

   TamanduaServer.Telemetry.event_schema(:process)

  1. Map fields correctly:

   # Use correct field mapping
   detection:
     selection:
       process.executable|endswith: '\powershell.exe'  # Not 'Image'

Regex Escaping Issues

Problem: Special characters not escaped Solution: 
# Escape backslashes in Sigma
detection:
  selection:
    CommandLine|contains: '\\Windows\\System32\\'  # Double backslash

# Or use re modifier for regex
detection:
  selection:
    CommandLine|re: '.*\\\\Windows\\\\System32\\\\.*'

Timeframe Not Met

Problem: Sigma rule with timeframe/count not triggering Solution:
  1. Check aggregation state:

   TamanduaServer.Detection.SigmaAggregator.get_state("rule_name")

  1. Verify timeframe configuration:

   detection:
     selection:
       EventID: 4625
     condition: selection | count() > 5
     timeframe: 5m

Incomplete Telemetry

Problem: Required fields not collected Solution:
  1. Enable additional collection:

   # Agent config
   [collectors.process]
   collect_command_line = true
   collect_parent_info = true
   collect_hashes = true

  1. Verify field is populated in events

ML Model Issues

Symptoms

  • Model not loading
  • Predictions failing
  • Inconsistent scoring
  • High inference latency

Diagnostic Steps

  1. Check model status

   curl http://localhost:8000/model/status

  1. Test prediction endpoint

   curl -X POST http://localhost:8000/predict \
     -H "Content-Type: application/json" \
     -d '{"file_path": "/tmp/test.exe"}'

Common Causes and Solutions

Model Not Loaded

Error: 
Model not loaded: file not found
RuntimeError: No model loaded
Solution:
  1. Check model path:

   ls -la /opt/tamandua-ml/models/

  1. Update configuration:

   export MODEL_PATH=/opt/tamandua-ml/models/malware_smell.pt

  1. Restart ML service

Model Version Mismatch

Error: 
RuntimeError: Model version incompatible
KeyError: 'unexpected key in state_dict'
Solution:
  1. Check model version:

   curl http://localhost:8000/model/version

  1. Download compatible model version
  2. Ensure training and inference use same architecture

CUDA/GPU Issues

Error: 
RuntimeError: CUDA out of memory
RuntimeError: CUDA not available
Solution:
  1. Check GPU status:

   nvidia-smi

  1. Fall back to CPU:

   export DEVICE=cpu

  1. Reduce batch size:

   config.batch_size = 8  # Reduce from default

High Inference Latency

Symptom: Predictions taking too long Solution:
  1. Check system resources:

   htop
   nvidia-smi

  1. Enable batching:

   config.batch_inference = True
   config.batch_size = 32
   config.batch_timeout_ms = 100

  1. Use GPU if available
  2. Consider model quantization

Feature Extraction Failure

Error: 
ValueError: Cannot extract features from file
KeyError: 'import_table'
Solution:
  1. Check file format support
  2. Verify file is not corrupted:

   file /tmp/sample.exe

  1. Add error handling for edge cases

Model Monitoring

# Check ML prediction metrics
TamanduaServer.Detection.ML.metrics()
# => %{
#   predictions_total: 10000,
#   predictions_malicious: 150,
#   avg_latency_ms: 45,
#   errors: 3
# }

YARA Compilation Errors

Symptoms

  • YARA rules fail to load
  • Compilation errors in logs
  • Some rules not available

Diagnostic Steps

  1. Validate rules

   yara -w /opt/tamandua/priv/yara_rules/*.yar /dev/null

  1. Check specific rule

   yara -w problematic_rule.yar /dev/null

Common Causes and Solutions

Syntax Error

Error: 
error: syntax error, unexpected identifier
line 10: strings:
Solution:
  1. Check rule syntax:

   rule example {
     meta:
       description = "Example"  // Use = not :
     strings:
       $s1 = "test"
     condition:
       $s1
   }

  1. Validate with yarac:

   yarac rule.yar compiled.yarc

Undefined Identifier

Error: 
error: undefined identifier "pe.imphash"
Solution:
  1. Add module import:

   import "pe"
   import "hash"
   import "math"

   rule example {
     condition:
       pe.imphash() == "abc123"
   }

  1. Verify module is available in YARA build

Duplicate Rule Name

Error: 
error: duplicate rule identifier "malware_generic"
Solution:
  1. Rename conflicting rules
  2. Use namespaces:

   rule vendor1_malware_generic { ... }
   rule vendor2_malware_generic { ... }

String Too Short

Error: 
warning: string "$s1" is too short
Solution: 
rule example {
  strings:
    $s1 = "ab"  // Too short, may cause FPs
  condition:
    // Add additional conditions
    $s1 and filesize < 1MB
}

Memory Limit Exceeded

Error: 
error: too many strings in rule
error: string exceeds maximum length
Solution:
  1. Split into multiple rules
  2. Use more efficient patterns
  3. Increase memory limits:

   config :tamandua_server, TamanduaServer.Detection.YARA,
     max_strings_per_rule: 20000,
     stack_size: 32768

YARA Best Practices

rule well_formed_rule {
  meta:
    author = "Tamandua Team"
    description = "Example of well-formed rule"
    severity = "high"
    reference = "https://example.com"

  strings:
    $mz = { 4D 5A }  // MZ header
    $s1 = "suspicious_string" ascii wide nocase
    $s2 = /evil[0-9]{2}\.exe/ nocase

  condition:
    $mz at 0 and
    filesize < 5MB and
    any of ($s*)
}

Sigma Rule Issues

Common Errors

Invalid Logsource

Error: 
Invalid logsource: product 'unknown' not mapped
Solution: 
logsource:
  product: windows  # Use valid product
  service: security
  category: process_creation

Unsupported Modifier

Error: 
Unsupported modifier: 'fieldref'
Solution:

Check supported modifiers and use alternatives:

  • contains, startswith, endswith
  • re (regex)
  • base64, base64offset
  • cidr
  • gt, gte, lt, lte
  • nocase, wide, ascii

Complex Condition Not Parsed

Error: 
Failed to parse condition: complex nested expression
Solution:

Simplify condition:

# Instead of complex nesting
detection:
  sel1:
    FieldA: value1
  sel2:
    FieldB: value2
  filter1:
    FieldC: value3
  condition: (sel1 or sel2) and not filter1

# Break into simpler rules if needed

Sigma Validation

# Validate Sigma rules
sigma check /opt/tamandua/priv/sigma_rules/*.yml

# Convert and test
sigma convert -t tamandua /opt/tamandua/priv/sigma_rules/rule.yml

Detection Engine Performance

Slow Detection

Symptoms:
  • High detection latency
  • Backlog of events

Solution:
  1. Check detection queue depth:

   TamanduaServer.Detection.queue_depth()

  1. Increase parallelism:

   config :tamandua_server, TamanduaServer.Detection.Engine,
     pool_size: 10,  # Increase workers
     batch_size: 100

  1. Optimize expensive rules
  2. Use rule priorities

Rule Optimization

# Profile rule performance
TamanduaServer.Detection.profile_rules()
# => [
#   {"expensive_regex_rule", avg_ms: 150, calls: 1000},
#   {"simple_hash_rule", avg_ms: 2, calls: 5000}
# ]
Optimization Tips:
  1. Put fast conditions first (AND short-circuit)
  2. Use hashes before regex
  3. Limit regex complexity
  4. Use YARA modules efficiently

Next Steps