Troubleshooting Overview

This guide helps you diagnose and resolve common issues with Tamandua Sentinel EDR. Whether you're experiencing agent connectivity problems, server errors, or detection issues, this section provides step-by-step troubleshooting procedures.

Quick Diagnostic Checklist

Before diving into specific troubleshooting guides, run through this quick checklist:

Check service status - Verify all components are running
Review recent changes - Configuration updates, deployments, network changes
Check resource utilization - CPU, memory, disk space
Examine logs - Look for error messages and patterns
Verify connectivity - Network paths between components

Common Issues

Issue Category	Common Symptoms	Guide
Agent Issues	Agent offline, high CPU, enrollment failures	Agent Troubleshooting
Server Issues	Server not starting, database errors, memory issues	Server Troubleshooting
Network Issues	Connection failures, timeout errors, certificate problems	Network Troubleshooting
Detection Issues	Missing alerts, false positives, rule errors	Detection Troubleshooting

Support Resources

Documentation

Agent Documentation - Agent installation and configuration
Server Documentation - Backend setup and management
API Reference - REST and WebSocket API documentation
Detection Rules - YARA, Sigma, and ML detection

Community Support

GitHub Issues: github.com/treant-lab/tamandua-community/issues
Discord Community: discord.gg/52Duv5GVDF
Discussion Forum: github.com/treant-lab/tamandua-community/discussions

Commercial Support

For enterprise customers with support contracts:

Support Portal: support.tamandua.io
Email: support@tamandua.io
Priority Hotline: Available for critical issues (SLA-dependent)

Log Locations

Agent Logs

Platform	Default Log Location
Windows	`C:\ProgramData\Tamandua\logs\tamandua-agent.log`
Linux	`/var/log/tamandua/tamandua-agent.log`
macOS	`/Library/Application Support/Tamandua/logs/tamandua-agent.log`

Agent logs rotate automatically at 100MB with 5 retained files.

Server Logs

Component	Location
Phoenix Application	`/var/log/tamandua-server/app.log`
Broadway Pipeline	`/var/log/tamandua-server/broadway.log`
Detection Engine	`/var/log/tamandua-server/detection.log`
Access Logs	`/var/log/tamandua-server/access.log`
Systemd Journal	`journalctl -u tamandua-server`

ML Service Logs

Component	Location
FastAPI	`/var/log/tamandua-ml/api.log`
Model Inference	`/var/log/tamandua-ml/inference.log`
Training Jobs	`/var/log/tamandua-ml/training.log`

Infrastructure Logs

Service	Command/Location
PostgreSQL	`journalctl -u postgresql` or `/var/log/postgresql/`
Redis	`journalctl -u redis` or `/var/log/redis/`
RabbitMQ	`/var/log/rabbitmq/`
Docker	`docker logs <container_name>`

Diagnostic Tools

Agent Diagnostics

# Check agent status
tamandua-agent --status

# Run connectivity test
tamandua-agent --test-connection

# Generate diagnostic bundle
tamandua-agent --diagnostics > diagnostics.json

# Verbose logging mode
RUST_LOG=debug tamandua-agent --server wss://your-server:4000/socket/agent

Server Diagnostics

# Check server health
curl https://your-server:4000/api/health

# Detailed health check
curl https://your-server:4000/api/health/detailed

# Check connected agents
curl -H "Authorization: Bearer $TOKEN" \
  https://your-server:4000/api/v1/agents/status

# Phoenix remote console
./bin/tamandua_server remote

Database Diagnostics

# Check PostgreSQL connections
psql -c "SELECT count(*) FROM pg_stat_activity WHERE datname = 'tamandua_prod';"

# Check table sizes
psql -c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid))
         FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC;"

# Check for locks
psql -c "SELECT * FROM pg_locks WHERE NOT granted;"

Network Diagnostics

# Test WebSocket connection
websocat wss://your-server:4000/socket/agent

# Check TLS certificate
openssl s_client -connect your-server:4000 -showcerts

# Test connectivity
curl -v https://your-server:4000/api/health

# DNS resolution
nslookup your-server
dig your-server

Redis Diagnostics

# Check Redis connectivity
redis-cli ping

# Monitor Redis commands
redis-cli monitor

# Check memory usage
redis-cli info memory

# List connected clients
redis-cli client list

RabbitMQ Diagnostics

# Check queue status
rabbitmqctl list_queues

# Check connections
rabbitmqctl list_connections

# Check consumers
rabbitmqctl list_consumers

# Management API health
curl -u guest:guest http://localhost:15672/api/healthchecks/node

Diagnostic Commands Reference

Agent Commands

Command	Description
`tamandua-agent --status`	Show agent status and configuration
`tamandua-agent --test-connection`	Test server connectivity
`tamandua-agent --diagnostics`	Generate diagnostic bundle
`tamandua-agent --version`	Show agent version
`tamandua-agent --list-collectors`	List active collectors
`tamandua-agent --validate-config`	Validate configuration file

Server Commands (IEx)

# Check agent registry
TamanduaServer.Agents.Registry.list_agents()

# Check detection engine status
TamanduaServer.Detection.Engine.status()

# Check Broadway pipeline
TamanduaServer.Telemetry.Ingestor.status()

# Reload detection rules
TamanduaServer.Detection.reload_rules()

# Check connected WebSocket channels
Phoenix.Presence.list("agents:lobby")

Environment Verification

# Required environment variables
echo "DATABASE_URL: $DATABASE_URL"
echo "REDIS_URL: $REDIS_URL"
echo "ML_SERVICE_URL: $ML_SERVICE_URL"
echo "SECRET_KEY_BASE: $(echo $SECRET_KEY_BASE | cut -c1-10)..."

# Check required services
systemctl status tamandua-server
systemctl status postgresql
systemctl status redis
systemctl status rabbitmq-server

Log Analysis Tips

Searching Logs

# Search for errors in agent logs
grep -i "error\|fail\|panic" /var/log/tamandua/tamandua-agent.log

# Search for connection issues
grep -i "websocket\|connect\|timeout" /var/log/tamandua/tamandua-agent.log

# Tail logs in real-time
tail -f /var/log/tamandua-server/app.log | grep -i error

# Search with context
grep -B5 -A5 "connection refused" /var/log/tamandua-server/app.log

Log Levels

Level	Description	When to Use
`error`	Critical failures requiring attention	Always review
`warn`	Potential issues that may need investigation	Regular review
`info`	Normal operational messages	General monitoring
`debug`	Detailed diagnostic information	Active troubleshooting
`trace`	Very detailed execution trace	Deep debugging

Enabling Debug Logging

Agent (Rust):

RUST_LOG=debug tamandua-agent --server wss://...
RUST_LOG=tamandua_agent::collectors=trace tamandua-agent --server wss://...

Server (Elixir):

# In config/runtime.exs
config :logger, level: :debug

ML Service (Python):

LOG_LEVEL=DEBUG uvicorn tamandua_ml.api:app

Collecting Support Information

When contacting support, collect the following information:

Required Information

Tamandua versions

Agent version (tamandua-agent --version)

Server version (mix tamandua.version)
ML Service version

Environment details

Operating system and version

Infrastructure (cloud provider, Docker, Kubernetes)
Database version

Logs

Relevant log files (last 24 hours)

Diagnostic bundle (tamandua-agent --diagnostics)

Issue description

When the issue started

Steps to reproduce
Error messages (exact text)
Screenshots if applicable

Generating Support Bundle

# Agent support bundle
tamandua-agent --diagnostics --output support-bundle.tar.gz

# Server support bundle (run in IEx)
TamanduaServer.Support.generate_bundle("/tmp/support-bundle")

Next Steps

Agent Troubleshooting - Resolve agent connectivity and performance issues
Server Troubleshooting - Fix server startup and database problems
Network Troubleshooting - Diagnose connectivity and certificate issues
Detection Troubleshooting - Address detection and rule problems
Frequently Asked Questions - Common questions and answers