8 min read Updated May 9, 2026

Server Troubleshooting

This guide covers common Tamandua server issues and their solutions. The server is built on Elixir/Phoenix with Broadway pipelines for telemetry processing.

Server Not Starting

Symptoms

  • Server process exits immediately
  • Service fails to start
  • Health endpoint not responding

Diagnostic Steps

  1. Check service status

   systemctl status tamandua-server
   journalctl -u tamandua-server -n 100

  1. Check logs

   tail -100 /var/log/tamandua-server/app.log

  1. Verify configuration

   ./bin/tamandua_server eval "IO.inspect(Application.get_all_env(:tamandua_server))"

Common Causes and Solutions

Missing Environment Variables

Error: 
** (RuntimeError) environment variable DATABASE_URL is missing
Solution:

Set required environment variables:

export DATABASE_URL="ecto://user:pass@localhost/tamandua_prod"
export REDIS_URL="redis://localhost:6379"
export SECRET_KEY_BASE="your-64-char-secret-key"
export ML_SERVICE_URL="http://localhost:8000"

# Or in /etc/tamandua-server/env
DATABASE_URL=ecto://user:pass@localhost/tamandua_prod

Port Already in Use

Error: 
** (Mint.TransportError) could not listen on port 4000: eaddrinuse
Solution: 
# Find process using port
lsof -i :4000
netstat -tlnp | grep 4000

# Kill conflicting process or change port
export PHX_PORT=4001

Database Connection Failed at Startup

Error: 
** (DBConnection.ConnectionError) tcp connect (localhost:5432): connection refused
Solution: 
# Verify PostgreSQL is running
systemctl status postgresql
pg_isready -h localhost -p 5432

# Check credentials
psql -h localhost -U tamandua_user -d tamandua_prod -c "SELECT 1"

Migration Pending

Error: 
** (Ecto.MigrationError) migrations pending
Solution: 
./bin/tamandua_server eval "TamanduaServer.Release.migrate()"
# Or
mix ecto.migrate

Invalid Configuration

Error: 
** (ArgumentError) invalid configuration for :tamandua_server
Solution:
  1. Validate configuration:

   ./bin/tamandua_server eval "TamanduaServer.Config.validate!()"

  1. Check config/runtime.exs syntax
  2. Verify all required keys are present

OTP Release Issues

Error: 
{"init terminating in do_boot", {undef,[{...}]}}
Solution:
  1. Rebuild release:

   MIX_ENV=prod mix release --overwrite

  1. Verify Erlang/Elixir versions match build environment

Database Connection Errors

Symptoms

  • Requests failing with database errors
  • Slow queries
  • Connection pool exhaustion

Diagnostic Steps

  1. Check PostgreSQL status

   pg_isready
   systemctl status postgresql

  1. Check connections

   SELECT count(*) FROM pg_stat_activity WHERE datname = 'tamandua_prod';
   SELECT * FROM pg_stat_activity WHERE datname = 'tamandua_prod' AND state != 'idle';

Common Causes and Solutions

Connection Pool Exhausted

Error: 
** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 5000ms
Solution:
  1. Increase pool size in configuration:

   # config/runtime.exs
   config :tamandua_server, TamanduaServer.Repo,
     pool_size: String.to_integer(System.get_env("POOL_SIZE") || "20")

  1. Check for connection leaks:

   SELECT pid, usename, application_name, state, query_start, query
   FROM pg_stat_activity
   WHERE datname = 'tamandua_prod' AND state != 'idle';

Too Many Connections

Error: 
** (Postgrex.Error) FATAL 53300 (too_many_connections) sorry, too many clients already
Solution:
  1. Check max_connections in PostgreSQL:

   SHOW max_connections;

  1. Increase limit in postgresql.conf:

   max_connections = 200

  1. Or reduce application pool sizes

Slow Queries

Symptom: High response times, timeout errors Solution:
  1. Enable query logging:

   config :tamandua_server, TamanduaServer.Repo,
     log: :debug

  1. Check for missing indexes:

   SELECT relname, seq_scan, idx_scan
   FROM pg_stat_user_tables
   WHERE seq_scan > idx_scan
   ORDER BY seq_scan DESC;

  1. Analyze query plans:

   EXPLAIN ANALYZE SELECT * FROM alerts WHERE status = 'open';

Connection Timeout

Error: 
** (DBConnection.ConnectionError) tcp connect (db.example.com:5432): timeout
Solution:
  1. Check network connectivity:

   telnet db.example.com 5432

  1. Increase timeout:

   config :tamandua_server, TamanduaServer.Repo,
     connect_timeout: 30_000,
     timeout: 60_000

  1. Check firewall/security group rules

SSL Required

Error: 
** (Postgrex.Error) FATAL 28000 (invalid_authorization_specification) no pg_hba.conf entry for host
Solution: 
config :tamandua_server, TamanduaServer.Repo,
  ssl: true,
  ssl_opts: [
    verify: :verify_peer,
    cacertfile: "/path/to/ca.pem"
  ]

Redis Connection Issues

Symptoms

  • Cache misses
  • Session errors
  • Rate limiting not working
  • PubSub failures

Diagnostic Steps

  1. Check Redis status

   redis-cli ping
   systemctl status redis

  1. Check connection info

   redis-cli info clients
   redis-cli client list

Common Causes and Solutions

Connection Refused

Error: 
** (Redix.ConnectionError) tcp connect (localhost:6379): connection refused
Solution: 
# Start Redis
systemctl start redis

# Check binding
grep "bind" /etc/redis/redis.conf

# Verify port
redis-cli -h localhost -p 6379 ping

Authentication Failed

Error: 
** (Redix.Error) NOAUTH Authentication required
Solution: 
config :tamandua_server, :redis,
  host: "localhost",
  port: 6379,
  password: System.get_env("REDIS_PASSWORD")

Memory Limit Reached

Error: 
** (Redix.Error) OOM command not allowed when used memory > 'maxmemory'
Solution:
  1. Check memory usage:

   redis-cli info memory

  1. Increase memory limit in redis.conf:

   maxmemory 2gb
   maxmemory-policy allkeys-lru

  1. Or clear unnecessary keys

Slow Redis Operations

Symptom: High latency on Redis operations Solution:
  1. Check slow log:

   redis-cli slowlog get 10

  1. Avoid expensive commands (KEYS, SMEMBERS on large sets)
  2. Enable pipelining for batch operations

Redis Cluster Connection

Error: 
** (Redix.Error) MOVED 12345 node2:6379
Solution:

Use cluster-aware client:

config :tamandua_server, :redis,
  cluster: [
    [host: "node1", port: 6379],
    [host: "node2", port: 6379],
    [host: "node3", port: 6379]
  ]

Memory Issues

Symptoms

  • Server process killed by OOM
  • Degraded performance
  • Swap usage high

Diagnostic Steps

  1. Check memory usage

   free -m
   ps aux --sort=-%mem | head -10

  1. Check BEAM memory

   :erlang.memory()
   :recon.bin_leak(10)

Common Causes and Solutions

ETS Table Growth

Symptom: Memory steadily increasing Solution:
  1. Check ETS table sizes:

   :ets.all() |> Enum.map(fn t -> {t, :ets.info(t, :size), :ets.info(t, :memory)} end) |> Enum.sort_by(&elem(&1, 2), :desc) |> Enum.take(10)

  1. Implement TTL for cache tables
  2. Review agent registry cleanup

Process Heap Growth

Symptom: Individual processes consuming large memory Solution:
  1. Find large processes:

   :recon.proc_count(:memory, 10)

  1. Check for mailbox backlog:

   :recon.proc_count(:message_queue_len, 10)

  1. Implement process recycling

Binary Memory

Symptom: High binary memory, not being collected Solution:
  1. Check binary memory:

   :erlang.memory(:binary)

  1. Force GC on suspected processes:

   :erlang.garbage_collect(pid)

  1. Add periodic GC triggers

Atom Table Exhaustion

Error: 
** (SystemLimitError) a][] system limit has been reached
Solution:
  1. Increase atom limit:

   # In vm.args or release
   +t 5000000

  1. Avoid dynamic atom creation from user input

Large Queries

Symptom: Memory spikes during queries Solution:
  1. Use streaming for large result sets:

   Repo.stream(query) |> Stream.each(&process/1) |> Stream.run()

  1. Add LIMIT to queries
  2. Use pagination

Broadway Pipeline Errors

Symptoms

  • Telemetry not being processed
  • Events queued but not consumed
  • Detection delays

Diagnostic Steps

  1. Check pipeline status

   # In IEx console
   TamanduaServer.Telemetry.Ingestor.status()
   Broadway.all_running()

  1. Check RabbitMQ queues

   rabbitmqctl list_queues name messages consumers

Common Causes and Solutions

RabbitMQ Connection Failed

Error: 
** (AMQP.ConnectionError) cannot connect to RabbitMQ: econnrefused
Solution: 
# Start RabbitMQ
systemctl start rabbitmq-server

# Check status
rabbitmqctl status

# Verify credentials
rabbitmqctl authenticate_user tamandua password123

Queue Not Declared

Error: 
** (AMQP.Error) no queue 'telemetry' in vhost '/'
Solution: 
# Declare queue manually
rabbitmqadmin declare queue name=telemetry durable=true

# Or run setup
./bin/tamandua_server eval "TamanduaServer.Release.setup_queues()"

Consumer Crashed

Error: 
[error] Broadway processor crashed: ** (ArgumentError) invalid telemetry event
Solution:
  1. Check dead letter queue for failed messages:

   rabbitmqctl list_queues name=telemetry.dlq messages

  1. Review message format
  2. Add error handling in processor

Backpressure/Slow Processing

Symptom: Queue depth increasing Solution:
  1. Check consumer count:

   Broadway.producer_names(TamanduaServer.Telemetry.Ingestor)

  1. Increase concurrency:

   config :tamandua_server, TamanduaServer.Telemetry.Ingestor,
     processors: [
       default: [concurrency: 10]
     ],
     batchers: [
       default: [concurrency: 5, batch_size: 100]
     ]

  1. Scale horizontally with more nodes

Message Acknowledgment Issues

Error: 
[warning] Message not acknowledged within timeout
Solution:
  1. Increase ack timeout:

   config :tamandua_server, TamanduaServer.Telemetry.Ingestor,
     producers: [
       default: [
         module: {BroadwayRabbitMQ.Producer, [
           ack_timeout: 60_000
         ]}
       ]
     ]

  1. Check for blocking operations in processor

Pipeline Restart Loop

Symptom: Pipeline continuously restarting Solution:
  1. Check supervisor logs:

   grep "restarting" /var/log/tamandua-server/app.log

  1. Review crash reasons in logs
  2. Implement circuit breaker pattern

Phoenix Channel Issues

Symptoms

  • Agents cannot connect via WebSocket
  • Channels crash
  • Message delivery failures

Common Causes and Solutions

Channel Process Crashed

Error: 
[error] GenServer #PID<0.1234.0> terminating
** (RuntimeError) agent channel crashed
Solution:
  1. Review channel logs
  2. Add error handling in channel callbacks
  3. Implement supervision for channels

Too Many Connections

Error: 
[warning] Too many connections, rejecting new WebSocket
Solution:
  1. Check connection limit:

   length(Phoenix.Tracker.list(TamanduaServer.AgentTracker, "agents:lobby"))

  1. Increase limits:

   config :tamandua_server, TamanduaServerWeb.Endpoint,
     http: [
       transport_options: [max_connections: 100_000]
     ]

  1. Scale horizontally

Presence/Tracker Issues

Error: 
[error] Phoenix.Tracker failed to sync
Solution:
  1. Check node connectivity:

   Node.list()

  1. Verify PubSub configuration
  2. Check Redis PubSub adapter

Performance Optimization

General Optimization Tips

  1. Enable Erlang scheduler binding

   # In vm.args
   +sbt db

  1. Tune garbage collection

   +hms 8192
   +hmbs 8192

  1. Enable JIT (OTP 24+)

   ERL_FLAGS="+JMsingle true"

Monitoring Recommendations

# Add telemetry handlers
:telemetry.attach_many("tamandua-metrics", [
  [:tamandua, :repo, :query],
  [:tamandua, :broadway, :processor, :stop],
  [:phoenix, :channel_handled]
], &handle_event/4, nil)

Next Steps