Network Troubleshooting

This guide covers network-related issues affecting communication between Tamandua components, including WebSocket connections, TLS/mTLS, firewalls, and proxies.

WebSocket Connection Failures

Symptoms

• Agents show as offline
• Connection drops frequently
• WebSocket upgrade fails

Diagnostic Steps

1. Test WebSocket connectivity

   # Using websocat
   websocat wss://server:4000/socket/agent

   # Using wscat
   wscat -c wss://server:4000/socket/agent
   

1. Check HTTP upgrade

   curl -v -H "Connection: Upgrade" -H "Upgrade: websocket" \
     https://server:4000/socket/agent
   

Common Causes and Solutions

WebSocket Upgrade Rejected

Error:

HTTP/1.1 400 Bad Request
error: missing upgrade header

Solution:

1. Verify request headers include:

   Connection: Upgrade
   Upgrade: websocket
   Sec-WebSocket-Key: <base64-key>
   Sec-WebSocket-Version: 13
   

1. Check reverse proxy passes WebSocket headers

Load Balancer Not Supporting WebSockets

Error:

HTTP/1.1 502 Bad Gateway
WebSocket connection failed: upstream closed connection

Solution (AWS ALB):

# Target group settings
stickiness.enabled: true
stickiness.type: lb_cookie

Solution (nginx):

location /socket {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_read_timeout 86400;
}

Solution (HAProxy):

frontend https
    bind *:443 ssl crt /etc/ssl/tamandua.pem
    acl is_websocket hdr(Upgrade) -i websocket
    use_backend websocket_backend if is_websocket
    default_backend http_backend

backend websocket_backend
    timeout server 86400s
    timeout tunnel 86400s
    server srv1 backend1:4000 check

Connection Timeout

Error:

WebSocket connection timeout after 30s

Solution:

1. Increase timeout in agent configuration:

   [transport]
   connect_timeout_secs = 60
   

1. Check for network latency:

   ping server.example.com
   traceroute server.example.com
   

1. Verify no rate limiting on connection establishment

Frequent Disconnections

Error:

WebSocket disconnected: ping timeout
Connection lost, attempting reconnect (attempt 5/10)

Solution:

1. Adjust ping/pong intervals:

   # Server config
   config :tamandua_server, TamanduaServerWeb.Endpoint,
     http: [
       transport_options: [
         idle_timeout: 120_000
       ]
     ]
   

   # Agent config
   [transport]
   ping_interval_secs = 30
   ping_timeout_secs = 60
   

1. Check for aggressive NAT timeouts
2. Verify load balancer idle timeout settings

Firewall Configuration

Required Ports

Firewall Rules

Linux (iptables)

# Allow outbound from agent to server
iptables -A OUTPUT -p tcp --dport 4000 -j ACCEPT

# Allow inbound to server
iptables -A INPUT -p tcp --dport 4000 -j ACCEPT

Linux (firewalld)

# On server
firewall-cmd --permanent --add-port=4000/tcp
firewall-cmd --reload

Windows Firewall

# Allow Tamandua agent outbound
New-NetFirewallRule -DisplayName "Tamandua Agent" `
  -Direction Outbound -Program "C:\Program Files\Tamandua\tamandua-agent.exe" `
  -Action Allow

# Allow server inbound
New-NetFirewallRule -DisplayName "Tamandua Server" `
  -Direction Inbound -LocalPort 4000 -Protocol TCP `
  -Action Allow

AWS Security Groups

{
  "SecurityGroupIngress": [
    {
      "IpProtocol": "tcp",
      "FromPort": 4000,
      "ToPort": 4000,
      "CidrIp": "10.0.0.0/8",
      "Description": "Tamandua agent connections"
    }
  ]
}

Troubleshooting Firewall Issues

Symptoms:

• Connection refused or timeout
• Works from some networks but not others

Diagnostic Steps:

# Test connectivity
telnet server.example.com 4000
nc -zv server.example.com 4000

# Check if port is listening
netstat -tlnp | grep 4000

# Test from agent network
curl -v https://server.example.com:4000/api/health

Proxy Issues

HTTP Proxy Configuration

Agent Configuration

[transport]
proxy_url = "http://proxy.corp.com:8080"
proxy_username = "user"
proxy_password = "pass"

Environment Variables

# HTTP proxy
export HTTP_PROXY="http://proxy.corp.com:8080"
export HTTPS_PROXY="http://proxy.corp.com:8080"
export NO_PROXY="localhost,127.0.0.1,.internal.corp.com"

Common Proxy Issues

Proxy Authentication Required

Error:

HTTP/1.1 407 Proxy Authentication Required

Solution:

1. Configure proxy credentials:

   [transport]
   proxy_url = "http://user:pass@proxy.corp.com:8080"
   

1. Or use NTLM authentication:

   [transport]
   proxy_url = "http://proxy.corp.com:8080"
   proxy_auth = "ntlm"
   proxy_domain = "CORP"
   proxy_username = "user"
   proxy_password = "pass"
   

Proxy Not Supporting WebSocket

Error:

WebSocket upgrade failed through proxy
HTTP/1.1 400 Bad Request

Solution:

1. Use CONNECT method (HTTPS tunneling):

   [transport]
   proxy_tunnel = true
   

1. Configure proxy to allow CONNECT to port 4000
2. Consider direct connection bypass for internal servers

Proxy SSL Interception

Error:

Certificate verification failed: issuer not trusted
SSL certificate signed by: "Corp Proxy CA"

Solution:

1. Add corporate CA to trust store:

   # Linux
   sudo cp corp-ca.pem /usr/local/share/ca-certificates/corp-ca.crt
   sudo update-ca-certificates

   # Windows
   certutil -addstore -f "ROOT" corp-ca.pem
   

1. Or specify CA in agent config:

   [transport]
   ca_cert_path = "/etc/tamandua/corp-ca.pem"
   

Proxy Timeout

Error:

Proxy connection timeout

Solution:

1. Increase proxy timeout:

   [transport]
   proxy_connect_timeout_secs = 60
   

1. Check proxy server health
2. Verify proxy allows long-lived connections

Transparent Proxy Detection

# Check if connections are being proxied
curl -v https://server.example.com:4000/api/health 2>&1 | grep -i "proxy\|via"

# Compare certificate chains
openssl s_client -connect server.example.com:4000 -showcerts

mTLS Handshake Failures

Symptoms

• TLS handshake fails
• Certificate rejected by server
• Agent cannot authenticate

Diagnostic Steps

1. Test TLS connection

   openssl s_client -connect server:4000 \
     -cert client.pem -key client-key.pem -CAfile ca.pem
   

1. Verify certificate

   openssl x509 -in client.pem -noout -text
   openssl verify -CAfile ca.pem client.pem
   

Common Causes and Solutions

Client Certificate Not Presented

Error:

Server: SSL peer did not return a certificate
Agent: SSL handshake failed

Solution:

Configure client certificate:

[transport]
client_cert_path = "/etc/tamandua/agent.pem"
client_key_path = "/etc/tamandua/agent-key.pem"

Certificate Chain Incomplete

Error:

verify error:num=21:unable to verify the first certificate

Solution:

1. Include intermediate certificates in cert file:

   cat agent.pem intermediate.pem > agent-chain.pem
   

1. Configure full chain:

   [transport]
   client_cert_path = "/etc/tamandua/agent-chain.pem"
   

Certificate Expired

Error:

verify error:num=10:certificate has expired

Solution:

1. Check expiration:

   openssl x509 -in agent.pem -noout -dates
   

1. Renew certificate and redeploy
2. Set up certificate rotation automation

Certificate Revoked

Error:

verify error:num=23:certificate revoked

Solution:

1. Check CRL/OCSP:

   openssl verify -crl_check -CAfile ca.pem -CRLfile crl.pem agent.pem
   

1. Issue new certificate
2. Review revocation reason

Wrong CA

Error:

verify error:num=20:unable to get local issuer certificate

Solution:

1. Verify certificate was signed by expected CA:

   openssl verify -CAfile server-ca.pem agent.pem
   

1. Update CA certificate on server or agent

Key Mismatch

Error:

error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch

Solution:

1. Verify key matches certificate:

   openssl x509 -noout -modulus -in agent.pem | md5sum
   openssl rsa -noout -modulus -in agent-key.pem | md5sum
   # Must match
   

1. Regenerate certificate with correct key

CN/SAN Mismatch

Error:

Agent certificate CN does not match agent_id

Solution:

1. Check certificate CN and SAN:

   openssl x509 -in agent.pem -noout -subject -ext subjectAltName
   

1. Generate certificate with correct CN matching agent_id

Certificate Generation

# Generate agent certificate
openssl req -new -nodes -keyout agent-key.pem -out agent.csr \
  -subj "/CN=agent-uuid-here/O=Tamandua"

# Sign with CA
openssl x509 -req -in agent.csr -CA ca.pem -CAkey ca-key.pem \
  -CAcreateserial -out agent.pem -days 365

DNS Resolution

Symptoms

• Hostname resolution fails
• Intermittent connectivity based on DNS
• Wrong server IP returned

Diagnostic Steps

1. Test resolution

   nslookup server.example.com
   dig server.example.com
   host server.example.com
   

1. Check resolv.conf

   cat /etc/resolv.conf
   

Common Causes and Solutions

DNS Server Unreachable

Error:

Failed to resolve hostname: server.example.com
connection timed out; no servers could be reached

Solution:

1. Check DNS server:

   dig @8.8.8.8 server.example.com
   

1. Update DNS configuration:

   # /etc/resolv.conf
   nameserver 8.8.8.8
   nameserver 8.8.4.4
   

NXDOMAIN (Host Not Found)

Error:

Host server.example.com not found: NXDOMAIN

Solution:

1. Verify hostname spelling
2. Check if internal DNS is required
3. Add to hosts file as workaround:

   # /etc/hosts
   10.0.1.100 server.example.com
   

DNS Cache Poisoning/Stale

Error:

Resolved to wrong IP address

Solution:

1. Flush DNS cache:

   # Linux (systemd-resolved)
   resolvectl flush-caches

   # Windows
   ipconfig /flushdns

   # macOS
   sudo dscacheutil -flushcache
   

1. Lower TTL for entries that change frequently

Split-Horizon DNS Issues

Error:

Different results from internal vs external

Solution:

1. Configure agent to use correct DNS server
2. Use IP address directly in configuration
3. Verify DNS zone configuration

DNS Configuration for Agent

# Use specific DNS servers
[network]
dns_servers = ["10.0.0.53", "10.0.0.54"]

# Or use system DNS with fallback
[network]
dns_fallback = ["8.8.8.8", "1.1.1.1"]

Network Debugging Tools

tcpdump

# Capture WebSocket traffic
tcpdump -i eth0 -w capture.pcap port 4000

# Capture with filter
tcpdump -i eth0 -n 'host server.example.com and port 4000'

Wireshark Filters

# WebSocket traffic
websocket

# TLS handshake issues
ssl.handshake.type == 1 || ssl.alert_message

# Filter by server
ip.addr == 10.0.1.100 and tcp.port == 4000

ss/netstat

# Check established connections
ss -tnp | grep 4000

# Check connection states
netstat -an | grep 4000 | awk '{print $6}' | sort | uniq -c

curl Verbose Mode

# Full connection debug
curl -v --trace-ascii /dev/stdout https://server:4000/api/health

# With client certificate
curl -v --cert agent.pem --key agent-key.pem https://server:4000/api/health

Network Performance Issues

High Latency

Symptoms:

• Slow response times
• Frequent timeouts

Solution:

1. Measure latency:

   ping server.example.com
   mtr server.example.com
   

1. Increase timeouts
2. Consider geographic proximity
3. Use CDN for static content

Packet Loss

Symptoms:

• Intermittent failures
• Retransmissions

Solution:

1. Measure packet loss:

   ping -c 100 server.example.com | grep loss
   mtr --report server.example.com
   

1. Investigate network path
2. Contact network team/ISP

MTU Issues

Error:

Packet too large, fragmentation needed

Solution:

1. Discover MTU:

   ping -M do -s 1472 server.example.com
   

1. Configure MSS clamping
2. Adjust MTU on interfaces

Next Steps

• Agent Troubleshooting - Agent-specific issues
• Server Troubleshooting - Backend server issues
• Detection Troubleshooting - Detection rule issues
• Troubleshooting Overview - General diagnostic tools