Remote Syslog Troubleshooting

Connectivity Issues

Problem Definition

Error Remote syslog server is not accessible, syslog transport disabled

When starting the server, you see a warning message in the logs:

Remote syslog server syslog.example.com:514 is not accessible, syslog transport disabled

Or:

Failed to check syslog server accessibility (syslog.example.com:514): Connection timeout, syslog transport disabled

Solution

1. Verify Syslog Server is Running

Ensure the remote syslog server is running and accessible:

# Check if syslog server is listening on port 514
netstat -tlnp | grep 514
# or
ss -tlnp | grep 514

2. Test TCP Connectivity

For TCP protocol, test the connection manually:

# Using telnet (if installed)
telnet syslog.example.com 514

# Using nc (netcat)
nc -vz syslog.example.com 514

# Using curl
curl -v telnet://syslog.example.com:514

3. Test UDP Connectivity

For UDP protocol, send a test packet:

# Using echo and nc (UDP mode)
echo "test" | nc -u -v syslog.example.com 514

# Using logger (if available)
logger -n syslog.example.com -P 514 "test message"

4. Check Firewall Rules

Ensure no firewall is blocking the connection:

On the server sending logs:

# Check outbound rules
sudo ufw status
sudo iptables -L OUTPUT -n -v

On the syslog server:

# Check inbound rules
sudo ufw status
sudo iptables -L INPUT -n -v

# Allow syslog traffic (UDP/TCP port 514)
sudo ufw allow 514/udp
sudo ufw allow 514/tcp

5. Verify Configuration

Check your REMOTE_SYSLOG_* environment variables:

# Check if variables are set
env | grep REMOTE_SYSLOG

# Should show:
# REMOTE_SYSLOG_ENABLED=true
# REMOTE_SYSLOG_HOST=syslog.example.com
# REMOTE_SYSLOG_PORT=514
# REMOTE_SYSLOG_PROTOCOL=udp

6. Test with Docker Compose

If using the development setup with built-in syslog server:

# Start with syslog server profile
docker compose --profile serverClient -f docker-compose.dev.yml up

# Check if syslog server is running
docker ps | grep monitor-syslog

# Check syslog server logs
docker logs monitor-syslog

# Check server logs for syslog transport
docker logs monitor-server | grep -i syslog

7. Enable Verbose Logging

Set higher log level to see detailed syslog transport logs:

# In your .env file
LOG_LEVEL=debug

8. Check Syslog Server Configuration

Verify the syslog server is configured to accept connections:

For rsyslog (/etc/rsyslog.conf):

# Ensure UDP module is loaded
$ModLoad imudp
$UDPServerAddress 0.0.0.0
$UDPServerRun 514

# Ensure TCP module is loaded (if using TCP)
$ModLoad imtcp
$InputTCPServerAddress 0.0.0.0
$InputTCPServerRun 514

For syslog-ng (/etc/syslog-ng/syslog-ng.conf):

# Ensure UDP source is configured
source s_udp {
    udp(port(514));
};

# Ensure TCP source is configured (if using TCP)
source s_tcp {
    tcp(port(514));
};

9. Verify Server Restart

After changing REMOTE_SYSLOG_* variables, restart the entire server:

# Docker Compose
docker compose restart server

# Or full restart
docker compose --profile serverClient -f docker-compose.dev.yml down
docker compose --profile serverClient -f docker-compose.dev.yml up -d

Logs Not Appearing in Syslog Server

Problem Definition

Server logs are not appearing in the remote syslog server, even though no error messages are shown.

Solution

1. Verify Transport is Enabled

Check server startup logs for confirmation:

# Check if syslog transport was initialized
docker logs monitor-server | grep -i "syslog"

You should see either:

• Success message (if using custom logging)
• No warnings about syslog being disabled

2. Generate Test Log

Force the server to generate a log entry:

# Make a request to the server
curl http://localhost:5000/health

# Check if logs are being generated
docker logs monitor-server | tail -50

3. Check Syslog Message Format

Verify the syslog server is receiving formatted messages:

# For UDP with netcat
nc -l -u -p 514

# For TCP with netcat
nc -l -p 514

# For rsyslog, check the log file
tail -f /var/log/syslog
tail -f /var/log/messages

4. Verify Facility and Severity

Check if messages are being filtered by facility or severity:

# Check syslog configuration for facility filtering
grep -r "local0" /etc/rsyslog.d/

# If using local0 facility, ensure it's not filtered
# Add to /etc/rsyslog.conf or /etc/rsyslog.d/*.conf:
local0.* /var/log/monitor_syslog.log

5. Check Message Size Limits

Syslog has a 1024-byte limit for RFC3164. If messages are too large:

# Or reduce log verbosity
LOG_LEVEL=warn

6. Monitor UDP Packet Loss

For UDP protocol, packets may be lost:

# On syslog server, monitor UDP traffic
tcpdump -i any -n udp port 514

# Check for dropped packets
netstat -s | grep -i udp

Recommendation: Switch to TCP for reliable delivery:

# In .env file
REMOTE_SYSLOG_PROTOCOL=tcp

7. Verify Application Logs

Check if the server is actually logging:

# Check application log files
tail -f /var/log/monitor/monitor_server*

# Check if logs contain expected entries
grep "info" /var/log/monitor/monitor_server* | tail -20

8. Test Direct Syslog Output

Send a test message directly to the syslog server:

# Using logger command
logger -p local0.info "Test message from server"

# Using echo and nc
echo "Test message" | nc -v syslog.example.com 514

9. Check for Rate Limiting

Some syslog servers rate-limit incoming messages:

# Check rsyslog configuration for rate limiting
grep -r "rate-limit" /etc/rsyslog.d/

# Temporarily disable rate limiting for testing
# Add to /etc/rsyslog.conf:
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0

Common Errors

ERR_SOCKET_DGRAM_NOT_RUNNING

Error ERR_SOCKET_DGRAM_NOT_RUNNING

Error [ERR_SOCKET_DGRAM_NOT_RUNNING]: Not running
    at healthCheck (node:dgram:965:11)
    at Socket.close (node:dgram:790:3)

Solution

This error occurs when closing a UDP socket that has already been closed or is not running.

Fix: The logger implementation has been updated to prevent this error by:

1. Using a closed flag to track socket state
2. Only closing the socket once (either on send callback or timeout)

No action required if you're using the latest version. If you see this error, ensure you're running the latest code from the repository.

Connection timeout

Error connection timeout

Remote syslog server syslog.example.com:514 is not accessible, syslog transport disabled

Solution

This means the server could not connect to the syslog server within 2 seconds.

Possible causes:

1. Syslog server is not running
2. Firewall is blocking the connection
3. Wrong host or port configuration
4. Network connectivity issues

Troubleshooting steps:

1. Verify syslog server is running
2. Check firewall rules
3. Verify REMOTE_SYSLOG_HOST and REMOTE_SYSLOG_PORT configuration
4. Test connectivity manually (see Connectivity Issues)

Protocol mismatch

Error protocol mismatch

Messages not received when using UDP, but work with TCP (or vice versa).

Solution

Check syslog server configuration:

1. For UDP:

   • Ensure UDP module is loaded in rsyslog/syslog-ng
   • Verify UDP port 514 is open

2. For TCP:

   • Ensure TCP module is loaded
   • Verify TCP port 514 is open

Recommendation: Use UDP for high-volume logging, TCP for reliable delivery.

# Test both protocols
# UDP
echo "test" | nc -u syslog.example.com 514

# TCP
echo "test" | nc syslog.example.com 514

Debug Checklist

Debug Checklist

Use this checklist to systematically troubleshoot remote syslog issues:

• 1. Verify Environment Variables

  env | grep REMOTE_SYSLOG

  Should show:

  • REMOTE_SYSLOG_ENABLED=true
  • REMOTE_SYSLOG_HOST=<your-syslog-server>
  • REMOTE_SYSLOG_PORT=514
  • REMOTE_SYSLOG_PROTOCOL=udp|tcp

• 2. Check Server Logs

  docker logs monitor-server | grep -i syslog

  Look for:

  • No warnings about syslog being disabled
  • No connection errors

• 3. Test Connectivity

  nc -vz <syslog-host> 514

• 4. Verify Syslog Server

  docker ps | grep monitor-syslog
  docker logs monitor-syslog

• 5. Check Firewall

  sudo ufw status
  sudo iptables -L INPUT -n -v

• 6. Test Direct Logging

  logger -n <syslog-host> -P 514 "test"

• 7. Verify Server Restart

  • After any configuration change, restart the entire server
  • Logging is initialized at startup and cannot be changed dynamically

• 8. Check Syslog Server Logs

  tail -f /var/log/syslog
  tail -f /var/log/messages

• 9. Monitor UDP Traffic (if using UDP)

  tcpdump -i any -n udp port 514

• 10. Enable Verbose Logging

  LOG_LEVEL=debug

Additional Resources

• RFC 3164 - BSD Syslog