Network Troubleshooting with ss, netstat, and tcpdump

"The app can't connect to the database" could mean a dozen things: nothing is listening, it's listening on the wrong address, a firewall is dropping packets, DNS points at the wrong host, or the problem is somewhere else entirely. I used to jump straight to tcpdump and waste an hour. These days I work through three tools in a fixed order: ss and netstat for what's listening and connected on a host, tcpdump for what's actually on the wire.

This is the order I use, on CentOS 7 and Ubuntu 16.04/18.04. Install what's missing (sudo yum install -y net-tools tcpdump on CentOS, sudo apt-get install -y net-tools tcpdump on Ubuntu; ss ships with iproute and is already there) and use root or sudo, or you won't see process names.

Is anything listening?

ss is the modern replacement for netstat and is faster on hosts with many connections. The flags I type most often are -tlnp: TCP, listening, numeric ports, process:

sudo ss -tlnp

State   Recv-Q Send-Q Local Address:Port  Peer Address:Port
LISTEN  0      128    127.0.0.1:3306      *:*     users:(("mysqld",pid=1187,fd=30))
LISTEN  0      128    *:22                *:*     users:(("sshd",pid=956,fd=3))

That output already answers a frequent question: MySQL is bound to 127.0.0.1, so nothing on another host can reach it regardless of firewall rules. Add -u for UDP. The same information from netstat, for muscle memory or older systems:

sudo netstat -tlnp

What is connected right now?

Drop the -l to see established connections, and filter with a state or an expression:

sudo ss -tnp state established '( dport = :3306 or sport = :3306 )'
sudo ss -tn state syn-sent
sudo ss -tan | awk '{print $1}' | sort | uniq -c

The last line gives a count per state, which is how you notice several thousand TIME-WAIT or CLOSE-WAIT sockets, each pointing at a different kind of problem (short-lived connections without pooling, or an application not closing sockets, respectively). ss -s gives a one-screen summary.

To find which process holds a port when -p shows nothing useful:

sudo lsof -i :8080

Rule out routing and DNS

Before capturing packets, rule out the boring causes. It's embarrassing how often it's DNS:

ip route get 10.0.0.5
getent hosts db.example.com

ip route get shows the interface and source address the kernel will use; getent shows what the resolver actually returns, including /etc/hosts entries that dig would ignore.

Now tcpdump

When the socket view says the client is trying but nothing connects, watch the wire. I always start with a tight filter, since an unfiltered capture on a busy box is unreadable:

sudo tcpdump -i eth0 -nn port 3306

-nn disables name and port lookups, which otherwise slow the capture and clutter the output. Common variations:

sudo tcpdump -i any -nn host 10.0.0.5 and port 3306
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn) != 0 and dst port 3306'
sudo tcpdump -i eth0 -nn -c 100 -w /tmp/db.pcap port 3306

The second shows only SYN packets, so you see connection attempts without the noise of the data. The third writes 100 packets to a file you can open in Wireshark on your workstation; -s 0 used to be needed to capture full packets, but it has been the default for years.

Reading a SYN with no SYN-ACK in reply tells you the packet reached this host and something dropped it (firewall) or nothing was listening (you would then see a RST). Not seeing the SYN at all means it never arrived, so look upstream: a security group, a router ACL, or the client's own routing.

Then the local firewall

If tcpdump sees the SYN but there is no reply, the firewall is the usual suspect:

sudo firewall-cmd --list-all               # CentOS 7
sudo ufw status verbose                    # Ubuntu
sudo iptables -L INPUT -n -v --line-numbers

The -v counters in iptables show which rule is matching, which settles arguments quickly.

Confirm from the client side

After a fix, test from the client, not the server:

nc -zv db.example.com 3306

Connection to db.example.com 3306 port [tcp/mysql] succeeded! means the TCP handshake completed. Anything at the application layer after that is a different investigation.

Small annoyances

• ss: command not found on a minimal install. Install iproute (CentOS) or iproute2 (Ubuntu). netstat is in net-tools, which is no longer installed by default on CentOS 7.
• tcpdump: eth0: You don't have permission to capture on that device. Run with sudo. On CentOS 7 the interface is more likely called ens192 or eno1; ip link lists them, and -i any sidesteps the question.
• ss -tlnp shows users:(...) only for some sockets. The process column requires root to see other users' processes. Run with sudo.
• tcpdump shows traffic but the application still times out. Check for asymmetric routing: the reply may be leaving via a different interface, which -i any will reveal and ip route get will explain.

The order matters more than the flags. Most problems are solved before tcpdump ever comes out.