Packet Capture & Network Diagnostics with tcpdump

Q: How do I track down intermittent timeouts?

Run continuous measurements, such as mtr or a scheduled probe, and a rotating capture filtered to the affected traffic, then correlate the failure times with flow logs, load balancer metrics, and application logs. Intermittent problems often line up with deploys, scaling events, idle timeouts, or traffic peaks.

Packet Capture and Network Diagnostics

"It's the network" and "it's not the network" are both claims that need evidence. When a request times out, a connection resets, or a service is reachable from one place but not another, the fastest route to an answer is usually to stop guessing and look: probe each layer of the connection in turn, and when probes are not enough, capture the packets and see exactly what was sent, what came back, and what never arrived.

Packet capture is the ground truth of networking. Logs show what applications believe happened; a capture shows what actually crossed the wire — the SYN that was never answered, the reset from a firewall, the TLS alert explaining why a handshake failed, the retransmissions that turn a 50 ms request into a 3-second one. This guide covers a method for isolating network failures, the probes to use at each layer, and how to capture and read traffic safely.

TL;DR

Work layer by layer: name resolution, reachability and routing, transport, TLS, then the application.
Start with simple probes — dig, nc, curl with timings, mtr — before reaching for captures.
Capture with tcpdump using tight filters, write to a file, and analyze in Wireshark.
Learn the signatures: unanswered SYNs mean drops, a RST means rejection, retransmissions mean loss, a TLS alert explains a failed handshake.
In containers and Kubernetes, capture inside the pod's network namespace; in the cloud, use flow logs and traffic mirroring.
Captures contain secrets and personal data: capture only what you need, store them securely, and delete them afterwards.

Quick Example

"Requests to the payments API time out." Four commands narrow it down:

If nc fails and the capture shows SYN packets leaving with no reply — repeated at roughly 1, 2, and 4 seconds — something between the client and server is dropping them: a security group, a firewall, or a missing route. The application is not involved at all.

Core Concepts

A Layered Method

Confirm each layer before moving up. Change one thing at a time, and compare a working client with a failing one: the difference between them is usually the answer.

Probes and What They Prove

ping proves little when it fails: many networks block ICMP. A failed ping with a working TCP connection is normal.
traceroute and mtr show the path and where loss begins. Asterisks often mean a router does not answer probes, not that traffic is lost there; loss that starts at one hop and continues to the destination is the meaningful pattern.
nc -vz answers the most common question directly: can this port be reached from here?
curl timings split a slow request into DNS, connect, TLS, and server time.
ss -tanp shows local sockets: what is listening, and which connections are stuck in which state.

Where to Capture

A capture on one side shows only what that side saw. Capturing on both the client and the server shows where packets disappear: if a SYN leaves the client and never reaches the server, the problem is in between. Load balancers, NAT gateways, and proxies change addresses and ports, so filter on each hop's view of the connection.

Capturing With tcpdump

Useful habits: -nn avoids slow name lookups, -w writes raw packets for later analysis instead of printing them, and a precise filter keeps captures small and limits how much sensitive data you collect. The filter language is documented in the pcap-filter manual.

Reading Captures in Wireshark

Open the file in Wireshark and narrow down with display filters:

"Follow TCP Stream" reconstructs one conversation, and the Expert Information and Conversations views summarize problems across the whole capture.

Failure Signatures

DNS failures have their own signatures: NXDOMAIN for names that do not exist, SERVFAIL for upstream or DNSSEC problems, timeouts for unreachable resolvers, and different answers from different resolvers when split-horizon DNS is involved. See DNS.

Containers, Kubernetes, and the Cloud

Containers

Every container has its own network namespace, so a capture on the host's main interface may never see its traffic. Capture inside the namespace:

Kubernetes

All containers in a pod share one network namespace, so an ephemeral debug container sees the pod's traffic without changing the pod's image:

Also check the usual Kubernetes suspects: cluster DNS, network policies silently dropping traffic, and the difference between pod and service addresses. Clusters using eBPF-based networking often provide flow visibility tools that show dropped flows and the policy responsible. See Kubernetes Networking.

Cloud Networks

You cannot run tcpdump on a cloud provider's network itself. Use what the platform offers: flow logs record accepted and rejected connections per interface, traffic mirroring copies packets to an analysis instance, and reachability analyzers evaluate routes, security groups, and network ACLs between two points. A REJECT in a flow log next to an unanswered SYN in a capture is about as clear as evidence gets.

Captures Are Sensitive Data

Packet captures can contain passwords sent over plaintext protocols, session tokens, API keys in headers, and personal data in requests. Treat them accordingly:

Capture only what you need: tight filters, and a short snapshot length (-s 128) when headers are enough.
Get approval before capturing in production, following your organization's policy.
Store captures securely and delete them when the investigation ends.
Never upload captures to public analysis sites or attach them to public tickets without sanitizing them.
Decrypt TLS only for clients you control, for example by having a test client write session keys to a key-log file — and protect that file like a credential.

Best Practices

Reproduce With the Simplest Client

Before capturing an application's traffic, reproduce the failure with curl or nc from the same host. If the simple client works, the problem is likely in the application's configuration or connection handling; if it fails too, the network path is the suspect.

Keep a Toolbox Ready

Have a debug container image with dig, curl, mtr, tcpdump, and openssl, and permissions to use it, prepared before an incident. Installing tools on a production host mid-incident is slow and risky.

Synchronize Clocks

Comparing captures from two machines depends on accurate timestamps. Keep hosts synchronized with NTP so the sequence of events across captures is trustworthy.

Record Healthy Baselines

A capture of a healthy connection — handshake timing, typical segment sizes, normal round-trip times — makes anomalies obvious when something breaks.

Common Mistakes

Treating a Failed Ping as Proof

Capturing in the Wrong Place

Unbounded Captures

Sharing Raw Captures

FAQ

What is the difference between tcpdump and Wireshark?

tcpdump is a command-line capture tool available on almost every Unix-like server, ideal for collecting packets with precise filters. Wireshark is a graphical analyzer with protocol decoders, statistics, and stream reconstruction. A common workflow captures with tcpdump on the server and analyzes the file in Wireshark.

How do I capture traffic for a Kubernetes pod?

Attach an ephemeral debug container that includes tcpdump with kubectl debug, which shares the pod's network namespace, or enter the namespace from the node with nsenter. Capturing on the node's main interface may miss pod-to-pod traffic.

Can I see the contents of HTTPS traffic in a capture?

Not without the session keys. A capture shows the handshake, the server name the client requested, timing, and sizes. For clients you control, you can export session keys to a file that Wireshark uses to decrypt the traffic; protect that file carefully.

What does a TCP reset mean?

That one side, or a device in between, abruptly refused or terminated the connection. A reset in reply to a SYN usually means nothing is listening on the port or a firewall is rejecting it; a reset in the middle of a connection often comes from an idle timeout, a crashed process, or a middlebox.

How do I track down intermittent timeouts?

Run continuous measurements, such as mtr or a scheduled probe, and a rotating capture filtered to the affected traffic, then correlate the failure times with flow logs, load balancer metrics, and application logs. Intermittent problems often line up with deploys, scaling events, idle timeouts, or traffic peaks.