Kubernetes Troubleshooting: Pods, DNS & Services

Q: How do I debug a pod in CrashLoopBackOff?

Read the previous container's logs with kubectl logs --previous, check the last state's exit code in kubectl describe pod, and verify configuration, command, environment variables, mounted secrets, and memory limits. Exit code 137 usually points to an OOM kill or a liveness probe failure.

Q: How do I debug a container without a shell?

Use kubectl debug to attach an ephemeral container with debugging tools to the running pod. It shares the pod's network and, with --target, the process namespace of a specific container.

Kubernetes Troubleshooting

Kubernetes failures are rarely mysterious once you know where to look, but they are spread across many objects. A Deployment looks healthy while its pods crash. A Service exists, yet requests time out because its selector matches nothing. Pods start but never become ready because a probe points at the wrong port. DNS works from one namespace and not another. The cluster records nearly all of this — in pod status, events, logs, and endpoint objects — and a consistent order of investigation finds most problems in minutes.

This page walks through that order, the meaning of the most common failure states, and the commands that expose each layer: scheduling, image pulls, container startup, readiness, service routing, DNS, and the node underneath.

TL;DR

Start with kubectl get pods, then kubectl describe pod — the Events section explains most failures.
Logs: kubectl logs <pod> -c <container>, and --previous for the crashed instance.
ImagePullBackOff: wrong image name or tag, missing pull credentials, or registry limits.
CrashLoopBackOff: the process exits — read previous logs and the exit code (137 often means OOM).
Running but not Ready: readiness probe failing; check the probe path, port, and timing.
Service not reachable: check selector labels, EndpointSlices, targetPort, network policies, then DNS.

Quick Example

A first pass at a broken workload:

If the pod is fine but the service is not reachable:

An empty EndpointSlice means the Service selects no ready pods: compare its selector with the pod labels, then check readiness.

Core Concepts

The Investigation Order

Status: what state are pods in, how many restarts, on which nodes?
Events: what did the scheduler, kubelet, and controllers report?
Logs: what did the application say before failing?
Configuration: image, environment, mounts, probes, resources.
Networking: Service selectors, endpoints, ports, network policies, DNS.
Node: conditions, pressure, kubelet and runtime health.

Moving down the list only when the layer above looks healthy keeps investigations short.

Common Pod States

Exit Codes

Probes

Startup, liveness, and readiness probes are a frequent self-inflicted failure. A liveness probe with too short an initial delay kills slow-starting applications forever; a readiness probe on the wrong port keeps pods out of service; a liveness probe that checks dependencies restarts healthy pods when a database blips. Use a startup probe for slow starts, keep liveness checks local to the process, and put dependency checks in readiness, if anywhere.

Service Routing and DNS

A request to a Service passes through several objects, each of which can be wrong:

For DNS problems, test the fully qualified name first. If it resolves but the short name does not, the caller is in a different namespace. If nothing resolves, check CoreDNS pods, their logs, and whether network policies allow traffic to them on port 53. See Kubernetes Networking and DNS.

Debugging Tools

kubectl debug adds an ephemeral container with tools (for example netshoot or busybox) to a running pod, sharing its network namespace — essential for minimal or distroless images.
kubectl debug node/<name> -it --image=busybox opens a shell on a node with the host filesystem mounted at /host.
kubectl port-forward tests a pod or service directly from your machine, bypassing ingress.
kubectl get <obj> -o yaml shows the full, defaulted object, including status conditions controllers write.
kubectl auth can-i checks RBAC when controllers or service accounts report forbidden errors.

Best Practices

Read Events Before Guessing

Events from the scheduler, kubelet, and controllers usually name the exact problem. They expire after about an hour by default, so ship them to your logging system for later investigations.

Make Applications Explain Their Failures

Log configuration errors and failed dependency connections clearly at startup and exit with a non-zero code. A crash loop with a clear last log line is a five-minute fix.

Keep a Debug Image Ready

Pin an approved debugging image in your registry so engineers can use kubectl debug without pulling arbitrary images during an incident.

Write Runbooks for Recurring Failures

Document the checks for your platform's common failures — registry credentials, quota errors, certificate expiry — and link them from alerts.

Common Mistakes

Reading Logs From the Restarted Container

Liveness Probes That Check Dependencies

Selector and Label Drift

FAQ

How do I debug a pod in CrashLoopBackOff?

Read the previous container's logs with kubectl logs --previous, check the last state's exit code in kubectl describe pod, and verify configuration, command, environment variables, mounted secrets, and memory limits. Exit code 137 usually points to an OOM kill or a liveness probe failure.

What causes ImagePullBackOff?

The kubelet cannot pull the image: the name or tag is wrong, the registry requires credentials the pod lacks, the registry is rate limiting or unreachable, or the image has no build for the node's CPU architecture. The pod's Events show the exact error.

Why can't my pods reach a Service?

Usually the Service selector does not match ready pods (empty EndpointSlices), the targetPort is wrong, a NetworkPolicy blocks traffic, or the DNS name is wrong for the caller's namespace. Test from inside the cluster with a debug container.

How do I debug a container without a shell?

Use kubectl debug to attach an ephemeral container with debugging tools to the running pod. It shares the pod's network and, with --target, the process namespace of a specific container.

Why is my pod stuck in Terminating?

Common causes are finalizers waiting for a controller that is not running, a node that is unreachable so the kubelet cannot confirm termination, or a very long termination grace period. Inspect metadata.finalizers and the node's status before forcing deletion.