# The Ultimate Guide to Kubernetes Troubleshooting ### **Introduction** A critical pod fails. Your alerts are firing. What's next? Every DevOps/SRE engineer needs a clear plan to handle cluster issues. This guide provides a systematic strategy and the essential tools to diagnose any problem, from pod health to the control plane. ## The First Steps: Observing the Problem When a pod is failing, your first task is to become a detective. You need to gather clues and observations before you jump to conclusions. Fortunately, Kubernetes provides a suite of essential commands for this initial investigation. Mastering these fundamentals will save you a lot of time and frustration. ### **1\. The Overview: kubectl get pods** The most basic used command is kubectl get pods. It provides a quick, high-level snapshot of your pods’ current state. ```bash kubectl get pods -n ``` By running this, you can immediately see: * The pod's STATUS (Running, Pending, CrashLoopBackOff, etc.). * The number of times the pod has RESTARTED. * How long the pod has been running (AGE). A pod in a CrashLoopBackOff state is a clear sign of a repeated failure, while a pod stuck in Pending indicates a scheduling issue. ### **2\. The Details: kubectl describe pod** Once you have a suspicious-looking pod, the next step is to run a more detailed inspection with kubectl describe pod. ```bash kubectl describe pod -n ``` This command provides a wealth of information, including: * **Events:** A list of recent events related to the pod, which can show you exactly why a container failed to start or why a volume couldn't be mounted. * **Containers:** Details about the containers within the pod, including their image, state, and restartCount. * **Status & Exit Reason:** Details about the containers within the pod, their image, and most importantly, the State and Exit Code if a container has terminated. An exit code of 1 or a specific error message can be your first solid clue. ```plaintext Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m1s default-scheduler Successfully assigned my-app-pod to node-01 Normal Pulling 4m kubelet Pulling image "my-app:v1" Normal Pulled 3m59s kubelet Successfully pulled image "my-app:v1" in 987ms Normal Created 3m59s kubelet Created container my-app-container Normal Started 3m59s kubelet Started container my-app-container Warning Failed 3m1s (x3 over 3m1s) kubelet Failed to pull image "my-app:v2": rpc error: code = NotFound ``` ### 3. **The Output: kubectl logs** The kubectl logs command is your window into what's happening inside the container itself. ```bash kubectl logs -n ``` By default, this shows you the standard output and error streams from your container. It's the go-to command for finding application-level errors, startup failures, or unexpected behavior. ## Advanced Pod Troubleshooting: When a Pod is Running but Not Responding What do you do when kubectl get pods shows a pod is Running, but your application still isn't working? ### **1\. The First Test: Is the Application Itself Unresponsive?** The quickest way to check if your application is simply hung is by trying to access it from a shell inside the container. This tests the application directly, bypassing the Kubernetes service layer and network policies. Use `kubectl exec` to run a simple connectivity test. ```bash kubectl exec -it -n -- curl localhost: ``` If the `curl` command fails to connect or gets a timeout, you know the problem is within the container itself. If it returns a correct response, the issue is likely with the Kubernetes networking or service configuration. ### **2\. The Network Check: Can the Pod Reach Out?** If the pod is supposed to be making external connections but isn't, you need to check its network connectivity. Since many images don't contain standard network tools, you can use kubectl exec to run a few commands if available. * **DNS Resolution:** Use `nslookup` or `dig` to check if the pod can resolve a service name or an external domain. * **Connectivity:** Use `ping`, `nc`, or `curl` to test connectivity to a service, database, or external API. For example, to check DNS resolution for a service: ```bash # If the service is in the same namespace as the pod kubectl exec -it -n -- nslookup # If the service is in a different namespace, # use the fully qualified domain name (FQDN) for reliability. kubectl exec -it -n -- nslookup ..svc.cluster.local ``` If these tests fail, it points to a network configuration issue (like a misconfigured DNS or Network Policy) rather than an application-level bug. ### **3\. Process and Resource Checks** Even if the application is reachable, its internal state might be unhealthy. You can use process inspection tools to see if a process has a high CPU or is hung. ```bash # Check running processes and their status kubectl exec -it -- ps aux # Check resource usage (if available in the image) kubectl exec -it -- top ``` These commands give you a live view of the pod's workload and can help identify bottlenecks or application deadlocks. ### **4\. Filesystem and Configuration Inspection** Sometimes a pod fails because a required file is missing or a volume has the wrong permissions. ```bash # Check if a mount point is full on the pod kubectl exec -it -- df -h # List files and check permissions kubectl exec -it -- ls -la /app/config # View the contents of a configuration file kubectl exec -it -- cat /app/config/settings.yaml ``` These commands help you quickly rule out common configuration and filesystem-related issues. ### **5\. The Problem with kubectl exec** > While `kubectl exec` is a familiar command, its significant flaw is that it only works on containers that have a shell and tools. This makes it impossible for troubleshooting secure, distroless containers. **The Solution:** kubectl debug To solve this, we can use kubectl debug to inject a new, ephemeral container into the pod that contains the tools we need. The command to do this is as follows: This below command adds a temporary 'busybox' container to your pod and shares the process namespace, allowing you to debug the original container. ```bash kubectl debug -it --image=busybox --share-processes --container=debugger ``` In this command: * \--image=busybox tells Kubernetes to use a standard debug image that contains essential Linux utilities like ps and ls. * \--share-processes is a critical flag that allows the new debugger container to see and interact with the processes running in the original application container. * \--container=debugger gives our temporary debugging container a clear, identifiable name. Once the command runs, you will be dropped into a shell inside the busybox container, where you can run commands like ps aux to inspect the processes of the main application container. > Troubleshoot smarter. Not harder.