1. Why Do We Need Local Access to Cluster Resources?

In a secure Kubernetes architecture, most microservices run in an isolated internal network, meaning they are not directly exposed to the node's network or the public internet. This isolation is a fundamental layer of security and must be preserved, especially for high-security applications.

However, this strict isolation can make it difficult to troubleshoot internal connectivity and application behavior. The solution is to leverage Kubernetes-native tools to create secure, temporary tunnels that eliminate the need for external exposure.

The key advantage of using tools like kubectl port-forward and kubectl proxy is that they require no code modifications to the application container, preserving its immutability and reducing the risk of accidental exposure when debugging highly secure applications.

2. Prerequisite: Establishing Cluster Trust

First verifies that you have the necessary authentication (kubeconfig) and authorization (RBAC permissions) required for accessing the cluster. You must be able to successfully run a basic command (e.g., kubectl get nodes) from the machine where you intend to run kubectl port-forward or kubectl proxy.

3. What is kubectl proxy?

kubectl proxy acts as an HTTP reverse proxy between your local machine and the Kubernetes API server.
It enables you to access any Kubernetes resource exposed by the API server — including Pods, Services, and other API endpoints — directly from your local host.

kubectl proxy

How it works

1. Local HTTP server start – kubectl proxy starts a local HTTP server on your machine (by default at http://localhost:8001).

2. Authentication with API server – The proxy authenticates using your kubeconfig credentials, ensuring secure communication with the Kubernetes API server.

3. Request forwarding – When you make a local HTTP request (e.g., http://localhost:8001/api/...), the proxy securely forwards it to the Kubernetes API server over HTTPS.

4. API server processing – The API server receives the request, interacts with the appropriate Kubernetes resources (such as Pods, Services, Deployments or etc..), and prepares the response.

5. Response delivery – The proxy sends the API server’s response back to your local machine, making it appear as if you’re directly interacting with the cluster’s REST API from localhost.

6.Traffic flow – Local machine → kubectl proxy (HTTP server) → API server → Kubernetes resource → API server → kubectl proxy → Local machine

# You’ll see all the Pods in the default namespace directly fetched via the API server, not by connecting to the Pod network.
http://localhost:8001/api/v1/namespaces/default/pods

# To access a services
http://localhost:8001/api/v1/namespaces/<namespace>/services/<service-name>:<port-name>/proxy/
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

## Security aspects

# Most Explicit (Secure): Identical to the default behavior. Used to ensure the proxy is never accidentally exposed by other system configurations.
kubectl proxy --address=127.0.0.1 --accept-hosts='^127\.0\.0\.1$'

# Expose API to anyone who can reach your system.
kubectl proxy --address=0.0.0.0 --port=8001 

#Dangerous(Insecure): Binds to 0.0.0.0 (all network interfaces) and accepts requests from any host (^*\$).
kubectl proxy --address=0.0.0.0 --port=8001 --accept-hosts='^*$'

4. What is kubectl port-forward?

kubectl port-forward establishes a TCP tunnel that allows you to connect directly from your local machine to a Pod’s or Service’s port.This method provides application-specific access for debugging or testing purposes, without exposing the application externally through a LoadBalancer or Ingress.
You can perform port forwarding on Pods, Services, Deployments, StatefulSets, and ReplicaSets.

kubectl port-forward pod/<POD_NAME> <LOCAL_PORT>:<CONTAINER_PORT>
kubectl port-forward service/<SERVICE_NAME> <LOCAL_PORT>:<CONTAINER_PORT>
kubectl port-forward deployments/<DEPLOYMENT_NAME> <LOCAL_PORT>:<CONTAINER_PORT>
kubectl port-forward statefulsets/<STATEFULSET_NAME> <LOCAL_PORT>:<CONTAINER_PORT>

How it works

1. Connection to API server – kubectl connects securely to the Kubernetes API server using your kubeconfig credentials, similar to how kubectl proxy operates.

2. Data stream to kubelet – The API server identifies the node where the target Pod is running and securely streams the data to the kubelet on that node.

3. Pod connection and tunneling – The kubelet opens a connection to the Pod’s specified port and tunnels the traffic back through the same API server channel to your local machine.

4. Local access established – You can now access the Pod or Service locally through the forwarded port, without exposing it externally or modifying cluster networking.

5.Traffic flow – through this secure path: Local machine → API server → kubelet → Pod.

kubectl run flask-app --image=python:3.11 --port=5000
kubectl port-forward pod/flask-app 5000:5000
## Visit http://localhost:5000

# It takes random port on local machine
kubectl port-forward pod/my-pod :5000
kubectl port-forward --address 0.0.0.0 pod/my-pod 8888:5000

#To Run kubectl port forward command in background
kubectl port-forward pod/flask-app 5000:5000  &

5. Security Note — Binding to 0.0.0.0

By default, both kubectl proxy and kubectl port-forward bind to 127.0.0.1, making them accessible only from your local machine.

If you manually bind them to 0.0.0.0, they become accessible from any network interface — not just localhost.
While this can be helpful for remote debugging, it introduces a major security risk: Anyone on your network could potentially access your cluster through the open proxy or tunnel, especially if your kubeconfig remains active.

• Best Practice:

  • Keep the binding to 127.0.0.1 (default).

  • Only bind to 0.0.0.0 if you have:

    • Strong network isolation (e.g., VPN or private subnet).

    • Proper authentication and authorization controls in place.

    • Strict firewall rules limiting who can connect.

6. Authentication & Authorization

7. When to Use Which

For Local Testing and Debugging – Use kubectl proxy or port-forward to test applications, APIs, or services running inside the cluster from your local machine without exposing them externally.

For Accessing Cluster APIs (HTTP Traffic) – Use kubectl proxy when you need HTTP-based access to Kubernetes resources, dashboards, or metrics APIs.

For Application-Specific Access (TCP Traffic) – Use kubectl port-forward to connect directly to a Pod or Service for TCP-based access, such as web apps, databases, or message queues like Kafka, in a development or testing environment.

For Short-Lived Connections – Ideal for temporary operations like inspecting logs, running queries, or testing functionality, where permanent exposure is not required.

For Secure Local Access – Both commands keep your traffic encrypted and access restricted to your kubeconfig permissions, making them safe for local development.

8. When not to Use

In Production – These are temporary connections that break when the command or terminal is closed. Use Ingress, LoadBalancer, or proper service exposure instead.

For Automation or CI/CD – Port-forwarding can hang or fail randomly. Use ClusterIP or internal services for reliable automation.

For Heavy Traffic – All traffic passes through the API server, which can cause latency and performance bottlenecks. Use proper network routing or service exposure instead.

For Long-Running Streams – Database or message queue connections, such as PostgreSQL or Kafka, can drop easily. Use a VPN, Bastion host, or persistent service exposure instead.

On Shared Machines – Anyone with access to the machine could reach your forwarded ports. Keep bindings local (127.0.0.1) and secure your environment.

Without Access Control – Running with admin kubeconfig may expose sensitive APIs. Enforce RBAC, limit privileges, and enable audit logging.

💡 Question of the Day:

If you had to securely expose a Kubernetes service for your team without using Ingress or LoadBalancer, what would be your preferred approach and why?

Secure, local, temporary – debug smart, expose smart.

As DevOps engineers, we constantly battle a security game of whack-a-mole. Every week, a new vulnerability is discovered in an OS package we don't even use, forcing us to rebuild and redeploy our container images. Let's stop making your container a virtual machine; make it a specific, purposeful application. This is where distroless containers enter the picture. By stripping away everything but your application and its dependencies, they offer a direct path to a smaller, more secure, and more efficient production environment. This concept, which focuses on creating minimal and secure images, was pioneered by Google to reduce the attack surface of containerized applications.

1. The Core Principle: Less is More

Traditional container images (like those based on Ubuntu or Debian) include a full Linux distribution, complete with a shell, package manager (apt or yum), and hundreds of system libraries. While this is convenient for debugging, every one of those included packages is a potential entry point for an attacker. A distroless container, on the other hand, contains only your application code and its immediate runtime dependencies.

2. Reduced Attack Surface & Fewer CVEs

This is the single biggest security win. With a traditional image, a vulnerability scanner will flag dozens or even hundreds of CVEs (Common Vulnerabilities and Exposures) from packages you don't even use. With a distroless image, the number of CVEs drops dramatically, often to zero. Since there's less code, there are fewer bugs and fewer vulnerabilities for an attacker to exploit.

Ready for the proof? A simple Trivy scan reveals the security power of a distroless container.

Non-Distroless Image Scanning

Distroless Image Scanning

3. No Shell, No Problem

One of the most significant security benefits is the absence of a shell. If an attacker gains access to your container, they can't simply kubectl exec into it and start running commands. They have no shell to use. This prevents "living off the land" attacks where an attacker uses pre-installed system binaries to move laterally within your cluster.

4. No Package Manager

An attacker who compromises a traditional container could use the built-in package manager to install more tools. With a distroless image, this is impossible. The image is immutable, and no new software can be added at runtime, which greatly limits what an attacker can do after an initial breach.

5. The Size & Performance Benefit

While security is the primary driver, a significant secondary benefit is image size. distroless images are often an order of magnitude smaller than their full-distribution counterparts. Smaller images mean faster downloads and faster pod startup times in a scaled Kubernetes cluster.

6. Multi-Stage Builds

Please follow the practical demonstration of a distroless image on GitHub: Distroless Practical

Here is the some code snippet of that demo.

package main
import "fmt"
func main() {
    fmt.Println("Hello, World!")
}

# Build stage
FROM golang:1.21-alpine AS builder

WORKDIR /app

# Copy the Go source code
COPY app.go .

# Build the Go binary (statically linked)
RUN go build -o helloworld app.go

# Final stage: distroless image
FROM gcr.io/distroless/static-debian12:nonroot

WORKDIR /app

# Copy the statically built binary from the builder
COPY --from=builder /app/helloworld .

# Run the binary
ENTRYPOINT ["/app/helloworld"]

You might be asking, "Why don't we just start with a distroless image?" The reason is that we require some utilities, like a compiler and package manager, during the application's build time. For that, we'll use a minimal image in the first stage. But as you'll see in the next stage, we are only passing the compiled file to the distroless image, leaving all the messy build tools behind.

7. Minimal vs. Distroless: A Critical Distinction💡

While many people use the terms "minimal" and "distroless" interchangeably, there is an important difference that is crucial for container security.

A minimal image (like alpine or slim) is a simplified Linux distribution. It's designed to be small but still includes a shell, a package manager (apk, apt), and essential utilities. It's like a tiny Linux machine.

A distroless image, on the other hand, is not a Linux distribution. It contains only your application and its direct runtime dependencies, like libc and ca-certificates. It's simply a file system with the bare essentials needed to run your code, without a shell or package manager.

8. The Trade-Off: Debugging

Since you can't kubectl exec into a distroless container, you need a different mental model and tools. To learn the best way to troubleshoot, the-ultimate-guide-to-kubernetes-troubleshooting

Clean, lean, and mean.

A critical pod fails. Your alerts are firing. What's next? Every DevOps/SRE engineer needs a clear plan to handle cluster issues. This guide provides a systematic strategy and the essential tools to diagnose any problem, from pod health to the control plane.

The First Steps: Observing the Problem

When a pod is failing, your first task is to become a detective. You need to gather clues and observations before you jump to conclusions. Fortunately, Kubernetes provides a suite of essential commands for this initial investigation. Mastering these fundamentals will save you a lot of time and frustration.

1. The Overview: kubectl get pods

The most basic used command is kubectl get pods. It provides a quick, high-level snapshot of your pods’ current state.

kubectl get pods -n <namespace>

By running this, you can immediately see:

• The pod's STATUS (Running, Pending, CrashLoopBackOff, etc.).

• The number of times the pod has RESTARTED.

• How long the pod has been running (AGE).

A pod in a CrashLoopBackOff state is a clear sign of a repeated failure, while a pod stuck in Pending indicates a scheduling issue.

2. The Details: kubectl describe pod

Once you have a suspicious-looking pod, the next step is to run a more detailed inspection with kubectl describe pod.

kubectl describe pod <pod-name> -n <namespace>

This command provides a wealth of information, including:

• Events: A list of recent events related to the pod, which can show you exactly why a container failed to start or why a volume couldn't be mounted.

• Containers: Details about the containers within the pod, including their image, state, and restartCount.

• Status & Exit Reason: Details about the containers within the pod, their image, and most importantly, the State and Exit Code if a container has terminated. An exit code of 1 or a specific error message can be your first solid clue.

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  4m1s                 default-scheduler  Successfully assigned my-app-pod to node-01
  Normal   Pulling    4m                   kubelet            Pulling image "my-app:v1"
  Normal   Pulled     3m59s                kubelet            Successfully pulled image "my-app:v1" in 987ms
  Normal   Created    3m59s                kubelet            Created container my-app-container
  Normal   Started    3m59s                kubelet            Started container my-app-container
  Warning  Failed     3m1s (x3 over 3m1s)  kubelet            Failed to pull image "my-app:v2": rpc error: code = NotFound

3. The Output: kubectl logs

The kubectl logs command is your window into what's happening inside the container itself.

kubectl logs <pod-name> -n <namespace>

By default, this shows you the standard output and error streams from your container. It's the go-to command for finding application-level errors, startup failures, or unexpected behavior.

Advanced Pod Troubleshooting: When a Pod is Running but Not Responding

What do you do when kubectl get pods shows a pod is Running, but your application still isn't working?

1. The First Test: Is the Application Itself Unresponsive?

The quickest way to check if your application is simply hung is by trying to access it from a shell inside the container. This tests the application directly, bypassing the Kubernetes service layer and network policies.

Use kubectl exec to run a simple connectivity test.

kubectl exec -it <pod-name> -n <namespace> -- curl localhost:<app-port>

If the curl command fails to connect or gets a timeout, you know the problem is within the container itself. If it returns a correct response, the issue is likely with the Kubernetes networking or service configuration.

2. The Network Check: Can the Pod Reach Out?

If the pod is supposed to be making external connections but isn't, you need to check its network connectivity. Since many images don't contain standard network tools, you can use kubectl exec to run a few commands if available.

• DNS Resolution: Use nslookup or dig to check if the pod can resolve a service name or an external domain.

• Connectivity: Use ping, nc, or curl to test connectivity to a service, database, or external API.

For example, to check DNS resolution for a service:

# If the service is in the same namespace as the pod
kubectl exec -it <pod-name> -n <namespace> -- nslookup <service-name>

# If the service is in a different namespace,
# use the fully qualified domain name (FQDN) for reliability.
kubectl exec -it <pod-name> -n <namespace> -- nslookup <service-name>.<service-namespace>.svc.cluster.local

If these tests fail, it points to a network configuration issue (like a misconfigured DNS or Network Policy) rather than an application-level bug.

3. Process and Resource Checks

Even if the application is reachable, its internal state might be unhealthy. You can use process inspection tools to see if a process has a high CPU or is hung.

# Check running processes and their status
kubectl exec -it <pod-name> -- ps aux

# Check resource usage (if available in the image)
kubectl exec -it <pod-name> -- top

These commands give you a live view of the pod's workload and can help identify bottlenecks or application deadlocks.

4. Filesystem and Configuration Inspection

Sometimes a pod fails because a required file is missing or a volume has the wrong permissions.

# Check if a mount point is full on the pod
kubectl exec -it <pod-name> -- df -h

# List files and check permissions
kubectl exec -it <pod-name> -- ls -la /app/config

# View the contents of a configuration file
kubectl exec -it <pod-name> -- cat /app/config/settings.yaml

These commands help you quickly rule out common configuration and filesystem-related issues.

5. The Problem with kubectl exec

While kubectl exec is a familiar command, its significant flaw is that it only works on containers that have a shell and tools. This makes it impossible for troubleshooting secure, distroless containers.

The Solution: kubectl debug

To solve this, we can use kubectl debug to inject a new, ephemeral container into the pod that contains the tools we need. The command to do this is as follows:

This below command adds a temporary 'busybox' container to your pod and shares the process namespace, allowing you to debug the original container.

kubectl debug -it <pod-name> --image=busybox --share-processes --container=debugger

In this command:

• --image=busybox tells Kubernetes to use a standard debug image that contains essential Linux utilities like ps and ls.

• --share-processes is a critical flag that allows the new debugger container to see and interact with the processes running in the original application container.

• --container=debugger gives our temporary debugging container a clear, identifiable name.

Once the command runs, you will be dropped into a shell inside the busybox container, where you can run commands like ps aux to inspect the processes of the main application container.

Troubleshoot smarter. Not harder.