Qingular

Pod Network Connectivity

·CKAk8sPractice

CKA Domain 3 — Pod network communication mechanisms, kube-proxy modes and CNI plugin comparison

← Back to CKA Practice Index

Overview

Kubernetes networking model requirements:

  • All Pods can communicate directly with each other without NAT
  • All nodes can communicate directly with all Pods without NAT
  • The IP address a Pod sees for itself is the same as what other Pods see for it

1. Intra-Pod Container Communication

All containers within a Pod share the same network namespace.

┌─────────────────────────────┐
│          Pod                │
│  ┌─────────┐  ┌─────────┐  │
│  │ Container│  │ Container│  │
│  │   App A  │  │   App B  │  │
│  │ localhost:8080  │  │
│  └─────────┘  └─────────┘  │
│       lo interface          │
│       eth0: 10.244.1.5     │
└─────────────────────────────┘

apiVersion: v1
kind: Pod
metadata:
  name: multi-container-pod
spec:
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80
    - name: sidecar
      image: busybox:1.28
      command: ["sleep", "3600"]
      # sidecar can access nginx via localhost:80

Key Points:

  • Communicate via localhost
  • Share the same IP address and port space
  • Can access each other without needing a Service

2. Pod Communication on the Same Node

Pods on the same node communicate through the node's network bridge (typically cbr0 or a bridge created by the CNI).

┌───────────── Node ─────────────┐
│                                │
│  ┌─────────┐  ┌─────────┐     │
│  │ Pod A   │  │ Pod B   │     │
│  │ veth pair│  │ veth pair│    │
│  └────┼────┘  └────┼────┘     │
│       │           │          │
│    ┌──┴───────────┴──┐       │
│    │   cni0 (bridge)  │       │
│    └────────┬─────────┘       │
│             │ eth0            │
│    Node IP: 192.168.1.10      │
└────────────────────────────────┘
  • Pods connect to the bridge via virtual Ethernet pairs (veth pairs)
  • Communication happens directly through the bridge, no encapsulation needed
  • Low latency, good performance

3. Cross-Node Pod Communication

Cross-node Pod communication relies on the CNI plugin's overlay network implementation.

┌────── Node 1 ──────┐    ┌────── Node 2 ──────┐
│ Pod A: 10.244.1.5  │    │ Pod C: 10.244.2.7  │
│    │               │    │    │               │
│ ┌──┴──┐            │    │ ┌──┴──┐            │
│ │cni0 │            │    │ │cni0 │            │
│ └──┬──┘            │    │ └──┬──┘            │
│    │               │    │    │               │
│ eth0: 192.168.1.10 │    │ eth0: 192.168.2.10│
└────────┬───────────┘    └────────┬───────────┘
         │                        │
         └──── Overlay (VXLAN) ───┘
              or direct routing

4. Service ClusterIP Communication Mechanism

When Pods communicate via Service ClusterIP, kube-proxy is responsible for forwarding traffic to backend Pods.

kube-proxy Modes

iptables Mode (default)

# View iptables rules
kubectl get svc

# On the node, view iptables rules for the Service
sudo iptables -t nat -L -n | grep <cluster-ip>

# View the KUBE-SERVICES chain
sudo iptables -t nat -L KUBE-SERVICES -n

# View a specific Service rule chain
sudo iptables -t nat -L KUBE-SVC-XXXXXXXX -n

# View Endpoints load balancing rules
sudo iptables -t nat -L KUBE-SEP-XXXXXXXX -n

iptables Forwarding Flow:

  1. PREROUTING → KUBE-SERVICES
  2. Match ClusterIP destination → jump to the corresponding KUBE-SVC-* chain
  3. KUBE-SVC-* chain performs random load balancing → jumps to KUBE-SEP-* chain
  4. KUBE-SEP-* chain performs DNAT → replaces destination IP with Pod IP

Characteristics:

  • Default mode, mature and stable
  • Number of rules is proportional to the number of Services + Endpoints
  • Rule updates can become slow at large scale
# Check current kube-proxy mode
kubectl get configmap -n kube-system kube-proxy -o yaml | grep mode

# Switch to IPVS mode (modify ConfigMap)
kubectl edit configmap -n kube-system kube-proxy
# Set mode: ipvs

# Restart kube-proxy
kubectl rollout restart -n kube-system daemonset kube-proxy

# View IPVS rules
sudo ipvsadm -L -n

# View IPVS connections
sudo ipvsadm -L -n -c

Characteristics:

  • Based on the kernel's IPVS module
  • More efficient: hash table lookup, O(1) time complexity
  • Supports more load balancing algorithms (rr, wrr, lc, wlc, sh, dh, etc.)
  • Suitable for large-scale clusters (thousands of Services)

Load Balancing Algorithms:

AlgorithmDescription
rrRound Robin (default)
wrrWeighted Round Robin
lcLeast Connections
wlcWeighted Least Connections
shSource Hashing
dhDestination Hashing

5. CNI Plugin Comparison

FeatureCalicoFlannelWeaveCilium
Base TechnologyBGP + iptablesVXLAN/Host-GWVXLAN/Weave MesheBPF
NetworkPolicyFull supportLimited support (needs additional plugins)Full supportFull support
EncryptionWireGuard optionalNot built-inBuilt-in encryptionWireGuard optional
PerformanceHighMediumMedium-lowVery high
ComplexityMediumLowLowMedium-high
Recommended ScenarioProduction, needs NetworkPolicySmall cluster, simple networkingSmall to medium clustersHigh-performance production
eBPFSupported (eBPF mode)Not supportedNot supportedNative eBPF

Installing Calico (CKA Focus)

# Install Calico (via Tigera Operator)
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27/manifests/tigera-operator.yaml

# Configuration file
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27/manifests/custom-resources.yaml

# Wait for all Pods to be ready
kubectl get pods -n calico-system -w

Installing Flannel

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

6. Network Troubleshooting Commands

Connectivity Testing

# Use busybox to test network connectivity
kubectl run test-pod --image=busybox:1.28 --rm -it --restart=Never -- wget -qO- http://<service-ip>

# Use netshoot (recommended, includes more tools)
kubectl run netshoot --image=nicolaka/netshoot --rm -it --restart=Never -- curl http://my-svc:80

# Test DNS resolution
kubectl run dns-test --image=busybox:1.28 --rm -it --restart=Never -- nslookup my-svc

# Test direct Pod IP communication
kubectl run test-pod --image=busybox:1.28 --rm -it --restart=Never -- sh
# Inside the container
ping <target-pod-ip>
wget -qO- http://<target-pod-ip>:8080

Network Diagnostic Tool Pod

apiVersion: v1
kind: Pod
metadata:
  name: network-tools
spec:
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      command: ["sleep", "3600"]

# Enter the container after creation
kubectl exec -it network-tools -- sh

# The following tools are available inside the container
curl http://my-svc:80
dig my-svc.default.svc.cluster.local
nslookup kubernetes.default
ping 10.96.0.1
traceroute 10.244.1.5
mtr 10.244.1.5
tcpdump -i eth0
ip addr
ip route
ss -tunap

Node-Level Troubleshooting

# View kube-proxy logs
kubectl logs -n kube-system -l k8s-app=kube-proxy

# View node routing table
ip route show

# View bridges
brctl show
ip link show type bridge

# View network interfaces and veth pairs
ip link

# View iptables NAT rules (run on the node)
sudo iptables-save | grep <service-name>

🧪 Complete Hands-On Example: Verify Cross-Node Pod Connectivity

Scenario

In a multi-node cluster, create Pods on different nodes and verify they can communicate directly via Pod IP and also perform service discovery via Service ClusterIP.

Prerequisites

  • Cluster has at least 2 available nodes
  • CNI plugin is working properly on all nodes
  • kubectl is configured with cluster access

Steps

Step 1: View cluster node topology

# View node list and labels
kubectl get nodes -o wide
# NAME       STATUS   ROLES           INTERNAL-IP      OS-IMAGE
# node-1     Ready    <none>          192.168.1.10     Ubuntu 22.04
# node-2     Ready    <none>          192.168.1.20     Ubuntu 22.04

# View node labels, determine which labels can be used to schedule Pods
kubectl get nodes --show-labels
# Record node names for subsequent scheduling

Step 2: Create Pod on Node 1

# Create Pod on node-1 using nodeName to force scheduling
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pod-node1
  labels:
    app: test
    node: node1
spec:
  nodeName: node-1          # Force scheduling to node-1
  containers:
    - name: nginx
      image: nginx:1.25
      ports:
        - containerPort: 80
EOF
# Expected output: pod/pod-node1 created

# Verify Pod is running on node-1
kubectl get pod pod-node1 -o wide
# NAME        READY   STATUS    RESTARTS   AGE   IP              NODE
# pod-node1   1/1     Running   0          10s   10.244.1.10     node-1

Step 3: Create Pod on Node 2

# Create Pod on node-2
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pod-node2
  labels:
    app: test
    node: node2
spec:
  nodeName: node-2
  containers:
    - name: nginx
      image: nginx:1.25
      ports:
        - containerPort: 80
EOF
# Expected output: pod/pod-node2 created

# Verify Pod is running on node-2
kubectl get pod pod-node2 -o wide
# NAME        READY   STATUS    RESTARTS   AGE   IP              NODE
# pod-node2   1/1     Running   0          10s   10.244.2.20     node-2

Step 4: Cross-node direct communication via Pod IP

# Record both Pod IPs
POD1_IP=$(kubectl get pod pod-node1 -o jsonpath='{.status.podIP}')
POD2_IP=$(kubectl get pod pod-node2 -o jsonpath='{.status.podIP}')
echo "Pod1 IP: $POD1_IP, Pod2 IP: $POD2_IP"
# Expected output: Pod1 IP: 10.244.1.10, Pod2 IP: 10.244.2.20

# Access pod-node2 from pod-node1 (cross-node)
kubectl exec pod-node1 -- curl -s -m 5 http://$POD2_IP
# Expected output: nginx welcome page HTML

# Access pod-node1 from pod-node2 (reverse cross-node)
kubectl exec pod-node2 -- curl -s -m 5 http://$POD1_IP
# Expected output: nginx welcome page HTML

# Test ICMP connectivity with ping (some CNIs may block ICMP)
kubectl exec pod-node1 -- ping -c 2 $POD2_IP
# Expected output: 2 packets transmitted, 2 received (may timeout depending on CNI config)

Step 5: Cross-node communication via Service ClusterIP

# Create Service to expose both Pods
kubectl expose pod pod-node1 --name=test-svc --port=80 --target-port=80
# Or use label selector
kubectl create service clusterip test-svc --tcp=80:80

# Manually set selector to point to both Pods
kubectl patch svc test-svc -p '{"spec":{"selector":{"app":"test"}}}'
# Expected output: service/test-svc patched

# View Service ClusterIP
kubectl get svc test-svc
# NAME        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
# test-svc    ClusterIP   10.96.200.50    <none>        80/TCP    10s

SVC_IP=$(kubectl get svc test-svc -o jsonpath='{.spec.clusterIP}')
echo "Service ClusterIP: $SVC_IP"
# Expected output: Service ClusterIP: 10.96.200.50

# Access from pod-node1 via Service name (DNS resolution)
kubectl exec pod-node1 -- curl -s -m 5 http://test-svc
# Expected output: nginx welcome page HTML

# Access from pod-node2 via Service ClusterIP
kubectl exec pod-node2 -- curl -s -m 5 http://$SVC_IP
# Expected output: nginx welcome page HTML

Step 6: Complete network debugging workflow

# Create a diagnostic Pod with full network tools
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: network-debug
spec:
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      command: ["sleep", "3600"]
EOF
# Expected output: pod/network-debug created

# Enter the diagnostic Pod
kubectl exec -it network-debug -- sh

# Run the following debug commands inside the diagnostic container:
# 1. IP and routing information
ip addr
ip route

# 2. DNS resolution test
dig test-svc.default.svc.cluster.local
nslookup kubernetes.default

# 3. Connectivity test
curl -m 5 http://$POD1_IP          # Access pod-node1
curl -m 5 http://$POD2_IP          # Access pod-node2
curl -m 5 http://test-svc          # Access via Service DNS

# 4. Trace route
traceroute $POD2_IP

# 5. Packet capture (optional)
tcpdump -i eth0 -c 10

# Exit container
exit

Verification

# Check Endpoints to confirm Service backends
kubectl get endpoints test-svc
# NAME        ENDPOINTS                           AGE
# test-svc    10.244.1.10:80,10.244.2.20:80      2m

# On the node, view iptables forwarding rules (requires sudo)
# sudo iptables -t nat -L KUBE-SERVICES -n | grep $SVC_IP

# View kube-proxy logs (when troubleshooting Service issues)
kubectl logs -n kube-system -l k8s-app=kube-proxy --tail=10

# Clean up resources
kubectl delete pod pod-node1 pod-node2 network-debug
kubectl delete svc test-svc

Exam Tips