Kubelet Informer/Lister Synchronization Guide

Understanding the Concept

What is an Informer/Lister?

Informer: A client-side component that watches Kubernetes API resources and maintains a local cache Lister: An interface that reads from the local cache (not directly from API server)

Think of it like this:

• API Server = Central database
• Informer = Local sync agent that keeps a copy
• Lister = Local query interface that reads the copy

The Synchronization Process

API Server                    Kubelet
┌─────────────┐              ┌──────────────────┐
│   Nodes     │              │    Informer      │
│  ┌───────┐  │    Watch     │  ┌─────────────┐ │
│  │Node-A │  │◄────────────►│  │ Local Cache │ │
│  │Node-B │  │   Events     │  │  ┌───────┐  │ │
│  │Node-C │  │              │  │  │Node-A │  │ │
│  └───────┘  │              │  │  │Node-B │  │ │
└─────────────┘              │  │  └───────┘  │ │
                             │  └─────────────┘ │
                             └──────────────────┘
                                      ▲
                                      │ Query
                                      ▼
                             ┌──────────────────┐
                             │     Lister       │
                             │ (Read from cache)│
                             └──────────────────┘

Example Scenario: Node Registration Race Condition

Step-by-Step Breakdown

1. Kubelet starts up

   Kubelet: "I need to register myself as node 'worker-01'"
   

2. Node registration succeeds

   API Server: "OK, node 'worker-01' created"
   Kubelet Log: "Successfully registered node worker-01"
   

3. Kubelet tries to update node status

   Kubelet: "Let me get node 'worker-01' from my local cache"
   Lister: "Sorry, I don't see 'worker-01' in my cache yet"
   

4. The problem

   • API Server has the node
   • Local cache doesn't have it yet (sync delay)
   • Kubelet gets "node not found" error

Why This Happens

Timing Issue Example:

Time 0ms:  Kubelet sends CREATE node request
Time 50ms: API Server creates node
Time 60ms: API Server responds "SUCCESS" to kubelet
Time 65ms: Kubelet tries to read node from local cache
Time 65ms: ERROR - node not in cache yet!
Time 100ms: Informer receives WATCH event about new node
Time 110ms: Local cache updated with new node

How to Trace in Logs

1. Increase Kubelet Logging Verbosity

# Add these flags to kubelet
--v=4  # For informer sync logs
--v=6  # For detailed API calls

2. Key Log Patterns to Look For

A. Informer Startup and Sync

# Informer starting
I0715 kubelet.go:xxx] "Starting informer" controller="node"

# Waiting for cache sync
I0715 shared_informer.go:xxx] "Waiting for caches to sync" controller="node"

# Cache sync completed
I0715 shared_informer.go:xxx] "Caches are synced" controller="node"

B. Node Registration Sequence

# 1. Registration attempt
I0715 kubelet_node_status.go:72] "Attempting to register node" node="worker-01"

# 2. Registration success
I0715 kubelet_node_status.go:75] "Successfully registered node" node="worker-01"

# 3. Immediate status update attempt (PROBLEM HERE)
E0715 kubelet_node_status.go:535] "Error updating node status, will retry" 
     err="error getting node \"worker-01\": node \"worker-01\" not found"

C. Lister Access Patterns

# Lister trying to find node
E0715 kubelet_node_status.go:453] "Error getting the current node from lister" 
     err="node \"worker-01\" not found"

# Eventually, cache catches up
I0715 kubelet_node_status.go:xxx] "Successfully updated node status" node="worker-01"

3. Diagnostic Commands

Check Current Node Cache State

# Get kubelet metrics
curl -s http://localhost:10255/metrics | grep informer

# Check if node exists in API server
kubectl get nodes worker-01

# Check kubelet's view
crictl info | grep -A 5 node

Monitor Real-time Logs

# Follow kubelet logs with filtering
journalctl -u kubelet -f | grep -E "(informer|lister|register|sync)"

# Or for containerized kubelet
kubectl logs -n kube-system kubelet-xxx -f | grep -E "(informer|lister)"

Complete Log Analysis Example

Healthy Sequence

I0715 16:20:10.100 kubelet.go:xxx] "Starting node informer"
I0715 16:20:10.150 shared_informer.go:xxx] "Waiting for caches to sync"
I0715 16:20:10.200 shared_informer.go:xxx] "Caches are synced"
I0715 16:20:15.100 kubelet_node_status.go:72] "Attempting to register node" node="worker-01"
I0715 16:20:15.200 kubelet_node_status.go:75] "Successfully registered node" node="worker-01"
I0715 16:20:15.250 kubelet_node_status.go:xxx] "Successfully updated node status" node="worker-01"

Problematic Sequence (Your Case)

I0715 16:24:50.100 shared_informer.go:xxx] "Caches are synced"  # Cache thinks it's ready
I0715 16:24:50.191 kubelet_node_status.go:72] "Attempting to register node" node="nuvlrknpcimn01"
I0715 16:24:50.198 kubelet_node_status.go:75] "Successfully registered node" node="nuvlrknpcimn01"
                                                                # ↑ API call succeeds
I0715 16:24:50.198 kubelet_node_status.go:535] "Error updating node status, will retry" 
     err="error getting node \"nuvlrknpcimn01\": node \"nuvlrknpcimn01\" not found"
                                                                # ↑ Lister call fails (7ms later!)

Troubleshooting Steps

1. Check Informer Health

# Look for informer sync issues
kubectl logs -n kube-system kubelet-xxx | grep -A 5 -B 5 "sync"

2. Verify Network Connectivity

# Test API server connectivity
curl -k https://$KUBERNETES_API_SERVER/api/v1/nodes

# Check kubelet to API server latency
time kubectl get nodes --server=$KUBERNETES_API_SERVER

3. Monitor Cache Refresh

# Watch for informer resync events
kubectl logs -n kube-system kubelet-xxx | grep -E "(resync|refresh|list.*nodes)"

4. Resource Constraints

# Check if kubelet is under resource pressure
top -p $(pgrep kubelet)
kubectl top nodes

Prevention and Solutions

1. Increase Kubelet Timeouts

# kubelet config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
nodeStatusUpdateFrequency: 10s
nodeStatusReportFrequency: 1m

2. Add Retry Logic Monitoring

# Monitor retry attempts
journalctl -u kubelet | grep "will retry" | wc -l

3. API Server Optimization

# API server flags for better performance
--max-requests-inflight=400
--max-mutating-requests-inflight=200

This synchronization issue typically resolves itself, but understanding the logs helps you distinguish between transient startup issues and persistent problems.