Rancher Provisioning Internals 

o deeply understand Rancher’s Cluster API (CAPI) integration, its v1/v2/v3 provisioning system, and how to debug provisioning issues, let's break this down step-by-step:

---

🔧 1. Rancher Provisioning Internals Overview

Rancher supports multiple versions of cluster provisioning mechanisms internally, and each represents different architecture and capability changes over time:

---

📌 v1 Provisioning (Legacy - RKE1)

• Used for: Rancher-managed clusters using RKE1 (Rancher Kubernetes Engine v1)

• Implemented via: Cluster, NodeTemplate, NodePool CRDs

• Provisioning handled by: Rancher controller logic, not based on Cluster API

• Where: /v3/cluster API

• Cloud Drivers: Node drivers using Docker Machine (rancher/machine)

🔍 Debug:

• Check logs of rancher pod

• Look at Cluster, NodePool, and NodeTemplate via Rancher API or kubectl

• SSH and Docker Machine logs

---

📌 v2 Provisioning (CAPI-based – RKE2/K3s – Rancher 2.5+)

• Used for: Rancher-managed clusters using RKE2 or K3s

• Implemented via: Cluster API (CAPI) style resource hierarchy

• Custom CRDs:

  • Cluster

  • MachineDeployment → MachineSet → Machine

  • RKECluster, RKEControlPlane, RKEBootstrap, MachineConfig

• API Group: provisioning.cattle.io/v1

• Machine driver moved to: rancher/system-agent, and provisioning handled by Fleet and CAPI

🔍 Debug:

• Check cattle-cluster-agent, rancher-system-agent, and rancher pod logs

• Inspect CRDs:

  bash
  kubectl get clusters.provisioning.cattle.io -A
  kubectl get rkecontrolplanes -A
  kubectl get machinedeployments.cluster.x-k8s.io -A
  

• Check the plan status in machineinventory, machineinventoryselector, and machineinventorybinding if Elemental is involved

• Events: kubectl get events -A | grep <cluster-name>

---

📌 v3 Provisioning (Rancher 2.7+ – Cluster API Native)

• Goal: Full Cluster API standard support with better plugability (more like upstream CAPI clusters)

• Introduced with: Enhanced multi-cloud, bare-metal, and edge provisioning

• Provisioning via: Cluster API upstream-compatible resources directly, not provisioning.cattle.io

🛠️ CRDs used:

• Native Cluster from cluster.x-k8s.io

• MachineDeployment, Machine, KubeadmConfig, InfrastructureProvider CRs

• capi-<provider> components like capi-aws, capi-vsphere, capi-openstack, etc.

🔍 Debug:

• Use upstream CAPI tools:

  • clusterctl describe cluster <name>

  • kubectl get machines.cluster.x-k8s.io -A

• Events and logs from the CAPI controllers

• Rancher UI still wraps this but doesn't abstract away everything

---

🧠 What is CAPI in Rancher?

CAPI is an upstream Kubernetes project for declarative Kubernetes cluster lifecycle management. Rancher leverages and extends it with:

• Bootstrap provider: RKE2Bootstrap or RKE1Bootstrap

• Infrastructure provider: RancherMachineProviderConfig or external (AWS, vSphere)

• Cluster & Control Plane CRDs

• Integration with Fleet and System Agent for orchestration

---

Rancher Cluster Provisioning Debugging Guide (v2 & v3)

🔄 Troubleshooting Flowchart (Text-Based)

[Start]
   |
   v
[1. Cluster stuck in "Provisioning"/"Waiting"]
   |
   v
[2. Get Cluster CRD Status]
   |
   v
kubectl get clusters.provisioning.cattle.io -A
kubectl describe cluster <name> -n <namespace>
   |
   v
[3. Check associated control plane + bootstrap + infra CRDs]
   |
   +--> kubectl get rkecontrolplanes -A
   +--> kubectl get rkemachinepools -A
   +--> kubectl get machineconfigs -A
   |
   v
[4. Check MachineDeployments and Machines]
   |
   +--> kubectl get machinedeployments.cluster.x-k8s.io -A
   +--> kubectl get machines.cluster.x-k8s.io -A
   |
   v
[5. Check Plan Delivery / Agent Sync]
   |
   +--> kubectl get plan -A
   +--> kubectl describe plan <name>
   +--> kubectl logs -n cattle-system -l app=rancher-system-agent -f
   |
   v
[6. Check Cluster Agent Communication]
   |
   +--> kubectl logs -n cattle-system -l app=cattle-cluster-agent -f
   |
   v
[7. Check Fleet GitOps Bundle Errors]
   |
   +--> kubectl get bundles -A
   +--> kubectl describe bundle <name>
   |
   v
[8. Check MachineInventory and Registration Tokens]
   |
   +--> kubectl get machineinventory -A
   +--> kubectl get machinetemplate -A
   |
   v
[End or Loop to Specific Failing CRD]

---

🧪 Sample Troubleshooting Script

# Set cluster namespace (usually cluster name in lowercase with random suffix)
CLUSTER_NS="fleet-default" # or find it dynamically using get clusters.provisioning.cattle.io

# 1. Get high-level cluster info
kubectl get clusters.provisioning.cattle.io -A
kubectl describe cluster <cluster-name> -n $CLUSTER_NS

# 2. Get CAPI cluster info (for v3 provisioning)
kubectl get cluster.cluster.x-k8s.io -A

# 3. Control plane and bootstrap components
kubectl get rkecontrolplanes -A
kubectl get rkemachinepools -A
kubectl get machineconfigs -A

# 4. Machine deployments and machines
kubectl get machinedeployments.cluster.x-k8s.io -A
kubectl get machines.cluster.x-k8s.io -A
kubectl describe machine <name> -n $CLUSTER_NS

# 5. Check Plan CRD
kubectl get plan -A
kubectl describe plan <plan-name>

# 6. Agent logs (machine plan application)
kubectl logs -n cattle-system -l app=rancher-system-agent -f

# 7. Cluster agent logs (communication)
kubectl logs -n cattle-system -l app=cattle-cluster-agent -f

# 8. Rancher manager logs
kubectl logs -n cattle-system -l app=rancher -f

# 9. Fleet GitOps troubleshooting
kubectl get bundles -A
kubectl describe bundle <bundle-name>
kubectl get gitrepos -A

# 10. Machine inventory and registration (esp. for Elemental)
kubectl get machineinventory -A
kubectl get machineinventorybinding -A
kubectl get machinetemplate -A

---

🧰 Optional Tools

🔍 Cluster API CLI: clusterctl

# Install
brew install clusterctl

# Describe cluster and see what is failing
clusterctl describe cluster <cluster-name> -n <namespace>

---

🧼 Common Cleanup Commands

# Delete stuck cluster
kubectl delete clusters.provisioning.cattle.io <cluster-name> -n $CLUSTER_NS --force --grace-period=0

# Force delete machine CRD
kubectl delete machine <name> -n <namespace> --force --grace-period=0

---

✅ Summary by Provisioning Version

Version	Engine	CRD Group	Notes
v1	RKE1	management.cattle.io/v3	Legacy Docker Machine
v2	RKE2	provisioning.cattle.io/v1	Rancher CAPI-style provisioning
v3	RKE2	cluster.x-k8s.io	Full upstream CAPI-native support