Longhorn Debugging & Concepts

This page contains information from multiple resources.

https://longhorn.io/kb/

Longhorn CSI-PLUGIN:

The Longhorn CSI plugin interfaces Kubernetes (the container Orchestration platform ) and Longhorn (the storage solution for persistent data).

In other words, it allows the Kubernetes engine to talk to the storage engine. If the CSI plugin is not starting, it can be caused by many issues but is usually symptomatic of a more fundamental problem.

Diagnosing

1. If the longhorn-csi-plugin pod is not starting, it is probably in a CrashLoopBackOff state. The first thing to do to is list to get more info about the state.
   1. kubectl -n longhorn-system get pods -l app=longhorn-csi-plugin -o wide
   2. In the output pay attention to the READY column. If the READY column is 1/2, it most likely means that the CSI-Plugin cannot talk to longhorn backend. This will be covered in later steps.
      • Example: longhorn-system longhorn-csi-plugin-jwvnj 1/2 CrashLoopBackOff
   3. Also pay attention to which node the failed pod is running on.
2. Next, do a describe of the pod to make sure that any other important information is identified
   1. kubectl -n longhorn-system get pods -l app=longhorn-csi-plugin --- This returns a list of the pods that are running in the cluster. Look for the one that is in a crashed state
   2. kubectl -n describe pod
   3. Alternatively run the command: kubectl -n longhorn-system describe pod -l app=longhorn-csi-plugin
   4. Look at the events associated with the Pod. They are at the end of the output and usually they include some helpful information.
   5. Most likely the error will be: Back-off restarting failed container
3. Try to get the logs from the crashed instance:
1. kubectl -n longhorn-system get pods -l app=longhorn-csi-plugin --- identity the pod that is in a crashed state
2. kubectl -n longhorn-system logs -c longhorn-csi-plugin (Note, the -p option might need to be added at the end of the command.)
   • Check for the error: Failed to initialize Longhorn API client: Get \"http://longhorn-backend:9500/v1\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
3. Get the logs for the node-driver-registrar
   • kubectl -n longhorn-system logs node-driver-registrar (Note, the -p option might need to be added at the end of the command.)
   • Check for the error: W0423 04:16:48.096462 11095 connection.go:173] Still connecting to unix:///csi/csi.sock
4. If the two errors mentioned above are seen, the issue is most likely that the different nodes in the network cannot talk to each other and something is wrong with the overlay network. Go to the Section Debugging Node Overlay Network

-- Also, Please check the node status in K8s as well as Longhorn UI.

In most cases the network issue might be the cause of the communication issue.