API audit logging in RKE2 & Other Rancher K8s distribution

Unlike RKE1, RKE2 does not automatically enable kube-audit logs with a default policy. We need to add additional parameters or create it manually and update it in the UI. 

Please check the documentation and the corresponding KB article for RKE2

https://www.suse.com/support/kb/doc/?id=000021171

How to collect the kube audit log along with Rancher logs:

https://www.suse.com/support/kb/doc/?id=000021022

Rancher doc:

https://ranchermanager.docs.rancher.com/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters

PLEASE refer this GitHub issues for the history,like :

https://github.com/rancher/rke2/issues/1183#issuecomment-926155570

The below procedure is required when you want to enable it from cli. Earlier, there was no support form UI so we recommend to follow below procedure:

How to enable Kube-apiserver Audit logs?

The audit log of the Kubernetes API Server (kube-apiserver) is an essential component of Kubernetes security. It keeps track of all API server requests, including who initiated the request, what action was taken, and when it was completed. RKE2, Rancher's lightweight Kubernetes deployment, makes enabling the kube-API server audit log simple. This article guides the procedures to allow the kube-apiserver audit log in RKE2.

Step 1: Update the RKE2 Configuration File 

The first step is to enable the audit log by updating the RKE2 config file. Accomplish this by executing the following command:

• echo "audit-policy-file: /etc/rancher/rke2/audit-policy.yaml" >> /etc/rancher/rke2/config.yaml

This command adds a new line to the RKE2 configuration file indicating the location of the audit policy file.

Step 2: Create an Audit Policy File

Create an audit policy file at the location specified in the RKE2 configuration file. Accomplish that by running the following command:

cat <<EOF > /etc/rancher/rke2/audit-policy.yaml

apiVersion: audit.k8s.io/v1

kind: Policy

rules:

- level: Metadata

EOF

This command creates a new file named audit-policy.yaml in the /etc/rancher/rke2/ directory with the provided audit policy. The above policy only records metadata-level audit events, which include request and response metadata.

Step 3: Restart RKE2 Server

Restart the RKE2 server after creating the audit policy file to apply the modifications.

• systemctl restart rke2-server

Step 4: Monitor Audit Log

Finally, monitor the audit log by running the following command:

• tail -f /var/lib/rancher/rke2/server/logs/audit.log

This command tails the audit log file in real-time, displaying the latest audit events as they occur.

 Longhorn Debugging & Concepts

This page contains information from multiple resources.

https://longhorn.io/kb/

Longhorn CSI-PLUGIN:

The Longhorn CSI plugin interfaces Kubernetes (the container Orchestration platform ) and Longhorn (the storage solution for persistent data).

In other words, it allows the Kubernetes engine to talk to the storage engine. If the CSI plugin is not starting, it can be caused by many issues but is usually symptomatic of a more fundamental problem.

Diagnosing

1. If the longhorn-csi-plugin pod is not starting, it is probably in a CrashLoopBackOff state. The first thing to do to is list to get more info about the state.
   1. kubectl -n longhorn-system get pods -l app=longhorn-csi-plugin -o wide
   2. In the output pay attention to the READY column. If the READY column is 1/2, it most likely means that the CSI-Plugin cannot talk to longhorn backend. This will be covered in later steps.
      • Example: longhorn-system longhorn-csi-plugin-jwvnj 1/2 CrashLoopBackOff
   3. Also pay attention to which node the failed pod is running on.
2. Next, do a describe of the pod to make sure that any other important information is identified
   1. kubectl -n longhorn-system get pods -l app=longhorn-csi-plugin --- This returns a list of the pods that are running in the cluster. Look for the one that is in a crashed state
   2. kubectl -n describe pod
   3. Alternatively run the command: kubectl -n longhorn-system describe pod -l app=longhorn-csi-plugin
   4. Look at the events associated with the Pod. They are at the end of the output and usually they include some helpful information.
   5. Most likely the error will be: Back-off restarting failed container
3. Try to get the logs from the crashed instance:
1. kubectl -n longhorn-system get pods -l app=longhorn-csi-plugin --- identity the pod that is in a crashed state
2. kubectl -n longhorn-system logs -c longhorn-csi-plugin (Note, the -p option might need to be added at the end of the command.)
   • Check for the error: Failed to initialize Longhorn API client: Get \"http://longhorn-backend:9500/v1\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
3. Get the logs for the node-driver-registrar
   • kubectl -n longhorn-system logs node-driver-registrar (Note, the -p option might need to be added at the end of the command.)
   • Check for the error: W0423 04:16:48.096462 11095 connection.go:173] Still connecting to unix:///csi/csi.sock
4. If the two errors mentioned above are seen, the issue is most likely that the different nodes in the network cannot talk to each other and something is wrong with the overlay network. Go to the Section Debugging Node Overlay Network

-- Also, Please check the node status in K8s as well as Longhorn UI.

In most cases the network issue might be the cause of the communication issue.