Knowledge Base:

## -- indicated KB

$$-- Important links

## How to check the orphaned object:

To check the orphan resources using following command 

on both namespaces, delete the those resources and then delete the namespaces:

kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

## How to calculate the memory occupied by Process at time oom kernal panic

MacBook-Pro systemlogs % grep kernel /var/log/messages | rev | cut -d"]" -f1 | rev | awk '{ print $3, $4, $5, $8 }' | grep '^[0-9].*[a-zA-Z][a-zA-Z]' | awk '{total_vm+=$1; rss+=$2; pgtables_bytes+=$3; db[$4]=db[$4]+$2;} END {for (name in db) printf("%.1fG %s\n",(db[name] * 4096)/(1024*1024*1024),name)}' | sort -g -r | head -n 10

8.1G mysql

15.9G beam.smp

0.0G tail

0.0G stats

0.0G start.sh

0.0G sleep

0.0G sh

0.0G run_erl

0.0G pause

0.0G out

0.0G oom-killer:

zsh: command not found: 8.1G

[ Ref https://stackoverflow.com/questions/9199731/understanding-the-linux-oom-killers-logs ]

## How to generate the SSH keys and copy to all nodes:

$$ How to generate ssh-key and copy to all the rke nodes & copy missing certs

https://jrisch.medium.com/how-to-force-a-certificate-rotation-with-rke-on-a-k8s-cluster-built-with-rancher-where-the-e73d89735d3f

$$ 

How to create separate ETCD and Controlplane nodes in RKE2 --
https://www.suse.com/support/kb/doc/?id=000020771

## How to recover the rancher when accidentally deleting the rancher deployment.

-- Make sure you have access to k8s cluster (i.e. get the kubeconfig file  first)

-- First, perform etcd snapshot restore

  #  /rke etcd snapshot-restore --config cluster.yml 2024-06-21T17:32:26Z_etcd

-- Then check the kubectl access 

export KUBECONFIG=kube_config_cluster.yml

#kubectl get nodes

# kubectl get pods -n cattle-system

NAME                              READY   STATUS    RESTARTS   AGE

rancher-webhook-ccf8c9784-vzmvv   1/1     Running   0          21d

--  Then restore the previous rancher app from the previous helm release

# helm ls -n cattle-system

helm history rancher  -n cattle-system

REVISION UPDATED                  STATUS    CHART        APP VERSION DESCRIPTION

1        Fri Apr 26 11:16:44 2024 superseded rancher-2.5.14 v2.5.14   

                                                                    .

2        Fri Apr 26 11:18:53 2024 superseded rancher-2.5.14 v2.5.14    Upgrade complete

3        Mon Jun  3 05:30:22 2024 superseded rancher-2.6.5 v2.6.5      Upgrade complete

4        Mon Jun  3 05:35:01 2024 deployed  rancher-2.6.8 v2.6.8      Upgrade complete

   # /home/ubuntu/rke# helm rollback rancher  -n cattle-system --revision 4

## How to re-generate the  rancher-webhook 

1. Remove rancher.cattle.io validating and mutating webhooks, as well as the webhook service:
kubectl delete mutatingwebhookconfigurations rancher.cattle.io 
kubectl delete validatingwebhookconfigurations rancher.cattle.io
kubectl -n cattle-system delete service webhook-service

2. Navigate to Apps & Marketplace in the local cluster Explorer, Installed Apps and

 perform an 'upgrade' of rancher-webhook to trigger the recreation of deleted resources 

and a new rancher-wehbook-tls certificate secret.

##How to regenerate Service Account tokens in Kubernetes

Situation

Task

At times Service Account tokens could be rendered invalid. 

This could be due to restoring etcd backups or regenerating CA certificates within

 your cluster.

This problem manifests in clients unable to authenticate against the Kubernetes API, for both pods within the cluster, or system components like the controller-manager or kube-proxy. A telling sign would be errors in these clients of the format connect: connection refused or within the Kubernetes API Server logs, showing a large number of clients failing TLS authentication.

Steps

With an admin kubeconfig sourced for the cluster facing issues, run the command below, to generate the list of kubectl commands required to delete all Service Account token secrets. After running the provided kubectl commands from the output, you will need to recreate pods, e.g. by deleting them, in order to regenerate the Service Account token.

kubectl get secret --all-namespaces | awk '{ if ($3 == "kubernetes.io/service-account-token") print "kubectl -n", $1 " delete secret", $2 }'

##Unable to access the service on /var/run/docker.sock

Problem:

WARN[0000] Failed to set up SSH tunneling for host [1.1.1.1]: Can't retrieve Docker Info: error during

 connect: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info": Unable to access the service 

on /var/run/docker.sock. 

The service might be still starting up. Error: ssh: rejected: connect failed (open failed)
WARN[0000] Removing host [1.1.1.1] from node lists
FATA[0000] Cluster must have at least one etcd plane host: failed to connect to the 

following etcd host(s) [1.1.1.1]

Solution:

ubuntu@ip-1.1.1.1~$ cat /etc/group | grep docker

docker:x:999:

ubuntu@ip-1.1.1.1:~$ sudo chmod 666 /var/run/docker.sock
ubuntu@ip-1.1.1.1:~$ docker ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

## Permission denied even after giving 777

Problem:

https://stackoverflow.com/questions/58128001/could-not-open-file-lol-json-permission-denied-using-jq

Stack Overflow

Could not open file lol.json: Permission denied using Jq?

I am facing some kind of permission issue here. I am able to read a file with cat, make changes to it using 

nano but for some reason, jq is having permission issues. Here is what I am facing: 

Solution

sudo snap remove jq
sudo apt install jq

Never install command line tools with snap without --classic. This error is indented behavior.

$$ RKE2 Debug:

https://gist.github.com/superseb/3b78f47989e0dbc1295486c186e944bf (Commands)

https://forum.uipath.com/t/rke2-stuck-in-activating/570680

$$ Networking:

https://www.rigacci.org/wiki/lib/exe/fetch.php/doc/appunti/linux/sa/iptables/conntrack.html

$$ nf_conntrack: table full, dropping packets

https://www.suse.com/support/kb/doc/?id=000020149

##  How nf_conntrack max value is calculated?

table size is calculated based on the memory allocated to the nod. like 

For systems with more than 4GB of memory it will be 65536 buckets. So in 8 GB RAM system the default value is 65536*2 (4GB*2=8GB)=131072

In a 8Gb node:

Max 

cat /proc/meminfo
MemTotal:        8019616 kB

cat  /proc/sys/net/netfilter/nf_conntrack_max
131072

What is the current value:

cat /proc/sys/net/netfilter/nf_conntrack_count
581

# How to add env variables in k3s and RKE2

rke2-service  ENV file:

/usr/local/lib/systemd/system/rke2-server.env

k3s-service ENV file:

/etc/systemd/system/k3s.service.env

For example, if the DNS is not pointing to right file then you can update this file and restart the agent/server.

If you're using k3s _or_ rke2, add

```shell

K3S_RESOLV_CONF=/etc/resolv.conf

RKE2_RESOLV_CONF=/etc/resolv.conf 

```

 to k3s/rke2 env files on all nodes, then restart the k3s/rke2 server/agent service. 

Note: If you're not using k3s, check the [Kubernetes DNS Resolution Known Issues](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues) to see how to set kubelet's `--resolv-conf` flag to point to the correct `resolv.conf`.

https://github.com/longhorn/website/pull/404/files (refer this)

## How to see  debug information  of the nslookup

~$ nslookup -debug google.com

Server: 127.0.0.53

Address: 127.0.0.53#53

------------

    QUESTIONS:

google.com, type = A, class = IN

    ANSWERS:

    ->  google.com

internet address = 142.250.77.110

ttl = 47

    AUTHORITY RECORDS:

    ADDITIONAL RECORDS:

------------

Non-authoritative answer:

Name: google.com

Address: 142.250.77.110

------------

    QUESTIONS:

google.com, type = AAAA, class = IN

    ANSWERS:

    ->  google.com

has AAAA address 2404:6800:4007:82a::200e

ttl = 111

    AUTHORITY RECORDS:

    ADDITIONAL RECORDS:

------------

Name: google.com

Address: 2404:6800:4007:82a::200e

$$ Node local DNS -->https://povilasv.me/kubernetes-node-local-dns-cache/

$$ https://www.geeksforgeeks.org/how-to-compress-and-extract-files-using-the-tar-command-on-linux/

## getting this error while running terrform init in MAC M1 

Problem:

"Template v2.2.0does not have a package available for your current platform, darwin_arm64"
Solution: 
working in latest version 1.3.7
brew uninstall terraform
brew install tfenv
TFENV_ARCH=amd64 tfenv install 1.3.7
tfenv use 1.3.7

.
6. If you're not using k3s, check the [Kubernetes DNS Resolution Known Issues](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues) to see how to set kubelet's `--resolv-conf` flag to point to the correct `resolv.conf`.