tcpdump -i eth0 -vvv -nn -s0 -SS -XX '(host 10.43.93.114)
https://github.com/projectcalico/calico/issues/3270 important.
tcpdump -ni eth0 icmp and src 10.15.32.11 or dst 10.15.32.11
we check the iptables rules it created? The basic steps should be:
- `docker inspect <container ID for ingress-nginx> | grep Pid`
- `nsenter -n -t <PID from above> iptables -nvL`
- `nsenter -n -t <PID from above> iptables -nvL -t nat`
https://github.com/rancherlabs/support-kb/wiki/Common-Errors#kernel-nf_conntrack-falling-back-to-vmalloc
Just so that this does not get lost, this is what should happen to all packets:
The ping path should be the following:
1 - echo request egresses on interface eth0 of the pod
2 - echo request reaches the other end of the veth (cali interface)
3 - linux kernel fowards the packet to the destination veth (cali interface)
4 - packet reaches the other end of the veth (eth0 of the destination pod)
5 - The other way around but echo reply
We should understand what step is not followed by the faulty packets to further debug. I suggest sniffing all interfaces with tcpdump (tcpdump -ni $INTERFACE icmp):
1 - eth0 of sourcePod
2 - cali interface of sourcePod
3 - cali interface of destPod
4 - eth0 of destPodThen start the ping that has problems^ And having the `ip r` output of the node and both pods will give us a lot of information too
####
## Why does my container have a route to 169.254.1.1?[](https://docs.tigera.io/calico/latest/reference/faq#why-does-my-container-have-a-route-to-16925411 "Direct link to Why does my container have a route to 169.254.1.1?")
In a Calico network, each host acts as a gateway router for the workloads that it hosts. In container deployments, Calico uses 169.254.1.1 as the address for the Calico router. By using a link-local address, Calico saves precious IP addresses and avoids burdening the user with configuring a suitable address.
While the routing table may look a little odd to someone who is used to configuring LAN networking, using explicit routes rather than subnet-local gateways is fairly common in WAN networking.
## Why can't I see the 169.254.1.1 address mentioned above on my host?[](https://docs.tigera.io/calico/latest/reference/faq#why-cant-i-see-the-16925411-address-mentioned-above-on-my-host "Direct link to Why can't I see the 169.254.1.1 address mentioned above on my host?")
Calico tries hard to avoid interfering with any other configuration on the host. Rather than adding the gateway address to the host side of each workload interface, Calico sets the `proxy_arp` flag on the interface. This makes the host behave like a gateway, responding to ARPs for 169.254.1.1 without having to actually allocate the IP address to the interface.
Why do all cali* interfaces have the MAC address ee:ee:ee:ee:ee:ee?[](https://docs.tigera.io/calico/latest/reference/faq#why-do-all-cali-interfaces-have-the-mac-address-eeeeeeeeeeee "Direct link to why-do-all-cali-interfaces-have-the-mac-address-eeeeeeeeeeee")
In some setups the kernel is unable to generate a persistent MAC address and so Calico assigns a MAC address itself. Since Calico uses point-to-point routed interfaces, traffic does not reach the data link layer so the MAC Address is never used and can therefore be the same for all the cali* interfaces.
##############
Try `tcpdump -ni any host 192.168.122.75`. Under normal circumstances it should show you
1. syn-synack-ack (if fw open), that's `[S]`-`[S.]`-`[.]` in tcpdump notation
2. syn-syn-syn-syn-... (if fw drops)
3. syn-rst (if fw rejects or port closed)
Your firewalld might also be set up weirdly to respond with ICMP unreachable or something. You'll see that in tcpdump.
nft list ruleset
Networking/DNS isssue
- You have mirrored prod cluster, and it is deployed 4 days back.
- 2 days back, the nodes were rebooted after which you started having this issue. Before reboot the cluster was working fine and service resolution was also working.
- Current issue is; The pod-to-service DNS resolution is not working. We performed the dnslookup/dnstest with 'kubernetes.default' service as well, but it's also failing.
- We verified the coredns config and it was correctly configured to use the /etc/resolve.conf.
- We verified the iptables and contrack table settings but did not find any issues. The firewalld as well as network manager services (`nm-cloud-setup.service` and `nm-cloud-setup.timer)` are disabled.
- The IP forwarding and netfilter settings are enabled. We verified the kernel as well as /var/log/messages' but did not find any error.
- You have another test cluster where the same setup is working fine.
- We did rollout restart of core-dns deployment as well as did rke2 service restart but it did not help. We checked the network policies but did not find any blocking/deny policies.
- We ran the network overlay test and found that; the nodes were unable to communicate with pod IP's running on other hosts where as the same test is working fine on dev/test cluster
- The cluster/network is managed by your customer and their team, and you do not have any control of it. So, you will check with your customer's networking team on below overlay network. failure error messages.
- Also, please ask them to verify the iptables rules and if there was any recent modification done. It would be good if you can compare and validate the configuration/settings with other working dev cluster.
you can retrive all the namespaces related to swiss-army-knife
########
buntu@testissue-pool1-697442e0-d99kc:~$ ps aux|grep swiss
root 960733 0.0 0.1 109272 6232 ? Ssl 08:22 0:00 swiss-army-knife
ubuntu 969023 0.0 0.0 8160 720 pts/0 S+ 08:37 0:00 grep --color=auto swiss
ubuntu@testissue-pool1-697442e0-d99kc:~$ sudo -i
root@testissue-pool1-697442e0-d99kc:~# lsns -p 960733
NS TYPE NPROCS PID USER COMMAND
4026531835 cgroup 188 1 root /lib/systemd/systemd --system --deserialize 32
4026531837 user 187 1 root /lib/systemd/systemd --system --deserialize 32
4026532650 net 7 960524 65535 /pause
4026532710 uts 7 960524 65535 /pause
4026532711 ipc 7 960524 65535 /pause
4026532713 mnt 6 960733 root swiss-army-knife
4026532714 pid 6 960733 root swiss-army-knife
find out the relevant ns:
, let's find the relevant network namespace.
ls -lt /var/run/netns
ip link | grep -A1 ^12
https://learnk8s.io/kubernetes-network-packets
https://learnk8s.io/setting-cpu-memory-limits-requests
https://learnk8s.io/kubernetes-services-and-load-balancing
######
**ip link show type veth**
12: cali4a8863346b5@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-ada9ce74-b522-142d-2e57-602338690daf
13: calie5e1d4ace9d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-f9dbbafc-db0b-2a05-0ba5-b734e1e971e4
14: cali1261a233ff9@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-9443a445-8fdf-45a7-2525-a9f38244eb0f
16: calie93d281bd10@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-974ea59d-3028-46ee-2f0f-696c317e3e97
19: cali816f506becb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-1adbeb5a-3421-ce62-71a5-86253899d537
23: cali9c0a6b9fdfa@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-10cf5578-f8a6-8e36-bea5-442ee793b577
here 23 is pod interface ********** see below pod ip details :
root@sw-7bb9d677-dw2c4:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc noqueue state UP group default qlen 1000
link/ether b2:84:ae:c0:c7:ae brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.42.127.144/32 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::b084:aeff:fec0:c7ae/64 scope link
ip link set veth2 up or down
bash
```
ip link show type veth
# output truncated
12: cali97e50e215bd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-0f226515-e28b-df13-9f16-dd79456825ac
```
> You can also use `ip -n cni-0f226515-e28b-df13-9f16-dd79456825ac link show type veth`.
Pay attention to the notation on both `3: eth0@if12` and `12: cali97e50e215bd@if3` interfaces.
From the pod namespace, the `eth0` interface connects to interface number 12 in the root namespace. Hence the `@if12`.
On the other end of the `veth` pair, the root namespace is connected to the pod namespace interface number 3.
Next comes the bridge that will connect each end of the `veth` pairs.
