ACS - CKS - Calico POD issue in Ubuntu 22.04 Image

[amitmvyas]

Hi,

I have downloaded Ubuntu 22.04 Image (CKS) from here. When deploying the CKS cluster using this image I am seeing that Calico POD on Master and Worker node is failing because of /etc/hosts file don't have anything configured in it.

amitvyas@LTQQ9JK7N3 .kube % kubectl get pods -A
NAMESPACE              NAME                                               READY   STATUS             RESTARTS          AGE
kube-system            calico-kube-controllers-7bfdc5b57c-m747t           1/1     Running            0                 25h
kube-system            calico-node-6459c                                  0/1     CrashLoopBackOff   21 (4m29s ago)    67m
kube-system            calico-node-vxddg                                  0/1     CrashLoopBackOff   435 (67s ago)     25h
kube-system            calico-node-w9d2n                                  0/1     CrashLoopBackOff   425 (30s ago)     25h
kube-system            calico-node-z84dj                                  0/1     CrashLoopBackOff   21 (4m44s ago)    67m
kube-system            calico-node-zfm5x                                  0/1     Running            434 (5m46s ago)   25h
kube-system            cloud-controller-manager-6887fc69c8-jpcxw          0/1     CrashLoopBackOff   13 (3m55s ago)    49m
kube-system            cloudstack-csi-controller-85f5b85cdb-2hbdd         7/7     Running            0                 25h
kube-system            cloudstack-csi-controller-85f5b85cdb-m25cb         7/7     Running            0                 25h
kube-system            cloudstack-csi-node-759zr                          3/3     Running            0                 25h
kube-system            cloudstack-csi-node-94k6s                          3/3     Running            0                 25h
kube-system            cloudstack-csi-node-q75pb                          3/3     Running            0                 25h
kube-system            cloudstack-csi-node-r9kdg                          3/3     Running            0                 25h
kube-system            cloudstack-csi-node-s68hl                          3/3     Running            0                 25h
kube-system            coredns-674b8bbfcf-qsv5h                           1/1     Running            0                 25h
kube-system            coredns-674b8bbfcf-s75qm                           1/1     Running            0                 25h
kube-system            etcd-cl01-control-19d8025b62c                      1/1     Running            0                 25h
kube-system            etcd-cl01-control-19d802668f0                      1/1     Running            0                 25h
kube-system            etcd-cl01-control-19d80271aad                      1/1     Running            0                 25h
kube-system            kube-apiserver-cl01-control-19d8025b62c            1/1     Running            0                 25h
kube-system            kube-apiserver-cl01-control-19d802668f0            1/1     Running            1 (25h ago)       25h
kube-system            kube-apiserver-cl01-control-19d80271aad            1/1     Running            1 (25h ago)       25h
kube-system            kube-controller-manager-cl01-control-19d8025b62c   1/1     Running            0                 25h
kube-system            kube-controller-manager-cl01-control-19d802668f0   1/1     Running            0                 25h
kube-system            kube-controller-manager-cl01-control-19d80271aad   1/1     Running            0                 25h
kube-system            kube-proxy-296bl                                   1/1     Running            0                 25h
kube-system            kube-proxy-5rzds                                   1/1     Running            0                 25h
kube-system            kube-proxy-dqqsc                                   1/1     Running            0                 25h

kube-system            kube-proxy-j2mmj                                   1/1     Running            0                 25h
kube-system            kube-proxy-p74bg                                   1/1     Running            0                 25h
kube-system            kube-scheduler-cl01-control-19d8025b62c            1/1     Running            0                 25h
kube-system            kube-scheduler-cl01-control-19d802668f0            1/1     Running            0                 25h
kube-system            kube-scheduler-cl01-control-19d80271aad            1/1     Running            0                 25h
kubernetes-dashboard   dashboard-metrics-scraper-5bd45c9dd6-hg8cm         1/1     Running            0                 25h
kubernetes-dashboard   kubernetes-dashboard-687457b9bd-46fsb              1/1     Running            1 (25h ago)       25h
amitvyas@LTQQ9JK7N3 .kube % 

amitvyas@LTQQ9JK7N3 .kube % kubectl logs calico-node-6459c -n kube-system 
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
2026-04-13 07:27:18.441 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:18.990 [INFO][90] felix/int_dataplane.go 2201: Received *proto.HostMetadataV4V6Update update from calculation graph. msg=hostname:"cl01-node-19d80288ac0" ipv4_addr:"10.1.0.107/24" labels:{key:"beta.kubernetes.io/arch" value:"amd64"} labels:{key:"beta.kubernetes.io/os" value:"linux"} labels:{key:"kubernetes.io/arch" value:"amd64"} labels:{key:"kubernetes.io/hostname" value:"cl01-node-19d80288ac0"} labels:{key:"kubernetes.io/os" value:"linux"} labels:{key:"topology.csi.cloudstack.apache.org/zone" value:"48e11a80-49e5-4e2b-bf11-f21030cd8221"}
2026-04-13 07:27:18.990 [INFO][90] felix/int_dataplane.go 2126: Dataplane updates no longer throttled
2026-04-13 07:27:19.450 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:20.458 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:21.467 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:22.475 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:23.484 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:24.494 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:25.503 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:26.510 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:27.517 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:28.526 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:29.535 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:29.586 [INFO][90] felix/int_dataplane.go 2201: Received *proto.HostMetadataV4V6Update update from calculation graph. msg=hostname:"cl01-control-19d8025b62c" ipv4_addr:"10.1.0.249/24" labels:{key:"beta.kubernetes.io/arch" value:"amd64"} labels:{key:"beta.kubernetes.io/os" value:"linux"} labels:{key:"kubernetes.io/arch" value:"amd64"} labels:{key:"kubernetes.io/hostname" value:"cl01-control-19d8025b62c"} labels:{key:"kubernetes.io/os" value:"linux"} labels:{key:"node-role.kubernetes.io/control-plane" value:""} labels:{key:"node.kubernetes.io/exclude-from-external-load-balancers" value:""} labels:{key:"topology.csi.cloudstack.apache.org/zone" value:"48e11a80-49e5-4e2b-bf11-f21030cd8221"}
2026-04-13 07:27:30.542 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:30.772 [INFO][90] felix/int_dataplane.go 2201: Received *proto.HostMetadataV4V6Update update from calculation graph. msg=hostname:"cl01-node-19d8027d736" ipv4_addr:"10.1.0.42/24" labels:{key:"beta.kubernetes.io/arch" value:"amd64"} labels:{key:"beta.kubernetes.io/os" value:"linux"} labels:{key:"kubernetes.io/arch" value:"amd64"} labels:{key:"kubernetes.io/hostname" value:"cl01-node-19d8027d736"} labels:{key:"kubernetes.io/os" value:"linux"} labels:{key:"topology.csi.cloudstack.apache.org/zone" value:"48e11a80-49e5-4e2b-bf11-f21030cd8221"}
2026-04-13 07:27:31.551 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:32.559 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:33.567 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:34.575 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:35.584 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:36.592 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:37.600 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:38.608 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:39.373 [INFO][90] felix/int_dataplane.go 2201: Received *proto.HostMetadataV4V6Update update from calculation graph. msg=hostname:"cl01-control-19d802668f0" ipv4_addr:"10.1.0.61/24" labels:{key:"beta.kubernetes.io/arch" value:"amd64"} labels:{key:"beta.kubernetes.io/os" value:"linux"} labels:{key:"kubernetes.io/arch" value:"amd64"} labels:{key:"kubernetes.io/hostname" value:"cl01-control-19d802668f0"} labels:{key:"kubernetes.io/os" value:"linux"} labels:{key:"node-role.kubernetes.io/control-plane" value:""} labels:{key:"node.kubernetes.io/exclude-from-external-load-balancers" value:""} labels:{key:"topology.csi.cloudstack.apache.org/zone" value:"48e11a80-49e5-4e2b-bf11-f21030cd8221"}
2026-04-13 07:27:39.614 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:40.622 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:41.628 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:42.636 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:43.644 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:44.652 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:45.659 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:46.666 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:47.675 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:48.705 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:49.715 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:50.722 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:51.730 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:52.738 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:53.745 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:54.755 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:55.762 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
bird: Mesh_10_1_0_249: State changed to start
2026-04-13 07:27:56.770 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:57.776 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:58.785 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:27:59.795 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:28:00.803 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:28:01.811 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:28:02.820 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:28:03.829 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:28:04.837 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:28:05.846 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
2026-04-13 07:28:06.854 [ERROR][90] felix/health.go 390: Health endpoint failed, trying to restart it... error=listen tcp: lookup localhost on 8.8.8.8:53: no such host
amitvyas@LTQQ9JK7N3 .kube % 

Once I login to either master / worker node and make an manual entry in /etc/hosts as 127.0.0.1 localhost then it start working as expected.

I haven't seen this happening when deploying CKS using system template.

Has any once faced such issue ?

Regards,
Amit Vyas