Сеть времени выполнения контейнера Kubernetes не готова

Я установил кластер Kubernetes из трех узлов, контрольный узел выглядел нормально, когда я попытался присоединиться к двум другим узлам, статус обоих: не готов.

На контрольном узле:

[root@alva-anstawx01 /]# kubectl get nodes     
NAME                             STATUS     ROLES                  AGE     VERSION
alva-anstawx01.<domain name>   Ready      control-plane,master   7d20h   v1.21.1
alva-anstawx02.<domain name>   NotReady   <none>                 22h     v1.21.1
alva-anstawx03.<domain name>   NotReady   <none>                 22h     v1.21.1

Модули POD выглядят нормально и все работают:

NAME                                                     READY   STATUS    RESTARTS   AGE
coredns-558bd4d5db-c8p97                                 1/1     Running   0          7d20h
coredns-558bd4d5db-hjb8g                                 1/1     Running   0          7d20h
etcd-alva-anstawx01.alv.autoliv.int                      1/1     Running   2          7d20h
kube-apiserver-alva-anstawx01.alv.autoliv.int            1/1     Running   2          7d20h
kube-controller-manager-alva-anstawx01.alv.autoliv.int   1/1     Running   2          7d20h
kube-proxy-b8ft2                                         1/1     Running   0          7d20h
kube-proxy-frr7c                                         1/1     Running   0          23h
kube-proxy-ztxbf                                         1/1     Running   0          23h
kube-scheduler-alva-anstawx01.alv.autoliv.int            1/1     Running   2          7d20h

При дальнейшей проверке журналов выясняется, что чего-то не хватает, поэтому плагин CNI запускается на этих узлах и не знает, как действовать дальше:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Wed, 09 Jun 2021 12:24:34 +0200   Tue, 08 Jun 2021 14:00:45 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 09 Jun 2021 12:24:34 +0200   Tue, 08 Jun 2021 14:00:45 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 09 Jun 2021 12:24:34 +0200   Tue, 08 Jun 2021 14:00:45 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Wed, 09 Jun 2021 12:24:34 +0200   Tue, 08 Jun 2021 14:00:45 +0200   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

У меня один сетевой интерфейс на каждом узле

На затронутом узле:

Jun 09 12:34:19 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:19.209657    1026 kubelet.go:2211] "Container runtime network not ready" networkReady="N
Jun 09 12:34:19 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:19.698034    1026 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"Sta
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817375    1026 remote_image.go:114] "PullImage from image service failed" err="rpc er
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817429    1026 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code =
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817627    1026 kuberuntime_manager.go:864] container &Container{Name:calico-typha,Ima
Jun 09 12:34:21 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:21.817706    1026 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"Sta
Jun 09 12:34:24 alva-anstawx02.alv.<Domain Name> kubelet[1026]: E0609 12:34:24.211195    1026 kubelet.go:2211] "Container runtime network not ready" networkReady="N

Я использовал конфигурацию Calico по умолчанию, и у меня один интерфейс на каждом узле:

Узел управления:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:ac:64:8c brd ff:ff:ff:ff:ff:ff
    inet 10.4.9.73/21 brd 10.4.15.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feac:648c/64 scope link 
       valid_lft forever preferred_lft forever
3: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default 
    link/ether 66:c5:7d:06:e5:fe brd ff:ff:ff:ff:ff:ff
    inet 192.168.228.192/32 scope global vxlan.calico
       valid_lft forever preferred_lft forever
    inet6 fe80::64c5:7dff:fe06:e5fe/64 scope link 
       valid_lft forever preferred_lft forever
4: cali5441eeb56bd@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
5: cali389c5f98ecc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
6: calicc306a285eb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever

Другие узлы:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:ac:67:61 brd ff:ff:ff:ff:ff:ff
    inet 10.4.9.80/21 brd 10.4.15.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feac:6761/64 scope link 
       valid_lft forever preferred_lft forever

Может ли кто-нибудь помочь мне, пожалуйста, как инициализировать ситцево на двух других узлах?

Редактировать:

Я решил проблему с / var space, но это не помогло:

[root@alva-anstawx03 ~]# df -kh
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                       1.9G     0  1.9G   0% /dev
tmpfs                          1.9G     0  1.9G   0% /dev/shm
tmpfs                          1.9G   60M  1.8G   4% /run
tmpfs                          1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/vg_system-lv_root  9.8G  2.1G  7.2G  23% /
/dev/sda1                      976M  206M  704M  23% /boot
/dev/mapper/vg_system-lv_var    19G  1.1G   17G   6% /var
/dev/mapper/vg_system-lv_opt   3.9G   72M  3.6G   2% /opt
tmpfs                          1.9G   12K  1.9G   1% /var/lib/kubelet/pods/77b1f7f0-8598-4640-af2f-e960c4c76120/volumes/kubernetes.io~projected/kube-api-access-7xnp8
tmpfs                          1.9G   12K  1.9G   1% /var/lib/kubelet/pods/4398eeeb-0f74-477c-a066-403ecab4abe1/volumes/kubernetes.io~projected/kube-api-access-9bh4m
shm                             64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/56cbc698b06f57937128eadc74cc098c4dfb9f5566e941d7a93baab9695ec22e/shm
shm                             64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/3cb246ac67ca0826ca86f8adb5c5c1b8802c4f96ca330456aea67aec02231f9c/shm
overlay                         19G  1.1G   17G   6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/56cbc698b06f57937128eadc74cc098c4dfb9f5566e941d7a93baab9695ec22e/rootfs
overlay                         19G  1.1G   17G   6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/3cb246ac67ca0826ca86f8adb5c5c1b8802c4f96ca330456aea67aec02231f9c/rootfs
overlay                         19G  1.1G   17G   6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/f3e2bedafb61411951557c6e66d037536240bf25a185e6b3e6da0b6ad0b91a38/rootfs
tmpfs                          378M     0  378M   0% /run/user/0
[root@alva-anstawx03 ~]#

То же самое на другом узле:

[root@alva-anstawx02 ~]# df -kh
Filesystem                     Size  Used Avail Use% Mounted on
devtmpfs                       1.9G     0  1.9G   0% /dev
tmpfs                          1.9G     0  1.9G   0% /dev/shm
tmpfs                          1.9G   68M  1.8G   4% /run
tmpfs                          1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/vg_system-lv_root  9.8G  2.1G  7.2G  23% /
/dev/sda1                      976M  206M  704M  23% /boot
/dev/mapper/vg_system-lv_opt   3.9G   72M  3.6G   2% /opt
/dev/mapper/vg_system-lv_var    19G  1.1G   17G   6% /var
tmpfs                          1.9G   12K  1.9G   1% /var/lib/kubelet/pods/31e01070-282a-453b-8e7f-fe0d93e359ec/volumes/kubernetes.io~projected/kube-api-access-4qhqs
tmpfs                          1.9G   12K  1.9G   1% /var/lib/kubelet/pods/4208e857-28e7-4005-bbe1-8bed0b08548b/volumes/kubernetes.io~projected/kube-api-access-bvjhg
shm                             64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/f3b43b5c1e46236e3e01536cff2089c788e0b39e34e43165608dbb2ea9906cb5/shm
shm                             64M     0   64M   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/285018acde46e519f9ae74551d06028304ca19ab76813ed1ca43a4b6e617e4f4/shm
overlay                         19G  1.1G   17G   6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/285018acde46e519f9ae74551d06028304ca19ab76813ed1ca43a4b6e617e4f4/rootfs
overlay                         19G  1.1G   17G   6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/f3b43b5c1e46236e3e01536cff2089c788e0b39e34e43165608dbb2ea9906cb5/rootfs
overlay                         19G  1.1G   17G   6% /run/containerd/io.containerd.runtime.v2.task/k8s.io/42aaae5f8c681ffa5fd0bf6ed3fcf4d8447962131459d4592d1bbb73a320edca/rootfs
tmpfs                          378M     0  378M   0% /run/user/0
[root@alva-anstawx02 ~]# 

Ниже показан результат работы kubectl:

[root@alva-anstawx01 ~]#  kubectl get node
NAME                             STATUS     ROLES                  AGE   VERSION
alva-anstawx01.<Domain Name>   Ready      control-plane,master   8d    v1.21.1
alva-anstawx02.<Domain Name>   NotReady   <none>                 43h   v1.21.1
alva-anstawx03.<Domain Name>   NotReady   <none>                 43h   v1.21.1
[root@alva-anstawx01 ~]# kubectl describe pod calico-node-dshv9 -n kube-system
Name:                 calico-node-dshv9
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 alva-anstawx03.<Domain Name>/10.4.9.96
Start Time:           Wed, 09 Jun 2021 20:39:52 +0200
Labels:               controller-revision-hash=c54f47b5c
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Pending
IP:                   10.4.9.96
IPs:
  IP:           10.4.9.96
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  
    Image:         docker.io/calico/cni:v3.19.1
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
  install-cni:
    Container ID:  
    Image:         docker.io/calico/cni:v3.19.1
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/install
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
  flexvol-driver:
    Container ID:   
    Image:          docker.io/calico/pod2daemon-flexvol:v3.19.1
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
Containers:
  calico-node:
    Container ID:   
    Image:          docker.io/calico/node:v3.19.1
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      kubernetes-services-endpoint  ConfigMap  Optional: true
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_HEALTHENABLED:                true
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /sys/fs/ from sysfs (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9bh4m (ro)
Conditions:
  Type              Status
  Initialized       False 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  sysfs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/
    HostPathType:  DirectoryOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:  
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  kube-api-access-9bh4m:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  13h                     default-scheduler  Successfully assigned kube-system/calico-node-dshv9 to alva-anstawx03.<Domain Name>
  Warning  Failed     13h (x2 over 13h)       kubelet            Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   Pulling    13h (x4 over 13h)       kubelet            Pulling image "docker.io/calico/cni:v3.19.1"
  Warning  Failed     13h (x4 over 13h)       kubelet            Error: ErrImagePull
  Warning  Failed     13h (x2 over 13h)       kubelet            Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed     13h (x6 over 13h)       kubelet            Error: ImagePullBackOff
  Normal   BackOff    12h (x61 over 13h)      kubelet            Back-off pulling image "docker.io/calico/cni:v3.19.1"
  Warning  Failed     12h (x2 over 12h)       kubelet            Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   Pulling    12h (x4 over 12h)       kubelet            Pulling image "docker.io/calico/cni:v3.19.1"
  Warning  Failed     12h (x4 over 12h)       kubelet            Error: ErrImagePull
  Warning  Failed     12h (x2 over 12h)       kubelet            Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed     12h (x6 over 12h)       kubelet            Error: ImagePullBackOff
  Normal   BackOff    12h (x18 over 12h)      kubelet            Back-off pulling image "docker.io/calico/cni:v3.19.1"
  Normal   Pulling    12h (x4 over 12h)       kubelet            Pulling image "docker.io/calico/cni:v3.19.1"
  Warning  Failed     12h (x4 over 12h)       kubelet            Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed     12h (x4 over 12h)       kubelet            Error: ErrImagePull
  Warning  Failed     12h (x6 over 12h)       kubelet            Error: ImagePullBackOff
  Normal   BackOff    12h (x81 over 12h)      kubelet            Back-off pulling image "docker.io/calico/cni:v3.19.1"
  Warning  Failed     12h (x2 over 12h)       kubelet            Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:f301171be0add870152483fcce71b28cafb8e910f61ff003032e9b1053b062c4: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Normal   Pulling    12h (x4 over 12h)       kubelet            Pulling image "docker.io/calico/cni:v3.19.1"
  Warning  Failed     12h (x2 over 12h)       kubelet            Failed to pull image "docker.io/calico/cni:v3.19.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.19.1": failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:51f294c56842e731fa8d7bdf6b9ba39771f69ba4eda28e186461be2662e599df: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed     12h (x4 over 12h)       kubelet            Error: ErrImagePull
  Warning  Failed     12h (x6 over 12h)       kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m45s (x3192 over 12h)  kubelet            Back-off pulling image "docker.io/calico/cni:v3.19.1"
[root@alva-anstawx01 ~]# kubectl get pods -o wide --all-namespaces
NAMESPACE     NAME                                                     READY   STATUS                  RESTARTS   AGE   IP                NODE                             NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-78d6f96c7b-wb96g                 1/1     Running                 1          13h   192.168.228.198   alva-anstawx01.<Domain Name>   <none>           <none>
kube-system   calico-node-dshv9                                        0/1     Init:ImagePullBackOff   0          13h   10.4.9.96         alva-anstawx03.<Domain Name>   <none>           <none>
kube-system   calico-node-rfrnq                                        0/1     Init:ImagePullBackOff   0          13h   10.4.9.80         alva-anstawx02.<Domain Name>   <none>           <none>
kube-system   calico-node-sl864                                        1/1     Running                 1          13h   10.4.9.73         alva-anstawx01.<Domain Name>   <none>           <none>
kube-system   coredns-558bd4d5db-c8p97                                 1/1     Running                 2          8d    192.168.228.200   alva-anstawx01.<Domain Name>   <none>           <none>
kube-system   coredns-558bd4d5db-hjb8g                                 1/1     Running                 2          8d    192.168.228.199   alva-anstawx01.<Domain Name>   <none>           <none>
kube-system   etcd-alva-anstawx01.<Domain Name>                      1/1     Running                 4          8d    10.4.9.73         alva-anstawx01.<Domain Name>   <none>           <none>
kube-system   kube-apiserver-alva-anstawx01.<Domain Name>            1/1     Running                 4          8d    10.4.9.73         alva-anstawx01.<Domain Name>   <none>           <none>
kube-system   kube-controller-manager-alva-anstawx01.<Domain Name>   1/1     Running                 4          8d    10.4.9.73         alva-anstawx01.<Domain Name>   <none>           <none>
kube-system   kube-proxy-b8ft2                                         1/1     Running                 2          8d    10.4.9.73         alva-anstawx01.<Domain Name>   <none>           <none>
kube-system   kube-proxy-frr7c                                         1/1     Running                 4          43h   10.4.9.80         alva-anstawx02.<Domain Name>   <none>           <none>
kube-system   kube-proxy-ztxbf                                         1/1     Running                 4          43h   10.4.9.96         alva-anstawx03.<Domain Name>   <none>           <none>
kube-system   kube-scheduler-alva-anstawx01.<Domain Name>            1/1     Running                 4          8d    10.4.9.73         alva-anstawx01.<Domain Name>   <none>           <none>