-
질문 & 답변
카테고리
-
세부 분야
데브옵스 · 인프라
-
해결 여부
미해결
pending 에서 진행이 되지 않습니다
23.04.21 08:28 작성 조회수 635
0
[질문 하기]
kubectl run nginx --image=nginx
이후 pending 상태에서 넘어가지 않습니다
kubectl describe nginx
kubectl get pods -A 에 대한 결과입니다
[root@m-k8s ~]# kubectl describe nginx
error: the server doesn't have a resource type "nginx"
[root@m-k8s ~]# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx 0/1 Pending 0 103s
kube-system calico-kube-controllers-6856fc6d6c-s8z7m 0/1 Pending 0 11h
kube-system calico-node-kjpsz 0/1 Init:ImagePullBackOff 0 10h
kube-system calico-node-kshdg 0/1 Init:ImagePullBackOff 0 11h
kube-system calico-node-zxbq7 0/1 Init:ImagePullBackOff 0 11h
kube-system coredns-565d847f94-tzrtt 0/1 Pending 0 11h
kube-system coredns-565d847f94-zbhvn 0/1 Pending 0 11h
kube-system etcd-m-k8s 1/1 Running 0 11h
kube-system kube-apiserver-m-k8s 1/1 Running 0 11h
kube-system kube-controller-manager-m-k8s 1/1 Running 0 11h
kube-system kube-proxy-8fhvk 1/1 Running 0 11h
kube-system kube-proxy-c5lt8 1/1 Running 0 11h
kube-system kube-proxy-sc7wg 1/1 Running 0 10h
kube-system kube-scheduler-m-k8s 1/1 Running 0 11h
답변을 작성해보세요.
0
조훈(Hoon Jo)
지식공유자2023.04.22
꼭 꺼야할 필요는 없긴 합니다...
그냥 마음 편하게 그리고 다시 구성해서 올라오는게 나으실꺼 같아서요.
계속 다시 시도할테니 다른 작업하다가 보시면 진행 완료되어 있을 거에요.
0
0
조훈(Hoon Jo)
지식공유자2023.04.21
안녕하세요
어떠한 이유인지 지금 바로 파악하긴 어려운데 Calico(CNI) 파드가 정상 동작하지 못해서 발생한 이슈이고요.
Image를 못 받아왔다고 하는데...
우선 간단하게는 전체 VM들을 다시 시작해 보시고 같은 문제가 발생하는지 파악해 보는게 좋을 것 같습니다.
이후에도 같은 문제가 발생하신다면 kubectl describe pod calico-node-<xxxxx> -n kube-system
kubectl get pod calico-node-<xxxxx> -o yaml -n kube-system | grep -i image:
의 내용을 공유해 주실 수 있을까요?
parkjsyj
질문자2023.04.21
[root@m-k8s ~]# kubectl describe pod calico-node-kjpsz -n kube-system
Name: calico-node-kjpsz
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: calico-node
Node: w2-k8s/192.168.1.102
Start Time: Thu, 20 Apr 2023 21:29:19 +0900
Labels: controller-revision-hash=5ddf5b4749
k8s-app=calico-node
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 192.168.1.102
IPs:
IP: 192.168.1.102
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID: containerd://a24084a73f6a4e538045b6b78b91b308357e6fb19671247c571185c776533e35
Image: docker.io/calico/cni:v3.17.1
Image ID: docker.io/calico/cni@sha256:3dc2506632843491864ce73a6e73d5bba7d0dc25ec0df00c1baa91d17549b068
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 21 Apr 2023 18:37:11 +0900
Finished: Fri, 21 Apr 2023 18:37:12 +0900
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrn8p (ro)
install-cni:
Container ID: containerd://760a259bfc133a1f53d91211f1d6a35e557992c69d7022b2d599f9b89224e8a8
Image: docker.io/calico/cni:v3.17.1
Image ID: docker.io/calico/cni@sha256:3dc2506632843491864ce73a6e73d5bba7d0dc25ec0df00c1baa91d17549b068
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 21 Apr 2023 18:37:12 +0900
Finished: Fri, 21 Apr 2023 18:37:16 +0900
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrn8p (ro)
flexvol-driver:
Container ID:
Image: docker.io/calico/pod2daemon-flexvol:v3.17.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/host/driver from flexvol-driver-host (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrn8p (ro)
Containers:
calico-node:
Container ID:
Image: docker.io/calico/node:v3.17.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_IPV4POOL_CIDR: 172.16.0.0/16
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_LOGSEVERITYSCREEN: info
FELIX_HEALTHENABLED: true
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/ from sysfs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrn8p (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sysfs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
flexvol-driver-host:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
HostPathType: DirectoryOrCreate
kube-api-access-xrn8p:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 9h (x14 over 21h) kubelet Failed to pull image "docker.io/calico/cni:v3.17.1": rpc error: code = Unknown desc = failed to p ull and unpack image "docker.io/calico/cni:v3.17.1": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/c alico/cni/manifests/sha256:3dc2506632843491864ce73a6e73d5bba7d0dc25ec0df00c1baa91d17549b068: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal BackOff 9h (x333 over 21h) kubelet Back-off pulling image "docker.io/calico/cni:v3.17.1"
Normal SandboxChanged 4m50s kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 4m50s kubelet Container image "docker.io/calico/cni:v3.17.1" already present on machine
Normal Created 4m50s kubelet Created container upgrade-ipam
Normal Started 4m50s kubelet Started container upgrade-ipam
Normal Pulled 4m49s kubelet Container image "docker.io/calico/cni:v3.17.1" already present on machine
Normal Created 4m49s kubelet Created container install-cni
Normal Started 4m49s kubelet Started container install-cni
Warning Failed 80s (x3 over 3m49s) kubelet Failed to pull image "docker.io/calico/pod2daemon-flexvol:v3.17.1": rpc error: code = Unknown des c = failed to pull and unpack image "docker.io/calico/pod2daemon-flexvol:v3.17.1": failed to copy: httpReadSeeker: failed open: unexpected status code htt ps://registry-1.docker.io/v2/calico/pod2daemon-flexvol/manifests/sha256:48f277d41c35dae051d7dd6f0ec8f64ac7ee6650e27102a41b0203a0c2ce6c6b: 429 Too Many Req uests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.do cker.com/increase-rate-limit
Warning Failed 80s (x3 over 3m49s) kubelet Error: ErrImagePull
Normal BackOff 42s (x4 over 2m30s) kubelet Back-off pulling image "docker.io/calico/pod2daemon-flexvol:v3.17.1"
Warning Failed 42s (x4 over 2m30s) kubelet Error: ImagePullBackOff
Normal Pulling 27s (x4 over 4m45s) kubelet Pulling image "docker.io/calico/pod2daemon-flexvol:v3.17.1"
parkjsyj
질문자2023.04.21
[root@m-k8s ~]# kubectl describe pod calico-node-kjpsz -n kube-system
Name: calico-node-kjpsz
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: calico-node
Node: w2-k8s/192.168.1.102
Start Time: Thu, 20 Apr 2023 21:29:19 +0900
Labels: controller-revision-hash=5ddf5b4749
k8s-app=calico-node
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 192.168.1.102
IPs:
IP: 192.168.1.102
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID: containerd://a24084a73f6a4e538045b6b78b91b308357e6fb19671247c571185c776533e35
Image: docker.io/calico/cni:v3.17.1
Image ID: docker.io/calico/cni@sha256:3dc2506632843491864ce73a6e73d5bba7d0dc25ec0df00c1baa91d17549b068
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 21 Apr 2023 18:37:11 +0900
Finished: Fri, 21 Apr 2023 18:37:12 +0900
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrn8p (ro)
install-cni:
Container ID: containerd://760a259bfc133a1f53d91211f1d6a35e557992c69d7022b2d599f9b89224e8a8
Image: docker.io/calico/cni:v3.17.1
Image ID: docker.io/calico/cni@sha256:3dc2506632843491864ce73a6e73d5bba7d0dc25ec0df00c1baa91d17549b068
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 21 Apr 2023 18:37:12 +0900
Finished: Fri, 21 Apr 2023 18:37:16 +0900
Ready: True
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrn8p (ro)
flexvol-driver:
Container ID:
Image: docker.io/calico/pod2daemon-flexvol:v3.17.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/host/driver from flexvol-driver-host (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrn8p (ro)
Containers:
calico-node:
Container ID:
Image: docker.io/calico/node:v3.17.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_IPV4POOL_CIDR: 172.16.0.0/16
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_LOGSEVERITYSCREEN: info
FELIX_HEALTHENABLED: true
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/ from sysfs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrn8p (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sysfs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
flexvol-driver-host:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
HostPathType: DirectoryOrCreate
kube-api-access-xrn8p:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 9h (x14 over 21h) kubelet Failed to pull image "docker.io/calico/cni:v3.17.1": rpc error: code = Unknown desc = failed to p ull and unpack image "docker.io/calico/cni:v3.17.1": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/c alico/cni/manifests/sha256:3dc2506632843491864ce73a6e73d5bba7d0dc25ec0df00c1baa91d17549b068: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal BackOff 9h (x333 over 21h) kubelet Back-off pulling image "docker.io/calico/cni:v3.17.1"
Normal SandboxChanged 4m50s kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 4m50s kubelet Container image "docker.io/calico/cni:v3.17.1" already present on machine
Normal Created 4m50s kubelet Created container upgrade-ipam
Normal Started 4m50s kubelet Started container upgrade-ipam
Normal Pulled 4m49s kubelet Container image "docker.io/calico/cni:v3.17.1" already present on machine
Normal Created 4m49s kubelet Created container install-cni
Normal Started 4m49s kubelet Started container install-cni
Warning Failed 80s (x3 over 3m49s) kubelet Failed to pull image "docker.io/calico/pod2daemon-flexvol:v3.17.1": rpc error: code = Unknown des c = failed to pull and unpack image "docker.io/calico/pod2daemon-flexvol:v3.17.1": failed to copy: httpReadSeeker: failed open: unexpected status code htt ps://registry-1.docker.io/v2/calico/pod2daemon-flexvol/manifests/sha256:48f277d41c35dae051d7dd6f0ec8f64ac7ee6650e27102a41b0203a0c2ce6c6b: 429 Too Many Req uests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.do cker.com/increase-rate-limit
Warning Failed 80s (x3 over 3m49s) kubelet Error: ErrImagePull
Normal BackOff 42s (x4 over 2m30s) kubelet Back-off pulling image "docker.io/calico/pod2daemon-flexvol:v3.17.1"
Warning Failed 42s (x4 over 2m30s) kubelet Error: ImagePullBackOff
Normal Pulling 27s (x4 over 4m45s) kubelet Pulling image "docker.io/calico/pod2daemon-flexvol:v3.17.1"
[root@m-k8s ~]# ^C
[root@m-k8s ~]# kubectl describe pod calico-node-kshdg -n kube-system
Name: calico-node-kshdg
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Service Account: calico-node
Node: m-k8s/192.168.1.10
Start Time: Thu, 20 Apr 2023 21:00:04 +0900
Labels: controller-revision-hash=5ddf5b4749
k8s-app=calico-node
pod-template-generation=1
Annotations: <none>
Status: Pending
IP: 192.168.1.10
IPs:
IP: 192.168.1.10
Controlled By: DaemonSet/calico-node
Init Containers:
upgrade-ipam:
Container ID:
Image: docker.io/calico/cni:v3.17.1
Image ID:
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/calico-ipam
-upgrade
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
Mounts:
/host/opt/cni/bin from cni-bin-dir (rw)
/var/lib/cni/networks from host-local-net-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zt7ch (ro)
install-cni:
Container ID:
Image: docker.io/calico/cni:v3.17.1
Image ID:
Port: <none>
Host Port: <none>
Command:
/opt/cni/bin/install
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
CNI_CONF_NAME: 10-calico.conflist
CNI_NETWORK_CONFIG: <set to the key 'cni_network_config' of config map 'calico-config'> Optional: false
KUBERNETES_NODE_NAME: (v1:spec.nodeName)
CNI_MTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
SLEEP: false
Mounts:
/host/etc/cni/net.d from cni-net-dir (rw)
/host/opt/cni/bin from cni-bin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zt7ch (ro)
flexvol-driver:
Container ID:
Image: docker.io/calico/pod2daemon-flexvol:v3.17.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/host/driver from flexvol-driver-host (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zt7ch (ro)
Containers:
calico-node:
Container ID:
Image: docker.io/calico/node:v3.17.1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 250m
Liveness: exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
Readiness: exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
Environment Variables from:
kubernetes-services-endpoint ConfigMap Optional: true
Environment:
DATASTORE_TYPE: kubernetes
WAIT_FOR_DATASTORE: true
NODENAME: (v1:spec.nodeName)
CALICO_NETWORKING_BACKEND: <set to the key 'calico_backend' of config map 'calico-config'> Optional: false
CLUSTER_TYPE: k8s,bgp
IP: autodetect
CALICO_IPV4POOL_IPIP: Always
CALICO_IPV4POOL_VXLAN: Never
FELIX_IPINIPMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_VXLANMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
FELIX_WIREGUARDMTU: <set to the key 'veth_mtu' of config map 'calico-config'> Optional: false
CALICO_IPV4POOL_CIDR: 172.16.0.0/16
CALICO_DISABLE_FILE_LOGGING: true
FELIX_DEFAULTENDPOINTTOHOSTACTION: ACCEPT
FELIX_IPV6SUPPORT: false
FELIX_LOGSEVERITYSCREEN: info
FELIX_HEALTHENABLED: true
Mounts:
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/ from sysfs (rw)
/var/lib/calico from var-lib-calico (rw)
/var/log/calico/cni from cni-log-dir (ro)
/var/run/calico from var-run-calico (rw)
/var/run/nodeagent from policysync (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zt7ch (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
var-run-calico:
Type: HostPath (bare host directory volume)
Path: /var/run/calico
HostPathType:
var-lib-calico:
Type: HostPath (bare host directory volume)
Path: /var/lib/calico
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
sysfs:
Type: HostPath (bare host directory volume)
Path: /sys/fs/
HostPathType: DirectoryOrCreate
cni-bin-dir:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType:
cni-net-dir:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
cni-log-dir:
Type: HostPath (bare host directory volume)
Path: /var/log/calico/cni
HostPathType:
host-local-net-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/cni/networks
HostPathType:
policysync:
Type: HostPath (bare host directory volume)
Path: /var/run/nodeagent
HostPathType: DirectoryOrCreate
flexvol-driver-host:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
HostPathType: DirectoryOrCreate
kube-api-access-zt7ch:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: :NoSchedule op=Exists
:NoExecute op=Exists
CriticalAddonsOnly op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 9h (x19 over 21h) kubelet Failed to pull image "docker.io/calico/cni:v3.17.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.17.1": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:3dc2506632843491864ce73a6e73d5bba7d0dc25ec0df00c1baa91d17549b068: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal BackOff 9h (x448 over 21h) kubelet Back-off pulling image "docker.io/calico/cni:v3.17.1"
Normal SandboxChanged 6m6s kubelet Pod sandbox changed, it will be killed and re-created.
Warning Failed 2m31s (x2 over 5m8s) kubelet Failed to pull image "docker.io/calico/cni:v3.17.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.17.1": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:3dc2506632843491864ce73a6e73d5bba7d0dc25ec0df00c1baa91d17549b068: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Normal Pulling 110s (x4 over 6m6s) kubelet Pulling image "docker.io/calico/cni:v3.17.1"
Warning Failed 48s (x4 over 5m8s) kubelet Error: ErrImagePull
Warning Failed 48s (x2 over 3m52s) kubelet Failed to pull image "docker.io/calico/cni:v3.17.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/cni:v3.17.1": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/calico/cni/manifests/sha256:4f8cbbaf93ef9c549021423ac804ac3e15e366c8a61cf6008b4737d924fe65e2: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
Warning Failed 22s (x6 over 5m8s) kubelet Error: ImagePullBackOff
Normal BackOff 8s (x7 over 5m8s) kubelet Back-off pulling image "docker.io/calico/cni:v3.17.1"
parkjsyj
질문자2023.04.21
[root@m-k8s ~]# kubectl describe pod calico-node-kjpsz yaml -n kube-system | grep -i image:
Image: docker.io/calico/cni:v3.17.1
Image: docker.io/calico/cni:v3.17.1
Image: docker.io/calico/pod2daemon-flexvol:v3.17.1
Image: docker.io/calico/node:v3.17.1
조훈(Hoon Jo)
지식공유자2023.04.21
도커 허브에서 6시간 내에 받을 수 있는 image 갯수의 한계가 있습니다.
고로 현재 클러스터의 VM을 끄시고 6-12시간에 다시 VM을 모두 키시면 될 겁니다.
조훈(Hoon Jo)
지식공유자2023.04.21
답변 3