7.7 실습문의

Question

7.7 실습 디렉토리 nginx-conf.yaml pod-adapter.yaml 실행 후 [root@m-k8s 7.7]# kubectl get pods -w NAME READY STATUS RESTARTS AGE net 1/1 Running 4 (5m57s ago) 2d2h nfs-client-provisioner-7884c4db4-qz75l 1/1 Running 7 (5m57s ago) 7d1h pod-adapter 2/2 Running 0 7s pod-adapter 1/2 Error 0 9s pod-adapter 2/2 Running 1 (1s ago) 10s pod-adapter 1/2 Error 1 (6s ago) 15s pod-adapter 1/2 CrashLoopBackOff 1 (16s ago) 30s pod-adapter 2/2 Running 2 (16s ago) 30s pod-adapter 1/2 Error 2 (21s ago) 35s pod-adapter 1/2 CrashLoopBackOff 2 (13s ago) 48s pod-adapter 2/2 Running 3 (28s ago) 63s pod-adapter 1/2 Error 3 (34s ago) 69s pod-adapter 1/2 CrashLoopBackOff 3 (11s ago) 79s 마스터노드(m-k8s) 수십분 먹통 마스터노드(m-k8s) 다시 접속후 : [root@m-k8s 7.7]# kubectl get pods NAME READY STATUS RESTARTS AGE net 1/1 Running 4 (12m ago) 2d3h nfs-client-provisioner-7884c4db4-qz75l 1/1 Running 7 (12m ago) 7d1h pod-adapter 1/2 CrashLoopBackOff 5 (2m30s ago) 6m10s 조치사항 : 버추얼박스 VM들 재실행, 노트북 재실행, 방화벽, 백신 프로세스 서비스 종료후 재실행, rerepo-k8s-learning.kit 다시 다운로드, 기존 파드 삭제후 다시 배포 --> 증상은 위와 같이 동일함 [root@m-k8s 7.7]# kubectl get pods -o wide [root@m-k8s 7.7]# curl 172.16.132.16/stub_status Active connections: 1 server accepts handled requests 2 2 2 Reading: 0 Writing: 1 Waiting: 0 [root@m-k8s 7.7]# curl 172.16.132.16:9113/metrics curl: (7) Failed connect to 172.16.132.16:9113; Connection refused

dukim73 · Answer

방금 전에 실습 했더니 정상적으로 실행 됩니다. 다음 부터는 문제가 발생되면 logs, describe 명령으로 확인해 보고 문의 드리겠습니다. ================================================ [root@m-k8s 7.7]# kubectl get pods NAME READY STATUS RESTARTS AGE net 1/1 Running 5 (3m46s ago) 3d12h nfs-client-provisioner-7884c4db4-qz75l 1/1 Running 8 (3m46s ago) 8d pod-adapter 2/2 Running 3 (2m9s ago) 2m43s [root@m-k8s 7.7]# kubectl logs pod-adapter web-page / docker-entrypoint.sh : /docker-entrypoint.d/ is not empty, will attempt to perform configuration / docker-entrypoint.sh : Looking for shell scripts in /docker-entrypoint.d/ / docker-entrypoint.sh : Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh : info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?) / docker-entrypoint.sh : Launching /docker-entrypoint.d/20-envsubst-on-templates.sh / docker-entrypoint.sh : Launching /docker-entrypoint.d/30-tune-worker-processes.sh / docker-entrypoint.sh : Configuration complete; ready for start up 2022/09/05 01:39:21 [notice] 1#1: using the "epoll" event method 2022/09/05 01:39:21 [notice] 1#1: nginx/1.23.1 2022/09/05 01:39:21 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 2022/09/05 01:39:21 [notice] 1#1: OS: Linux 3.10.0-1127.19.1.el7.x86_64 2022/09/05 01:39:21 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2022/09/05 01:39:21 [notice] 1#1: start worker processes 2022/09/05 01:39:21 [notice] 1#1: start worker process 23 127.0.0.1 - - [05/Sep/2022:01:40:19 +0000] "GET /stub_status HTTP/1.1" 200 97 "-" "NGINX-Prometheus-Exporter/v0.9.0" "-" [root@m-k8s 7.7]# kubectl logs pod-adapter adapter 2022/09/05 01:40:19 Starting NGINX Prometheus Exporter version=0.9.0 commit=5f88afbd906baae02edfbab4f5715e06d88538a0 date=2021-03-22T20:16:09Z 2022/09/05 01:40:19 Listening on :9113 2022/09/05 01:40:19 NGINX Prometheus Exporter has successfully started [root@m-k8s 7.7]# kubectl describe pod pod-adapter Name: pod-adapter Namespace: default Priority: 0 Node: w3-k8s/192.168.1.103 Start Time: Mon, 05 Sep 2022 10:39:18 +0900 Labels: app=nginx Annotations: cni.projectcalico.org/podIP : 172.16.132.38/32 cni.projectcalico.org/podIPs : 172.16.132.38/32 Status: Running IP: 172.16.132.38 IPs: IP: 172.16.132.38 Containers: web-page: Container ID: docker://6f15cef63e3bcf8982bf68dc19e3f43f0ab47032299eedfe294a13b122690cdb Image: nginx Image ID: docker-pullable://nginx@sha256:b95a99feebf7797479e0c5eb5ec0bdfa5d9f504bc94da550c2f58e839ea6914f Port: Host Port: State: Running Started: Mon, 05 Sep 2022 10:39:21 +0900 Ready: True Restart Count: 0 Environment: Mounts: /etc/nginx/conf.d from nginx-conf (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wj5gm (ro) adapter: Container ID: docker://09a94aad41f433bd97529142a4c8584c34d85f35a23e46d1f700e2846a0c53d8 Image: nginx/nginx-prometheus-exporter:0.9.0 Image ID: docker-pullable://nginx/nginx-prometheus-exporter@sha256:0caaabffc24204ca72f79c22d222bc7ddbc16355a65a3661c7d17a9a4991aa68 Port: 9113/TCP Host Port: 0/TCP State: Running Started: Mon, 05 Sep 2022 10:40:19 +0900 Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 05 Sep 2022 10:39:47 +0900 Finished: Mon, 05 Sep 2022 10:39:52 +0900 Ready: True Restart Count: 3 Environment: SCRAPE_URI: http://localhost/stub_status Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wj5gm (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: nginx-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: nginx-conf Optional: false kube-api-access-wj5gm: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 3m53s default-scheduler Successfully assigned default/pod-adapter to w3-k8s Normal Pulling 3m52s kubelet Pulling image "nginx" Normal Pulled 3m50s kubelet Successfully pulled image "nginx" in 2.45375911s Normal Created 3m50s kubelet Created container web-page Normal Started 3m50s kubelet Started container web-page Warning BackOff 3m5s (x3 over 3m37s) kubelet Back-off restarting failed container Normal Pulled 2m52s (x4 over 3m50s) kubelet Container image "nginx/nginx-prometheus-exporter:0.9.0" already present on machine Normal Created 2m52s (x4 over 3m50s) kubelet Created container adapter Normal Started 2m52s (x4 over 3m50s) kubelet Started container adapter [root@m-k8s 7.7]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME m-k8s Ready control-plane,master 19d v1.22.0 192.168.1.10 CentOS Linux 7 (Core) 3.10.0-1127.19.1.el7.x86_64 docker://20.10.8 w1-k8s Ready 19d v1.22.0 192.168.1.101 CentOS Linux 7 (Core) 3.10.0-1127.19.1.el7.x86_64 docker://20.10.8 w2-k8s Ready 19d v1.22.0 192.168.1.102 CentOS Linux 7 (Core) 3.10.0-1127.19.1.el7.x86_64 docker://20.10.8 w3-k8s Ready 19d v1.22.0 192.168.1.103 CentOS Linux 7 (Core) 3.10.0-1127.19.1.el7.x86_64 docker://20.10.8 [root@m-k8s 7.7]# curl 172.16.132.38/stub_status Active connections: 1 server accepts handled requests 2 2 2 Reading: 0 Writing: 1 Waiting: 0 [root@m-k8s 7.7]# curl 172.16.132.38:9113/metrics # HELP nginx_connections_accepted Accepted client connections # TYPE nginx_connections_accepted counter nginx_connections_accepted 3 # HELP nginx_connections_active Active client connections # TYPE nginx_connections_active gauge nginx_connections_active 1 # HELP nginx_connections_handled Handled client connections # TYPE nginx_connections_handled counter nginx_connections_handled 3 # HELP nginx_connections_reading Connections where NGINX is reading the request header # TYPE nginx_connections_reading gauge nginx_connections_reading 0 # HELP nginx_connections_waiting Idle client connections # TYPE nginx_connections_waiting gauge nginx_connections_waiting 0 # HELP nginx_connections_writing Connections where NGINX is writing the response back to the client # TYPE nginx_connections_writing gauge nginx_connections_writing 1 # HELP nginx_http_requests_total Total http requests # TYPE nginx_http_requests_total counter nginx_http_requests_total 3 # HELP nginx_up Status of the last metric scrape # TYPE nginx_up gauge nginx_up 1 # HELP nginxexporter_build_info Exporter build information # TYPE nginxexporter_build_info gauge nginxexporter_build_info{commit="5f88afbd906baae02edfbab4f5715e06d88538a0",date="2021-03-22T20:16:09Z",version="0.9.0"} 1

비모 · Answer

안녕하세요, dukim73님. 튜터 비모입니다. 해당 증상에 대해서는 좀더 상세한 내용 확인이 필요해 보입니다. 혹시 증상이 발생하셨을때 아래 명령어에 대한 결과값을 전달해주실 수 있으실까요? kubectl logs pod-adapter web-page kubectl logs pod-adapter adapter kubectl describe pod pod-adapter kubectl get nodes -o wide   혹시 추가로 궁금하신 점이 있으면 남겨주세요 감사합니다 :)