-
카테고리
-
세부 분야
데브옵스 · 인프라
-
해결 여부
미해결
k8s HA 구성중 kubeadm init 실패 증상 - serviceaccount 부분부터 실패
22.06.14 00:19 작성 조회수 900
0
답변을 작성해보세요.
0
gasbugs
2022.06.19
안녕하세요 강사 최일선입니다.
전에 올려두신 내용도 보긴했는데 평일에 시간이 없어서 이제야 답변을 답니다.
해당 내용은 kube-apiserver로 접속이 불가능한 경우에 뜨는 오류로 알려져 있습니다.
응답을 확인해보면 500을 받는 것으로 보아 서버는 정상적으로 오픈했으나 정상 동작하지 않는 것으로 생각됩니다.
동일한 문제로 이슈가 올린 케이스가 있는데 이를 확인해보시면 좋을 것 같습니다. 동일하게 "malformed header: missing HTTP content-type" 오류가 나타난 모습입니다. kubeadm의 버그가 아니라 ha-proxy 오류인 것 같다고 합니다.
https://github.com/kubernetes/kubeadm/issues/2699
감사합니다.
defacto
질문자2022.06.19
안녕하세요.
답변주신 haproxy 가 쿠버네티스 내부에 내장되어 있을까요?
처음에 HA를 위해 haproxy 서버를 따로 두어 구축하고 있었는데 오류가 발생하자 master1 서버에서 kubeadm init 으로만 실행해도 동일한 오류가 발생했습니다.
첨부한 로그에 적혀있는 10.1.10.101는 master1 서버의 IP입니다.
I0615 15:49:22.038672 1224 request.go:1073] Request Body: {"kind":"ClusterRoleBinding","apiVersion":"rbac.authorization.k8s.io/v1","metadata":{"name":"system:coredns","creationTimestamp":null},"subjects":[{"kind":"ServiceAccount","name":"coredns","namespace":"kube-system"}],"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"system:coredns"}}
I0615 15:49:22.038725 1224 round_trippers.go:466] curl -v -XPOST -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubeadm/v1.24.1 (linux/amd64) kubernetes/3ddd0f4" 'https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s'
I0615 15:49:22.063041 1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 24 milliseconds
I0615 15:49:22.063060 1224 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 24 ms Duration 24 ms
I0615 15:49:22.063070 1224 round_trippers.go:577] Response Headers:
I0615 15:49:22.063077 1224 round_trippers.go:580] Audit-Id: f52a46cb-6ecb-4b79-aa62-1a22300a8d07
I0615 15:49:22.063084 1224 round_trippers.go:580] Cache-Control: no-cache, private
I0615 15:49:22.063090 1224 round_trippers.go:580] Content-Type: application/json
I0615 15:49:22.063096 1224 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 4383fc49-8e46-40c9-8a5e-7031390e956f
I0615 15:49:22.063102 1224 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 5e6fba37-152d-4f54-bbf7-e6829d7f15b1
I0615 15:49:22.063109 1224 round_trippers.go:580] Content-Length: 604
I0615 15:49:22.063116 1224 round_trippers.go:580] Date: Wed, 15 Jun 2022 15:49:22 GMT
I0615 15:49:22.063503 1224 request.go:1073] Response Body: {"kind":"ClusterRoleBinding","apiVersion":"rbac.authorization.k8s.io/v1","metadata":{"name":"system:coredns","uid":"133b3a67-dfac-4244-b1a7-0f27a314cb58","resourceVersion":"266","creationTimestamp":"2022-06-15T15:49:22Z","managedFields":[{"manager":"kubeadm","operation":"Update","apiVersion":"rbac.authorization.k8s.io/v1","time":"2022-06-15T15:49:22Z","fieldsType":"FieldsV1","fieldsV1":{"f:roleRef":{},"f:subjects":{}}}]},"subjects":[{"kind":"ServiceAccount","name":"coredns","namespace":"kube-system"}],"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"system:coredns"}}
I0615 15:49:22.063860 1224 request.go:1073] Request Body: {"kind":"ServiceAccount","apiVersion":"v1","metadata":{"name":"coredns","namespace":"kube-system","creationTimestamp":null}}
I0615 15:49:22.063921 1224 round_trippers.go:466] curl -v -XPOST -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubeadm/v1.24.1 (linux/amd64) kubernetes/3ddd0f4" 'https://10.1.10.101:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s'
I0615 15:49:22.201998 1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s 500 Internal Server Error in 138 milliseconds
I0615 15:49:22.202021 1224 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 137 ms Duration 138 ms
I0615 15:49:22.202028 1224 round_trippers.go:577] Response Headers:
I0615 15:49:22.202035 1224 round_trippers.go:580] Date: Wed, 15 Jun 2022 15:49:22 GMT
I0615 15:49:22.202042 1224 round_trippers.go:580] Audit-Id: 58b00d5f-5f4f-4ab2-bbb3-c3479189f4bc
I0615 15:49:22.202048 1224 round_trippers.go:580] Cache-Control: no-cache, private
I0615 15:49:22.202054 1224 round_trippers.go:580] Content-Type: application/json
I0615 15:49:22.202060 1224 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 4383fc49-8e46-40c9-8a5e-7031390e956f
I0615 15:49:22.202128 1224 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: 5e6fba37-152d-4f54-bbf7-e6829d7f15b1
I0615 15:49:22.202134 1224 round_trippers.go:580] Content-Length: 169
I0615 15:49:22.202152 1224 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"rpc error: code = Unknown desc = malformed header: missing HTTP content-type","code":500}
rpc error: code = Unknown desc = malformed header: missing HTTP content-type
missing HTTP content-type오류가 발생한 시점의 로그 일부입니다.
앞서 500 오류 전에는 201가 발생한 작업이 몇개 있습니다.
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 108 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles?timeout=10s 201 Created in 2 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s 201 Created in 16 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 20 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles?timeout=10s 201 Created in 49 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s 201 Created in 167 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/secrets?timeout=10s 201 Created in 182 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?timeout=10s 201 Created in 10 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 18 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 36 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 109 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 50 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-public/configmaps?timeout=10s 201 Created in 223 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/roles?timeout=10s 201 Created in 12 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/rolebindings?timeout=10s 201 Created in 20 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 38 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?timeout=10s 201 Created in 16 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 24 milliseconds
1224 round_trippers.go:553] POST https://10.1.10.101:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s 500 Internal Server Error in 138 milliseconds
----
결국 ubuntu server 18.04 환경에서 kubeadm init가 성공하여 HA 구성을 완료하였습니다.
아직도 의문인게
local hyper-v에서는 ubuntu server 22 에서도 kubeadm init가 정상동작했습니다.
하지만 원격 hyper-v 서버는 ubuntu server 18 에서 kubeadm init가 왜 성공 했는지 원인을 찾지 못한 상태입니다.
gasbugs
2022.06.23
답변주신 haproxy 가 쿠버네티스 내부에 내장되어 있을까요?
아뇨 그렇지는 않습니다. malformed header: missing HTTP content-type" 에러로 생각되는게
앞단에 로드밸런서에서 뭔가 트래픽에 문제가 있었을 것으로 예상됩니다 ㅠ
중간에 용량이 큰 데이터를 넘길 때 데이터가 누락되지 않는가 조심스래 추측도해봅니다.
감사합니다.
답변 1