Skip to main content

워커노드 제외방법

사전 설명

k8s환경에서 특정 노드 제외할때 그냥 삭제해도 kube scheduling에 의해 재배치가 있긴하지만 계획된 정지 작업같은 작업이 있는경우

보다 안정적으로 노드를 제외하려면 cordon / drain 절차를 거치면 좋습니다.

cordon / drain은 약간의 차이가 있는데, cordon의 경우 단순 스케쥴링을 제외하는 절차, drain의 경우 제외할 노드를 모두 제거하는 절차를 수행합니다. (drain은 cordon절차를 포함합니다.)

작업절차

  1. cordon 설정
    1. 노드 정보 확인
      $>  kubectl get no
NAME    STATUS   ROLES                  AGE   VERSION
masr1   Ready    control-plane,master   37d   v1.23.7
work1   Ready    <none>                 37d   v1.23.7
work2   Ready    <none>                 37d   v1.23.7

    2. cordon 수행 (work1노드를 제외할껍니다.)
      $>  kubectl cordon work1
node/work1 cordoned

$>  kubectl get no
NAME    STATUS                         ROLES                  AGE   VERSION
masr1   Ready                          control-plane,master   37d   v1.23.7
work1   Ready                          <none>                 37d   v1.23.7
work2   Ready,SchedulingDisabled       <none>                 37d   v1.23.7

  2. drain 수행
    1. 해당 노드에 daemonset으로 구동되는 pod가 있거나 local storage가 구동되고 있는 경우 drain 작업이 수행되지 않습니다.
      kubectl drain work2
node/work2 already cordoned
error: unable to drain node "work2", aborting command...

There are pending nodes to be drained:
 k8sw2
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/calico-node-fb862, kube-system/kube-proxy-4qhsx, kube-system/nodelocaldns-6cptm
cannot delete Pods with local storage (use --delete-emptydir-data to override): kubernetes-dashboard/dashboard-metrics-scraper-66dd8bdd86-6cnbv, kubernetes-dashboard/kubernetes-dashboard-844749bcff-mmpq2

    2. 강제로 drain 처리
      kubectl drain work2  --ignore-daemonsets --delete-emptydir-data
node/work2 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-fb862, kube-system/kube-proxy-4qhsx, kube-system/nodelocaldns-6cptm
evicting pod kubernetes-dashboard/kubernetes-dashboard-844749bcff-mmpq2
evicting pod cert-manager/cert-manager-cainjector-75c94654d-r7trs
evicting pod cert-manager/cert-manager-webhook-d4fd4f479-cr6z2
evicting pod gitlab-agent/house-gitlab-agent-v1-586d6d6797-vm68t
evicting pod kube-system/coredns-657959df74-j2zbq
evicting pod kubernetes-dashboard/dashboard-metrics-scraper-66dd8bdd86-6cnbv
pod/cert-manager-cainjector-75c94654d-r7trs evicted
pod/cert-manager-webhook-d4fd4f479-cr6z2 evicted
pod/house-gitlab-agent-v1-586d6d6797-vm68t evicted
pod/kubernetes-dashboard-844749bcff-mmpq2 evicted
pod/dashboard-metrics-scraper-66dd8bdd86-6cnbv evicted
pod/coredns-657959df74-j2zbq evicted
node/work2 evicted

  3. 작업완료 
    1. 작업이 끝난노드를 다시 클러스터로 포함시키면 됩니다.
      kubectl uncordon work2 
node/work2 already uncordoned
    2. 스케쥴링 활성화 확인
      $>  kubectl get no
NAME    STATUS   ROLES                  AGE   VERSION
masr1   Ready    control-plane,master   37d   v1.23.7
work1   Ready    <none>                 37d   v1.23.7
work2   Ready    <none>                 37d   v1.23.7

Back to top