Panduan ini membahas langkah-langkah untuk mengupgrade Rancher Kubernetes Engine (RKE) ke versi terbaru. RKE adalah alat yang memudahkan pengelolaan cluster Kubernetes dengan menyederhanakan proses instalasi dan pemeliharaan. Memperbarui RKE adalah langkah penting untuk memastikan fitur terbaru dan perbaikan keamanan diterapkan.
Langkah-langkah Utama:
- Persiapan: Sebelum melakukan upgrade, pastikan versi RKE yang sedang digunakan dan rilis terbaru yang tersedia sudah diperiksa. Backup konfigurasi cluster untuk mencegah kehilangan data.
- Periksa Kompatibilitas: Verifikasi bahwa versi terbaru RKE kompatibel dengan versi Kubernetes yang digunakan dan sesuai dengan kebutuhan aplikasi.
- Download Versi Terbaru: Unduh versi terbaru RKE dari repositori resmi Rancher. Pastikan mendapatkan versi yang sesuai dengan sistem operasi dan arsitektur.
- Upgrade RKE: Gantikan file biner RKE lama dengan yang baru. Jalankan perintah upgrade menggunakan CLI RKE untuk memulai proses pembaruan.
- Verifikasi Upgrade: Setelah upgrade, periksa status cluster dan pastikan semua komponen berjalan dengan baik.
Panduan ini memudahkan proses upgrade RKE, menjaga agar cluster Kubernetes tetap terkini dan aman.
Versi RKE sebelum di upgrade
admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.20.8 master02 Ready controlplane,etcd,worker 67d v1.20.8 master03 Ready controlplane,etcd,worker 67d v1.20.8 worker01 Ready worker 67d v1.20.8 worker02 Ready worker 67d v1.20.8 worker03 Ready worker 67d v1.20.8 admin@master01:~$
Backup file binary rke sebelumnya
admin@master01:~$ mv rke_linux-amd64{,-back-210824}Selanjudnya, download binary rke yang baru sesuai kebutuhan, saat ini memilih versi 1.5.12, adapun versi lainnya bisa di peroleh pada github resmi rke dengan link sebagai berikut :
https://github.com/rancher/rke/releases
admin@master01:~$ wget https://github.com/rancher/rke/releases/download/v1.5.12/rke_linux-amd64 --2024-08-21 01:06:49-- https://github.com/rancher/rke/releases/download/v1.5.12/rke_linux-amd64 Resolving github.com (github.com)... 20.205.243.166 Connecting to github.com (github.com)|20.205.243.166|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/108337180/b1f41f9e-5d13-4e15 -a94c-9e705eb44b7b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240821%2Fus-east-1% 2Fs3%2Faws4_request&X-Amz-Date=20240821T010649Z&X-Amz-Expires=300&X-Amz-Signature=b7ef917241e3aabd97876c15bd4697288 e67b9d7996bb7f846ac32d6d220d877&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=108337180&response-content-dis position=attachment%3B%20filename%3Drke_linux-amd64&response-content-type=application%2Foctet-stream [following] --2024-08-21 01:06:49-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/108337180/b1f 41f9e-5d13-4e15-a94c-9e705eb44b7b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F202408 21%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240821T010649Z&X-Amz-Expires=300&X-Amz-Signature=b7ef917241e3aabd97 876c15bd4697288e67b9d7996bb7f846ac32d6d220d877&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=108337180&respo nse-content-disposition=attachment%3B%20filename%3Drke_linux-amd64&response-content-type=application%2Foctet-stream Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.19 9.110.133, ... Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 59513251 (57M) [application/octet-stream] Saving to: 'rke_linux-amd64' rke_linux-amd64 100%[=============================================>] 56.76M 19.7MB/s in 2.9s 2024-08-21 01:06:53 (19.7 MB/s) - 'rke_linux-amd64' saved [59513251/59513251] admin@master01:~$
Check file hasil download dan rubah permission di tambah izin exsekusi :
admin@master01:~$ ls -l total 100384 -rwxrwxr-x 1 masadmin masadmin 3338 Jun 24 04:37 agentTM.sh -rw-rw-r-- 1 masadmin masadmin 68 Aug 13 06:24 cert-manager-backup.yaml -rwxr-xr-x 1 root root 648 Aug 13 06:34 cert.sh -rw------- 1 masadmin masadmin 123836 Aug 21 00:59 cluster.rkestate -rw-r----- 1 masadmin masadmin 6920 Aug 21 00:58 cluster.yml -rw-r----- 1 masadmin masadmin 6901 Aug 21 00:38 cluster.yml-back-210824 -rw-r----- 1 masadmin masadmin 6901 Jun 14 08:34 cluster.yml.old -rw-rw-r-- 1 masadmin masadmin 473 Jun 19 04:54 csi-nfs.yaml drwxrwxr-x 11 masadmin masadmin 4096 Jun 19 07:07 k8s-config -rw-rw-r-- 1 masadmin masadmin 1047354 Jun 19 02:25 k8s-config_PROD.zip -rw------- 1 masadmin masadmin 5379 Aug 21 00:58 kube_config_cluster.yml drwxr-xr-x 2 root root 4096 Jun 20 15:30 master -rw-rw-r-- 1 masadmin masadmin 59513251 Aug 5 21:55 rke_linux-amd64 -rwxrwxr-x 1 masadmin masadmin 42042980 Dec 6 2021 rke_linux-amd64-back-210824 admin@master01:~$ admin@master01:~$ chmod +x rke_linux-amd64 admin@master01:~$
Check versi RKE
admin@master01:~$ ./rke_linux-amd64 -v rke version v1.5.12 admin@master01:~$ admin@master01:~$ rke -v rke version v1.2.9 admin@master01:~$
Backup dan mengganti binary rke dengan yang lebih baru
admin@master01:~$ ls -l /usr/local/bin/rke
-rwxr-xr-x 1 root root 42042980 Jun 14 06:16 /usr/local/bin/rke
admin@master01:~$
admin@master01:~$ sudo mv /usr/local/bin/rke{,-back}
[sudo] password for admin:
admin@master01:~$
Mengecek versi kubernetes yang tersedia pada rke
admin@master01:~$ rke config --list-version --all v1.25.16-rancher2-3 v1.27.16-rancher1-1 v1.28.12-rancher1-1 v1.26.15-rancher1-1 admin@master01:~$
Lakukan backup konfigurasi cluster.yml
admin@master01:~$ cp cluster.yml{,-back-210824}nano cluster.yml
Hapus bagian system_images:
addons_include: [] system_images: etcd: rancher/mirrored-coreos-etcd:v3.4.15-rancher1 alpine: rancher/rke-tools:v0.1.75 nginx_proxy: rancher/rke-tools:v0.1.75 cert_downloader: rancher/rke-tools:v0.1.75 kubernetes_services_sidecar: rancher/rke-tools:v0.1.75 kubedns: rancher/mirrored-k8s-dns-kube-dns:1.15.10 dnsmasq: rancher/mirrored-k8s-dns-dnsmasq-nanny:1.15.10 kubedns_sidecar: rancher/mirrored-k8s-dns-sidecar:1.15.10 kubedns_autoscaler: rancher/mirrored-cluster-proportional-autoscaler:1.8.1 coredns: rancher/mirrored-coredns-coredns:1.8.0 coredns_autoscaler: rancher/mirrored-cluster-proportional-autoscaler:1.8.1 nodelocal: rancher/mirrored-k8s-dns-node-cache:1.15.13 kubernetes: rancher/hyperkube:v1.20.8-rancher1 flannel: rancher/coreos-flannel:v0.13.0-rancher1 flannel_cni: rancher/flannel-cni:v0.3.0-rancher6 calico_node: rancher/mirrored-calico-node:v3.17.2 calico_cni: rancher/mirrored-calico-cni:v3.17.2 calico_controllers: rancher/mirrored-calico-kube-controllers:v3.17.2 calico_ctl: rancher/mirrored-calico-ctl:v3.17.2 calico_flexvol: rancher/mirrored-calico-pod2daemon-flexvol:v3.17.2 canal_node: rancher/mirrored-calico-node:v3.17.2 canal_cni: rancher/mirrored-calico-cni:v3.17.2 canal_controllers: rancher/mirrored-calico-kube-controllers:v3.17.2 canal_flannel: rancher/coreos-flannel:v0.13.0-rancher1 canal_flexvol: rancher/mirrored-calico-pod2daemon-flexvol:v3.17.2 weave_node: weaveworks/weave-kube:2.8.1 weave_cni: weaveworks/weave-npc:2.8.1 pod_infra_container: rancher/mirrored-pause:3.2 ingress: rancher/nginx-ingress-controller:nginx-0.43.0-rancher3 ingress_backend: rancher/mirrored-nginx-ingress-controller-defaultbackend:1.5-rancher1 metrics_server: rancher/mirrored-metrics-server:v0.4.1 windows_pod_infra_container: rancher/kubelet-pause:v0.1.6 aci_cni_deploy_container: noiro/cnideploy:5.1.1.0.1ae238a aci_host_container: noiro/aci-containers-host:5.1.1.0.1ae238a aci_opflex_container: noiro/opflex:5.1.1.0.1ae238a aci_mcast_container: noiro/opflex:5.1.1.0.1ae238a aci_ovs_container: noiro/openvswitch:5.1.1.0.1ae238a aci_controller_container: noiro/aci-containers-controller:5.1.1.0.1ae238a aci_gbp_server_container: noiro/gbp-server:5.1.1.0.1ae238a aci_opflex_server_container: noiro/opflex-server:5.1.1.0.1ae238a ssh_key_path: ~/.ssh/id_rsa ssh_cert_path: ""
Kondisi konfigurasi kubernetes_version
ignore_docker_version: null kubernetes_version: "" private_registries: []
Setelah perubahan adalah sebagai berikut :
Pada bagian system_images: sudah di hapus, agar bisa mendownload image terbaru sesuai versi kubernetes yang di tentukan
addons_include: [] ssh_key_path: ~/.ssh/id_rsa ssh_cert_path: ""
ignore_docker_version: null kubernetes_version: "v1.28.12-rancher1-1" private_registries: []
Memulai upgrade RKE
admin@master01:~$ rke up --config cluster.yml --ignore-docker-version
Proses Upgrade
admin@master01:~$ rke up --config cluster.yml --ignore-docker-version INFO[0000] Running RKE version: v1.5.12 INFO[0000] Initiating Kubernetes cluster INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0000] [certificates] Generating admin certificates and kubeconfig INFO[0000] Successfully Deployed state file at [./cluster.rkestate] INFO[0000] Building Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [master02] INFO[0000] [dialer] Setup tunnel for host [worker01] INFO[0000] [dialer] Setup tunnel for host [master01] INFO[0000] [dialer] Setup tunnel for host [worker03] INFO[0000] [dialer] Setup tunnel for host [master03] INFO[0000] [dialer] Setup tunnel for host [worker02] INFO[0000] [network] No hosts added existing cluster, skipping port check INFO[0000] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0000] Finding container [cert-deployer] on host [worker03], try #1 INFO[0000] Finding container [cert-deployer] on host [master02], try #1 INFO[0000] Finding container [cert-deployer] on host [worker01], try #1 INFO[0000] Finding container [cert-deployer] on host [worker02], try #1 INFO[0000] Finding container [cert-deployer] on host [master01], try #1 INFO[0000] Finding container [cert-deployer] on host [master03], try #1 INFO[0000] Pulling image [rancher/rke-tools:v0.1.100] on host [worker03], try #1 INFO[0000] Pulling image [rancher/rke-tools:v0.1.100] on host [worker01], try #1 INFO[0000] Pulling image [rancher/rke-tools:v0.1.100] on host [worker02], try #1 INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0000] Starting container [cert-deployer] on host [master03], try #1 INFO[0000] Starting container [cert-deployer] on host [master01], try #1 INFO[0001] Starting container [cert-deployer] on host [master02], try #1 INFO[0001] Finding container [cert-deployer] on host [master03], try #1 INFO[0001] Finding container [cert-deployer] on host [master01], try #1 INFO[0001] Finding container [cert-deployer] on host [master02], try #1 INFO[0006] Finding container [cert-deployer] on host [master03], try #1 INFO[0006] Removing container [cert-deployer] on host [master03], try #1 INFO[0006] Finding container [cert-deployer] on host [master01], try #1 INFO[0006] Removing container [cert-deployer] on host [master01], try #1 INFO[0006] Finding container [cert-deployer] on host [master02], try #1 INFO[0006] Removing container [cert-deployer] on host [master02], try #1 INFO[0015] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0015] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0015] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0016] Starting container [cert-deployer] on host [worker03], try #1 INFO[0016] Starting container [cert-deployer] on host [worker02], try #1 INFO[0016] Finding container [cert-deployer] on host [worker03], try #1 INFO[0016] Starting container [cert-deployer] on host [worker01], try #1 INFO[0016] Finding container [cert-deployer] on host [worker02], try #1 INFO[0016] Finding container [cert-deployer] on host [worker01], try #1 INFO[0021] Finding container [cert-deployer] on host [worker03], try #1 INFO[0021] Removing container [cert-deployer] on host [worker03], try #1 INFO[0021] Finding container [cert-deployer] on host [worker02], try #1 INFO[0021] Removing container [cert-deployer] on host [worker02], try #1 INFO[0021] Finding container [cert-deployer] on host [worker01], try #1 INFO[0021] Removing container [cert-deployer] on host [worker01], try #1 INFO[0021] [reconcile] Rebuilding and updating local kube config INFO[0021] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0021] [reconcile] host [master01] is a control plane node with reachable Kubernetes API endpoint in the cluste r INFO[0021] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0021] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [master01] INFO[0021] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0022] Starting container [file-deployer] on host [master01], try #1 INFO[0022] Successfully started [file-deployer] container on host [master01] INFO[0022] Waiting for [file-deployer] container to exit on host [master01] INFO[0022] Waiting for [file-deployer] container to exit on host [master01] INFO[0022] Container [file-deployer] is still running on host [master01]: stderr: [], stdout: [] INFO[0023] Removing container [file-deployer] on host [master01], try #1 INFO[0023] [remove/file-deployer] Successfully removed container on host [master01] INFO[0023] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [master02] INFO[0023] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0023] Starting container [file-deployer] on host [master02], try #1 INFO[0023] Successfully started [file-deployer] container on host [master02] INFO[0023] Waiting for [file-deployer] container to exit on host [master02] INFO[0023] Waiting for [file-deployer] container to exit on host [master02] INFO[0023] Container [file-deployer] is still running on host [master02]: stderr: [], stdout: [] INFO[0024] Removing container [file-deployer] on host [master02], try #1 INFO[0025] [remove/file-deployer] Successfully removed container on host [master02] INFO[0025] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [master03] INFO[0025] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0025] Starting container [file-deployer] on host [master03], try #1 INFO[0025] Successfully started [file-deployer] container on host [master03] INFO[0025] Waiting for [file-deployer] container to exit on host [master03] INFO[0025] Waiting for [file-deployer] container to exit on host [master03] INFO[0025] Container [file-deployer] is still running on host [master03]: stderr: [], stdout: [] INFO[0026] Removing container [file-deployer] on host [master03], try #1 INFO[0026] [remove/file-deployer] Successfully removed container on host [master03] INFO[0026] [/etc/kubernetes/admission.yaml] Successfully deployed admission control config to Cluster control nodes INFO[0026] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [master03] INFO[0026] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0026] Starting container [file-deployer] on host [master03], try #1 INFO[0027] Successfully started [file-deployer] container on host [master03] INFO[0027] Waiting for [file-deployer] container to exit on host [master03] INFO[0027] Waiting for [file-deployer] container to exit on host [master03] INFO[0027] Container [file-deployer] is still running on host [master03]: stderr: [], stdout: [] INFO[0028] Removing container [file-deployer] on host [master03], try #1 INFO[0028] [remove/file-deployer] Successfully removed container on host [master03] INFO[0028] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [master01] INFO[0028] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0028] Starting container [file-deployer] on host [master01], try #1 INFO[0028] Successfully started [file-deployer] container on host [master01] INFO[0028] Waiting for [file-deployer] container to exit on host [master01] INFO[0028] Waiting for [file-deployer] container to exit on host [master01] INFO[0028] Container [file-deployer] is still running on host [master01]: stderr: [], stdout: [] INFO[0029] Removing container [file-deployer] on host [master01], try #1 INFO[0030] [remove/file-deployer] Successfully removed container on host [master01] INFO[0030] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [master02] INFO[0030] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0030] Starting container [file-deployer] on host [master02], try #1 INFO[0030] Successfully started [file-deployer] container on host [master02] INFO[0030] Waiting for [file-deployer] container to exit on host [master02] INFO[0030] Waiting for [file-deployer] container to exit on host [master02] INFO[0030] Container [file-deployer] is still running on host [master02]: stderr: [], stdout: [] INFO[0031] Removing container [file-deployer] on host [master02], try #1 INFO[0031] [remove/file-deployer] Successfully removed container on host [master02] INFO[0031] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes INFO[0031] [reconcile] Reconciling cluster state INFO[0031] [reconcile] Check etcd hosts to be deleted INFO[0031] [reconcile] Check etcd hosts to be added INFO[0031] [reconcile] Rebuilding and updating local kube config INFO[0031] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0031] [reconcile] host [master01] is a control plane node with reachable Kubernetes API endpoint in the cluste r INFO[0031] [reconcile] Reconciled cluster state successfully INFO[0031] max_unavailable_worker got rounded down to 0, resetting to 1 INFO[0031] Setting maxUnavailable for worker nodes to: 1 INFO[0031] Setting maxUnavailable for controlplane nodes to: 1 INFO[0031] Pre-pulling kubernetes images INFO[0031] Pulling image [rancher/hyperkube:v1.28.12-rancher1] on host [master02], try #1 INFO[0031] Pulling image [rancher/hyperkube:v1.28.12-rancher1] on host [master01], try #1 INFO[0031] Pulling image [rancher/hyperkube:v1.28.12-rancher1] on host [worker03], try #1 INFO[0031] Pulling image [rancher/hyperkube:v1.28.12-rancher1] on host [worker01], try #1 INFO[0031] Pulling image [rancher/hyperkube:v1.28.12-rancher1] on host [worker02], try #1 INFO[0031] Pulling image [rancher/hyperkube:v1.28.12-rancher1] on host [master03], try #1 INFO[0085] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker03] INFO[0085] Pulling image [rancher/mirrored-pause:3.7] on host [worker03], try #1 INFO[0088] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master01] INFO[0088] Pulling image [rancher/mirrored-pause:3.7] on host [master01], try #1 INFO[0089] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker01] INFO[0089] Pulling image [rancher/mirrored-pause:3.7] on host [worker01], try #1 INFO[0089] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker02] INFO[0089] Pulling image [rancher/mirrored-pause:3.7] on host [worker02], try #1 INFO[0090] Image [rancher/mirrored-pause:3.7] exists on host [worker03] INFO[0091] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master03] INFO[0091] Pulling image [rancher/mirrored-pause:3.7] on host [master03], try #1 INFO[0092] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master02] INFO[0092] Pulling image [rancher/mirrored-pause:3.7] on host [master02], try #1 INFO[0093] Image [rancher/mirrored-pause:3.7] exists on host [master01] INFO[0093] Image [rancher/mirrored-pause:3.7] exists on host [worker01] INFO[0094] Image [rancher/mirrored-pause:3.7] exists on host [worker02] INFO[0096] Image [rancher/mirrored-pause:3.7] exists on host [master03] INFO[0097] Image [rancher/mirrored-pause:3.7] exists on host [master02] INFO[0097] Kubernetes images pulled successfully INFO[0097] [etcd] Building up etcd plane.. INFO[0097] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0098] Starting container [etcd-fix-perm] on host [master01], try #1 INFO[0099] Successfully started [etcd-fix-perm] container on host [master01] INFO[0099] Waiting for [etcd-fix-perm] container to exit on host [master01] INFO[0099] Waiting for [etcd-fix-perm] container to exit on host [master01] INFO[0099] Removing container [etcd-fix-perm] on host [master01], try #1 INFO[0099] [remove/etcd-fix-perm] Successfully removed container on host [master01] INFO[0099] Finding container [etcd] on host [master01], try #1 INFO[0099] Pulling image [rancher/mirrored-coreos-etcd:v3.5.10] on host [master01], try #1 INFO[0108] Image [rancher/mirrored-coreos-etcd:v3.5.10] exists on host [master01] INFO[0108] Finding container [old-etcd] on host [master01], try #1 INFO[0108] Stopping container [etcd] on host [master01] with stopTimeoutDuration [5s], try #1 INFO[0110] Waiting for [etcd] container to exit on host [master01] INFO[0110] Renaming container [etcd] to [old-etcd] on host [master01], try #1 INFO[0110] Starting container [etcd] on host [master01], try #1 INFO[0110] [etcd] Successfully updated [etcd] container on host [master01] INFO[0110] Removing container [old-etcd] on host [master01], try #1 INFO[0110] [etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [master01] INFO[0110] Removing container [etcd-rolling-snapshots] on host [master01], try #1 INFO[0110] [remove/etcd-rolling-snapshots] Successfully removed container on host [master01] INFO[0110] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0111] Starting container [etcd-rolling-snapshots] on host [master01], try #1 INFO[0111] [etcd] Successfully started [etcd-rolling-snapshots] container on host [master01] INFO[0116] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0116] Starting container [rke-bundle-cert] on host [master01], try #1 INFO[0116] [certificates] Successfully started [rke-bundle-cert] container on host [master01] INFO[0116] Waiting for [rke-bundle-cert] container to exit on host [master01] INFO[0117] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on hos t [master01] INFO[0117] Removing container [rke-bundle-cert] on host [master01], try #1 INFO[0117] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0117] Starting container [rke-log-linker] on host [master01], try #1 INFO[0117] [etcd] Successfully started [rke-log-linker] container on host [master01] INFO[0117] Removing container [rke-log-linker] on host [master01], try #1 INFO[0117] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0117] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0118] Starting container [rke-log-linker] on host [master01], try #1 INFO[0118] [etcd] Successfully started [rke-log-linker] container on host [master01] INFO[0118] Removing container [rke-log-linker] on host [master01], try #1 INFO[0118] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0118] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0120] Starting container [etcd-fix-perm] on host [master02], try #1 INFO[0120] Successfully started [etcd-fix-perm] container on host [master02] INFO[0120] Waiting for [etcd-fix-perm] container to exit on host [master02] INFO[0120] Waiting for [etcd-fix-perm] container to exit on host [master02] INFO[0120] Removing container [etcd-fix-perm] on host [master02], try #1 INFO[0120] [remove/etcd-fix-perm] Successfully removed container on host [master02] INFO[0120] Finding container [etcd] on host [master02], try #1 INFO[0120] Pulling image [rancher/mirrored-coreos-etcd:v3.5.10] on host [master02], try #1 INFO[0129] Image [rancher/mirrored-coreos-etcd:v3.5.10] exists on host [master02] INFO[0129] Finding container [old-etcd] on host [master02], try #1 INFO[0129] Stopping container [etcd] on host [master02] with stopTimeoutDuration [5s], try #1 INFO[0131] Waiting for [etcd] container to exit on host [master02] INFO[0131] Renaming container [etcd] to [old-etcd] on host [master02], try #1 INFO[0131] Starting container [etcd] on host [master02], try #1 INFO[0131] [etcd] Successfully updated [etcd] container on host [master02] INFO[0131] Removing container [old-etcd] on host [master02], try #1 INFO[0131] [etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [master02] INFO[0131] Removing container [etcd-rolling-snapshots] on host [master02], try #1 INFO[0131] [remove/etcd-rolling-snapshots] Successfully removed container on host [master02] INFO[0131] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0131] Starting container [etcd-rolling-snapshots] on host [master02], try #1 INFO[0131] [etcd] Successfully started [etcd-rolling-snapshots] container on host [master02] INFO[0136] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0137] Starting container [rke-bundle-cert] on host [master02], try #1 INFO[0137] [certificates] Successfully started [rke-bundle-cert] container on host [master02] INFO[0137] Waiting for [rke-bundle-cert] container to exit on host [master02] INFO[0137] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on hos t [master02] INFO[0137] Removing container [rke-bundle-cert] on host [master02], try #1 INFO[0137] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0137] Starting container [rke-log-linker] on host [master02], try #1 INFO[0138] [etcd] Successfully started [rke-log-linker] container on host [master02] INFO[0138] Removing container [rke-log-linker] on host [master02], try #1 INFO[0138] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0138] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0138] Starting container [rke-log-linker] on host [master02], try #1 INFO[0138] [etcd] Successfully started [rke-log-linker] container on host [master02] INFO[0138] Removing container [rke-log-linker] on host [master02], try #1 INFO[0138] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0138] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0139] Starting container [etcd-fix-perm] on host [master03], try #1 INFO[0139] Successfully started [etcd-fix-perm] container on host [master03] INFO[0139] Waiting for [etcd-fix-perm] container to exit on host [master03] INFO[0139] Waiting for [etcd-fix-perm] container to exit on host [master03] INFO[0139] Removing container [etcd-fix-perm] on host [master03], try #1 INFO[0139] [remove/etcd-fix-perm] Successfully removed container on host [master03] INFO[0139] Finding container [etcd] on host [master03], try #1 INFO[0139] Pulling image [rancher/mirrored-coreos-etcd:v3.5.10] on host [master03], try #1 INFO[0148] Image [rancher/mirrored-coreos-etcd:v3.5.10] exists on host [master03] INFO[0148] Finding container [old-etcd] on host [master03], try #1 INFO[0148] Stopping container [etcd] on host [master03] with stopTimeoutDuration [5s], try #1 INFO[0150] Waiting for [etcd] container to exit on host [master03] INFO[0150] Renaming container [etcd] to [old-etcd] on host [master03], try #1 INFO[0150] Starting container [etcd] on host [master03], try #1 INFO[0150] [etcd] Successfully updated [etcd] container on host [master03] INFO[0150] Removing container [old-etcd] on host [master03], try #1 INFO[0150] [etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [master03] INFO[0150] Removing container [etcd-rolling-snapshots] on host [master03], try #1 INFO[0150] [remove/etcd-rolling-snapshots] Successfully removed container on host [master03] INFO[0150] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0151] Starting container [etcd-rolling-snapshots] on host [master03], try #1 INFO[0151] [etcd] Successfully started [etcd-rolling-snapshots] container on host [master03] INFO[0156] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0156] Starting container [rke-bundle-cert] on host [master03], try #1 INFO[0156] [certificates] Successfully started [rke-bundle-cert] container on host [master03] INFO[0156] Waiting for [rke-bundle-cert] container to exit on host [master03] INFO[0157] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on hos t [master03] INFO[0157] Removing container [rke-bundle-cert] on host [master03], try #1 INFO[0157] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0157] Starting container [rke-log-linker] on host [master03], try #1 INFO[0157] [etcd] Successfully started [rke-log-linker] container on host [master03] INFO[0157] Removing container [rke-log-linker] on host [master03], try #1 INFO[0157] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0157] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0158] Starting container [rke-log-linker] on host [master03], try #1 INFO[0158] [etcd] Successfully started [rke-log-linker] container on host [master03] INFO[0158] Removing container [rke-log-linker] on host [master03], try #1 INFO[0158] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0158] [etcd] Successfully started etcd plane.. Checking etcd cluster health INFO[0159] [etcd] etcd host [master01] reported healthy=true INFO[0159] [controlplane] Now checking status of node master01, try #1 INFO[0159] [controlplane] Now checking status of node master02, try #1 INFO[0159] [controlplane] Now checking status of node master03, try #1 INFO[0159] [controlplane] Processing controlplane hosts for upgrade 1 at a time INFO[0159] Processing controlplane host master01 INFO[0159] [controlplane] Now checking status of node master01, try #1 INFO[0159] [controlplane] Getting list of nodes for upgrade INFO[0159] Upgrading controlplane components for control host master01 INFO[0159] Finding container [service-sidekick] on host [master01], try #1 INFO[0159] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0159] Removing container [service-sidekick] on host [master01], try #1 INFO[0159] [remove/service-sidekick] Successfully removed container on host [master01] INFO[0159] Finding container [kube-apiserver] on host [master01], try #1 INFO[0159] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master01] INFO[0159] Finding container [old-kube-apiserver] on host [master01], try #1 INFO[0159] Stopping container [kube-apiserver] on host [master01] with stopTimeoutDuration [5s], try #1 INFO[0161] Waiting for [kube-apiserver] container to exit on host [master01] INFO[0161] Renaming container [kube-apiserver] to [old-kube-apiserver] on host [master01], try #1 INFO[0161] Starting container [kube-apiserver] on host [master01], try #1 INFO[0161] [controlplane] Successfully updated [kube-apiserver] container on host [master01] INFO[0161] Removing container [old-kube-apiserver] on host [master01], try #1 INFO[0161] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [master01] INFO[0166] [healthcheck] service [kube-apiserver] on host [master01] is healthy INFO[0166] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0167] Starting container [rke-log-linker] on host [master01], try #1 INFO[0167] [controlplane] Successfully started [rke-log-linker] container on host [master01] INFO[0167] Removing container [rke-log-linker] on host [master01], try #1 INFO[0167] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0167] Finding container [kube-controller-manager] on host [master01], try #1 INFO[0167] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master01] INFO[0167] Finding container [old-kube-controller-manager] on host [master01], try #1 INFO[0167] Stopping container [kube-controller-manager] on host [master01] with stopTimeoutDuration [5s], try #1 INFO[0167] Waiting for [kube-controller-manager] container to exit on host [master01] INFO[0167] Renaming container [kube-controller-manager] to [old-kube-controller-manager] on host [master01], try #1 INFO[0167] Starting container [kube-controller-manager] on host [master01], try #1 INFO[0167] [controlplane] Successfully updated [kube-controller-manager] container on host [master01] INFO[0167] Removing container [old-kube-controller-manager] on host [master01], try #1 INFO[0167] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [master01] INFO[0173] [healthcheck] service [kube-controller-manager] on host [master01] is healthy INFO[0173] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0173] Starting container [rke-log-linker] on host [master01], try #1 INFO[0173] [controlplane] Successfully started [rke-log-linker] container on host [master01] INFO[0174] Removing container [rke-log-linker] on host [master01], try #1 INFO[0174] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0174] Finding container [kube-scheduler] on host [master01], try #1 INFO[0174] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master01] INFO[0174] Finding container [old-kube-scheduler] on host [master01], try #1 INFO[0174] Stopping container [kube-scheduler] on host [master01] with stopTimeoutDuration [5s], try #1 INFO[0174] Waiting for [kube-scheduler] container to exit on host [master01] INFO[0174] Renaming container [kube-scheduler] to [old-kube-scheduler] on host [master01], try #1 INFO[0174] Starting container [kube-scheduler] on host [master01], try #1 INFO[0174] [controlplane] Successfully updated [kube-scheduler] container on host [master01] INFO[0174] Removing container [old-kube-scheduler] on host [master01], try #1 INFO[0174] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [master01] INFO[0179] [healthcheck] service [kube-scheduler] on host [master01] is healthy INFO[0179] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0179] Starting container [rke-log-linker] on host [master01], try #1 INFO[0180] [controlplane] Successfully started [rke-log-linker] container on host [master01] INFO[0180] Removing container [rke-log-linker] on host [master01], try #1 INFO[0180] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0180] Upgrading workerplane components for control host master01 INFO[0180] Finding container [service-sidekick] on host [master01], try #1 INFO[0180] [sidekick] Sidekick container already created on host [master01] INFO[0180] Finding container [kubelet] on host [master01], try #1 INFO[0180] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master01] INFO[0180] Finding container [old-kubelet] on host [master01], try #1 INFO[0180] Stopping container [kubelet] on host [master01] with stopTimeoutDuration [5s], try #1 INFO[0180] Waiting for [kubelet] container to exit on host [master01] INFO[0180] Renaming container [kubelet] to [old-kubelet] on host [master01], try #1 INFO[0180] Starting container [kubelet] on host [master01], try #1 INFO[0180] [worker] Successfully updated [kubelet] container on host [master01] INFO[0180] Removing container [old-kubelet] on host [master01], try #1 INFO[0180] [healthcheck] Start Healthcheck on service [kubelet] on host [master01] INFO[0196] [healthcheck] service [kubelet] on host [master01] is healthy INFO[0196] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0197] Starting container [rke-log-linker] on host [master01], try #1 INFO[0197] [worker] Successfully started [rke-log-linker] container on host [master01] INFO[0197] Removing container [rke-log-linker] on host [master01], try #1 INFO[0197] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0197] Finding container [kube-proxy] on host [master01], try #1 INFO[0197] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master01] INFO[0197] Finding container [old-kube-proxy] on host [master01], try #1 INFO[0197] Stopping container [kube-proxy] on host [master01] with stopTimeoutDuration [5s], try #1 INFO[0197] Waiting for [kube-proxy] container to exit on host [master01] INFO[0197] Renaming container [kube-proxy] to [old-kube-proxy] on host [master01], try #1 INFO[0197] Starting container [kube-proxy] on host [master01], try #1 INFO[0198] [worker] Successfully updated [kube-proxy] container on host [master01] INFO[0198] Removing container [old-kube-proxy] on host [master01], try #1 INFO[0198] [healthcheck] Start Healthcheck on service [kube-proxy] on host [master01] INFO[0203] [healthcheck] service [kube-proxy] on host [master01] is healthy INFO[0203] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0203] Starting container [rke-log-linker] on host [master01], try #1 INFO[0203] [worker] Successfully started [rke-log-linker] container on host [master01] INFO[0204] Removing container [rke-log-linker] on host [master01], try #1 INFO[0204] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0204] [controlplane] Now checking status of node master01, try #1 INFO[0204] Processing controlplane host master02 INFO[0204] [controlplane] Now checking status of node master02, try #1 INFO[0204] [controlplane] Getting list of nodes for upgrade INFO[0204] Upgrading controlplane components for control host master02 INFO[0204] Finding container [service-sidekick] on host [master02], try #1 INFO[0204] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0204] Removing container [service-sidekick] on host [master02], try #1 INFO[0204] [remove/service-sidekick] Successfully removed container on host [master02] INFO[0204] Finding container [kube-apiserver] on host [master02], try #1 INFO[0204] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master02] INFO[0204] Finding container [old-kube-apiserver] on host [master02], try #1 INFO[0204] Stopping container [kube-apiserver] on host [master02] with stopTimeoutDuration [5s], try #1 INFO[0205] Waiting for [kube-apiserver] container to exit on host [master02] INFO[0205] Renaming container [kube-apiserver] to [old-kube-apiserver] on host [master02], try #1 INFO[0205] Starting container [kube-apiserver] on host [master02], try #1 INFO[0205] [controlplane] Successfully updated [kube-apiserver] container on host [master02] INFO[0205] Removing container [old-kube-apiserver] on host [master02], try #1 INFO[0205] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [master02] INFO[0210] [healthcheck] service [kube-apiserver] on host [master02] is healthy INFO[0210] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0211] Starting container [rke-log-linker] on host [master02], try #1 INFO[0211] [controlplane] Successfully started [rke-log-linker] container on host [master02] INFO[0211] Removing container [rke-log-linker] on host [master02], try #1 INFO[0211] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0211] Finding container [kube-controller-manager] on host [master02], try #1 INFO[0211] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master02] INFO[0211] Finding container [old-kube-controller-manager] on host [master02], try #1 INFO[0211] Stopping container [kube-controller-manager] on host [master02] with stopTimeoutDuration [5s], try #1 INFO[0211] Waiting for [kube-controller-manager] container to exit on host [master02] INFO[0211] Renaming container [kube-controller-manager] to [old-kube-controller-manager] on host [master02], try #1 INFO[0211] Starting container [kube-controller-manager] on host [master02], try #1 INFO[0211] [controlplane] Successfully updated [kube-controller-manager] container on host [master02] INFO[0211] Removing container [old-kube-controller-manager] on host [master02], try #1 INFO[0211] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [master02] INFO[0217] [healthcheck] service [kube-controller-manager] on host [master02] is healthy INFO[0217] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0217] Starting container [rke-log-linker] on host [master02], try #1 INFO[0217] [controlplane] Successfully started [rke-log-linker] container on host [master02] INFO[0217] Removing container [rke-log-linker] on host [master02], try #1 INFO[0217] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0217] Finding container [kube-scheduler] on host [master02], try #1 INFO[0217] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master02] INFO[0217] Finding container [old-kube-scheduler] on host [master02], try #1 INFO[0217] Stopping container [kube-scheduler] on host [master02] with stopTimeoutDuration [5s], try #1 INFO[0217] Waiting for [kube-scheduler] container to exit on host [master02] INFO[0217] Renaming container [kube-scheduler] to [old-kube-scheduler] on host [master02], try #1 INFO[0217] Starting container [kube-scheduler] on host [master02], try #1 INFO[0217] [controlplane] Successfully updated [kube-scheduler] container on host [master02] INFO[0217] Removing container [old-kube-scheduler] on host [master02], try #1 INFO[0217] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [master02] INFO[0223] [healthcheck] service [kube-scheduler] on host [master02] is healthy INFO[0223] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0223] Starting container [rke-log-linker] on host [master02], try #1 INFO[0223] [controlplane] Successfully started [rke-log-linker] container on host [master02] INFO[0223] Removing container [rke-log-linker] on host [master02], try #1 INFO[0224] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0224] Upgrading workerplane components for control host master02 INFO[0224] Finding container [service-sidekick] on host [master02], try #1 INFO[0224] [sidekick] Sidekick container already created on host [master02] INFO[0224] Finding container [kubelet] on host [master02], try #1 INFO[0224] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master02] INFO[0224] Finding container [old-kubelet] on host [master02], try #1 INFO[0224] Stopping container [kubelet] on host [master02] with stopTimeoutDuration [5s], try #1 INFO[0224] Waiting for [kubelet] container to exit on host [master02] INFO[0224] Renaming container [kubelet] to [old-kubelet] on host [master02], try #1 INFO[0224] Starting container [kubelet] on host [master02], try #1 INFO[0224] [worker] Successfully updated [kubelet] container on host [master02] INFO[0224] Removing container [old-kubelet] on host [master02], try #1 INFO[0224] [healthcheck] Start Healthcheck on service [kubelet] on host [master02] INFO[0239] [healthcheck] service [kubelet] on host [master02] is healthy INFO[0239] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0240] Starting container [rke-log-linker] on host [master02], try #1 INFO[0240] [worker] Successfully started [rke-log-linker] container on host [master02] INFO[0240] Removing container [rke-log-linker] on host [master02], try #1 INFO[0240] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0240] Finding container [kube-proxy] on host [master02], try #1 INFO[0240] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master02] INFO[0240] Finding container [old-kube-proxy] on host [master02], try #1 INFO[0240] Stopping container [kube-proxy] on host [master02] with stopTimeoutDuration [5s], try #1 INFO[0240] Waiting for [kube-proxy] container to exit on host [master02] INFO[0240] Renaming container [kube-proxy] to [old-kube-proxy] on host [master02], try #1 INFO[0240] Starting container [kube-proxy] on host [master02], try #1 INFO[0240] [worker] Successfully updated [kube-proxy] container on host [master02] INFO[0240] Removing container [old-kube-proxy] on host [master02], try #1 INFO[0240] [healthcheck] Start Healthcheck on service [kube-proxy] on host [master02] INFO[0246] [healthcheck] service [kube-proxy] on host [master02] is healthy INFO[0246] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0246] Starting container [rke-log-linker] on host [master02], try #1 INFO[0246] [worker] Successfully started [rke-log-linker] container on host [master02] INFO[0246] Removing container [rke-log-linker] on host [master02], try #1 INFO[0247] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0247] [controlplane] Now checking status of node master02, try #1 INFO[0247] Processing controlplane host master03 INFO[0247] [controlplane] Now checking status of node master03, try #1 INFO[0247] [controlplane] Getting list of nodes for upgrade INFO[0247] Upgrading controlplane components for control host master03 INFO[0247] Finding container [service-sidekick] on host [master03], try #1 INFO[0247] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0247] Removing container [service-sidekick] on host [master03], try #1 INFO[0247] [remove/service-sidekick] Successfully removed container on host [master03] INFO[0247] Finding container [kube-apiserver] on host [master03], try #1 INFO[0247] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master03] INFO[0247] Finding container [old-kube-apiserver] on host [master03], try #1 INFO[0247] Stopping container [kube-apiserver] on host [master03] with stopTimeoutDuration [5s], try #1 INFO[0248] Waiting for [kube-apiserver] container to exit on host [master03] INFO[0248] Renaming container [kube-apiserver] to [old-kube-apiserver] on host [master03], try #1 INFO[0248] Starting container [kube-apiserver] on host [master03], try #1 INFO[0248] [controlplane] Successfully updated [kube-apiserver] container on host [master03] INFO[0248] Removing container [old-kube-apiserver] on host [master03], try #1 INFO[0248] [healthcheck] Start Healthcheck on service [kube-apiserver] on host [master03] INFO[0253] [healthcheck] service [kube-apiserver] on host [master03] is healthy INFO[0253] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0254] Starting container [rke-log-linker] on host [master03], try #1 INFO[0254] [controlplane] Successfully started [rke-log-linker] container on host [master03] INFO[0254] Removing container [rke-log-linker] on host [master03], try #1 INFO[0254] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0254] Finding container [kube-controller-manager] on host [master03], try #1 INFO[0254] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master03] INFO[0254] Finding container [old-kube-controller-manager] on host [master03], try #1 INFO[0254] Stopping container [kube-controller-manager] on host [master03] with stopTimeoutDuration [5s], try #1 INFO[0254] Waiting for [kube-controller-manager] container to exit on host [master03] INFO[0254] Renaming container [kube-controller-manager] to [old-kube-controller-manager] on host [master03], try #1 INFO[0254] Starting container [kube-controller-manager] on host [master03], try #1 INFO[0254] [controlplane] Successfully updated [kube-controller-manager] container on host [master03] INFO[0254] Removing container [old-kube-controller-manager] on host [master03], try #1 INFO[0254] [healthcheck] Start Healthcheck on service [kube-controller-manager] on host [master03] INFO[0260] [healthcheck] service [kube-controller-manager] on host [master03] is healthy INFO[0260] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0260] Starting container [rke-log-linker] on host [master03], try #1 INFO[0260] [controlplane] Successfully started [rke-log-linker] container on host [master03] INFO[0260] Removing container [rke-log-linker] on host [master03], try #1 INFO[0260] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0260] Finding container [kube-scheduler] on host [master03], try #1 INFO[0260] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master03] INFO[0260] Finding container [old-kube-scheduler] on host [master03], try #1 INFO[0260] Stopping container [kube-scheduler] on host [master03] with stopTimeoutDuration [5s], try #1 INFO[0260] Waiting for [kube-scheduler] container to exit on host [master03] INFO[0261] Renaming container [kube-scheduler] to [old-kube-scheduler] on host [master03], try #1 INFO[0261] Starting container [kube-scheduler] on host [master03], try #1 INFO[0261] [controlplane] Successfully updated [kube-scheduler] container on host [master03] INFO[0261] Removing container [old-kube-scheduler] on host [master03], try #1 INFO[0261] [healthcheck] Start Healthcheck on service [kube-scheduler] on host [master03] INFO[0266] [healthcheck] service [kube-scheduler] on host [master03] is healthy INFO[0266] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0266] Starting container [rke-log-linker] on host [master03], try #1 INFO[0267] [controlplane] Successfully started [rke-log-linker] container on host [master03] INFO[0267] Removing container [rke-log-linker] on host [master03], try #1 INFO[0267] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0267] Upgrading workerplane components for control host master03 INFO[0267] Finding container [service-sidekick] on host [master03], try #1 INFO[0267] [sidekick] Sidekick container already created on host [master03] INFO[0267] Finding container [kubelet] on host [master03], try #1 INFO[0267] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master03] INFO[0267] Finding container [old-kubelet] on host [master03], try #1 INFO[0267] Stopping container [kubelet] on host [master03] with stopTimeoutDuration [5s], try #1 INFO[0267] Waiting for [kubelet] container to exit on host [master03] INFO[0267] Renaming container [kubelet] to [old-kubelet] on host [master03], try #1 INFO[0267] Starting container [kubelet] on host [master03], try #1 INFO[0267] [worker] Successfully updated [kubelet] container on host [master03] INFO[0267] Removing container [old-kubelet] on host [master03], try #1 INFO[0267] [healthcheck] Start Healthcheck on service [kubelet] on host [master03] INFO[0283] [healthcheck] service [kubelet] on host [master03] is healthy INFO[0283] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0283] Starting container [rke-log-linker] on host [master03], try #1 INFO[0284] [worker] Successfully started [rke-log-linker] container on host [master03] INFO[0284] Removing container [rke-log-linker] on host [master03], try #1 INFO[0284] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0284] Finding container [kube-proxy] on host [master03], try #1 INFO[0284] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master03] INFO[0284] Finding container [old-kube-proxy] on host [master03], try #1 INFO[0284] Stopping container [kube-proxy] on host [master03] with stopTimeoutDuration [5s], try #1 INFO[0284] Waiting for [kube-proxy] container to exit on host [master03] INFO[0284] Renaming container [kube-proxy] to [old-kube-proxy] on host [master03], try #1 INFO[0284] Starting container [kube-proxy] on host [master03], try #1 INFO[0284] [worker] Successfully updated [kube-proxy] container on host [master03] INFO[0284] Removing container [old-kube-proxy] on host [master03], try #1 INFO[0284] [healthcheck] Start Healthcheck on service [kube-proxy] on host [master03] INFO[0290] [healthcheck] service [kube-proxy] on host [master03] is healthy INFO[0290] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0290] Starting container [rke-log-linker] on host [master03], try #1 INFO[0291] [worker] Successfully started [rke-log-linker] container on host [master03] INFO[0291] Removing container [rke-log-linker] on host [master03], try #1 INFO[0291] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0291] [controlplane] Now checking status of node master03, try #1 INFO[0291] [controlplane] Successfully upgraded Controller Plane.. INFO[0291] [authz] Creating rke-job-deployer ServiceAccount INFO[0291] [authz] rke-job-deployer ServiceAccount created successfully INFO[0291] [authz] Creating system:node ClusterRoleBinding INFO[0291] [authz] system:node ClusterRoleBinding created successfully INFO[0291] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding INFO[0291] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully INFO[0291] Successfully Deployed state file at [./cluster.rkestate] INFO[0291] [state] Saving full cluster state to Kubernetes INFO[0291] [worker] Now checking status of node worker01, try #1 INFO[0291] [worker] Now checking status of node worker02, try #1 INFO[0291] [worker] Now checking status of node worker03, try #1 INFO[0291] [worker] Upgrading Worker Plane.. INFO[0291] Now checking and upgrading worker components on nodes with only worker role 1 at a time INFO[0291] [workerplane] Processing host worker01 INFO[0291] [worker] Now checking status of node worker01, try #1 INFO[0291] [worker] Getting list of nodes for upgrade INFO[0291] Finding container [nginx-proxy] on host [worker01], try #1 INFO[0291] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0291] Finding container [old-nginx-proxy] on host [worker01], try #1 INFO[0291] Stopping container [nginx-proxy] on host [worker01] with stopTimeoutDuration [5s], try #1 INFO[0297] Waiting for [nginx-proxy] container to exit on host [worker01] INFO[0297] Renaming container [nginx-proxy] to [old-nginx-proxy] on host [worker01], try #1 INFO[0297] Starting container [nginx-proxy] on host [worker01], try #1 INFO[0297] [worker] Successfully updated [nginx-proxy] container on host [worker01] INFO[0297] Removing container [old-nginx-proxy] on host [worker01], try #1 INFO[0297] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0297] Starting container [rke-log-linker] on host [worker01], try #1 INFO[0298] [worker] Successfully started [rke-log-linker] container on host [worker01] INFO[0298] Removing container [rke-log-linker] on host [worker01], try #1 INFO[0298] [remove/rke-log-linker] Successfully removed container on host [worker01] INFO[0298] Finding container [service-sidekick] on host [worker01], try #1 INFO[0298] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0298] Removing container [service-sidekick] on host [worker01], try #1 INFO[0298] [remove/service-sidekick] Successfully removed container on host [worker01] INFO[0298] Finding container [kubelet] on host [worker01], try #1 INFO[0298] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker01] INFO[0298] Finding container [old-kubelet] on host [worker01], try #1 INFO[0298] Stopping container [kubelet] on host [worker01] with stopTimeoutDuration [5s], try #1 INFO[0299] Waiting for [kubelet] container to exit on host [worker01] INFO[0299] Renaming container [kubelet] to [old-kubelet] on host [worker01], try #1 INFO[0299] Starting container [kubelet] on host [worker01], try #1 INFO[0299] [worker] Successfully updated [kubelet] container on host [worker01] INFO[0299] Removing container [old-kubelet] on host [worker01], try #1 INFO[0299] [healthcheck] Start Healthcheck on service [kubelet] on host [worker01] INFO[0314] [healthcheck] service [kubelet] on host [worker01] is healthy INFO[0314] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0314] Starting container [rke-log-linker] on host [worker01], try #1 INFO[0315] [worker] Successfully started [rke-log-linker] container on host [worker01] INFO[0315] Removing container [rke-log-linker] on host [worker01], try #1 INFO[0315] [remove/rke-log-linker] Successfully removed container on host [worker01] INFO[0315] Finding container [kube-proxy] on host [worker01], try #1 INFO[0315] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker01] INFO[0315] Finding container [old-kube-proxy] on host [worker01], try #1 INFO[0315] Stopping container [kube-proxy] on host [worker01] with stopTimeoutDuration [5s], try #1 INFO[0315] Waiting for [kube-proxy] container to exit on host [worker01] INFO[0315] Renaming container [kube-proxy] to [old-kube-proxy] on host [worker01], try #1 INFO[0315] Starting container [kube-proxy] on host [worker01], try #1 INFO[0315] [worker] Successfully updated [kube-proxy] container on host [worker01] INFO[0315] Removing container [old-kube-proxy] on host [worker01], try #1 INFO[0315] [healthcheck] Start Healthcheck on service [kube-proxy] on host [worker01] INFO[0321] [healthcheck] service [kube-proxy] on host [worker01] is healthy INFO[0321] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0321] Starting container [rke-log-linker] on host [worker01], try #1 INFO[0321] [worker] Successfully started [rke-log-linker] container on host [worker01] INFO[0321] Removing container [rke-log-linker] on host [worker01], try #1 INFO[0322] [remove/rke-log-linker] Successfully removed container on host [worker01] INFO[0322] [worker] Now checking status of node worker01, try #1 INFO[0322] [workerplane] Processing host worker02 INFO[0322] [worker] Now checking status of node worker02, try #1 INFO[0322] [worker] Getting list of nodes for upgrade INFO[0322] Finding container [nginx-proxy] on host [worker02], try #1 INFO[0322] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0322] Finding container [old-nginx-proxy] on host [worker02], try #1 INFO[0322] Stopping container [nginx-proxy] on host [worker02] with stopTimeoutDuration [5s], try #1 INFO[0327] Waiting for [nginx-proxy] container to exit on host [worker02] INFO[0327] Renaming container [nginx-proxy] to [old-nginx-proxy] on host [worker02], try #1 INFO[0328] Starting container [nginx-proxy] on host [worker02], try #1 INFO[0328] [worker] Successfully updated [nginx-proxy] container on host [worker02] INFO[0328] Removing container [old-nginx-proxy] on host [worker02], try #1 INFO[0328] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0328] Starting container [rke-log-linker] on host [worker02], try #1 INFO[0328] [worker] Successfully started [rke-log-linker] container on host [worker02] INFO[0328] Removing container [rke-log-linker] on host [worker02], try #1 INFO[0328] [remove/rke-log-linker] Successfully removed container on host [worker02] INFO[0328] Finding container [service-sidekick] on host [worker02], try #1 INFO[0328] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0328] Removing container [service-sidekick] on host [worker02], try #1 INFO[0328] [remove/service-sidekick] Successfully removed container on host [worker02] INFO[0329] Finding container [kubelet] on host [worker02], try #1 INFO[0329] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker02] INFO[0329] Finding container [old-kubelet] on host [worker02], try #1 INFO[0329] Stopping container [kubelet] on host [worker02] with stopTimeoutDuration [5s], try #1 INFO[0329] Waiting for [kubelet] container to exit on host [worker02] INFO[0329] Renaming container [kubelet] to [old-kubelet] on host [worker02], try #1 INFO[0329] Starting container [kubelet] on host [worker02], try #1 INFO[0329] [worker] Successfully updated [kubelet] container on host [worker02] INFO[0329] Removing container [old-kubelet] on host [worker02], try #1 INFO[0329] [healthcheck] Start Healthcheck on service [kubelet] on host [worker02] INFO[0345] [healthcheck] service [kubelet] on host [worker02] is healthy INFO[0345] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0345] Starting container [rke-log-linker] on host [worker02], try #1 INFO[0346] [worker] Successfully started [rke-log-linker] container on host [worker02] INFO[0346] Removing container [rke-log-linker] on host [worker02], try #1 INFO[0346] [remove/rke-log-linker] Successfully removed container on host [worker02] INFO[0346] Finding container [kube-proxy] on host [worker02], try #1 INFO[0346] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker02] INFO[0346] Finding container [old-kube-proxy] on host [worker02], try #1 INFO[0346] Stopping container [kube-proxy] on host [worker02] with stopTimeoutDuration [5s], try #1 INFO[0346] Waiting for [kube-proxy] container to exit on host [worker02] INFO[0346] Renaming container [kube-proxy] to [old-kube-proxy] on host [worker02], try #1 INFO[0346] Starting container [kube-proxy] on host [worker02], try #1 INFO[0346] [worker] Successfully updated [kube-proxy] container on host [worker02] INFO[0346] Removing container [old-kube-proxy] on host [worker02], try #1 INFO[0346] [healthcheck] Start Healthcheck on service [kube-proxy] on host [worker02] INFO[0352] [healthcheck] service [kube-proxy] on host [worker02] is healthy INFO[0352] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0352] Starting container [rke-log-linker] on host [worker02], try #1 INFO[0353] [worker] Successfully started [rke-log-linker] container on host [worker02] INFO[0353] Removing container [rke-log-linker] on host [worker02], try #1 INFO[0353] [remove/rke-log-linker] Successfully removed container on host [worker02] INFO[0353] [worker] Now checking status of node worker02, try #1 INFO[0353] [workerplane] Processing host worker03 INFO[0353] [worker] Now checking status of node worker03, try #1 INFO[0353] [worker] Getting list of nodes for upgrade INFO[0354] Finding container [nginx-proxy] on host [worker03], try #1 INFO[0354] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0354] Finding container [old-nginx-proxy] on host [worker03], try #1 INFO[0354] Stopping container [nginx-proxy] on host [worker03] with stopTimeoutDuration [5s], try #1 INFO[0359] Waiting for [nginx-proxy] container to exit on host [worker03] INFO[0359] Renaming container [nginx-proxy] to [old-nginx-proxy] on host [worker03], try #1 INFO[0359] Starting container [nginx-proxy] on host [worker03], try #1 INFO[0359] [worker] Successfully updated [nginx-proxy] container on host [worker03] INFO[0359] Removing container [old-nginx-proxy] on host [worker03], try #1 INFO[0359] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0360] Starting container [rke-log-linker] on host [worker03], try #1 INFO[0360] [worker] Successfully started [rke-log-linker] container on host [worker03] INFO[0360] Removing container [rke-log-linker] on host [worker03], try #1 INFO[0360] [remove/rke-log-linker] Successfully removed container on host [worker03] INFO[0360] Finding container [service-sidekick] on host [worker03], try #1 INFO[0360] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0360] Removing container [service-sidekick] on host [worker03], try #1 INFO[0360] [remove/service-sidekick] Successfully removed container on host [worker03] INFO[0360] Finding container [kubelet] on host [worker03], try #1 INFO[0360] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker03] INFO[0360] Finding container [old-kubelet] on host [worker03], try #1 INFO[0360] Stopping container [kubelet] on host [worker03] with stopTimeoutDuration [5s], try #1 INFO[0361] Waiting for [kubelet] container to exit on host [worker03] INFO[0361] Renaming container [kubelet] to [old-kubelet] on host [worker03], try #1 INFO[0361] Starting container [kubelet] on host [worker03], try #1 INFO[0361] [worker] Successfully updated [kubelet] container on host [worker03] INFO[0361] Removing container [old-kubelet] on host [worker03], try #1 INFO[0361] [healthcheck] Start Healthcheck on service [kubelet] on host [worker03] INFO[0376] [healthcheck] service [kubelet] on host [worker03] is healthy INFO[0376] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0377] Starting container [rke-log-linker] on host [worker03], try #1 INFO[0377] [worker] Successfully started [rke-log-linker] container on host [worker03] INFO[0377] Removing container [rke-log-linker] on host [worker03], try #1 INFO[0377] [remove/rke-log-linker] Successfully removed container on host [worker03] INFO[0377] Finding container [kube-proxy] on host [worker03], try #1 INFO[0377] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker03] INFO[0377] Finding container [old-kube-proxy] on host [worker03], try #1 INFO[0378] Stopping container [kube-proxy] on host [worker03] with stopTimeoutDuration [5s], try #1 INFO[0378] Waiting for [kube-proxy] container to exit on host [worker03] INFO[0378] Renaming container [kube-proxy] to [old-kube-proxy] on host [worker03], try #1 INFO[0378] Starting container [kube-proxy] on host [worker03], try #1 INFO[0378] [worker] Successfully updated [kube-proxy] container on host [worker03] INFO[0378] Removing container [old-kube-proxy] on host [worker03], try #1 INFO[0378] [healthcheck] Start Healthcheck on service [kube-proxy] on host [worker03] INFO[0383] [healthcheck] service [kube-proxy] on host [worker03] is healthy INFO[0383] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0384] Starting container [rke-log-linker] on host [worker03], try #1 INFO[0384] [worker] Successfully started [rke-log-linker] container on host [worker03] INFO[0384] Removing container [rke-log-linker] on host [worker03], try #1 INFO[0384] [remove/rke-log-linker] Successfully removed container on host [worker03] INFO[0384] [worker] Now checking status of node worker03, try #1 INFO[0384] [worker] Successfully upgraded Worker Plane.. INFO[0384] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0384] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0384] Starting container [rke-log-cleaner] on host [worker03], try #1 INFO[0384] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0385] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0385] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0385] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0385] Starting container [rke-log-cleaner] on host [worker02], try #1 INFO[0385] [cleanup] Successfully started [rke-log-cleaner] container on host [worker03] INFO[0385] Starting container [rke-log-cleaner] on host [master03], try #1 INFO[0385] Starting container [rke-log-cleaner] on host [master02], try #1 INFO[0385] Starting container [rke-log-cleaner] on host [worker01], try #1 INFO[0385] [cleanup] Successfully started [rke-log-cleaner] container on host [worker02] INFO[0385] Removing container [rke-log-cleaner] on host [worker03], try #1 INFO[0385] [remove/rke-log-cleaner] Successfully removed container on host [worker03] INFO[0385] Starting container [rke-log-cleaner] on host [master01], try #1 INFO[0385] [cleanup] Successfully started [rke-log-cleaner] container on host [master03] INFO[0385] Removing container [rke-log-cleaner] on host [master03], try #1 INFO[0385] Removing container [rke-log-cleaner] on host [worker02], try #1 INFO[0385] [remove/rke-log-cleaner] Successfully removed container on host [worker02] INFO[0385] [cleanup] Successfully started [rke-log-cleaner] container on host [master02] INFO[0385] Removing container [rke-log-cleaner] on host [master02], try #1 INFO[0385] [cleanup] Successfully started [rke-log-cleaner] container on host [worker01] INFO[0386] [remove/rke-log-cleaner] Successfully removed container on host [master03] INFO[0386] [cleanup] Successfully started [rke-log-cleaner] container on host [master01] INFO[0386] Removing container [rke-log-cleaner] on host [master01], try #1 INFO[0386] [remove/rke-log-cleaner] Successfully removed container on host [master02] INFO[0386] Removing container [rke-log-cleaner] on host [worker01], try #1 INFO[0386] [remove/rke-log-cleaner] Successfully removed container on host [worker01] INFO[0386] [remove/rke-log-cleaner] Successfully removed container on host [master01] INFO[0386] [sync] Syncing nodes Labels and Taints INFO[0386] [sync] Successfully synced nodes Labels and Taints INFO[0386] [network] Setting up network plugin: canal INFO[0386] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes INFO[0386] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes INFO[0386] [addons] Executing deploy job rke-network-plugin INFO[0421] [addons] Setting up coredns INFO[0421] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0421] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0421] [addons] Executing deploy job rke-coredns-addon INFO[0451] [addons] CoreDNS deployed successfully INFO[0451] [dns] DNS provider coredns deployed successfully INFO[0451] [addons] Setting up Metrics Server INFO[0451] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0451] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0451] [addons] Executing deploy job rke-metrics-addon INFO[0462] [addons] Metrics Server deployed successfully INFO[0462] [ingress] Setting up nginx ingress controller INFO[0462] [ingress] removing admission batch jobs if they exist INFO[0462] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0462] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0462] [addons] Executing deploy job rke-ingress-controller INFO[0477] [ingress] removing default backend service and deployment if they exist INFO[0477] [ingress] ingress controller nginx deployed successfully INFO[0477] [addons] Setting up user addons INFO[0477] [addons] no user addons defined INFO[0477] Finished building Kubernetes cluster successfully masadmin@master01:~$ masadmin@master01:~$ nano cluster.yml masadmin@master01:~$ rke up --config cluster.yml --ignore-docker-version INFO[0000] Running RKE version: v1.5.12 INFO[0000] Initiating Kubernetes cluster INFO[0000] [certificates] GenerateServingCertificate is disabled, checking if there are unused kubelet certificates INFO[0000] [certificates] Generating admin certificates and kubeconfig INFO[0000] Successfully Deployed state file at [./cluster.rkestate] INFO[0000] Building Kubernetes cluster INFO[0000] [dialer] Setup tunnel for host [master03] INFO[0000] [dialer] Setup tunnel for host [worker02] INFO[0000] [dialer] Setup tunnel for host [worker01] INFO[0000] [dialer] Setup tunnel for host [worker03] INFO[0000] [dialer] Setup tunnel for host [master01] INFO[0000] [dialer] Setup tunnel for host [master02] INFO[0000] [network] No hosts added existing cluster, skipping port check INFO[0000] [certificates] Deploying kubernetes certificates to Cluster nodes INFO[0000] Finding container [cert-deployer] on host [worker01], try #1 INFO[0000] Finding container [cert-deployer] on host [worker02], try #1 INFO[0000] Finding container [cert-deployer] on host [worker03], try #1 INFO[0000] Finding container [cert-deployer] on host [master03], try #1 INFO[0000] Finding container [cert-deployer] on host [master02], try #1 INFO[0000] Finding container [cert-deployer] on host [master01], try #1 INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0000] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0000] Starting container [cert-deployer] on host [master02], try #1 INFO[0001] Starting container [cert-deployer] on host [worker01], try #1 INFO[0001] Starting container [cert-deployer] on host [worker02], try #1 INFO[0001] Finding container [cert-deployer] on host [master02], try #1 INFO[0001] Starting container [cert-deployer] on host [worker03], try #1 INFO[0001] Starting container [cert-deployer] on host [master03], try #1 INFO[0001] Starting container [cert-deployer] on host [master01], try #1 INFO[0001] Finding container [cert-deployer] on host [worker01], try #1 INFO[0001] Finding container [cert-deployer] on host [worker02], try #1 INFO[0001] Finding container [cert-deployer] on host [worker03], try #1 INFO[0001] Finding container [cert-deployer] on host [master03], try #1 INFO[0001] Finding container [cert-deployer] on host [master01], try #1 INFO[0006] Finding container [cert-deployer] on host [master02], try #1 INFO[0006] Removing container [cert-deployer] on host [master02], try #1 INFO[0006] Finding container [cert-deployer] on host [worker01], try #1 INFO[0006] Removing container [cert-deployer] on host [worker01], try #1 INFO[0006] Finding container [cert-deployer] on host [worker02], try #1 INFO[0006] Removing container [cert-deployer] on host [worker02], try #1 INFO[0006] Finding container [cert-deployer] on host [worker03], try #1 INFO[0006] Removing container [cert-deployer] on host [worker03], try #1 INFO[0006] Finding container [cert-deployer] on host [master03], try #1 INFO[0006] Removing container [cert-deployer] on host [master03], try #1 INFO[0006] Finding container [cert-deployer] on host [master01], try #1 INFO[0006] Removing container [cert-deployer] on host [master01], try #1 INFO[0006] [reconcile] Rebuilding and updating local kube config INFO[0006] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0006] [reconcile] host [master01] is a control plane node with reachable Kubernetes API endpoint in the cluste r INFO[0006] [certificates] Successfully deployed kubernetes certificates to Cluster nodes INFO[0006] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [master01] INFO[0006] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0006] Starting container [file-deployer] on host [master01], try #1 INFO[0007] Successfully started [file-deployer] container on host [master01] INFO[0007] Waiting for [file-deployer] container to exit on host [master01] INFO[0007] Waiting for [file-deployer] container to exit on host [master01] INFO[0007] Container [file-deployer] is still running on host [master01]: stderr: [], stdout: [] INFO[0008] Removing container [file-deployer] on host [master01], try #1 INFO[0008] [remove/file-deployer] Successfully removed container on host [master01] INFO[0008] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [master02] INFO[0008] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0008] Starting container [file-deployer] on host [master02], try #1 INFO[0008] Successfully started [file-deployer] container on host [master02] INFO[0008] Waiting for [file-deployer] container to exit on host [master02] INFO[0008] Waiting for [file-deployer] container to exit on host [master02] INFO[0008] Container [file-deployer] is still running on host [master02]: stderr: [], stdout: [] INFO[0009] Removing container [file-deployer] on host [master02], try #1 INFO[0009] [remove/file-deployer] Successfully removed container on host [master02] INFO[0009] [file-deploy] Deploying file [/etc/kubernetes/admission.yaml] to node [master03] INFO[0009] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0010] Starting container [file-deployer] on host [master03], try #1 INFO[0010] Successfully started [file-deployer] container on host [master03] INFO[0010] Waiting for [file-deployer] container to exit on host [master03] INFO[0010] Waiting for [file-deployer] container to exit on host [master03] INFO[0010] Removing container [file-deployer] on host [master03], try #1 INFO[0010] [remove/file-deployer] Successfully removed container on host [master03] INFO[0010] [/etc/kubernetes/admission.yaml] Successfully deployed admission control config to Cluster control nodes INFO[0010] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [master01] INFO[0010] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0010] Starting container [file-deployer] on host [master01], try #1 INFO[0011] Successfully started [file-deployer] container on host [master01] INFO[0011] Waiting for [file-deployer] container to exit on host [master01] INFO[0011] Waiting for [file-deployer] container to exit on host [master01] INFO[0011] Container [file-deployer] is still running on host [master01]: stderr: [], stdout: [] INFO[0012] Removing container [file-deployer] on host [master01], try #1 INFO[0012] [remove/file-deployer] Successfully removed container on host [master01] INFO[0012] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [master02] INFO[0012] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0012] Starting container [file-deployer] on host [master02], try #1 INFO[0012] Successfully started [file-deployer] container on host [master02] INFO[0012] Waiting for [file-deployer] container to exit on host [master02] INFO[0012] Waiting for [file-deployer] container to exit on host [master02] INFO[0012] Removing container [file-deployer] on host [master02], try #1 INFO[0012] [remove/file-deployer] Successfully removed container on host [master02] INFO[0012] [file-deploy] Deploying file [/etc/kubernetes/audit-policy.yaml] to node [master03] INFO[0012] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0013] Starting container [file-deployer] on host [master03], try #1 INFO[0013] Successfully started [file-deployer] container on host [master03] INFO[0013] Waiting for [file-deployer] container to exit on host [master03] INFO[0013] Waiting for [file-deployer] container to exit on host [master03] INFO[0013] Container [file-deployer] is still running on host [master03]: stderr: [], stdout: [] INFO[0014] Removing container [file-deployer] on host [master03], try #1 INFO[0014] [remove/file-deployer] Successfully removed container on host [master03] INFO[0014] [/etc/kubernetes/audit-policy.yaml] Successfully deployed audit policy file to Cluster control nodes INFO[0014] [reconcile] Reconciling cluster state INFO[0014] [reconcile] Check etcd hosts to be deleted INFO[0014] [reconcile] Check etcd hosts to be added INFO[0014] [reconcile] Rebuilding and updating local kube config INFO[0014] Successfully Deployed local admin kubeconfig at [./kube_config_cluster.yml] INFO[0014] [reconcile] host [master01] is a control plane node with reachable Kubernetes API endpoint in the cluste r INFO[0014] [reconcile] Reconciled cluster state successfully INFO[0014] max_unavailable_worker got rounded down to 0, resetting to 1 INFO[0014] Setting maxUnavailable for worker nodes to: 1 INFO[0014] Setting maxUnavailable for controlplane nodes to: 1 INFO[0014] Pre-pulling kubernetes images INFO[0014] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master03] INFO[0014] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker01] INFO[0014] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker03] INFO[0014] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master01] INFO[0014] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [worker02] INFO[0014] Image [rancher/hyperkube:v1.28.12-rancher1] exists on host [master02] INFO[0014] Image [rancher/mirrored-pause:3.7] exists on host [master03] INFO[0014] Image [rancher/mirrored-pause:3.7] exists on host [worker01] INFO[0014] Image [rancher/mirrored-pause:3.7] exists on host [worker03] INFO[0014] Image [rancher/mirrored-pause:3.7] exists on host [master01] INFO[0014] Image [rancher/mirrored-pause:3.7] exists on host [master02] INFO[0014] Image [rancher/mirrored-pause:3.7] exists on host [worker02] INFO[0014] Kubernetes images pulled successfully INFO[0014] [etcd] Building up etcd plane.. INFO[0014] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0014] Starting container [etcd-fix-perm] on host [master01], try #1 INFO[0014] Successfully started [etcd-fix-perm] container on host [master01] INFO[0014] Waiting for [etcd-fix-perm] container to exit on host [master01] INFO[0014] Waiting for [etcd-fix-perm] container to exit on host [master01] INFO[0015] Removing container [etcd-fix-perm] on host [master01], try #1 INFO[0015] [remove/etcd-fix-perm] Successfully removed container on host [master01] INFO[0015] [etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [master01] INFO[0015] Removing container [etcd-rolling-snapshots] on host [master01], try #1 INFO[0015] [remove/etcd-rolling-snapshots] Successfully removed container on host [master01] INFO[0015] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0015] Starting container [etcd-rolling-snapshots] on host [master01], try #1 INFO[0015] [etcd] Successfully started [etcd-rolling-snapshots] container on host [master01] INFO[0020] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0020] Starting container [rke-bundle-cert] on host [master01], try #1 INFO[0021] [certificates] Successfully started [rke-bundle-cert] container on host [master01] INFO[0021] Waiting for [rke-bundle-cert] container to exit on host [master01] INFO[0021] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on hos t [master01] INFO[0021] Removing container [rke-bundle-cert] on host [master01], try #1 INFO[0021] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0021] Starting container [rke-log-linker] on host [master01], try #1 INFO[0021] [etcd] Successfully started [rke-log-linker] container on host [master01] INFO[0021] Removing container [rke-log-linker] on host [master01], try #1 INFO[0021] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0021] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0022] Starting container [rke-log-linker] on host [master01], try #1 INFO[0022] [etcd] Successfully started [rke-log-linker] container on host [master01] INFO[0022] Removing container [rke-log-linker] on host [master01], try #1 INFO[0022] [remove/rke-log-linker] Successfully removed container on host [master01] INFO[0022] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0022] Starting container [etcd-fix-perm] on host [master02], try #1 INFO[0023] Successfully started [etcd-fix-perm] container on host [master02] INFO[0023] Waiting for [etcd-fix-perm] container to exit on host [master02] INFO[0023] Waiting for [etcd-fix-perm] container to exit on host [master02] INFO[0023] Removing container [etcd-fix-perm] on host [master02], try #1 INFO[0023] [remove/etcd-fix-perm] Successfully removed container on host [master02] INFO[0023] [etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [master02] INFO[0023] Removing container [etcd-rolling-snapshots] on host [master02], try #1 INFO[0023] [remove/etcd-rolling-snapshots] Successfully removed container on host [master02] INFO[0023] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0023] Starting container [etcd-rolling-snapshots] on host [master02], try #1 INFO[0023] [etcd] Successfully started [etcd-rolling-snapshots] container on host [master02] INFO[0028] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0029] Starting container [rke-bundle-cert] on host [master02], try #1 INFO[0029] [certificates] Successfully started [rke-bundle-cert] container on host [master02] INFO[0029] Waiting for [rke-bundle-cert] container to exit on host [master02] INFO[0029] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on hos t [master02] INFO[0029] Removing container [rke-bundle-cert] on host [master02], try #1 INFO[0029] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0029] Starting container [rke-log-linker] on host [master02], try #1 INFO[0030] [etcd] Successfully started [rke-log-linker] container on host [master02] INFO[0030] Removing container [rke-log-linker] on host [master02], try #1 INFO[0030] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0030] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0030] Starting container [rke-log-linker] on host [master02], try #1 INFO[0030] [etcd] Successfully started [rke-log-linker] container on host [master02] INFO[0031] Removing container [rke-log-linker] on host [master02], try #1 INFO[0031] [remove/rke-log-linker] Successfully removed container on host [master02] INFO[0031] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0031] Starting container [etcd-fix-perm] on host [master03], try #1 INFO[0031] Successfully started [etcd-fix-perm] container on host [master03] INFO[0031] Waiting for [etcd-fix-perm] container to exit on host [master03] INFO[0031] Waiting for [etcd-fix-perm] container to exit on host [master03] INFO[0031] Removing container [etcd-fix-perm] on host [master03], try #1 INFO[0031] [remove/etcd-fix-perm] Successfully removed container on host [master03] INFO[0031] [etcd] Running rolling snapshot container [etcd-rolling-snapshots] on host [master03] INFO[0031] Removing container [etcd-rolling-snapshots] on host [master03], try #1 INFO[0031] [remove/etcd-rolling-snapshots] Successfully removed container on host [master03] INFO[0031] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0031] Starting container [etcd-rolling-snapshots] on host [master03], try #1 INFO[0032] [etcd] Successfully started [etcd-rolling-snapshots] container on host [master03] INFO[0037] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0037] Starting container [rke-bundle-cert] on host [master03], try #1 INFO[0037] [certificates] Successfully started [rke-bundle-cert] container on host [master03] INFO[0037] Waiting for [rke-bundle-cert] container to exit on host [master03] INFO[0037] [certificates] successfully saved certificate bundle [/opt/rke/etcd-snapshots//pki.bundle.tar.gz] on hos t [master03] INFO[0037] Removing container [rke-bundle-cert] on host [master03], try #1 INFO[0037] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0038] Starting container [rke-log-linker] on host [master03], try #1 INFO[0038] [etcd] Successfully started [rke-log-linker] container on host [master03] INFO[0038] Removing container [rke-log-linker] on host [master03], try #1 INFO[0038] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0038] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0038] Starting container [rke-log-linker] on host [master03], try #1 INFO[0039] [etcd] Successfully started [rke-log-linker] container on host [master03] INFO[0039] Removing container [rke-log-linker] on host [master03], try #1 INFO[0039] [remove/rke-log-linker] Successfully removed container on host [master03] INFO[0039] [etcd] Successfully started etcd plane.. Checking etcd cluster health INFO[0039] [etcd] etcd host [master01] reported healthy=true INFO[0039] [controlplane] Now checking status of node master01, try #1 INFO[0039] [controlplane] Now checking status of node master02, try #1 INFO[0039] [controlplane] Now checking status of node master03, try #1 INFO[0039] [controlplane] Processing controlplane hosts for upgrade 1 at a time INFO[0039] Processing controlplane host master01 INFO[0039] [controlplane] Now checking status of node master01, try #1 INFO[0039] [controlplane] Getting list of nodes for upgrade INFO[0039] Upgrade not required for controlplane and worker components of host master01 INFO[0039] Processing controlplane host master02 INFO[0039] [controlplane] Now checking status of node master02, try #1 INFO[0039] [controlplane] Getting list of nodes for upgrade INFO[0039] Upgrade not required for controlplane and worker components of host master02 INFO[0039] Processing controlplane host master03 INFO[0039] [controlplane] Now checking status of node master03, try #1 INFO[0039] [controlplane] Getting list of nodes for upgrade INFO[0039] Upgrade not required for controlplane and worker components of host master03 INFO[0039] [controlplane] Successfully upgraded Controller Plane.. INFO[0039] [authz] Creating rke-job-deployer ServiceAccount INFO[0039] [authz] rke-job-deployer ServiceAccount created successfully INFO[0039] [authz] Creating system:node ClusterRoleBinding INFO[0039] [authz] system:node ClusterRoleBinding created successfully INFO[0039] [authz] Creating kube-apiserver proxy ClusterRole and ClusterRoleBinding INFO[0039] [authz] kube-apiserver proxy ClusterRole and ClusterRoleBinding created successfully INFO[0039] Successfully Deployed state file at [./cluster.rkestate] INFO[0039] [state] Saving full cluster state to Kubernetes INFO[0039] [worker] Now checking status of node worker01, try #1 INFO[0039] [worker] Now checking status of node worker02, try #1 INFO[0039] [worker] Now checking status of node worker03, try #1 INFO[0039] [worker] Upgrading Worker Plane.. INFO[0040] Now checking and upgrading worker components on nodes with only worker role 1 at a time INFO[0040] [workerplane] Processing host worker01 INFO[0040] [worker] Now checking status of node worker01, try #1 INFO[0040] [worker] Getting list of nodes for upgrade INFO[0040] [workerplane] Upgrade not required for worker components of host worker01 INFO[0040] [workerplane] Processing host worker02 INFO[0040] [worker] Now checking status of node worker02, try #1 INFO[0040] [worker] Getting list of nodes for upgrade INFO[0040] [workerplane] Upgrade not required for worker components of host worker02 INFO[0040] [workerplane] Processing host worker03 INFO[0040] [worker] Now checking status of node worker03, try #1 INFO[0040] [worker] Getting list of nodes for upgrade INFO[0040] [workerplane] Upgrade not required for worker components of host worker03 INFO[0040] [worker] Successfully upgraded Worker Plane.. INFO[0040] Image [rancher/rke-tools:v0.1.100] exists on host [master02] INFO[0040] Image [rancher/rke-tools:v0.1.100] exists on host [master01] INFO[0040] Image [rancher/rke-tools:v0.1.100] exists on host [master03] INFO[0040] Image [rancher/rke-tools:v0.1.100] exists on host [worker03] INFO[0040] Image [rancher/rke-tools:v0.1.100] exists on host [worker01] INFO[0040] Image [rancher/rke-tools:v0.1.100] exists on host [worker02] INFO[0041] Starting container [rke-log-cleaner] on host [master01], try #1 INFO[0041] Starting container [rke-log-cleaner] on host [master02], try #1 INFO[0041] Starting container [rke-log-cleaner] on host [worker02], try #1 INFO[0041] Starting container [rke-log-cleaner] on host [worker03], try #1 INFO[0041] Starting container [rke-log-cleaner] on host [worker01], try #1 INFO[0041] Starting container [rke-log-cleaner] on host [master03], try #1 INFO[0041] [cleanup] Successfully started [rke-log-cleaner] container on host [master02] INFO[0041] [cleanup] Successfully started [rke-log-cleaner] container on host [master01] INFO[0041] Removing container [rke-log-cleaner] on host [master02], try #1 INFO[0041] Removing container [rke-log-cleaner] on host [master01], try #1 INFO[0041] [cleanup] Successfully started [rke-log-cleaner] container on host [worker02] INFO[0041] Removing container [rke-log-cleaner] on host [worker02], try #1 INFO[0041] [cleanup] Successfully started [rke-log-cleaner] container on host [worker01] INFO[0041] [cleanup] Successfully started [rke-log-cleaner] container on host [worker03] INFO[0041] Removing container [rke-log-cleaner] on host [worker03], try #1 INFO[0041] [cleanup] Successfully started [rke-log-cleaner] container on host [master03] INFO[0041] [remove/rke-log-cleaner] Successfully removed container on host [master02] INFO[0041] Removing container [rke-log-cleaner] on host [worker01], try #1 INFO[0041] [remove/rke-log-cleaner] Successfully removed container on host [master01] INFO[0041] [remove/rke-log-cleaner] Successfully removed container on host [worker02] INFO[0041] [remove/rke-log-cleaner] Successfully removed container on host [worker01] INFO[0041] [remove/rke-log-cleaner] Successfully removed container on host [worker03] INFO[0041] Removing container [rke-log-cleaner] on host [master03], try #1 INFO[0041] [remove/rke-log-cleaner] Successfully removed container on host [master03] INFO[0041] [sync] Syncing nodes Labels and Taints INFO[0041] [sync] Successfully synced nodes Labels and Taints INFO[0041] [network] Setting up network plugin: canal INFO[0041] [addons] Saving ConfigMap for addon rke-network-plugin to Kubernetes INFO[0041] [addons] Successfully saved ConfigMap for addon rke-network-plugin to Kubernetes INFO[0041] [addons] Executing deploy job rke-network-plugin INFO[0041] [addons] Setting up coredns INFO[0041] [addons] Saving ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0041] [addons] Successfully saved ConfigMap for addon rke-coredns-addon to Kubernetes INFO[0041] [addons] Executing deploy job rke-coredns-addon INFO[0041] [addons] CoreDNS deployed successfully INFO[0041] [dns] DNS provider coredns deployed successfully INFO[0041] [addons] Setting up Metrics Server INFO[0041] [addons] Saving ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0041] [addons] Successfully saved ConfigMap for addon rke-metrics-addon to Kubernetes INFO[0041] [addons] Executing deploy job rke-metrics-addon INFO[0041] [addons] Metrics Server deployed successfully INFO[0041] [ingress] Setting up nginx ingress controller INFO[0041] [ingress] removing admission batch jobs if they exist INFO[0042] [addons] Saving ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0042] [addons] Successfully saved ConfigMap for addon rke-ingress-controller to Kubernetes INFO[0042] [addons] Executing deploy job rke-ingress-controller INFO[0042] [ingress] removing default backend service and deployment if they exist INFO[0042] [ingress] ingress controller nginx deployed successfully INFO[0042] [addons] Setting up user addons INFO[0042] [addons] no user addons defined INFO[0042] Finished building Kubernetes cluster successfully admin@master01:~$
Proses upgrade versi dari node kubernetesnya, node kubernetes akan mengalami upgrade secara bertahap untuk meminimalisir downtime
admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.20.8 master02 Ready controlplane,etcd,worker 67d v1.20.8 master03 Ready controlplane,etcd,worker 67d v1.20.8 worker01 Ready worker 67d v1.20.8 worker02 Ready worker 67d v1.20.8 worker03 Ready worker 67d v1.20.8 admin@master01:~$ admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.28.12 master02 Ready controlplane,etcd,worker 67d v1.28.12 master03 Ready controlplane,etcd,worker 67d v1.28.12 worker01 Ready worker 67d v1.28.12 worker02 Ready,SchedulingDisabled worker 67d v1.20.8 worker03 Ready worker 67d v1.20.8 admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.28.12 master02 Ready controlplane,etcd,worker 67d v1.28.12 master03 Ready controlplane,etcd,worker 67d v1.28.12 worker01 Ready worker 67d v1.28.12 worker02 Ready,SchedulingDisabled worker 67d v1.28.12 worker03 Ready worker 67d v1.20.8 admin@master01:~$ admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.28.12 master02 Ready controlplane,etcd,worker 67d v1.28.12 master03 Ready controlplane,etcd,worker 67d v1.28.12 worker01 Ready worker 67d v1.28.12 worker02 Ready worker 67d v1.28.12 worker03 Ready,SchedulingDisabled worker 67d v1.20.8 admin@master01:~$ admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.28.12 master02 Ready controlplane,etcd,worker 67d v1.28.12 master03 Ready controlplane,etcd,worker 67d v1.28.12 worker01 Ready worker 67d v1.28.12 worker02 Ready worker 67d v1.28.12 worker03 Ready,SchedulingDisabled worker 67d v1.20.8 admin@master01:~$ admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.28.12 master02 Ready controlplane,etcd,worker 67d v1.28.12 master03 Ready controlplane,etcd,worker 67d v1.28.12 worker01 Ready worker 67d v1.28.12 worker02 Ready worker 67d v1.28.12 worker03 NotReady,SchedulingDisabled worker 67d v1.28.12 admin@master01:~$ admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.28.12 master02 Ready controlplane,etcd,worker 67d v1.28.12 master03 Ready controlplane,etcd,worker 67d v1.28.12 worker01 Ready worker 67d v1.28.12 worker02 Ready worker 67d v1.28.12 worker03 NotReady,SchedulingDisabled worker 67d v1.28.12 admin@master01:~$ admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.28.12 master02 Ready controlplane,etcd,worker 67d v1.28.12 master03 Ready controlplane,etcd,worker 67d v1.28.12 worker01 Ready worker 67d v1.28.12 worker02 Ready worker 67d v1.28.12 worker03 NotReady,SchedulingDisabled worker 67d v1.28.12 admin@master01:~$ admin@master01:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready controlplane,etcd,worker 67d v1.28.12 master02 Ready controlplane,etcd,worker 67d v1.28.12 master03 Ready controlplane,etcd,worker 67d v1.28.12 worker01 Ready worker 67d v1.28.12 worker02 Ready worker 67d v1.28.12 worker03 Ready worker 67d v1.28.12 admin@master01:~$