Unable to add new master/etcd node to cluster

[adoo123]

Current:

master1
master2
master3

[etcd]
master1
master2
master3

[kube-node]
node1
node2
node3

Added node3 to group etcd:

master1
master2
master3
master4
master5

[etcd]
master1
master2
master3
master4
master5

[kube-node]
node1
node2
node3

Now i have 3 master/etcd and 45 nodes, i've already refrenced #1122 but couldn't fix it.I extended etcd success but master failed.It shows "kubectl" error:

Unable to connect to the server: x509: certificate is valid for "new master ip"

And my extend command is：

ansible-playbook -i inventory/mycluster/host.ini cluster.yml -l master1,master2,master3,master4,master5

My kubernetes cluster version is 1.9.3,how to fix it?

[ykfq]

The feature of scaling master nodes seems imperfect, but it's possible to scale etcd cluster seperatly, to do so, just add etcd nodes under [etcd] and rerun cluster.yml.

[juliohm1978]

I'm facing the same issue with adding new masters. I'm using Kubespray v2.10.x and the reason it fails is that Kubespray does not update the apiserver certificates to add the new master to the SAN list.

You can check your certificate with

openssl x509 -text -noout -in /etc/kubernetes/ssl/apiserver.crt

... and the new master IP and hostname should be listed in the Subject Alternative Name section.

X509v3 Subject Alternative Name: 
                DNS:infra00-lab, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, DNS:infra00-lab, DNS:lb-apiserver.kubernetes.local, IP Address:10.233.0.1, IP Address:172.31.134.110, IP Address:172.31.134.110, IP Address:10.233.0.1, IP Address:127.0.0.1, IP Address:172.31.134.110

The execution of cluster.yml adds the new master IP and hostname to /etc/kubernetes/kubeadm-config.yaml as expected. It seems, however, that Kubespray is not calling kubeadm to replace the certificate before trying to join the new master node. We fixed this by using kubeadm manually to recreate the certificate.

NOTE: The works for v2.10.x. I never tested this in older versions of Kubespray.

In your first master, recreate the apiserver certificate.

cd /etc/kubernetes/ssl
mv apiserver.crt apiserver.crt.old
mv apiserver.key apiserver.key.old

cd /etc/kubernetes
kubeadm init phase certs apiserver --config kubeadm-config.yaml

If you are doing this after you ended up with a broken master, be sure to run reset.yml using the parameter --limit=<broken_master_hostname> before continuing. If you take the precaution of recreating the certificate before adding the new master node, you won't need this.

Run cluster.yml to include the new master node. You should end up with a working cluster.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ppcololo]

is it possible to add master? or replace failed master for new one?

[juliohm1978]

You should be able to. In the past, we managed to replace all nodes in the cluster: master, etcd and workers. But.... there are some misteps you need to be carefull along the way. After a lot of experiments and retries in our lab environment, we came up with a few guidelines.

Adding/replacing a master node

1) Recreate apiserver certs manually to include the new master node in the cert SAN field.

For some reason, Kubespray will not update the apiserver certificate.

Edit /etc/kubernetes/kubeadm-config.yaml, include new host in certSANs list.

Use kubeadm to recreate the certs.

cd /etc/kubernetes/ssl
mv apiserver.crt apiserver.crt.old
mv apiserver.key apiserver.key.old

cd /etc/kubernetes
kubeadm init phase certs apiserver --config kubeadm-config.yaml

Check the certificate, new host needs to be there.

openssl x509 -text -noout -in /etc/kubernetes/ssl/apiserver.crt

2) Run cluster.yml

Add the new host to the inventory and run cluster.yml.

3) Restart kube-system/nginx-proxy

In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserver. Kubespray will update its static config, but it needs to be restarted in order to reload.

# run in every host
docker ps | grep k8s_nginx-proxy_nginx-proxy | awk '{print $1}' | xargs docker restart

4) Remove old master nodes

If you are replacing a node, remove the old one from the inventory, and remove from the cluster runtime.

kubectl drain --force --ignore-daemonsets --grace-period 300 --timeout 360s --delete-local-data NODE_NAME

kubectl delete node NODE_NAME

After that, the old node can be safely shutdown. Also, make sure to restart nginx-proxy in all remaining nodes (step 3)

From any active master that remains in the cluster, re-upload kubeadm-config.yaml

kubeadm config upload from-file --config /etc/kubernetes/kubeadm-config.yaml

Adding/replacing a worker node

This should be the easiest.

1) Add new node to the inventory.

2) Run upgrade-cluster.yml

You can use --limit=node1 to limit Kubespray to avoid disturbing other nodes in the cluster.

3) Drain the node that will be removed

kubectl drain --force=true --grace-period=10 --ignore-daemonsets=true --timeout=0s --delete-local-data NODE_NAME

4) Run the remove-node.yml playbook

With the old node still in the inventory, run remove-node.yml. You need to pass -e node=NODE_NAME to the playbook to limit the execution to the node being removed.

5) Remove the node from the inventory

That's it.

Adding/Replacing an etcd node

You need to make sure there are always an odd number of etcd nodes in the cluster. In such a way, this is always a replace or scale up operation. Either add two new nodes or remove an old one.

1) Add the new node running cluster.yml.

Update the inventory and run cluster.yml passing --limit=etcd,kube-master -e ignore_assert_errors=yes.

Run upgrade-cluster.yml also passing --limit=etcd,kube-master -e ignore_assert_errors=yes. This is necessary to update all etcd configuration in the cluster.

At this point, you will have an even number of nodes. Everything should still be working, and you should only have problems if the cluster decides to elect a new etcd leader before you remove a node. Even so, running applications should continue to be available.

2) Remove an old etcd node

With the node still in the inventory, run remove-node.yml passing -e node=NODE_NAME as the name of the node that should be removed.

3) Make sure the remaining etcd members have their config updated

In each etcd host that remains in the cluster:

cat /etc/etcd.env | grep ETCD_INITIAL_CLUSTER

Only active etcd members should be in that list.

4) Remove old etcd members from the cluster runtime

Acquire a shell prompt into one of the etcd containers and use etcdctl to remove the old member.

# list all members
etcdctl member list 

# remove old member
etcdctl member remove MEMBER_ID

# careful!!! if you remove a wrong member you will be in trouble

# note: these command lines are actually much bigger, since you need to pass all certificates to etcdctl.

5) Make sure the apiserver config is correctly updated.

In every master node, edit /etc/kubernetes/manifests/kube-apiserver.yaml. Make sure only active etcd nodes are still present in the apiserver command line parameter --etcd-servers=....

6) Shutdown the old instance

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

Details

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[yujunz]

Can https://github.com/kubernetes-sigs/kubespray/blob/48a182844c9c3438e36c78cbc4518c962e0a9ab2/docs/recover-control-plane.md be applied for adding new master/etcd nodes? @qvicksilver

[qvicksilver]

@yujunz Not sure, haven't really tried that use case. Also I'm a bit unsure of the state of that playbook. Haven't had time to add it to CI. But please do try.