Clustering etcd

etcd stores network information and states in Kubernetes. Any data loss could be crucial. Clustering is strongly recommended in etcd. etcd comes with support for clustering; a cluster of N members can tolerate up to (N-1)/2 failures. There are three mechanisms for creating an etcd cluster. They are as follows:

  • Static
  • etcd discovery
  • DNS discovery

In this recipe, we will discuss how to bootstrap an etcd cluster by static and etcd discovery.

Getting ready

Before you start building an etcd cluster, you have to decide how many members you need. How big the etcd cluster should be really depends on the environment you want to create. In the production environment, at least three members are recommended. Then, the cluster can tolerate at least one permanent failure. In this recipe, we will use three members as an example of the development environment:

Name/Hostname

IP address

ip-172-31-0-1

172.31.0.1

ip-172-31-0-2

172.31.0.2

ip-172-31-0-3

172.31.0.3

How to do it…

A static mechanism is the easiest way to set up a cluster. However, the IP address of every member should be known beforehand. It means that if you bootstrap an etcd cluster in some cloud provider environment, the static mechanism might not be so practical. Therefore, etcd also provides a discovery mechanism to bootstrap itself from the existing cluster.

Static

With a static mechanism, you have to know the address information of each member:

Parameters

Meaning

-name

The name of this member

-initial-advertise-peer-urls

Used to peer with other members, should be the same as the one listing in -initial-cluster

-listen-peer-urls

The URL to accept peer traffic

-listen-client-urls

The URL to accept client traffic

-advertise-client-urls

etcd member used to advertise to other members

-initial-cluster-token

A unique token for distinguishing different clusters

-initial-cluster

Advertised peer URLs of all the members

-initial-cluster-state

Specifies the state of the initial cluster

Use the etcd command-line tool to bootstrap a cluster with additional parameters on each member:

// on the host ip-172-31-0-1, running etcd command to make it peer with ip-172-31-0-2 and ip-172-31-0-3, advertise and listen other members via port 2379, and accept peer traffic via port 2380
# etcd -name ip-172-31-0-1 
       -initial-advertise-peer-urls http://172.31.0.1:2380 
       -listen-peer-urls http://172.31.0.1:2380 
       -listen-client-urls http://0.0.0.0:2379 
       -advertise-client-urls http://172.31.0.1:2379 
       -initial-cluster-token mytoken 
       -initial-cluster ip-172-31-0-1=http://172.31.0.1:2380,ip-172-31-0-2=http://172.31.0.2:2380,ip-172-31-0-3=http://172.31.0.3:2380 
       -initial-cluster-state new
...
2016-05-01 18:57:26.539787 I | etcdserver: starting member e980eb6ff82d4d42 in cluster 8e620b738845cd7
2016-05-01 18:57:26.551610 I | etcdserver: starting server... [version: 2.2.5, cluster version: to_be_decided]
2016-05-01 18:57:26.553100 N | etcdserver: added member 705d980456f91652 [http://172.31.0.3:2380] to cluster 8e620b738845cd7
2016-05-01 18:57:26.553192 N | etcdserver: added member 74627c91d7ab4b54 [http://172.31.0.2:2380] to cluster 8e620b738845cd7
2016-05-01 18:57:26.553271 N | etcdserver: added local member e980eb6ff82d4d42 [http://172.31.0.1:2380] to cluster 8e620b738845cd7
2016-05-01 18:57:26.553349 E | rafthttp: failed to dial 705d980456f91652 on stream MsgApp v2 (dial tcp 172.31.0.3:2380: getsockopt: connection refused)
2016-05-01 18:57:26.553392 E | rafthttp: failed to dial 705d980456f91652 on stream Message (dial tcp 172.31.0.3:2380: getsockopt: connection refused)
2016-05-01 18:57:26.553424 E | rafthttp: failed to dial 74627c91d7ab4b54 on stream Message (dial tcp 172.31.0.2:2380: getsockopt: connection refused)
2016-05-01 18:57:26.553450 E | rafthttp: failed to dial 74627c91d7ab4b54 on stream MsgApp v2 (dial tcp 172.31.0.2:2380: getsockopt: connection refused)

The etcd daemon on ip-172-31-0-1 will then start checking whether all the members are online. The logs show connection refused since ip-172-31-0-2 and ip-172-31-0-3 are still offline. Let's go to the next member and run the etcd command:

// on the host ip-172-31-0-2, running etcd command to make it peer with ip-172-31-0-1 and ip-172-31-0-3, advertise and listen other members via port 2379, and accept peer traffic via port 2380
# etcd -name ip-172-31-0-2 
       -initial-advertise-peer-urls http://172.31.0.2:2380 
       -listen-peer-urls http://172.31.0.2:2380 
       -listen-client-urls http://0.0.0.0:2379 
       -advertise-client-urls http://172.31.0.2:2379 
       -initial-cluster-token mytoken 
       -initial-cluster ip-172-31-0-1=http://172.31.0.1:2380,ip-172-31-0-2=http://172.31.0.2:2380, ip-172-31-0-3=http://172.31.0.3:2380 -initial-cluster-state new
...
2016-05-01 22:59:55.696357 I | etcdserver: starting member 74627c91d7ab4b54 in cluster 8e620b738845cd7
2016-05-01 22:59:55.696397 I | raft: 74627c91d7ab4b54 became follower at term 0
2016-05-01 22:59:55.696407 I | raft: newRaft 74627c91d7ab4b54 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-05-01 22:59:55.696411 I | raft: 74627c91d7ab4b54 became follower at term 1
2016-05-01 22:59:55.706552 I | etcdserver: starting server... [version: 2.2.5, cluster version: to_be_decided]
2016-05-01 22:59:55.707627 E | rafthttp: failed to dial 705d980456f91652 on stream MsgApp v2 (dial tcp 172.31.0.3:2380: getsockopt: connection refused)
2016-05-01 22:59:55.707690 N | etcdserver: added member 705d980456f91652 [http://172.31.0.3:2380] to cluster 8e620b738845cd7
2016-05-01 22:59:55.707754 N | etcdserver: added local member 74627c91d7ab4b54 [http://172.31.0.2:2380] to cluster 8e620b738845cd7
2016-05-01 22:59:55.707820 N | etcdserver: added member e980eb6ff82d4d42 [http://172.31.0.1:2380] to cluster 8e620b738845cd7
2016-05-01 22:59:55.707873 E | rafthttp: failed to dial 705d980456f91652 on stream Message (dial tcp 172.31.0.3:2380: getsockopt: connection refused)
2016-05-01 22:59:55.708433 I | rafthttp: the connection with e980eb6ff82d4d42 became active
2016-05-01 22:59:56.196750 I | raft: 74627c91d7ab4b54 is starting a new election at term 1
2016-05-01 22:59:56.196903 I | raft: 74627c91d7ab4b54 became candidate at term 2
2016-05-01 22:59:56.196946 I | raft: 74627c91d7ab4b54 received vote from 74627c91d7ab4b54 at term 2
2016-05-01 22:59:56.949201 I | raft: raft.node: 74627c91d7ab4b54 elected leader e980eb6ff82d4d42 at term 112
2016-05-01 22:59:56.961883 I | etcdserver: published {Name:ip-172-31-0-2 ClientURLs:[http://10.0.0.2:2379]} to cluster 8e620b738845cd7
2016-05-01 22:59:56.966981 N | etcdserver: set the initial cluster version to 2.1

After starting member 2, we can see that the current cluster version is 2.1. The following error message shows the connection to peer 705d980456f91652 is unhealthy. By observing the log, we can find that member 705d980456f91652 is pointing to http://172.31.0.3:2380. Let's start up the last member ip-172-31-0-3:

# etcd -name ip-172-31-0-3 
       -initial-advertise-peer-urls http://172.31.0.3:2380 
       -listen-peer-urls http://172.31.0.3:2380 
       -listen-client-urls http://0.0.0.0:2379 
       -advertise-client-urls http://172.31.0.3:2379 
       -initial-cluster-token mytoken 
       -initial-cluster ip-172-31-0-1=http://172.31.0.1:2380,ip-172-31-0-2=http://172.31.0.2:2380, ip-172-31-0-3=http://172.31.0.3:2380 -initial-cluster-state new
2016-05-01 19:02:19.106540 I | etcdserver: starting member 705d980456f91652 in cluster 8e620b738845cd7
2016-05-01 19:02:19.106590 I | raft: 705d980456f91652 became follower at term 0
2016-05-01 19:02:19.106608 I | raft: newRaft 705d980456f91652 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2016-05-01 19:02:19.106615 I | raft: 705d980456f91652 became follower at term 1
2016-05-01 19:02:19.118330 I | etcdserver: starting server... [version: 2.2.5, cluster version: to_be_decided]
2016-05-01 19:02:19.120729 N | etcdserver: added local member 705d980456f91652 [http://10.0.0.75:2380] to cluster 8e620b738845cd7
2016-05-01 19:02:19.120816 N | etcdserver: added member 74627c91d7ab4b54 [http://10.0.0.204:2380] to cluster 8e620b738845cd7
2016-05-01 19:02:19.120887 N | etcdserver: added member e980eb6ff82d4d42 [http://10.0.0.205:2380] to cluster 8e620b738845cd7
2016-05-01 19:02:19.121566 I | rafthttp: the connection with 74627c91d7ab4b54 became active
2016-05-01 19:02:19.121690 I | rafthttp: the connection with e980eb6ff82d4d42 became active
2016-05-01 19:02:19.143351 I | raft: 705d980456f91652 [term: 1] received a MsgHeartbeat message with higher term from e980eb6ff82d4d42 [term: 112]
2016-05-01 19:02:19.143380 I | raft: 705d980456f91652 became follower at term 112
2016-05-01 19:02:19.143403 I | raft: raft.node: 705d980456f91652 elected leader e980eb6ff82d4d42 at term 112
2016-05-01 19:02:19.146582 N | etcdserver: set the initial cluster version to 2.1
2016-05-01 19:02:19.151353 I | etcdserver: published {Name:ip-172-31-0-3 ClientURLs:[http://10.0.0.75:2379]} to cluster 8e620b738845cd7
2016-05-01 19:02:22.022578 N | etcdserver: updated the cluster version from 2.1 to 2.2

We can see, on member 3, we successfully initiated the etcd cluster without any errors and the current cluster version is 2.2. How about member 1 now?

2016-05-01 19:02:19.118910 I | rafthttp: the connection with 705d980456f91652 became active
2016-05-01 19:02:22.014958 I | etcdserver: updating the cluster version from 2.1 to 2.2
2016-05-01 19:02:22.018530 N | etcdserver: updated the cluster version from 2.1 to 2.2

With member 2 and 3 online, member 1 can now get connected and go online too. When observing the log, we can see the leader election took place in the etcd cluster:

ip-172-31-0-1: raft: raft.node: e980eb6ff82d4d42 (ip-172-31-0-1) elected leader e980eb6ff82d4d42 (ip-172-31-0-1) at term 112
ip-172-31-0-2: raft: raft.node: 74627c91d7ab4b54 (ip-172-31-0-2) elected leader e980eb6ff82d4d42 (ip-172-31-0-1) at term 112
ip-172-31-0-3: 2016-05-01 19:02:19.143380 I | raft: 705d980456f91652 became follower at term 112

The etcd cluster will send the heartbeat to the members in the cluster to check the health status. Note that when you need to add or remove any members in the cluster, the preceding etcd command needs to be rerun on all the members in order to notify that there are new members joining the cluster. In this way, all the members in the cluster are aware of all the online members; if one node goes offline, the other members will poll the failure member until it refreshes members by the etcd command. If we set a message from a member, we could get the same message from the other members too. If one member becomes unhealthy, the other members in the etcd cluster will still be in service and elect for a new leader.

etcd discovery

Before using the etcd discovery, you should have a discovery URL that is used to bootstrap a cluster. If you want to add or remove a member, you should use the etcdctl command as the runtime reconfiguration. The command line is pretty much the same as the static mechanism. What we need to do is change –initial-cluster to –discovery, which is used to specify the discovery service URL. We could use the etcd discovery service (https://discovery.etcd.io) to request a discovery URL:

// get size=3 cluster url from etcd discovery service
# curl -w "
" 'https://discovery.etcd.io/new?size=3'
https://discovery.etcd.io/be7c1938bbde83358d8ae978895908bd
// Init a cluster via requested URL
# etcd -name ip-172-31-0-1 -initial-advertise-peer-urls http://172.31.43.209:2380 
 -listen-peer-urls http://172.31.0.1:2380 
 -listen-client-urls http://0.0.0.0:2379 
 -advertise-client-urls http://172.31.0.1:2379 
 -discovery https://discovery.etcd.io/be7c1938bbde83358d8ae978895908bd
...
2016-05-02 00:28:08.545651 I | etcdmain: listening for peers on http://172.31.0.1:2380
2016-05-02 00:28:08.545756 I | etcdmain: listening for client requests on http://127.0.0.1:2379
2016-05-02 00:28:08.545807 I | etcdmain: listening for client requests on http://172.31.0.1:2379
2016-05-02 00:28:09.199987 N | discovery: found self e980eb6ff82d4d42 in the cluster
2016-05-02 00:28:09.200010 N | discovery: found 1 peer(s), waiting for 2 more

The first member has joined the cluster; wait for the other two peers. Let's start etcd in the second node:

# etcd -name ip-172-31-0-2 -initial-advertise-peer-urls http://172.31.0.2:2380 
 -listen-peer-urls http://172.31.0.2:2380 
 -listen-client-urls http://0.0.0.0:2379 
 -advertise-client-urls http://172.31.0.2:2379 
 -discovery https://discovery.etcd.io/be7c1938bbde83358d8ae978895908bd
...
2016-05-02 00:30:12.919005 I | etcdmain: listening for peers on http://172.31.0.2:2380
2016-05-02 00:30:12.919074 I | etcdmain: listening for client requests on http://0.0.0.0:2379
2016-05-02 00:30:13.018160 N | discovery: found self 25fc8075ab1ed17e in the cluster
2016-05-02 00:30:13.018235 N | discovery: found 1 peer(s), waiting for 2 more
2016-05-02 00:30:22.985300 N | discovery: found peer e980eb6ff82d4d42 in the cluster
2016-05-02 00:30:22.985396 N | discovery: found 2 peer(s), waiting for 1 more

We know there are two members in etcd already and they are waiting for the last one to join. The following code starts the last node:

# etcd -name ip-172-31-0-3 -initial-advertise-peer-urls http://172.31.0.3:2380 
 -listen-peer-urls http://172.31.0.3:2380 
 -listen-client-urls http://0.0.0.0:2379 
 -advertise-client-urls http://172.31.0.3:2379 
 -discovery https://discovery.etcd.io/be7c1938bbde83358d8ae978895908bd

After new nodes join, we can check from the logs that there is a new election taking place:

2016-05-02 00:31:01.152215 I | raft: e980eb6ff82d4d42 is starting a new election at term 308
2016-05-02 00:31:01.152272 I | raft: e980eb6ff82d4d42 became candidate at term 309
2016-05-02 00:31:01.152281 I | raft: e980eb6ff82d4d42 received vote from e980eb6ff82d4d42 at term 309
2016-05-02 00:31:01.152292 I | raft: e980eb6ff82d4d42 [logterm: 304, index: 9739] sent vote request to 705d980456f91652 at term 309
2016-05-02 00:31:01.152302 I | raft: e980eb6ff82d4d42 [logterm: 304, index: 9739] sent vote request to 74627c91d7ab4b54 at term 309
2016-05-02 00:31:01.162742 I | rafthttp: the connection with 74627c91d7ab4b54 became active
2016-05-02 00:31:01.197820 I | raft: e980eb6ff82d4d42 received vote from 74627c91d7ab4b54 at term 309
2016-05-02 00:31:01.197852 I | raft: e980eb6ff82d4d42 [q:2] has received 2 votes and 0 vote rejections
2016-05-02 00:31:01.197882 I | raft: e980eb6ff82d4d42 became leader at term 309

With the discovery method, we can see that the cluster can be launched without knowing the others' IPs beforehand. etcd will start a new election if new nodes join or leave, and always keep the service online with the multi-nodes setting.

See also

To understand the installation of a single etcd server, refer to the Building datastore recipe in Chapter 1, Building Your Own Kubernetes.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset