Running etcd cluster in Docker Swarm

by Thomas Urban

Credits

This guide is extending example provided by etcd team on its website.

tl;dr

In this tutorial a custom overlay network is created to connect stand-alone containers with static IP addresses each running a single member of etcd cluster on another host of swarm. Overlay network is encrypted and neither node is exposing any ports to the public, thus resulting cluster is sufficiently save for production use.

Primer

Running a cluster of etcd nodes requires to set up either node with a static IP address so all nodes are capable of relying on either other node's IP address for identifying who is an eligible member of cluster and who is not. It is possible to have a private etcd cluster in a Docker Swarm your stacks of services might gain access on exclusively without risking to expose the etcd cluster to the public itself.

Create a Network

First of all, you need to create a dedicated private network in your Docker Swarm to be shared by all the nodes in etcd cluster as well as all the services meant to access this cluster.

docker create network \
    --driver overlay \
    --scope swarm \
    --opt encrypted \
    --subnet 10.100.100.0/24 \
    --attachable \
    etcd-net

This command is creating an overlay network named etcd-net which is working across all nodes in your swarm. Make sure to encrypt its traffic for security reasons by providing according option as demonstrated. The network is created to manage a particular subnet. It's okay to pick a different subnet here. Eventually, you need to enable regular containers on either node to attach to it (for the etcd cluster members will be running in regular containers, only). Services are capable of attaching to this network implicitly.

Select Configurations

Cluster Token

All nodes of etcd cluster share a common token which is meant to be some internal secret. Usually you pick a random string of sufficient size. In example given below this token is provided as value of shell variable ${TOKEN} and its the same for every container to be started.

Per-Node Addresses and Names

After picking a subnet for the overlay network connecting your etcd nodes with each other you have to pick static IP addresses for your nodes in scope of that subnet. Either container to be created below will get its own unique IP address as well as a unique name. Let's assume there are variables ${IP1} and ${NAME1} storing IP address and name of first etcd node, ${IP2} and ${NAME2} storing IP address and name of second etcd node and so on.

Cluster Size

Eventually, you have to decide how many nodes are participating in your etcd cluster. You can go with one node for development reasons. But for production setups you should pick at least three nodes to benefit from this cluster keeping its data even if one of the nodes is down.

Basically, the number of nodes does not matter except for two reasons:

  • Every decision of cluster requires a majority of members casting vote in its favour. A very high number of members results in more complex communication probably causing slightly more delay to gain consensus.
  • Split votes are just like lost votes. Thus, using an even number of nodes isn't beneficial over using one node less with regards to fail safety and stability. That's why it is suggested to always run an odd number of nodes.

Usually, sizes range from three to nine.

Cluster ID

Listing your cluster's initial member nodes is another parameter you need to provide on running either node. This way every node knows how many nodes are meant to cast a vote and how to reach them for that.

This cluster ID is a comma-separated concatenation of assignment associating either node's unique name with its unique peer URL. The latter is derived from its unique internal IP address by wrapping it with the scheme http:// and its port used to listen for incoming peer traffic which is 2380 by default.

So for example, when ${IP1} is representing your first node's IP address its related URL is http://${IP1}:2380. The resulting cluster ID is something like

${NAME1}=http://${IP1}:2380,${NAME2}=http://${IP2}:2380,...

Let's assume this information is available in variable ${CLUSTER_ID} below.

Start Cluster Nodes

At this point you should have:

  • a number of etcd nodes to be active members of your cluster
  • a secret token to be shared by all those nodes
  • a separate IP addresses and hostnames for every node
  • a cluster ID listing all the member nodes.

Important

It is essential that neither parameter is changing while running or even restarting your etcd cluster in parts or as a whole.

Since picking static IP addresses isn't supported for Docker services you need to log into every node of your swarm which is meant to host one of the etcd cluster nodes and start a container using a command according to this template:

docker run -d \
    --restart always \
    --volume etcd-data:/etcd-data \
    --name "${NAMEx}" \
    --net etcd-net \
    --ip "${IPx}" \
    gcr.io/etcd-development/etcd:v3.4.3 \
    /usr/local/bin/etcd \
    --data-dir /etcd-data --name "${NAMEx}" \
    --initial-advertise-peer-urls "http://${IPx}:2380" \
    --listen-peer-urls "http://0.0.0.0:2380" \
    --advertise-client-urls "http://${IPx}:2379" \
    --listen-client-urls "http://0.0.0.0:2379" \
    --initial-cluster "${CLUSTER_ID}" \
    --initial-cluster-state new \
    --initial-cluster-token "${TOKEN}"

On first host ${IPx} must be replaced with ${IP1} and ${NAMEx} must be replaced with ${NAME1}. On second host they are replaced with ${IP2} and ${NAME2} and so on.

Important

This guide assumes there is a different host for every container started that way. This way is most beneficial for etcd cluster's fail safety depending on underlying Docker Swarm. If you need to run multiple etcd member containers on same host you need to assign different volumes in line 3 of provided command.

Check Status

Basically, that's it. If either container was starting properly you should have a running etcd cluster which is available for use by stacks in your Docker Swarm or any other container there is. And it's private in that no external client is capable of accessing this cluster. Since using encrypted overlay network you don't even have to set up certificates for encrypting etcd traffic. Sticking with unencrypted traffic is just fine.

For monitoring cluster's state you may execute etcdctl in context of either running etcd container started before, e.g. on first container you may run commands like these:

docker exec -it ${NAME1} etcdctl endpoint health

... will show short information on whether cluster is healthy or not.

docker exec -it ${NAME1} etcdctl member list

... is listing all member nodes of etcd cluster.

docker exec -it ${NAME1} etcdctl put foo bar

... will write record named foo with value bar into cluster.

docker exec -it ${NAME2} etcdctl get foo

... should display the record named foo with value bar previously on first node. Obey this command is run on second node for illustration purposes.

docker exec -it ${NAME1} etcdctl check perf

... can be used to test your etcd cluster's performance.

Accessing the Cluster

Using Services

The following excerpt of a service definition is illustrating how to access this cluster:

version: "3.7"

services:
  your-service:
    ...
    networks:
      - etcd-net

networks:
  etcd-net:
    external: true

In this partial example a service named your-service has access on etcd members using the latter ones' container names in endpoint URLs like http://${NAME1}:2379, http://${NAME2}:2379 etc.

Using Containers

Due to creating an attachable overlay network in first step you can attach any container on any host of your swarm using a command like this one:

docker run \
    -it --rm \
    -v /some/host/folder:/backup
    --net etcd-net \
    gcr.io/etcd-development/etcd:v3.4.3 \
    etcdctl --endpoints=http://${NAME1}:2379,http://${NAME2}:2379,http://${NAME3}:2379 \
    snapshot save /backup/saved

For illustration purposes, this one-shot container is run to fetch a snapshot of your cluster e.g. for backup.

Go back