As we've seen already, Flannel supports two types of overlay network. Using either UDP or VXLAN encapsulation, Flannel can build an overlay network between Docker hosts. The obvious advantage to this is that you can provision networks across disparate Docker nodes without having to touch the physical underlay network. However, some types of overlay networks also introduce a significant performance penalty, especially for processes that perform encapsulation in user space. Host gateway mode aims to solve that problem by not using an overlay network. This, however, comes with its own limitations. In this recipe, we'll review what host gateway mode can provide as well as show how to configure it.
In this recipe, we'll be slightly modifying the lab we've been using up until this point. The lab topology will look like this:
In this case, the hosts docker3 and docker4 now have IP addresses on the same subnet as docker1 and docker2. That is, all of the hosts are now layer 2 adjacent to each other and can talk directly without the need to route through a gateway. Once you have your hosts reconfigured in this topology, we'll want to wipe the Flannel configuration. To do that, perform these steps:
- On the host running the
etcdservice:sudo systemctl stop etcd sudo rm -rf /var/lib/etcd/default sudo systemctl start etcd
- On all of the hosts running the Flannel service:
sudo systemctl stop flanneld sudo ip link delete flannel.1 sudo systemctl --no-block start flanneld
Note
You'll note that we passed the systemctl command with the --no-block parameter when we started flanneld. Since we deleted the Flannel configuration from etcd, the Flannel service is searching for configuration to use for initialization. Due to the way the service is defined (as type notify), passing this parameter is required to prevent the command from hanging on the CLI.
At this point, your Flannel nodes will be searching for their configuration. Since we deleted the etcd data store, the key that tells the Flannel nodes how to configure the service is currently missing, and the Flannel service will continue to poll the etcd host until we make the appropriate configuration. We can verify this by checking the logs on one of the hosts:
user@docker4:~$ journalctl -f -u flanneld -- Logs begin at Wed 2016-10-12 12:39:35 CDT. – Oct 12 12:39:36 docker4 flanneld[873]: I1012 12:39:36.843784 00873 manager.go:163] Using 10.10.10.104 as external interface Oct 12 12:39:36 docker4 flanneld[873]: I1012 12:39:36.844160 00873 manager.go:164] Using 10.10.10.104 as external endpoint Oct 12 12:41:22 docker4 flanneld[873]: E1012 12:41:22.102872 00873 network.go:106] failed to retrieve network config: 100: Key not found (/coreos.com) [4] Oct 12 12:41:23 docker4 flanneld[873]: E1012 12:41:23.104904 00873 network.go:106] failed to retrieve network config: 100: Key not found (/coreos.com) [4]
It's important to note that at this point Flannel has already decided what its external endpoint IP address will be by seeing which interface supports the default route for the host:
user@docker4:~$ ip route
default via 10.10.10.1 dev eth0
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.104
user@docker4:~$Since that happens to be eth0, Flannel picks that interface's IP address as its external address. To configure host gateway mode, we can put the following configuration into etcd:
{
"Network":"10.100.0.0/16",
"Backend":{
"Type":"host-gw"
}
}As we've seen before, we still specify a network. The only difference is that we supply a type of host-gw. The command to insert this into etcd looks like this:
user@docker1:~$ etcdctl set /coreos.com/network/config
'{"Network":"10.100.0.0/16", "Backend": {"Type": "host-gw"}}'After we insert this configuration, the Flannel nodes should all pick up the new configuration. Let's examine the service logs for Flannel on the host docker4 to verify this:
user@docker4:~$ journalctl -r -u flanneld -- Logs begin at Wed 2016-10-12 12:39:35 CDT, end at Wed 2016-10-12 12:55:38 CDT. -- Oct 12 12:55:06 docker4 flanneld[873]: I1012 12:55:06.797289 00873 network.go:83] Subnet added: 10.100.23.0/24 via 10.10.10.103 Oct 12 12:55:06 docker4 flanneld[873]: I1012 12:55:06.796982 00873 network.go:83] Subnet added: 10.100.20.0/24 via 10.10.10.101 Oct 12 12:55:06 docker4 flanneld[873]: I1012 12:55:06.796468 00873 network.go:83] Subnet added: 10.100.43.0/24 via 10.10.10.102 Oct 12 12:55:06 docker4 flanneld[873]: I1012 12:55:06.785464 00873 network.go:51] Watching for new subnet leases Oct 12 12:55:06 docker4 flanneld[873]: I1012 12:55:06.784436 00873 manager.go:246] Lease acquired: 10.100.3.0/24 Oct 12 12:55:06 docker4 flanneld[873]: I1012 12:55:06.779349 00873 local_manager.go:179] Picking subnet in range 10.100.1.0 ... 10.100.255.0
Note
The journalctl command is useful for seeing all the logs related to a service being managed by systemd. In the preceding example, we passed the -r parameter to show the logs in reverse order ( the most current on top). We also passed the -u parameter to specify which service we want to see the logs for.
The oldest log entry we see is this host's Flannel service picking and registering a scope within the 10.100.0.0/16 subnet. This works in the same manner as it did with both the UDP and the VXLAN backend. The next three log entries show Flannel detecting the registrations of the other three Flannel nodes scopes. Since etcd is tracking each Flannel node's external IP address, as well as their registered scope, all of the Flannel hosts now know what external IP address can be used to reach each registered Flannel scope. In overlay mode (UDP or VXLAN), this external IP address was used as the destination for encapsulated traffic. In host gateway mode, this external IP address is used as route destination. If we inspect the routing table, we can see a route entry for each host:
user@docker4:~$ ip route default via 10.10.10.1 dev eth0 onlink 10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.104 10.100.20.0/24 via 10.10.10.101 dev eth0 10.100.23.0/24 via 10.10.10.103 dev eth0 10.100.43.0/24 via 10.10.10.102 dev eth0 user@docker4:~$
In this configuration, Flannel simply relies on basic routing to provide reachability to all the Flannel registered scopes. In this case, the host docker4 has routes to all the other Docker hosts in order to reach their Flannel network scope:
Not only is this far less complex than dealing with overlay networks, but it can also be much more performant than requiring each host to do encapsulation for the overlay network. The downside of this approach is that each host needs to have an interface on the same network in order for this to work. If the hosts are not on the same network, Flannel cannot add these routes because it would require the upstream network device (the host's default gateway) to also have routing information about how to reach the remote host. And while the Flannel node can point a static route at its default gateway, the physical network has no knowledge of the 10.100.0.0/16 network and will fail to deliver the traffic. The net effect is that host gateway mode limits the location of where you can place Flannel-enabled Docker hosts.
Finally, it's important to point out that Flannel may have changed state after the Docker service was already running. If that's the case, you'll want to restart Docker to make sure that it picks up the new variables from Flannel. If you rebooted your hosts when you reconfigured their network interfaces, you probably just need to start the Docker service. The service likely failed to load when the system booted because of the lack of Flannel configuration information which should now be present.