We will keep building off the lab in the previous chapter where we configured the host gateway backend. However, the lab topology is going to revert to the previous configuration with Docker hosts docker3 and docker4 being in the 192.168.50.0/24 subnet:

Once you have your hosts configured in this topology, we'll want to wipe out the Flannel configuration. To do that, perform these steps:

In some cases, the changes we make may require you to have root-level access to the system.

The previous recipes showed several examples of how to specify an overall Flannel network or global scope as well change the backend network type. We also saw some backend network types allowed for additional configuration options. In addition to the options we've seen, we can also configure other parameters globally that dictate how Flannel works overall. There are three other main parameters that can influence the scopes assigned to the Flannel nodes:

Using these options in combination with the Network flag gives us quite a bit of flexibility when we assign networks. For instance, let's use this configuration:

{  
   "Network":"10.100.0.0/16",
   "SubnetLen":25,
   "SubnetMin":"10.100.0.0",
   "SubnetMax":"10.100.1.0",
   "Backend":{  
      "Type":"host-gw"
   }
}

This defines that each Flannel node should get a scope allocation of a /25, the first subnet should start at 10.100.0.0, and the last subnet should end at 10.100.1.0. You might have noticed that, in this case, we only have space for three subnets within that allocation:

This was done intentionally to show what happens when Flannel runs out of space in the global scope. Let's now put this configuration in etcd using this command:

user@docker1:~$ etcdctl set /coreos.com/network/config 
 '{"Network":"10.100.0.0/16","SubnetLen": 25, "SubnetMin": "10.100.0.0", "SubnetMax": "10.100.1.0", "Backend": {"Type": "host-gw"}}'

Once in place, you should see that the majority of the hosts receive local scope allocations. However, if we check our hosts, we'll see that one has failed to receive an allocation. In my case, that's the host docker4. We can see this within the Flannel services logs:

user@docker4:~$ journalctl -r -u flanneld
-- Logs begin at Wed 2016-10-12 12:39:35 CDT, end at Wed 2016-10-12 13:17:42 CDT. --
Oct 12 13:17:42 docker4 flanneld[1422]: E1012 13:17:42.650086 01422 network.go:106] failed to register network: failed to acquire lease: out of subnets
Oct 12 13:17:42 docker4 flanneld[1422]: I1012 13:17:42.649604 01422 local_manager.go:179] Picking subnet in range 10.100.0.0 ... 10.100.1.0

Since we only allowed space for three allocations in the global scope, the fourth host is unable to receive a local scope and will continue to request one until one becomes available. This could be remedied by updating the SubnetMax parameter to 10.100.1.128 and restarting the Flannel service on the host that failed to receive a local scope allocation.

As I mentioned, there are also configuration parameters that we can pass to the Flannel service on each host.

From a network perspective, perhaps the most valuable configuration option would be the --iface parameter, which allows you to specify which host interface you wish to use as Flannel's external endpoint. To view the importance of this, let's look at a quick example of our multihost lab topology:

If you recall, in host gateway mode Flannel requires that all the Flannel nodes be layer 2 adjacent or on the same network. In this case, there are two hosts on the 10.10.10.0/24 network on the left and two hosts on the 192.168.50.0/24 network on the right. In order to talk to each other, they need to route through the multilayer switch. A scenario like this typically calls for an overlay backend mode that would tunnel the container traffic across the multilayer switch. However, if host gateway mode is a requirement for performance or other reasons, you might still be able to use it if you can provide additional interfaces to your hosts. For instance, imagine that these hosts were really virtual machines, and it was relatively easy for us to provision another interface on each host, call it eth1:

This interface could be dedicated to Flannel traffic allowing each host to still be layer 2 adjacent for the sake of Flannel traffic while still maintaining their existing default route through eth0. However, just provisioning the interface is not enough. Recall that Flannel by default picks its external endpoint interface by referencing the default route of the host. Since the default route is unchanged in this model, Flannel will be unable to add the appropriate routes:

user@docker4:~$ journalctl -ru flanneld
-- Logs begin at Wed 2016-10-12 14:24:51 CDT, end at Wed 2016-10-12 14:31:14 CDT. --
Oct 12 14:31:14 docker4 flanneld[1491]: E1012 14:31:14.463106 01491 network.go:116] Error adding route to 10.100.1.128/25 via 10.10.10.102: network is unreachable
Oct 12 14:31:14 docker4 flanneld[1491]: I1012 14:31:14.462801 01491 network.go:83] Subnet added: 10.100.1.128/25 via 10.10.10.102
Oct 12 14:31:14 docker4 flanneld[1491]: E1012 14:31:14.462589 01491 network.go:116] Error adding route to 10.100.0.128/25 via 10.10.10.101: network is unreachable
Oct 12 14:31:14 docker4 flanneld[1491]: I1012 14:31:14.462008 01491 network.go:83] Subnet added: 10.100.0.128/25 via 10.10.10.101

Since Flannel is still using the eth0 interface as its external endpoint IP address, it knows that the hosts on the other subnet aren't directly reachable. We can fix this by telling Flannel to use the eth1 interface by passing the --iface option to the Flannel service.

For instance, we can change the Flannel configuration by updating the Flannel service definition (/lib/systemd/system/flanneld.service) to look like this:

[Unit]
Description=Flannel Network Fabric
Documentation=https://github.com/coreos/flannel
Before=docker.service
After=etcd.service

[Service]
Environment= 'DAEMON_ARGS=--etcd-endpoints=http://10.10.10.101:2379 --iface=eth1'
Type=notify
ExecStart=/usr/bin/flanneld $DAEMON_ARGS
Restart=always
RestartSec=10s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

With this configuration, Flannel will use the eth1 interface for its external endpoint, allowing all the hosts to communicate directly across the 10.11.12.0/24 network. You can then load the new configuration by reloading the systemd configuration and restarting the service on all hosts:

sudo systemctl daemon-reload
sudo systemctl restart flanneld

Keep in mind that Flannel uses the external endpoint IP address to keep track of Flannel nodes. Changing this means that Flannel will allocate a new scope to each Flannel node. It's best to configure these options before joining a Flannel node. In our case, since etcd is already configured, we'll want to once again delete the existing etcd configuration and reconfigure it in order for the scope to become available:

user@docker1:~$ sudo systemctl stop etcd
user@docker1:~$ sudo rm -rf /var/lib/etcd/default
user@docker1:~$ sudo systemctl start etcd
user@docker1:~$ etcdctl set /coreos.com/network/config 
 '{"Network":"10.100.0.0/16","SubnetLen": 25, "SubnetMin": "10.100.0.0", "SubnetMax": "10.100.1.128", "Backend": {"Type": "host-gw"}}'

If you check a host, you should now see that it has three Flannel routes—one for each assigned scope of the other three hosts:

user@docker1:~$ ip route
default via 10.10.10.1 dev eth0 onlink
10.10.10.0/24 dev eth0  proto kernel  scope link src 10.10.10.101
10.11.12.0/24 dev eth1  proto kernel  scope link src 10.11.12.101
10.100.0.0/25 via 10.11.12.102 dev eth1
10.100.1.0/25 via 10.11.12.104 dev eth1
10.100.1.128/25 via 10.11.12.103 dev eth1
10.100.0.128/25 dev docker0  proto kernel  scope link src 10.100.75.1 
user@docker1:~$

In addition, if you'll be using Flannel through NAT, you might also want to look at the --public-ip option, which allows you to define a node's public IP address. This is particularly relevant in cloud environments where the server's real IP address may be hidden behind NAT.