According to the second requirement, all nodes must communicate with all containers. Kubernetes delegates implementation to the container network interface (CNI). Users can choose different implementations, by L2, L3, or overlay. Overlay networking, also known as packet encapsulation, is one of the most common solutions. This wraps a message before leaving the source, delivers it, and unwraps the message at its destination. This leads to a situation in which the overlay increases the network latency and complexity. As long as all the containers can access each other across nodes, you're free to use any technology, such as L2 adjacency or the L3 gateway. For more information about CNI, refer to its spec (https://github.com/containernetworking/cni/blob/master/SPEC.md):

Let's say we have a packet moving from pod 1 to pod 4. The packet leaves the container interface and reaches the veth pair, and then passes through the bridge and the node's network interface. Network implementation comes into play in step 4. As long as the packet can be routed to the target node, you are free to use any options. In the following example, we'll launch minikube with the --network-plugin=cni option. With CNI enabled, the parameters will be passed through kubelet in the node. Kubelet has a default network plugin, but you can probe any supported plugin when it starts up. Before starting minikube, you can use minikube stop first if it has been started or minikube delete to delete the whole cluster thoroughly before doing anything else. Although minikube is a single node environment that might not completely represent the production scenario we'll encounter, this just gives you a basic idea of how all of this works. We will learn about the deployment of networking options in the real world in Chapter 9, Continuous Delivery, and Chapter 10, Kubernetes on AWS:

// start minikube with cni option
# minikube start --network-plugin=cni
...
Loading cached images from config file.
Everything looks great. Please enjoy minikube!

When we specify the network-plugin option, minikube will use the directory specified in --network-plugin-dir for plugins on startup. In the CNI plugin, the default plugin directory is /opt/cni/net.d. After the cluster comes up, we can log into the node and look at the network interface configuration inside the minikube via minikube ssh:

# minikube ssh
$ ifconfig 
...
mybridge  Link encap:Ethernet  HWaddr 0A:58:0A:01:00:01
          inet addr:10.1.0.1  Bcast:0.0.0.0  
          Mask:255.255.0.0
...

We will find that there is one new bridge in the node, and if we create the example pod again by using 5-1-1_pod.yml, we will find that the IP address of the pod becomes 10.1.0.x, which is attaching to mybridge instead of docker0:

# kubectl create -f 6-1-1_pod.yaml
pod/example created
# kubectl describe po example
Name:       example
Namespace:  default
Node:       minikube/10.0.2.15
Start Time: Sun, 23 Jul 2017 14:24:24 -0400
Labels:           <none>
Annotations:      <none>
Status:           Running
IP:         10.1.0.4  

This is because we have specified that we'll use CNI as the network plugin, and docker0 will not be used (also known as the container network model, or libnetwork). The CNI creates a virtual interface, attaches it to the underlay network, sets the IP address, and eventually routes and maps it to the pods' namespace. Let's take a look at the configuration that's located at /etc/cni/net.d/k8s.conf in minikube:

# cat /etc/cni/net.d/k8s.conf
{
  "name": "rkt.kubernetes.io",
  "type": "bridge",
  "bridge": "mybridge",
  "mtu": 1460,
  "addIf": "true",
  "isGateway": true,
  "ipMasq": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.1.0.0/16",
    "gateway": "10.1.0.1",
    "routes": [
      {
       "dst": "0.0.0.0/0"
      }
    ]
  }
}

In this example, we are using the bridge CNI plugin to reuse the L2 bridge for pod containers. If the packet is from 10.1.0.0/16, and its destination is to anywhere, it'll go through this gateway. Just like the diagram we saw earlier, we could have another node with CNI enabled with the 10.1.2.0/16 subnet so that ARP packets can go out to the physical interface on the node where the target pod is located. This then achieves pod-to-pod communication across nodes.

Let's check the rules in iptables:

// check the rules in iptables 
# sudo iptables -t nat -nL
... 
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0
CNI-25df152800e33f7b16fc085a  all  --  10.1.0.0/16          0.0.0.0/0            /* name: "rkt.kubernetes.io" id: "328287949eb4d4483a3a8035d65cc326417ae7384270844e59c2f4e963d87e18" */
CNI-f1931fed74271104c4d10006  all  --  10.1.0.0/16          0.0.0.0/0            /* name: "rkt.kubernetes.io" id: "08c562ff4d67496fdae1c08facb2766ca30533552b8bd0682630f203b18f8c0a" */

All the related rules have been switched to 10.1.0.0/16 CIDR.