The public cloud platforms such as AWS, GCE, and Azure are great and provide many benefits, but they are not cheap. Many large organizations have invested a lot in their own data centers. Other organizations work with private service providers such as OVS, Rackspace, or Digital Ocean. If you have the operational capacity to manage and operate infrastructure on your own it makes a lot of economic sense to run your Kubernetes cluster on your infrastructure rather than in the cloud. But what if some of your workloads fluctuate and for a relatively short amount of time require a lot more capacity?

For example, your system maybe hit especially hard on the weekends or maybe during holidays. The traditional approach is to just provision extra capacity. But in many dynamic situations, it is not easy. With capacity overflow, you can run the bulk of your work in a Kubernetes cluster running on an on-premise data center or with a private service provider and have a secondary cloud-based Kubernetes cluster running on one of the big platform providers. Most of the time, the cloud-based cluster will be shut down (stopped instances), but when the need arises you can elastically add capacity to your system by starting some stopped instances. Kubernetes cluster federation can make this configuration relatively straightforward. It eliminates a lot of headaches about capacity planning and paying for hardware that's not used most of the time.

This approach is sometimes called Cloud bursting.

This is almost the opposite of capacity overflow. Maybe you've embraced the cloud native lifestyle and your entire system runs on the cloud, but some data or workloads deal with sensitive information. Regulatory compliance or your organization's security policies may dictate that those data and workloads must run in an environment that's fully controlled by you. Your sensitive data and workloads may be subject to external auditing. It may be critical to ensure no information ever leaks from the private Kubernetes cluster to the cloud-based Kubernetes cluster. But it may be desirable to have visibility into the public cluster and the ability to launch non-sensitive workloads from the private cluster to the cloud-based cluster. If the nature of a workload can change dynamically from non-sensitive to sensitive then it needs to be addressed by coming up with a proper policy and implementation. For example, you may prevent workloads from changing their nature. Alternatively, you may migrate a workload that suddenly became sensitive and ensure that it doesn't run on the cloud-based cluster anymore. Another important instance is national compliance, where certain data is required by law to remain and be accessed only from a designated geographical region (typically a country). In this case, a cluster must be created in that geographical region.

Large organizations often prefer to have options and not be tied to a single provider. The risk is often too great, because the provider may shut down or be unable to provide the same level of service. Having multiple providers is often good for negotiating prices, too. Kubernetes is designed to be vendor-agnostic. You can run it on different cloud platforms, private service providers, and on-premise data centers.

However, this is not trivial. If you want to be sure that you are able to switch providers quickly or shift some workloads from one provider to the next, you should already be running your system on multiple providers. You can do it yourself or there are some companies that provide the service of running Kubernetes transparently on multiple providers. Since different providers run different data centers, you automatically get some redundancy and protection from vendor-wide outages.

High availability means that a service will remain available to users even when some parts of the system fail. In the context of a federated Kubernetes cluster, the scope of failure is an entire cluster, which is typically due to problems with the physical data center hosting the cluster, or perhaps a wider issue with the platform provider. The key to high-availability is redundancy. Geo-distributed redundancy means having multiple clusters running in different locations. It may be different availability zones of the same cloud provider, different regions of the same cloud provider, or even different cloud providers altogether (see the Avowing vendor lock-in section). There are many issues to address when it comes to running a cluster federation with redundancy. We'll discuss some of these issues later. Assuming that the technical and organizational issues have been resolved, high availability will allow the switching of traffic from a failed cluster to another cluster. This should be transparent to the users up to a point (delay during switchover and some in-flight requests or tasks may disappear or fail). The system administrators may need to take extra steps to support the switchover and to deal with the original cluster failure.

The federation API server is managing the Kubernetes clusters that together comprise the federation. It manages the federation state (which clusters are part of the federation) in an etcd database the same as a regular Kubernetes cluster, but the state it keeps is just which clusters are members of the federation. The state of each cluster is stored in the etcd database of that cluster. The main purpose of the federation API server job is to interact with the federation controller manager and route requests to the federation member clusters. The federation members don't need to know they are part of a federation: they just work the same.

The following diagram demonstrates the relationships between the federation API server, the federation replication controllers, and the Kubernetes clusters in the federation:

The federation controller manager makes sure the federation's desired state matches the actual state. It forwards any necessary changes to the relevant cluster or clusters. The federated controller manager binary contains multiple controllers for all the different federated resources we'll cover later in the chapter. The control logic is similar, though: observes changes and brings cluster state to the desired state when they deviate. This is done for each member in the cluster federation.

The following diagram demonstrates this perpetual control loop:

Federated ConfigMaps are very useful because they help centralize the configuration of applications that may be spread across multiple clusters.

> kubectl --context=federation-cluster create -f configmap.yaml

> kubectl --context=cluster-1 get configmap configmap.yaml

> kubectl --context=federation-cluster delete configmap

> kubectl --context=cluster-1 delete configmap
> kubectl --context=cluster-2 delete configmap
> kubectl --context=cluster-3 delete configmap

A federated DaemonSet is pretty much the same as a regular Kubernetes DaemonSet. You create it and interact with it via the control plane, and the control plane propagates it to all the member clusters. At the end of the day, you can be sure that your Daemons run on every node in every cluster of the federation.

Federated deployments are a little smarter. When you create a federated deployment with X replicas and you have N clusters, the replicas will be distributed evenly between the clusters by default. If you have three clusters and the federated deployment has 15 pods, then each cluster will run five replicas. As other federated resources, the control plane will store the federated deployment with 15 replicas and then create three deployments (one for each cluster) with five replicas each. You can control the number of replicas per cluster by adding an annotation: federation.kubernetes.io/deployment-preferences. Federated deployment is still in alpha as of Kubernetes 1.5. In the future, the annotation will become a proper field in the federated deployment configuration.

Federated events are different than the other federated resources. They are only stored in the control plane and are not propagated to the underlying Kubernetes member clusters.

You can query the federation events with --context=federation-cluster as usual:

> kubectl --context=federation-cluster get events

The federated ingress does more than just create matching ingress objects in each cluster. One of the main features of federated ingress is that if a whole cluster goes down it can direct traffic to other clusters. As of Kubernetes 1.4, federated ingress is supported on Google Cloud Platform, both on GKE and GCE. In the future, hybrid cloud support for federated ingress will be added.

The federated ingress performs the following roles:

> kubectl --context=federation-cluster create -f ingress.yaml

> kubectl --context=cluster-1 get ingress myingress
NAME        HOSTS     ADDRESS           PORTS     AGE
ingress      *         157.231.15.33    80, 443   1m

Kubernetes namespaces are used within a cluster to isolate independent areas and support multi-tenant deployments. Federated namespaces provide the same capabilities across a cluster federation. The API is identical. When a client is accessing the federation control plane, they will only get access to the namespaces they requested and are authorized to access across all the clusters in the federation.

You use the same commands and add --context=federation-cluster:

> kubectl --context=federation-cluster create -f namespace.yaml
> kubectl --context=cluster-1 get namespaces namespace
> kubectl --context=federation-cluster create -f namespace.yaml

It is best to use deployments and federated deployments to manage the replicas in your cluster or federation. However, if for some reason you prefer to work directly with ReplicaSets, then Kubernetes supports a federated ReplicaSet. There is no federated replication controller because ReplicaSets supersede replication controllers.

When you create a federated ReplicaSets, the job of the control plane is to ensure that the number of replicas across the cluster matches your federated ReplicaSets configuration. The control plane will create a regular ReplicaSet in each federation member. Each cluster will get, by default, an equal (or as close as possible) number of replicas so that the total will add up to the specified number of replicas.

You can control the number of replicas per cluster by specifying using the following annotation: federation.kubernetes.io/replica-set-preferences.

The corresponding data structure is as follows:

type FederatedReplicaSetPreferences struct {
  Rebalance bool
  Clusters map[string]ClusterReplicaSetPreferences
}

If Rebalance is true, then running replicas may be moved between clusters as necessary. The clusters map determines the ReplicaSets preferences per cluster. If * is specified as the key, then all unspecified clusters will use that set of preferences. If there is no * entry, then replicas will only run on clusters that show up in the map. Clusters that belong to the federation but don't have an entry will not have pods scheduled (for that pod template).

The individual ReplicaSets preferences per cluster are specified using the following data structure:

type ClusterReplicaSetPreferences struct {
  MinReplicas int64
  MaxReplicas *int64
  Weight int64
}

MinReplicas is 0 by default. MaxReplicas is unbounded by default. Weight expresses the preference to add an additional replica to this ReplicaSets and defaults to 0.

Federated secrets are simple. When you create a federated secret as usual through the control plane it gets propagated to the whole cluster. That's it.

The unit of work in a Kubernetes cluster is the pod. You can't break a pod in Kubernetes. The entire pod will always be deployed together and be subject to the same lifecycle treatment. Should the pod remain the unit of work for a cluster federation? Maybe it makes more sense to be able to associate a bigger unit, such as a whole ReplicaSet deployment, or service with a specific cluster. If the cluster fails, the entire ReplicaSet deployment, or service is scheduled to a different cluster. How about a collection of tightly coupled ReplicaSets? The answers to these questions are not always easy and may even change dynamically as the system evolves.

Location affinity is a major concern. When can pods be distributed across clusters? What are the relationships between those pods? Are there any requirements for affinity between pods or pods and other resources, such as storage? There are several major categories:

When designing the system and how to allocate and schedule services and pods across the federation it's important to make sure the location affinity requirements are always respected.

Cross-cluster scheduling goes hand-in-hand with location affinity. When a new pod is created or an existing pod fails and a replacement needs to be scheduled, where should it go? The current cluster federation doesn't handle all the scenarios and options for location affinity we mentioned earlier. At this point, cluster federation handles the loosely-coupled (including weighted distribution) and strictly-coupled (by making sure the number of replicas matches the number of clusters) categories well. Anything else will require that you don't use cluster federation. You'll have to add your own custom federation layer that takes more specialized concerns into account and can accommodate more intricate scheduling use cases.

This is a tough problem. If you have a lot of data and pods running in multiple clusters (possibly on different continents) and need to access it quickly, then you have several unpleasant options:

There is currently no support for federated auto-calling. There are two dimensions of scaling that can be utilized, as well as a combination:

Consider the relatively simple scenario of a loosely coupled application running on three clusters with five pods in each cluster. At some point, 15 pods can't handle the load anymore. We need to add more capacity. We can increase the number of pods per cluster, but if we do it at the federation level than we will have six pods running in each cluster. We've increased the federation capacity by three pods, when only one pod is needed. Of course, if you have more clusters the problem gets worse. Another option is to pick a cluster and just change its capacity. This is possible with annotations, but now we're explicitly managing capacity across the federation. It can get complicated very quickly if we have lots of clusters running hundreds of services with dynamically changing requirements.

Adding a whole new cluster is even more complicated. Where should we add the new cluster? There is no requirement for extra availability that can guide the decision. It is just about extra capacity. Creating a new cluster also often requires complicated first time setup that may take days to approve various quotas on public cloud platforms. The hybrid approach increases the capacity of existing clusters in the federation until reaching some threshold and then starts adding new clusters. The benefit of this approach is that when you're getting closer to capacity limit per cluster you start preparing new clusters that will be ready to go when necessary. Other than that, it requires a lot effort and you pay in increased complexity for the flexibility and scalability.