An admission control plugin must implement the admission.Interface interface (yes, it's a little confusing that the interface name is Interface):

type Interface interface {
  Admit(a Attributes) (err error)
  Handles(operation Operation) bool
}

The interface is pretty simple. The Admit() function accepts an Attributes interface and, based on those Attributes, make a decision if the request should be admitted or not. If it returns nil, the request id is admitted. Otherwise it is rejected.

The Handles() function returns the operations that the admission control plugin handles. If an admission controller doesn't support an operation, it is considered admitted (for this plugin).

The whole workflow of going through the chain of registered admission control plugins and determining if an operation is admitted is just a few lines:

func (admissionHandler chainAdmissionHandler) Admit(a Attributes) error {
    for _, handler := range admissionHandler {
        if !handler.Handles(a.GetOperation()) {
            continue
        }
        err := handler.Admit(a)
        if err != nil {
            return err
        }
    }
    return nil
}

Let's look at the simplest example – the alwaysDeny admission control plugin. It is designed for testing and will reject any request. You can find it here:

https://github.com/kubernetes/kubernetes/tree/master/plugin/pkg/admission/deny.

The Admit() function always returns a non-nil result and Handles() always returns true, so it handles every operation and Admit() rejects it:

type alwaysDeny struct{}

func (alwaysDeny) Admit(a admission.Attributes) (err error) {
  return admission.NewForbidden(a, errors.New("Admission control is denying all modifications"))
}

func (alwaysDeny) Handles(operation admission.Operation) bool {
  return true
}
// NewAlwaysDeny creates an always deny admission handler
func NewAlwaysDeny() admission.Interface {
  return new(alwaysDeny)
}

Every admission control plugin has its own init() function, which is called when the plugin is imported. In this method you should register your plugin, so it's available. Here is the init() function of the AlwaysDeny admission control plugin:

func init() {
    admission.RegisterPlugin(
        "AlwaysDeny", 
        func(config io.Reader) (admission.Interface, error) {
            return NewAlwaysDeny(), nil
        })
}

It just calls the RegisterPlugin() function of the admission package, passing the name of the plugin and a factory function that accepts a configuration reader and returns a plugin instance.

Go supports only static plugins. Every custom plugin must be linked into the API server executable in order to be imported and registered. The key file is here:

https://github.com/kubernetes/kubernetes/tree/master/cmd/kube-apiserver/app/plugins.go.

Here is part of the file. When you add your plugin, it will be imported later, which will invoke its init() function to register the plugin:

package app

import (
  // Cloud providers
  _ "k8s.io/kubernetes/pkg/cloudprovider/providers"

  // Admission policies
  _ "k8s.io/kubernetes/plugin/pkg/admission/admit"
  _ "k8s.io/kubernetes/plugin/pkg/admission/alwayspullimages"
    ...
  _ "k8s.io/kubernetes/plugin/pkg/admission/serviceaccount"
  )

Another critical file is the build file at:

https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-api-server/app/BUILD.

Here is a snippet that shows some admission plugins:

go_library(
    name = "go_default_library",
    srcs = [
        "plugins.go",
        "server.go",
    ],
    tags = ["automanaged"],
    deps = [
    .
    .
    .
         "//plugin/pkg/admission/admit:go_default_library",
      "//plugin/pkg/admission/deny:go_default_library",
"//plugin/pkg/admission/exec:go_default_library",
  "//plugin/pkg/admission/gc:go_default_library",
    .
    .
    .

You must add a line for your admission control plugin.

The pods to be scaled must have cAdvisor-specific custom (aka application) metrics endpoint configured. The configuration format is described here. Kubernetes expects the configuration to be placed in definition.json mounted via a config map in /etc/custom-metrics. A sample ConfigMap may look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-config
data:
  definition.json: "{"endpoint" : "http://localhost:8080/metrics"}"

Due to the way cAdvisor currently works, localhost refers to the node itself, not to the running pod. Thus, the appropriate container in the pod must ask for a node port:

    ports:
    - hostPort: 8080
      containerPort: 8080

Horizontal pod auto-scaling using custom metrics is configured via an annotation. The value in the annotation is interpreted as a target metric value averaged over all running pods:

    annotations:
      alpha/target.custom-metrics.podautoscaler.kubernetes.io: '{"items":[{"name":"qps", "value": "10"}]}'

In this case, if there are four pods running and each of them reports the qps metric to be equal to 15, HPA will start two additional pods, so there will be six pods in total. If there are multiple metrics passed in the annotation or the CPU is configured as well, then HPA will use the biggest number of replicas that come from the calculations.

Even if the target CPU utilization is not specified, a default of 80% will be used. To calculate the number of desired replicas based only on custom metrics, the CPU utilization target should be set to a very large value (for example, 100,000%). Then CPU-related logic will want only one replica, leaving the decision about a higher replica count to custom metrics (and min/max limits).

Volume plugins are complicated entities. If you need to implement a new volume plugin you'll have to dig in deeper as there are many details to get right. We'll just go over the pieces and interfaces here. The main interfaces are defined here:

https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/plugins.go.

Here is the bare-bones VolumePlugin interface (represent non-persistent volume):

type VolumePlugin interface {
  Init(host VolumeHost) error
  GetPluginName() string
  GetVolumeName(spec *Spec) (string, error)
  CanSupport(spec *Spec) bool
  RequiresRemount() bool
  NewMounter(spec *Spec, 
              podRef *v1.Pod, 
              opts VolumeOptions) (Mounter, error)
  NewUnmounter(name string, 
                podUID types.UID) (Unmounter, error)
  ConstructVolumeSpec(volumeName, 
                       mountPath string) (*Spec, error)
  SupportsMountOption() bool
  SupportsBulkVolumeVerification() bool
}

The various interface functions accept or return several other interfaces and data types such as Spec, Mounter and Unmounter.

Here is the Spec, which is an internal representation of an API Volume:

type Spec struct {
  Volume           *v1.Volume
  PersistentVolume *v1.PersistentVolume
  ReadOnly         bool
}

There are several other interfaces that derive from the VolumePlugin and bestow some extra properties on the volumes they represent. Here is a list of the available interfaces:

Registering a Kubelet volume plugin is a little different again. It is done by calling ProbeVolumePlugins() on each plugin, which returns a list of plugins that are appended together. Here is a snippet:

func ProbeVolumePlugins(pluginDir string) []volume.VolumePlugin {
  allPlugins := []volume.VolumePlugin{}

  allPlugins = append(allPlugins, 
                       aws_ebs.ProbeVolumePlugins()...)
  allPlugins = append(allPlugins, 
                       empty_dir.ProbeVolumePlugins()...)
  allPlugins = append(allPlugins, 
                       gce_pd.ProbeVolumePlugins()...)
.
.
.
  return allPlugins
}

Check out the complete source code here:

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/plugins.go.

Here is an example of the probeVolumePlugins() function of the aws_elb volume plugin:

func ProbeVolumePlugins() []volume.VolumePlugin {
  return []volume.VolumePlugin{&awsElasticBlockStorePlugin{nil}}
}

In general, multiple plugins may be returned, and not just one.

A custom volume plugin must be linked into the Kubelet executable. You must add a line for your custom volume plugin in the deps section to the build file at https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/BUILD.

Here is a snippet from the file that shows other volume plugins:

go_library(
    name = "go_default_library",
    srcs = [
        "auth.go",
        "bootstrap.go",
        "plugins.go",
        "server.go",
        "server_linux.go",
    ],
    tags = ["automanaged"],
    deps = [
        "//cmd/kubelet/app/options:go_default_library",
        "//pkg/api:go_default_library",
    .
    .
    .
        "//pkg/volume:go_default_library",
        "//pkg/volume/aws_ebs:go_default_library",
        "//pkg/volume/azure_dd:go_default_library",
    .
    .
    .