In this chapter you’ll learn how to use the Kubernetes code generators in Go projects to write custom resources in a natural way. Code generators are used a lot in the implementation of native Kubernetes resources, and we’ll use the very same generators here.
Go is a simple language by design. It lacks higher-level or even metaprogramming-like mechanisms to express algorithms on different data types in a generic (i.e., type-independent) way. The “Go way” is to use external code generation instead.
Very early in the Kubernetes development process, more and more code had to be rewritten as more resources were added to the system. Code generation made the maintenance of this code much easier. Very early on, the Gengo library was created, and eventually, based on Gengo, k8s.io/code-generator was developed as an externally usable collection of generators. We will use these generators in the following sections for CRs.
Usually, the code generators are called in mostly the same way in every controller project. Only packages, group names, and API versions differ. Calling the script k8s.io/code-generator/generate-groups.sh or a bash script like hack/update-codegen.sh is the easiest way to add code generation to CR Go types from the build system (see the book’s GitHub repository).
Note that some projects call the generator binaries directly due to very special requirements and often historic reasons. For the use case of building a controller for CRs, it is much easier to just call the generate-groups.sh script from the k8s.io/code-generator repository:
$
vendor/k8s.io/code-generator/generate-groups.sh allgithub.com/programming-kubernetes/cnat/cnat-client-go/pkg/generated github.com/programming-kubernetes/cnat/cnat-client-go/pkg/apis
cnat:v1alpha1
--output-base
"
${
GOPATH
}
/src"
--go-header-file
"hack/boilerplate.go.txt"
Here, all
means to call all four standard code generators for CRs:
deepcopy-gen
Generates func
(t *T)
DeepCopy()
*T
and func
(t *T)
DeepCopyInto(*T)
methods.
client-gen
Creates typed client sets.
informer-gen
Creates informers for CRs that offer an event-based interface to react to changes of CRs on the server.
lister-gen
Creates listers for CRs that offer a read-only caching layer for GET
and LIST
requests.
The last two are the basis for building controllers (see “Controllers and Operators”). These four code generators make up a powerful basis for building full-featured, production-ready controllers using the same mechanisms and packages that the Kubernetes upstream controllers are using.
There are some more generators in k8s.io/code-generator, mostly for other contexts. For example, if you build your own aggregated API server (see Chapter 8), you will work with internal types in addition to versioned types, and you have to define defaulting functions. Then these two generators, which you can access by calling the generate-internal-groups.sh script from k8s.io/code-generator, will become relevant:
Now let’s look in detail at the parameters to generate-groups.sh
:
The second parameter is the target package name for the generated clients, listers, and informers.
The third parameter is the base package for the API group.
The fourth parameter is a space-separated list of API groups with their versions.
--output-base
is passed as a flag to all generators to define the base directory where the given packages are found.
--go-header-file
enables us to put copyright headers into generated code.
Some generators, like deepcopy-gen
, create files directly inside the API group packages. Those files follow a standard naming scheme with a zz_generated. prefix such that it is easy to exclude them from version control systems (e.g., via .gitignore file), though most projects decide to check generated files in because the Go tooling around code generators is not well developed.1
If the project follows the pattern of k8s.io/sample-controller—the sample-controller
is a blueprint project replicating the patterns established by the many controllers built in Kubernetes itself—then the code generation starts with:
$
hack/update-codegen.sh
The cnat
example in the sample-controller+client-go
variant in “Following sample-controller” goes this route.
Usually, in addition to the hack/update-codegen.sh
script, there is a second script called hack/verify-codegen.sh
.
This script calls the hack/update-codegen.sh
script and checks whether anything changed, and then it terminates with a nonzero return code if any of the generated files is not up-to-date.
This is very helpful in a continuous integration (CI) script: if a developer modified the files by accident or if the files are just outdated, CI will notice and complain.
While some of the code-generator behavior is controlled via command-line flags as described earlier (especially the packages to process), a lot more properties are controlled via tags in your Go files. A tag is a specially formatted Go comment in the following form:
// +some-tag
// +some-other-tag=value
There are two kind of tags:
Global tags above the package
line in a file called doc.go
Local tags above a type declaration (e.g., above a struct definition)
Depending on the tags in question, the position of the comment might be important.
Global tags are written into a package’s doc.go. A typical pkg/apis/group
/version
/doc.go file looks like this:
// +k8s:deepcopy-gen=package
// Package v1 is the v1alpha1 version of the API.
// +groupName=cnat.programming-kubernetes.info
package
v1alpha1
The first line of this file tells deepcopy-gen
to create deep-copy methods by default for every type in that package. If you have types where deep copy is not necessary, not desired, or even not possible, you can opt out for them with the local tag // +k8s:deepcopy-gen=false
. If you do not enable package-wide deep copy, you have to opt in to deep copy for each desired type via // +k8s:deepcopy-gen=true
.
The second tag, // +groupName=example.com
, defines the fully qualified API group name. This tag is necessary if the Go parent package name does not match the group name.
The file shown here actually comes from the cnat client-go
example pkg/apis/cnat/v1alpha1/doc.go file (see “Following sample-controller”). There, cnat
is the parent package, but cnat.programming-kubernetes.info
is the group name.
With the // +groupName
tag, the client generator (see “Typed client created via client-gen”) will generate a client using the correct HTTP path /apis/foo.project.example.com. Besides +groupName
there is also +groupGoName
, which defines a custom Go identifier (for variable and type names) to be used instead of the parent package name. For example, the generators will use the uppercase parent package name for identifies by default, which in our example is Cnat
. A better identifier would be CNAt
for “Cloud Native At.” With // +groupGoName=CNAt
we could use that instead of Cnat
(though we don’t do that in this example—we’ve stayed with Cnat
), and the client-gen
result would look like the following:
type
Interface
interface
{
Discovery
()
discovery
.
DiscoveryInterface
CNatV1
()
atv1alpha1
.
CNatV1alpha1Interface
}
Local tags are written either directly above an API type or in the second comment block above it. Here are the main types in the types.go file of the cnat
example:
// AtSpec defines the desired state of At
type
AtSpec
struct
{
// Schedule is the desired time the command is supposed to be executed.
// Note: the format used here is UTC time https://www.utctime.net
Schedule
string
`json:"schedule,omitempty"`
// Command is the desired command (executed in a Bash shell) to be executed.
Command
string
`json:"command,omitempty"`
// Important: Run "make" to regenerate code after modifying this file
}
// AtStatus defines the observed state of At
type
AtStatus
struct
{
// Phase represents the state of the schedule: until the command is executed
// it is PENDING, afterwards it is DONE.
Phase
string
`json:"phase,omitempty"`
// Important: Run "make" to regenerate code after modifying this file
}
// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// At runs a command at a given schedule.
type
At
struct
{
metav1
.
TypeMeta
`json:",inline"`
metav1
.
ObjectMeta
`json:"metadata,omitempty"`
Spec
AtSpec
`json:"spec,omitempty"`
Status
AtStatus
`json:"status,omitempty"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// AtList contains a list of At
type
AtList
struct
{
metav1
.
TypeMeta
`json:",inline"`
metav1
.
ListMeta
`json:"metadata,omitempty"`
Items
[]
At
`json:"items"`
}
In the following sections we’ll walk through the tags of this example.
In this example, the API documentation is in the first comment block, while we put the tags into the second comment block. This helps to keep the tags out of the API documentation, if you use some tool to extract the Go doc comments for that purpose.
Deep-copy method generation is usually enabled for all types by default via the global // +k8s:deepcopy-gen=package
tag (see “Global Tags”)—that is, with possible opt-out. However, in the preceding example file (and actually the whole package), all API types need deep-copy methods. Hence, we don’t have to opt out locally.
If we had a helper struct in the API types package (this is usually discouraged to keep API packages clean), we would have to disable deep-copy generation. For example:
// +k8s:deepcopy-gen=false
//
// Helper is a helper struct, not an API type.
type
Helper
struct
{
...
}
There is a special deep-copy tag that needs more explanation:
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
In “Kubernetes Objects in Go” we saw that runtime.Object
s have to implement the DeepCopyObject() runtime.Object
method. The reason is that generic code within Kubernetes has to be able to create deep copies of objects. This method allows that.
The DeepCopyObject()
method does nothing other than calling the generated DeepCopy
method. The signature of the latter varies from type to type (DeepCopy()
*T
depends on T
). The signature of the former is always DeepCopyObject()
runtime.Object
:
func
(
in
*
T
)
DeepCopyObject
()
runtime
.
Object
{
if
c
:=
in
.
DeepCopy
();
c
!=
nil
{
return
c
}
else
{
return
nil
}
}
Put the local tag //
+k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
above your top-level API types to generate this method with deepcopy-gen
. This tells deepcopy-gen
to create such a method for runtime.Object
, called DeepCopyObject()
.
In the previous example, both At
and AtList
are top-level types because they are used as runtime.Object
s.
As a rule of thumb, top-level types are those that have metav1.TypeMeta
embedded.
It happens that other interfaces need a way to be deep-copied. This is usually the case if, for example, API types have a field of interface type Foo
:
type
SomeAPIType
struct
{
Foo
Foo
`json:"foo"`
}
As we have seen, API types must be deep-copyable, and hence the field Foo
must be deep-copied too. How could you do that in a generic way (without type-casts) without adding DeepCopyFoo() Foo
to the Foo
interface?
type
Foo
interface
{
...
DeepCopyFoo
()
Foo
}
In that case the same tag can be used:
// +k8s:deepcopy-gen:interfaces=<package>.Foo
type
FooImplementation
struct
{
...
}
There are a few examples beyond runtime.Object
in the Kubernetes source where this tag is actually used:
// +k8s:deepcopy-gen:interfaces=.../pkg/registry/rbac/reconciliation.RuleOwner
// +k8s:deepcopy-gen:interfaces=.../pkg/registry/rbac/reconciliation.RoleBinding
Finally, there are a number of tags to control client-gen
, one of which we saw in the earlier example for At
and AtList
:
// +genclient
It tells client-gen
to create a client for this type (this is always opt-in). Note that you don’t have to and indeed must not put it above the List
type of the API objects.
In our cnat
example, we use the /status subresource and update the status of the CRs with the UpdateStatus
method of the client (see “Status subresource”). There are instances of CRs without a status or without a spec-status split. In those cases, the following tag avoids the generation of that UpdateStatus()
method:
// +genclient:noStatus
Without this tag, client-gen
will blindly generate the UpdateStatus()
method. It is important to understand, however, that the spec-status split works only if the /status subresource is actually enabled in the CustomResourceDefinition manifest (see “Subresources”).
The existence of the method alone in the client has no effect. Requests to it without the change in the manifest will even fail.
The client generator has to choose the right HTTP path, either with or without a namespace. For cluster-wide resources, you have to use the tag:
// +genclient:nonNamespaced
The default is to generate a namespaced client. Again, this has to match the scope setting in the CRD manifest. For special-purpose clients, you might also want to control in detail which HTTP methods are offered. You can do this by using a couple of tags, for example:
// +genclient:noVerbs
// +genclient:onlyVerbs=create,delete
// +genclient:skipVerbs=get,list,create,update,patch,delete,watch
// +genclient:method=Create,verb=create,
// result=k8s.io/apimachinery/pkg/apis/meta/v1.Status
The first three should be pretty self-explanatory, but the last one warrants some explanation.
The type this tag is written above will be create-only and will not return the API type itself, but a metav1.Status
. For CRs this does not make much sense, but for user-provided API servers written in Go (see Chapter 8) those resources can exist, and they do in practice.
One common case for the // +genclient:method=
tag is the addition of a method to scale a resource. In “Scale subresource” we describe how the /scale subresource can be enabled for CRs. The following tags create the corresponding client methods:
// +genclient:method=GetScale,verb=get,subresource=scale,
// result=k8s.io/api/autoscaling/v1.Scale
// +genclient:method=UpdateScale,verb=update,subresource=scale,
// input=k8s.io/api/autoscaling/v1.Scale,result=k8s.io/api/autoscaling/v1.Scale
The first tag creates the getter GetScale
. The second creates the setter UpdateScale
.
All CR /scale subresources receive and output the Scale
type from the autoscaling/v1 group. In the Kubernetes API there are resources that use other types for historic reasons.
Both informer-gen
and lister-gen
process the // +genclient
tag of client-gen
. There is nothing else to configure. Each type that opted in to client generation gets informers and listers automatically that match the client (if the whole suite of generators is called via the k8s.io/code-generator/generate-group.sh script).
The documentation of the Kubernetes generators has a lot of room for improvement and will certainly be refined slowly over time. For more information about the different generators, it is often helpful to look at examples inside Kubernetes itself—for example, k8s.io/api and OpenShift API types. Both repositories have many advanced use cases.
Moreover, don’t hesitate to look into the generators themselves. deepcopy-gen
has some documentation available inside its main.go file. client-gen
has some documentation available in the Kubernetes contributor documentation. informer-gen
and lister-gen
currently don’t have further documentation, but generate-groups.sh shows how each is invoked.
1 The Go tools do not run the generation automatically when needed and lack a way to define dependencies between source and generated files.