Databases controller on Humus

The Database resource

Mon, 01 Jan 0001 00:00:00 +0000

The api/v1alpha1 package declares the resource as Go values. kube reflects its CRD from them.

The types

The spec names the database, and says what deleting the Database does to it. The status names the Secret, and holds the conditions:

type DeletionPolicy string

const (
	Retain DeletionPolicy = "Retain"
	Delete DeletionPolicy = "Delete"
)

type DatabaseSpec struct {
	Name string `json:"name"`
	DeletionPolicy DeletionPolicy `json:"deletionPolicy,omitempty"`
}

type DatabaseStatus struct {
	SecretName string `json:"secretName,omitempty"`
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

The resource and its rules

Database is the kube.Resource, with a rule for each field:

Talking to PostgreSQL

Mon, 01 Jan 0001 00:00:00 +0000

The postgres package provisions and drops a Database’s role and database, through one admin connection to the server.

The admin interface

The Reconciler and its finalizer call PostgreSQL through an interface, so their tests can fake it:

type Admin interface {
	Provision(ctx context.Context, name, password string) error
	Drop(ctx context.Context, name string) error
}

Both are safe to repeat: a reconcile that fails partway is run again.

The `Server`

Server implements Admin on a pool. It traces each statement, and injects faults before each operation:

Reconciling

Mon, 01 Jan 0001 00:00:00 +0000

The controller package’s Reconciler brings a Database to its spec, whatever state it finds. A reconcile that fails is retried, with backoff, so each step is safe to repeat.

The `Reconciler`

It holds a clientset for the Secret, the Admin, and the server’s address, which the Secret hands on:

const fieldManager = "databases-controller"

type database = kube.Object[v1alpha1.DatabaseSpec, v1alpha1.DatabaseStatus]

type Reconciler struct {
	Kube kubernetes.Interface
	Admin postgres.Admin
	Host string
	Port int
}

var _ kube.Reconciler[v1alpha1.DatabaseSpec, v1alpha1.DatabaseStatus] = Reconciler{}

The steps

Reconcile reads the password, provisions the role and the database, applies the Secret, then sets Ready. An apply the API server refuses with a conflict means the Secret was deleted after it was read, so the password is rotated there and then:

Cleaning up

Mon, 01 Jan 0001 00:00:00 +0000

The database and the role are outside the cluster, so no owner reference can garbage collect them. A finalizer keeps a deleted Database until the controller has cleaned up after it.

The finalizer

Init registers the Reconciler’s Finalize under the finalizer’s name:

const (
	serviceAccount = "/var/run/secrets/kubernetes.io/serviceaccount/"
	finalizer = "postgres.example.com/finalizer"
	scenarioAnnotation = "postgres.example.com/failure-scenario"
)

kube.Finalize(finalizer, rec.Finalize),

The controller adds the finalizer to each Database. Once one is deleted, it calls Finalize, and removes the finalizer when Finalize succeeds, which lets the API server delete the Database.

Init

Mon, 01 Jan 0001 00:00:00 +0000

The settings

Everything Init reads, in one struct:

type settings struct {
	Connection kube.Connection
	Controller kube.Settings
	Admin *pgxpool.Config
	Host string
	Port int
	Probes httphealth.Settings
	TrustedPrefix string
}

config.Struct sources and checks each one. A bad one makes Init fail, and the controller exits 78, naming its variable:

func readSettings(ctx context.Context, s *humus.Setup) (settings, error) {
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }
	read := readFile(s.Host.FS)

	return config.Struct(func(r *config.Resolver) settings {
		tokenFile := r.Get(env("KUBERNETES_TOKEN_FILE").Default(serviceAccount + "token").Validate(func(name string) error {
			_, err := readToken(s.Host.FS, name)
			return err
		}))

		k := kube.DefaultSettings()
		k.Name = r.Get(env("CONTROLLER_NAME").Default("databases-controller").Validate(kube.CheckName))
		k.Namespace = r.Get(env("CONTROLLER_NAMESPACE").Default(k.Namespace).Validate(kube.CheckWatchNamespace))
		k.Workers = r.Get(env("CONTROLLER_WORKERS").Parse(strconv.Atoi).Default(k.Workers).Validate(kube.CheckWorkers))
		k.Resync = r.Get(env("CONTROLLER_RESYNC").Parse(time.ParseDuration).Default(k.Resync).Validate(kube.CheckResync))
		k.LeaderElection.Namespace = r.Get(env("POD_NAMESPACE").Validate(kube.CheckNamespace))
		k.LeaderElection.Identity = r.Get(env("POD_NAME").Validate(kube.CheckIdentity))

		return settings{
			Connection: kube.Connection{
				Server: net.JoinHostPort(
					r.Get(env("KUBERNETES_SERVICE_HOST").Validate(checkAPIServerHost)),
					r.Get(env("KUBERNETES_SERVICE_PORT").Validate(checkAPIServerPort)),
				),
				TLS: r.Get(env("KUBERNETES_CA_FILE").Default(serviceAccount + "ca.crt").Parse(read).Parse(certPool).Map(trusting).Validate(kube.CheckTLS)),
				Token: func(context.Context) (string, error) {
					return readToken(s.Host.FS, tokenFile)
				},
			},
			Controller: k,
			Admin: r.Get(env("POSTGRES_ADMIN_DSN_FILE").Parse(read).Parse(adminDSN)),
			Host: r.Get(env("POSTGRES_HOST").Validate(notEmpty)),
			Port: r.Get(env("POSTGRES_PORT").Parse(strconv.Atoi).Default(5432).Validate(checkPostgresPort)),
			Probes: httphealth.Settings{
				Addr: r.Get(env("PROBES_ADDR").Validate(httphealth.CheckAddr)),
				TLS: r.Get(env("PROBES_TLS_PEM_FILE").Parse(pemTLS(s.Host.FS)).Validate(httphealth.CheckTLS)),
			},
			TrustedPrefix: r.Get(env("FAULT_TRUSTED_NAMESPACE_PREFIX").Default("")),
		}
	}).Read(ctx)
}

Variable	What it is	Default
`KUBERNETES_SERVICE_HOST`, `KUBERNETES_SERVICE_PORT`	the API server, which the kubelet sets in every container	required
`KUBERNETES_CA_FILE`, `KUBERNETES_TOKEN_FILE`	the API server’s CA, and the service account’s token	the service account’s, mounted in the pod
`CONTROLLER_NAME`	the controller’s name, its Lease’s, and its field manager	`databases-controller`
`CONTROLLER_NAMESPACE`	the one namespace it watches	every namespace
`CONTROLLER_WORKERS`, `CONTROLLER_RESYNC`	how many `Database`s are reconciled at once, and how often every one is reconciled again	`kube.DefaultSettings()`'
`POD_NAMESPACE`, `POD_NAME`	the Lease’s namespace, and this replica’s identity in it	required
`POSTGRES_ADMIN_DSN_FILE`	a file holding the admin connection’s DSN, its password included	required
`POSTGRES_HOST`, `POSTGRES_PORT`	the server’s address, as a client reaches it, which each Secret holds	required, and 5432
`PROBES_ADDR`, `PROBES_TLS_PEM_FILE`	the probes’ address, and a file holding their certificate and key, PEM	required
`FAULT_TRUSTED_NAMESPACE_PREFIX`	the namespace prefix whose `Database`s’ scenarios are trusted	none
`FAULT_MAX_DELAY`	the longest delay a scenario injects	10s

The checks of their own:

Testing

Mon, 01 Jan 0001 00:00:00 +0000

A unit test of the Reconciler

The Reconciler takes a clientset and the Admin interface, so its unit tests give it client-go’s fake clientset and a fake Admin, which records what it was given:

type fakeAdmin struct {
	provisions []provision
	provisionErr error
	databases map[string]bool
	drops []string
	dropErr error
}

func (a *fakeAdmin) Provision(_ context.Context, name, password string) error {
	a.provisions = append(a.provisions, provision{name: name, password: password})
	return a.provisionErr
}

func (a *fakeAdmin) Drop(_ context.Context, name string) error {
	a.drops = append(a.drops, name)
	if a.dropErr != nil {
		return a.dropErr
	}
	delete(a.databases, name)
	return nil
}

func newReconciler(kc *kubefake.Clientset, admin *fakeAdmin) controller.Reconciler {
	return controller.Reconciler{Kube: kc, Admin: admin, Host: "postgres.databases.svc", Port: 5432}
}

Each test reconciles a Database, and checks what the Admin was given, the Secret, and the status:

Deploying

Mon, 01 Jan 0001 00:00:00 +0000

The image

A Go stage builds the controller static, and the image copies it to /databases on a distroless base, to run as the non-root user 65532 with /databases as its entrypoint. example/kube/databases/Dockerfile names the exact images and their versions.

Build it from the repository’s root, since the module replaces humus and kube with their directories:

docker build --file example/kube/databases/Dockerfile --tag databases .

The kustomization

deploy holds the CRD, the namespace, the ServiceAccount, RBAC, the Deployment, its disruption budget and the settings:

Observing

Mon, 01 Jan 0001 00:00:00 +0000

Opening Grafana

up.sh generates Grafana’s admin password into the grafana-admin Secret. Read it, then forward Grafana’s port:

kubectl --context kind-kind --namespace databases get secret grafana-admin \
 --output jsonpath='{.data.password}' | base64 --decode
kubectl --context kind-kind --namespace databases port-forward service/grafana 3000:3000

A reconcile’s trace

Apply the Database in deploy/kind/database.yaml, as Deploying shows. In Explore, choose Tempo and find its reconciles by its namespace and name:

{resource.service.name="databases" && span.k8s.namespace.name="default" && span.humus.kube.object.name="orders"}

Span	What it is
`reconcile Database`	the reconcile, a root span, with `k8s.namespace.name` and `humus.kube.object.name`
`SELECT pg_roles`, `CREATE ROLE`, `ALTER ROLE`, `SELECT pg_database`, `CREATE DATABASE`, `ALTER DATABASE`	the `Server`’s statements as it provisions, with `db.query.summary` and no SQL text
`DROP DATABASE`, `DROP ROLE`	the statements of a `Finalize` whose policy is `Delete`
`GET`, `PATCH`	each request to the API server, from `otelhttp`, such as reading the Secret and applying it

So a reconcile’s trace holds everything it did. A test of the App checks it:

Production-ready

Mon, 01 Jan 0001 00:00:00 +0000

The controller meets each item. Check your own controller against them:

Item	How	Where
Every setting sourced and checked; a bad one exits 78, naming its variable	`config.Struct`, `kube`’s checks and the controller’s own	The settings
No secret in the environment, the logs or the spans	the admin DSN, the service account’s token and CA, and the probes’ key are files read through `s.Host.FS`. A role’s password reaches PostgreSQL as a SCRAM verifier, and a statement’s span carries its summary, never its SQL. A test of the App checks that no password, DSN or token is in a span, a record or a metric	The secrets, as files, Keeping the password out of the SQL, Keeping credentials out of the spans
Certificates renewed without a restart	the probes’ certificate is served through `GetCertificate`, by a small helper of the walkthrough’s own that reads its file again when it changes. The token’s file is read each time the Client asks, so a rotated one is picked up	The probes, The Connection
Available through a lost replica	two replicas share the Lease, and the standby takes over when the leader stops. A rollout starts a new replica before it stops an old one, and a PodDisruptionBudget of `minAvailable: 1`	The Settings and the Lease, The Deployment, The disruption budget
Ready only once its caches have synced	`Synced` as readiness, on an HTTPS probes port	The probes
A rollout loses nothing	SIGTERM finishes the reconciles in flight and releases the Lease, in a grace period of 20s, and `terminationGracePeriodSeconds` is 30. The next leader lists every `Database` again	The grace period, `main`
Reconciling again is harmless	the Secret is the password’s source of truth, so every reconcile sets the same one. The role and the database are ensured, and the drops are `IF EXISTS`	The password, Provisioning, Dropping
Deleting cleans up, as the `Database` says	the finalizer holds a deleted `Database` until `Finalize` has dropped what its `deletionPolicy` says to. The Secret is deleted with it, as the `Database` controls it	The finalizer, The Secret
Least privilege	a ClusterRole that watches and patches `Database`s and applies Secrets, and a Role for its own Lease and no other. A non-root container, on a read-only root filesystem, with every capability dropped	RBAC, The Deployment
Observable	traces, logs and metrics over OTLP, and rules for failing, panicking and stuck reconciles, a backlog and restarts, on humus’s own instruments, and a dashboard	Observing
Failures say why	a failed step sets `Ready` to `False`, with a reason naming the step, `PostgresUnavailable` or `SecretApplyFailed`	The conditions
Failures are rehearsed	the failure injection cases	The failure injection cases
Safe by default	`deletionPolicy` defaults to `Retain`, and `name` is immutable. No namespace’s scenarios are trusted	The resource and its rules, Trusting a namespace’s scenarios

Databases controller on Humus

The Database resource

The types

The resource and its rules

Talking to PostgreSQL

The admin interface

The Server

Reconciling

The Reconciler

The steps

Cleaning up

The finalizer

Init

The settings

Testing

A unit test of the Reconciler

Deploying

The image

The kustomization

Observing

Opening Grafana

A reconcile’s trace

Production-ready

The `Server`

The `Reconciler`