Order status processor on Humus

The topics

Mon, 01 Jan 0001 00:00:00 +0000

The processor creates no topic. The cluster’s owner creates the topics and their ACLs before it starts. The kind overlay does it with a Job that runs deploy/kind/kafka/topics.sh.

Logging in as the admin

Kafka’s tools read the admin’s login from a properties file. Over mTLS, it is a certificate and key in one PEM file, and the cluster’s CA:

security.protocol=SSL
ssl.keystore.type=PEM
ssl.keystore.location=/etc/kafka-admin/tls/tls.pem
ssl.truststore.type=PEM
ssl.truststore.location=/etc/kafka-admin/tls/ca.crt

kafka runs one of the tools with it:

The projection

Mon, 01 Jan 0001 00:00:00 +0000

The projection keeps every event it applied, and each order’s status, folded from its events. The store package reads and writes it, and the projector package folds it.

The schema

store/schema.sql:

CREATE TABLE order_events (
 source_partition integer NOT NULL,
 source_offset bigint NOT NULL,
 order_id text NOT NULL,
 type text NOT NULL,
 at timestamptz NOT NULL,
 PRIMARY KEY (source_partition, source_offset)
);

CREATE INDEX order_events_by_order ON order_events (order_id, at);

CREATE TABLE order_status (
 order_id text PRIMARY KEY,
 status text NOT NULL,
 at timestamptz NOT NULL
);

An event is keyed by the partition and offset it was consumed from on orders.events, so a redelivery is found by them.

Projecting an event

Mon, 01 Jan 0001 00:00:00 +0000

The projector

The projector package’s Projector is the queue.Processor of kafka.Messages that the Service runs. It takes the store and its Producers as interfaces, so its unit tests can give it their own:

type Store interface {
	Applied(ctx context.Context, partition int32, offset int64) (bool, error)
	Events(ctx context.Context, orderID string) ([]store.Event, error)
	Apply(ctx context.Context, e store.Event, status store.Status) error
}

type Producer interface {
	Produce(ctx context.Context, records ...kafka.Record) error
}

type Settings struct {
	EventsTopic string
	StatusTopic string
	RetryTopic string
	DeadLetterTopic string
	Attempts int
	MinDelay time.Duration
	MaxDelay time.Duration
}

The Service consumes under queue.AtLeastOnce, so a record is committed only once Process returns nil:

Retries and dead letters

Mon, 01 Jan 0001 00:00:00 +0000

Processing a record

One Process handles both orders.events and orders.retry, and keeps no state of its own:

func (p *Projector) Process(ctx context.Context, m kafka.Message) error {
	attempt, from, err := p.origin(m)
	if err != nil {
		return p.deadLetter(ctx, m, from, reasonMalformed, err, 0)
	}

	e, reason, err := decode(m, from)
	if err != nil {
		return p.deadLetter(ctx, m, from, reason, err, 0)
	}

	err = p.wait(ctx, m, attempt)
	if err != nil {
		return err
	}

	work, cancel := context.WithTimeout(ctx, workDeadline)
	defer cancel()
	reason, err = p.project(work, m, e)
	if err == nil {
		return nil
	}
	if ctx.Err() != nil {
		return fmt.Errorf("projecting %s, partition %d, offset %d: %w", from.topic, from.partition, from.offset, context.Cause(ctx))
	}
	return p.retry(ctx, m, from, attempt, reason, err)
}

The record	Then
malformed, or of an unknown type	a dead letter, at once
projected, or applied already	committed
`unavailable` or `invalid-transition`, below `RETRY_ATTEMPTS`	produced to `orders.retry`, one attempt higher
the same, at `RETRY_ATTEMPTS`	a dead letter, with its last attempt’s reason
its context ended, at shutdown or a revoke	`Process` returns the error, and the record is processed again
neither `orders.retry` nor `orders.dlq` stores it	`Process` returns the error, and the App exits 1

const (
	reasonMalformed = "malformed"
	reasonUnknownType = "unknown-type"
	reasonInvalidTransition = "invalid-transition"
	reasonUnavailable = "unavailable"
)

The retry headers

A retried record carries its attempt, why it failed, and where it came from on orders.events:

Init

Mon, 01 Jan 0001 00:00:00 +0000

The settings

Everything Init reads, in one struct:

type settings struct {
	Consumer kafka.Settings
	Producer kafka.ProducerSettings
	Projector projector.Settings
	Database *pgxpool.Config
	Probes httphealth.Settings
}

config.Struct sources and checks each one, with what it needs to build the rest. A bad one makes Init fail, and the processor exits 78, naming its variable:

type postgres struct {
	URL *url.URL
	Password string
	CA *x509.CertPool
}

type sourced struct {
	settings
	KafkaCA *x509.CertPool
	KafkaTLS *tls.Config
	Postgres postgres
	TrustedTopics []string
}

func readSettings(ctx context.Context, s *humus.Setup) (settings, error) {
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }
	file := func(name string) config.Value[[]byte] { return env(name).Parse(readFile(s.Host.FS)) }
	list := func(v string) []string { return strings.Split(v, ",") }

	got, err := config.Struct(func(r *config.Resolver) sourced {
		consumer := kafka.DefaultSettings()
		consumer.Brokers = r.Get(env("KAFKA_BROKERS").Map(list).Validate(kafka.CheckBrokers))
		consumer.GroupID = r.Get(env("KAFKA_GROUP_ID").Validate(kafka.CheckGroupID))
		consumer.Delivery = queue.AtLeastOnce
		consumer.SessionTimeout = r.Get(env("KAFKA_SESSION_TIMEOUT").Parse(time.ParseDuration).Default(consumer.SessionTimeout).Validate(kafka.CheckSessionTimeout))

		producer := kafka.DefaultProducerSettings()
		producer.DeliveryTimeout = r.Get(env("KAFKA_DELIVERY_TIMEOUT").Parse(time.ParseDuration).Default(producer.DeliveryTimeout).Validate(kafka.CheckDeliveryTimeout))
		producer.Breaker.OpenFor = r.Get(env("KAFKA_BREAKER_OPEN_FOR").Parse(time.ParseDuration).Default(producer.Breaker.OpenFor).Validate(kafka.CheckBreakerOpenFor))

		return sourced{
			settings: settings{
				Consumer: consumer,
				Producer: producer,
				Projector: projector.Settings{
					EventsTopic: r.Get(env("KAFKA_EVENTS_TOPIC").Validate(notEmpty)),
					StatusTopic: r.Get(env("KAFKA_STATUS_TOPIC").Validate(notEmpty)),
					RetryTopic: r.Get(env("KAFKA_RETRY_TOPIC").Validate(notEmpty)),
					DeadLetterTopic: r.Get(env("KAFKA_DEAD_LETTER_TOPIC").Validate(notEmpty)),
					Attempts: r.Get(env("RETRY_ATTEMPTS").Parse(strconv.Atoi).Default(defaultRetryAttempts).Validate(positive[int])),
					MinDelay: r.Get(env("RETRY_MIN_DELAY").Parse(time.ParseDuration).Default(defaultRetryMinDelay).Validate(positive[time.Duration])),
					MaxDelay: r.Get(env("RETRY_MAX_DELAY").Parse(time.ParseDuration).Default(defaultRetryMaxDelay).Validate(positive[time.Duration])),
				},
				Probes: httphealth.Settings{
					Addr: r.Get(env("PROBES_ADDR").Validate(httphealth.CheckAddr)),
					TLS: r.Get(env("PROBES_TLS_PEM_FILE").Parse(pemTLS(s.Host.FS)).Validate(httphealth.CheckTLS)),
				},
			},
			KafkaCA: r.Get(file("KAFKA_TLS_CA_FILE").Parse(certPool)),
			KafkaTLS: r.Get(env("KAFKA_TLS_CERT_FILE").Parse(pemTLS(s.Host.FS))),
			Postgres: postgres{
				URL: r.Get(env("POSTGRES_URL").Parse(parsePostgresURL).Validate(checkPostgresURL)),
				Password: r.Get(file("POSTGRES_PASSWORD_FILE").Parse(secret)),
				CA: r.Get(file("POSTGRES_TLS_CA_FILE").Parse(certPool)),
			},
			TrustedTopics: r.Get(env("FAULT_TRUSTED_TOPICS").Parse(topicList).Default(nil)),
		}
	}).Read(ctx)
	if err != nil {
		return settings{}, err
	}

const (
	defaultRetryAttempts = 10
	defaultRetryMinDelay = time.Second
	defaultRetryMaxDelay = 5 * time.Minute
)

Variable	What it is	Default
`KAFKA_BROKERS`	the brokers’ `host:port`s, separated by commas	required
`KAFKA_GROUP_ID`	the consumer group	required
`KAFKA_EVENTS_TOPIC`, `KAFKA_STATUS_TOPIC`, `KAFKA_RETRY_TOPIC`, `KAFKA_DEAD_LETTER_TOPIC`	the four topics, each a different one	required
`KAFKA_TLS_CA_FILE`, `KAFKA_TLS_CERT_FILE`	the brokers’ CA, and a file holding the processor’s certificate and key, PEM	required
`KAFKA_SESSION_TIMEOUT`	`kafka.Settings`’ `SessionTimeout`	`kafka.DefaultSettings()`'
`KAFKA_DELIVERY_TIMEOUT`, `KAFKA_BREAKER_OPEN_FOR`	`kafka.ProducerSettings`’ `DeliveryTimeout` and `Breaker.OpenFor`	`kafka.DefaultProducerSettings()`'
`RETRY_ATTEMPTS`	attempts before a record is dead-lettered	10
`RETRY_MIN_DELAY`, `RETRY_MAX_DELAY`	the backoff of a record’s first retry, and the longest	1s and 5m
`POSTGRES_URL`	PostgreSQL’s URL, with its user and database, and no password	required
`POSTGRES_PASSWORD_FILE`, `POSTGRES_TLS_CA_FILE`	a file holding the password, and PostgreSQL’s CA	required
`PROBES_ADDR`, `PROBES_TLS_PEM_FILE`	the probes’ address, and a file holding their certificate and key, PEM	required
`FAULT_TRUSTED_TOPICS`	the topics whose records’ scenarios are trusted, separated by commas	none
`FAULT_MAX_DELAY`	the longest delay a scenario injects	10s

The four topics must differ, and the backoff can’t start above its longest:

Testing

Mon, 01 Jan 0001 00:00:00 +0000

A unit test of the projector

The projector takes the store and its Producers as interfaces, so its unit tests give it fakes. A store in memory holds what it was given, and fails when the test says so:

type memoryStore struct {
	mu sync.Mutex
	events []store.Event
	applied []applied

	appliedErrs []error
	eventsErrs []error
	applyErrs []error

	appliedErr error
	eventsErr error

	appliedCalls int
	eventsCalls int
	applyCalls int
}

func (s *memoryStore) Apply(ctx context.Context, e store.Event, status store.Status) error {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.applyCalls++
	if err := next(&s.applyErrs); err != nil {
		return err
	}
	s.events = append(s.events, e)
	s.applied = append(s.applied, applied{event: e, status: status})
	return nil
}

A fake Producer records what it was given:

Deploying

Mon, 01 Jan 0001 00:00:00 +0000

The image

One image holds orderstatus and orderevents. A Go stage builds each of them static, and the image copies them to /orderstatus and /orderevents on a distroless base, to run as the non-root user 65532 with /orderstatus as its entrypoint. example/kafka/orderstatus/Dockerfile names the exact images and their versions.

Build it from the repository’s root, since the module replaces humus with its directory:

docker build --file example/kafka/orderstatus/Dockerfile --tag orderstatus .

The kustomization

deploy/base holds a ServiceAccount, the Deployment, its disruption budget and the settings:

Observing

Mon, 01 Jan 0001 00:00:00 +0000

Opening Grafana

up.sh generates Grafana’s admin password into the grafana-admin Secret. Read it, then forward Grafana’s port:

kubectl --context kind-kind --namespace orderstatus get secret grafana-admin \
 --output jsonpath='{.data.password}' | base64 --decode
kubectl --context kind-kind --namespace orderstatus port-forward service/grafana 3000:3000

A record’s trace

Produce an order’s events with orderevents, as Deploying shows. In Explore, choose Tempo and find the latest records’ traces:

{resource.service.name="orderstatus" && name="process orders.events"}

Span	What it is
`process orders.events`	the record’s processing, with its partition and offset
`send orders.status`	each change of status it produced
`send orders.retry`	the record, produced for another attempt
`process orders.retry`	that attempt, a child of the `send orders.retry` that produced it
`send orders.dlq`	the dead letter

So every attempt at a record is in one trace. A test of the App checks it, for a record retried once, then dead-lettered:

Production-ready

Mon, 01 Jan 0001 00:00:00 +0000

The processor meets each item. Check your own processor against them:

Item	How	Where
Every setting sourced and checked; a bad one exits 78, naming its variable	`config.Struct` and the packages’ checks	The settings
No secret in the environment, the logs or the spans	the client certificate, PostgreSQL’s password and the probes’ key are files read through `s.Host.FS`. The record’s key, an order’s ID, isn’t recorded, as `queue/kafka` doesn’t record it	The secrets, as files, A record’s trace
Certificates renewed without a restart	the Kafka client certificate is served through `GetClientCertificate`, and the probes’ through `GetCertificate`, by a small helper of the walkthrough’s own that reads their files again when they change	mTLS to Kafka
Concurrency within the partitions	three replicas on twelve partitions; a PodDisruptionBudget of `maxUnavailable: 1`	Partitions, The disruption budget
Ready only once it has joined its group	`Joined` as readiness, on an HTTPS probes port	The probes
A rollout loses nothing	the revoke commits the record in flight. A record’s work fits in 15s, under the 20s grace period, and `terminationGracePeriodSeconds` is 30. A retried record still waiting at SIGTERM is cut short when the grace period runs out, and processed again by its partition’s next owner. The rollout case checks it	The deadline, `main`, The cases, in process
Redelivery is harmless	every event applied is kept, by its partition and offset, and skipped when it comes again	Skipping what was applied
No record stops it	records that are malformed or of an unknown type are dead letters at once. The rest go through `orders.retry`, and then to `orders.dlq`	Processing a record
An outage blocks neither topic, and keeps order	its records wait in `orders.retry`. An order’s later events follow its earlier ones there, since an event out of order is retried too	Retrying a record
Least privilege	mTLS with ACLs that let it read `orders.events` and `orders.retry`, and write `orders.status`, `orders.retry` and `orders.dlq`. A role with three privileges on two tables. No service account token. A non-root, read-only container	The ACLs, The schema, The Deployment
Observable	traces from producer to `orders.status`, logs and metrics over OTLP, and rules for lag, a stalled group, a crash loop, dead letters, retries, an open breaker and slow processing	Observing
Failures are rehearsed	the failure injection cases	The failure injection cases
Safe by default	no record’s scenario trusted. Only the producer to `orders.retry` forwards one	Trusting a topic’s scenarios, The `Producer`s