Order status processor

A walkthrough of a Kafka processor on humus that projects orders’ events into their statuses in PostgreSQL, produces each change of status, retries what fails for now through a retry topic, and dead-letters what never will. These pages build it step by step.

The order status processor consumes orders’ lifecycle events from Kafka. For each one, it:

folds the order’s events into its status, through a state machine;
produces each change of status to orders.status, for the services that want them;
stores the event and the status in PostgreSQL.

An event that fails for now goes to orders.retry, and is tried again after a wait. One that never will succeed goes to orders.dlq, with why. No record stops it, and a record processed twice changes nothing.

The topics

Topic	Key	Value	Written by
`orders.events`	the order’s ID	an event, `type` and `at`	the end user’s other services
`orders.status`	the order’s ID	a change of status, `order_id`, `status`, `previous` and `at`	the processor
`orders.retry`	the order’s ID	the consumed record’s value, unchanged	the processor, for each record it will try again
`orders.dlq`	the order’s ID	the consumed record’s value, unchanged	the processor, for each record it gives up on

An event’s type is one of the statuses:

const (
	Placed    = "placed"
	Paid      = "paid"
	Shipped   = "shipped"
	Delivered = "delivered"
	Cancelled = "cancelled"
)

An event, and a change of status, as JSON:

type event struct {
	Type string    `json:"type"`
	At   time.Time `json:"at"`
}

type change struct {
	OrderID  string    `json:"order_id"`
	Status   string    `json:"status"`
	Previous string    `json:"previous"`
	At       time.Time `json:"at"`
}

The files it ends with

Its code is a module of its own, at example/kafka/orderstatus:

example/kafka/orderstatus/
├── main.go             humus.Main(app.Init, ...)
├── app/                Init, the settings, BuildTelemetry and Faults
├── store/              the projection in PostgreSQL, and its schema.sql
├── projector/          the Processor: decode, fold, produce, store, retry
├── internal/certfile/  the certificates, read again when they're renewed
├── cmd/orderevents/    a CLI that produces an order's events, to try the
│                       processor by hand
├── cases/              the test cases the in-process tests and kind share
├── functional/         the cases against kind
├── deploy/             the kustomization, the alerts and dashboard, and the
│                       kind overlay
└── Dockerfile          the orderstatus and orderevents image

These pages create the topics, then build store, projector, then app and main, then test it, deploy it and observe it, and end with the production-ready checklist.

The topics

The order status processor’s four topics: creating them and their ACLs with Kafka’s CLI, their keys, their partitions and their retention.

The projection

The order status projection in PostgreSQL: schema.sql, the events it keeps, and the status folded from them through a state machine, stored in one transaction with the event.

Projecting an event

The order status processor’s Processor: decoding a record, retrying the store’s calls, skipping an event already applied, folding the order’s history with it, and producing its changes before storing it.

Retries and dead letters

How the order status processor retries a record that failed for now through orders.retry, with its attempt in a header and a backoff derived from it, dead-letters a record that never will succeed to orders.dlq with why, and what to do with orders.dlq.

Init

The order status processor’s Init: its settings, sourced and checked; its secrets, read from files; mTLS to Kafka; the PostgreSQL pool; the two Producers; the probes; BuildTelemetry and Faults; and main.

Testing

How the order status processor is tested: its projector as a unit, on fakes of the store and the Producers; the store against PostgreSQL; the shared cases, their Target and their barrier, run in process on kfake and on kind; and the failure injection cases.

Deploying

How the order status processor is deployed: its image, the kustomization, the Deployment and its disruption budget, and the kind overlay. Then running it on kind with up.sh, and producing an order’s events to it with orderevents.

Observing

What the running order status processor tells you in Grafana on kind: a record’s trace across the topics in Tempo, its log records in Loki, the processor’s, humus’s, the group’s lag and the containers’ restarts in Mimir, the alert rules in deploy/alerts.yaml, and the dashboard in deploy/dashboard.json.

Production-ready

The checklist the order status processor meets, and how it meets each item, with the page that builds it. Check your own Kafka processor against it.