Mailer on Humus

The topology

Mon, 01 Jan 0001 00:00:00 +0000

The mailer declares nothing. Its topology is deploy/rabbitmq/definitions.json, a definitions file the broker’s team imports. It holds no users and no passwords.

The exchanges

Four exchanges, each durable:

"exchanges": [
 {
 "name": "mail",
 "vhost": "/",
 "type": "direct",
 "durable": true,
 "auto_delete": false,
 "internal": false,
 "arguments": {}
 },
 {
 "name": "mail.retry",
 "vhost": "/",
 "type": "direct",
 "durable": true,
 "auto_delete": false,
 "internal": false,
 "arguments": {}
 },
 {
 "name": "mail.dead",
 "vhost": "/",
 "type": "fanout",
 "durable": true,
 "auto_delete": false,
 "internal": false,
 "arguments": {}
 },
 {
 "name": "mail.events",
 "vhost": "/",
 "type": "topic",
 "durable": true,
 "auto_delete": false,
 "internal": false,
 "arguments": {}
 }
],

The queues

Three quorum queues. The queue type is their only argument:

Calling the provider

Mon, 01 Jan 0001 00:00:00 +0000

The provider is invented for the walkthrough, and is one endpoint: POST /v1/messages, with a bearer key, an Idempotency-Key, and the sender, recipient, subject and text as JSON. It answers 2xx with the email’s id, or a 4xx with a reason.

Its settings and its result

Start the provider package with what Init hands it, and what a send hands back:

type Settings struct {
	URL *url.URL
	CA *x509.CertPool
	Timeout time.Duration
	AttemptTimeout time.Duration
	MaxAttempts int
	From string
	APIKey string
}

type Result struct {
	Refused bool
	ID string
	Reason string
}

The client

New checks the settings, then builds an httpclient client for the provider’s URL, trusting CA, or the system’s roots when it’s nil:

Processing a request

Mon, 01 Jan 0001 00:00:00 +0000

The request

The mail package decodes a delivery into a request:

const MaxBodySize = 256 << 10

var ErrMalformed = errors.New("malformed request")

type Request struct {
	MessageID string
	CorrelationID string
	To string
	Subject string
	Text string
}

type body struct {
	To string `json:"to"`
	Subject string `json:"subject"`
	Text string `json:"text"`
}

Decoding

A request with no MessageId, a body over 256 KiB or not JSON, or a field missing is malformed:

Retries and dead letters

Mon, 01 Jan 0001 00:00:00 +0000

Parking a request

A request that failed for now is published to mail.retry, with its attempt one higher. A returned error would stop the Service instead, and every replica would crash-loop through a provider’s outage:

func (w *Worker) park(ctx context.Context, m rabbitmq.Message, attempt int) error {
	p := request(m, mail.RetryExchange, map[string]any{
		mail.AttemptHeader: int64(attempt),
	})
	p.UserID = w.user
	if err := w.retries.Publish(ctx, p); err != nil {
		return fmt.Errorf("worker: parking the request in %s: %w", mail.RetryExchange, err)
	}
	return nil
}

The parked request is the request as it was published, plus its headers:

Init

Mon, 01 Jan 0001 00:00:00 +0000

The settings

Everything Init reads, in one struct:

type settings struct {
	Consumer rabbitmq.Settings
	Publisher rabbitmq.PublisherSettings
	Login string
	TrustedPublishers []string
	MaxAttempts int
	Provider provider.Settings
	Probes httphealth.Settings
}

config.Struct sources and checks each one. A bad one makes Init fail, and the mailer exits 78, naming its variable:

func readSettings(ctx context.Context, s *humus.Setup) (settings, error) {
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }
	file := readFile(s.Host.FS)

	read, err := config.Struct(func(r *config.Resolver) settings {
		consumer := rabbitmq.DefaultSettings()
		conn := consumer.Connection
		conn.Addr = r.Get(env("RABBITMQ_ADDR").Validate(rabbitmq.CheckAddr))
		conn.Vhost = r.Get(env("RABBITMQ_VHOST").Default(conn.Vhost).Validate(rabbitmq.CheckVhost))
		conn.TLS = r.Get(env("RABBITMQ_CA_FILE").Parse(file).Parse(certPool).Map(trusting).Default(nil))
		login := r.Get(env("RABBITMQ_USERNAME").Validate(notEmpty))
		conn.SASL = &amqp.PlainAuth{
			Username: login,
			Password: r.Get(env("RABBITMQ_PASSWORD_FILE").Parse(file).Parse(secret)),
		}
		consumer.Connection = conn
		consumer.Queue = mail.SendQueue
		consumer.Delivery = queue.AtLeastOnce
		consumer.Prefetch = r.Get(env("RABBITMQ_PREFETCH").Parse(strconv.Atoi).Default(defaultPrefetch).Validate(rabbitmq.CheckPrefetch))
		consumer.Concurrency = r.Get(env("RABBITMQ_CONCURRENCY").Parse(strconv.Atoi).Default(defaultConcurrency).Validate(rabbitmq.CheckConcurrency))

		publisher := rabbitmq.DefaultPublisherSettings()
		publisher.Connection = conn

		client := httpclient.DefaultSettings()
		return settings{
			Consumer: consumer,
			Publisher: publisher,
			Login: login,
			TrustedPublishers: r.Get(env("MAIL_TRUSTED_PUBLISHERS").Parse(logins).Default(nil)),
			MaxAttempts: r.Get(env("MAIL_MAX_ATTEMPTS").Parse(strconv.Atoi).Default(defaultMaxAttempts).Validate(positive)),
			Provider: provider.Settings{
				URL: r.Get(env("PROVIDER_URL").Parse(url.Parse).Validate(httpclient.CheckBaseURL)),
				CA: r.Get(env("PROVIDER_CA_FILE").Parse(file).Parse(certPool).Default(nil)),
				Timeout: r.Get(env("PROVIDER_TIMEOUT").Parse(time.ParseDuration).Default(client.Timeout).Validate(httpclient.CheckTimeout)),
				AttemptTimeout: r.Get(env("PROVIDER_ATTEMPT_TIMEOUT").Parse(time.ParseDuration).Default(client.AttemptTimeout).Validate(httpclient.CheckAttemptTimeout)),
				MaxAttempts: r.Get(env("PROVIDER_MAX_ATTEMPTS").Parse(strconv.Atoi).Default(client.MaxAttempts).Validate(httpclient.CheckMaxAttempts)),
				From: r.Get(env("MAIL_FROM").Validate(notEmpty)),
				APIKey: r.Get(env("PROVIDER_API_KEY_FILE").Parse(file).Parse(secret)),
			},
			Probes: httphealth.Settings{
				Addr: r.Get(env("PROBES_ADDR").Validate(httphealth.CheckAddr)),
				TLS: r.Get(env("PROBES_TLS_PEM_FILE").Parse(pemTLS(s.Host.FS)).Validate(httphealth.CheckTLS)),
			},
		}
	}).Read(ctx)
	if err != nil {
		return settings{}, err
	}

The consumer and every Publisher share one Connection, so they log in as one user.

Testing

Mon, 01 Jan 0001 00:00:00 +0000

A unit test of the Processor

The worker takes the provider and its Publishers as interfaces:

type Provider interface {
	Send(ctx context.Context, r mail.Request) (provider.Result, error)
}

type Publisher interface {
	Publish(ctx context.Context, m rabbitmq.Publishing) error
}

So its unit tests give it fakes. A fake Publisher records what it was given, and fails when the test says so:

type fakePublisher struct {
	mu sync.Mutex
	fail func(rabbitmq.Publishing) error
	published []rabbitmq.Publishing
}

func (p *fakePublisher) Publish(_ context.Context, m rabbitmq.Publishing) error {
	p.mu.Lock()
	defer p.mu.Unlock()
	p.published = append(p.published, m)
	if p.fail != nil {
		return p.fail(m)
	}
	return nil
}

func failing(exchange string, err error) func(rabbitmq.Publishing) error {
	return func(m rabbitmq.Publishing) error {
		if m.Exchange == exchange {
			return err
		}
		return nil
	}
}

A harness builds the worker on the fakes, with humustest’s telemetry:

Deploying

Mon, 01 Jan 0001 00:00:00 +0000

The image

One image holds mailer and mailrequest. A Go stage builds each of them static, and the image copies them to /mailer and /mailrequest on a distroless base, to run as the non-root user 65532 with /mailer as its entrypoint. example/rabbitmq/mailer/Dockerfile names the exact images and their versions.

Build it from the repository’s root, since the module replaces humus with its directory:

docker build --file example/rabbitmq/mailer/Dockerfile --tag mailer .

The kustomization

deploy/base holds a ServiceAccount, the Deployment, its disruption budget and the settings:

Observing

Mon, 01 Jan 0001 00:00:00 +0000

Opening Grafana

up.sh generates Grafana’s admin password into the grafana-admin Secret. Read it, then forward Grafana’s port:

kubectl --context kind-kind --namespace mailer get secret grafana-admin \
 --output jsonpath='{.data.password}' | base64 --decode
kubectl --context kind-kind --namespace mailer port-forward service/grafana 3000:3000

A request’s trace

Publish a request with mailrequest, as Deploying shows, and note the MessageId it prints. In Explore, choose Tempo and find the request’s trace by it:

{resource.service.name="mailer" && span.messaging.message.id="<MessageId>"}

Span	What it is
`process mail.send`	the delivery, a child of the span that published the request
`POST`	each attempt at the provider, under `process mail.send`
`send mail.events`	the `mail.sent` or `mail.refused` event
`send mail.retry`	the request, parked for another attempt
`send mail.dead`	the dead letter

The worker puts the attempt on the process span:

Production-ready

Mon, 01 Jan 0001 00:00:00 +0000

The mailer meets each item. Check your own worker against them:

Item	How	Where
Every setting sourced and checked; a bad one exits 78, naming its variable	`config.Struct` and the packages’ checks	The settings
No secret in the environment, the logs or the spans	the RabbitMQ password, the provider’s key and the probes’ certificate are files read through `s.Host.FS`	The secrets, as files
Certificates renewed without a restart	the probes’ certificate is served through `GetCertificate`, by a small helper of the walkthrough’s own that reads its files again when they change	The probes
No personal data in the telemetry	addresses and bodies are never logged or put on a span	What it records
Two replicas, competing for one queue	a PodDisruptionBudget of `minAvailable: 1`	The disruption budget
Ready only once it consumes	`Consuming` as readiness, on a probes server	The probes
A rollout loses nothing, and sends nothing twice	SIGTERM cancels the consumer, finishes the deliveries in flight within 25s, and nacks the rest, which counts nothing against them. `terminationGracePeriodSeconds: 35`	The grace period
A broker’s rolling restart costs no restart	the Service reconnects. The deliveries that were in flight are redelivered, and the idempotency key sends them once	The request and its key
A dependency’s outage is ridden out	the client’s retries, then the delay queue, then the parking lot after `MAIL_MAX_ATTEMPTS`. The breaker sheds calls while the provider keeps failing	Retries and dead letters
No request stops the worker	malformed requests go to the parking lot at once. The rest are parked in `mail.send.retry`, an event that can’t be published included, and dead-lettered after `MAIL_MAX_ATTEMPTS`. Only a request that can go to neither `mail.retry` nor `mail.dead` stops the worker, and the delivery limit dead-letters it after 5 deliveries	The delivery limit
Least privilege	no configure permission on the broker; write and read on its own exchanges and queue only, granted with `rabbitmqctl set_permissions`. No service account token. A non-root, read-only container	The logins, The Deployment
Observable	traces, logs and metrics over OTLP, and the alerts in `deploy/alerts.yaml`	Observing
Failures say why	each dead letter’s reason headers, and the `mail.refused` event	Dead letters, The events
Failures are rehearsed	the failure injection cases	The failure injection cases
Safe by default	`TrustScenario` trusts no publisher; the sender’s address is a setting	`TrustScenario`