Mailer

A walkthrough of a RabbitMQ worker on humus that sends the emails other services ask for through a provider’s API, retries what fails for now after a delay, and dead-letters what never will. These pages build it step by step.

The mailer consumes requests for emails from RabbitMQ. For each one, it:

  1. calls the provider’s API, which sends the email;
  2. publishes what became of it, mail.sent or mail.refused, for the services that asked;
  3. parks a request that failed for now, which comes back after a delay, and dead-letters one that never will succeed, with why.

No request stops it, and a request delivered twice sends one email, because its MessageId is the provider’s idempotency key.

The contract

A request is published to the mail exchange with the routing key mail.send:

  • its MessageId is required, and is the provider’s idempotency key;
  • its CorrelationId is optional, and the events carry it back;
  • its body is JSON, to, subject and text, up to 256 KiB.

The sender’s address is the mailer’s setting, so no publisher can send as anyone else.

An event is published to the mail.events topic exchange, with the request’s MessageId and CorrelationId:

  • mail.sent, with the provider’s ID for the email, as provider_id;
  • mail.refused, with the provider’s reason, for a request the provider will never accept.

A service that wants the events binds a queue of its own to mail.events.

The names

The mail package holds the contract and the topology’s names:

const (
	Exchange       = "mail"
	RetryExchange  = "mail.retry"
	DeadExchange   = "mail.dead"
	EventsExchange = "mail.events"
)

const (
	SendQueue  = "mail.send"
	RetryQueue = "mail.send.retry"
	DeadQueue  = "mail.send.dead"
)

const (
	SendKey    = "mail.send"
	SentKey    = "mail.sent"
	RefusedKey = "mail.refused"
)
NameKindWhat it does
maildirect exchangewhere requests are published
mail.sendquorum queue, bound to mail by mail.sendthe work queue the mailer consumes
mail.retrydirect exchangewhere the mailer parks a request that failed for now
mail.send.retryquorum queue, bound to mail.retryholds each parked request for the delay, then dead-letters it back to mail
mail.deadfanout exchangedead letters, from the mailer and from mail.send’s delivery limit
mail.send.deadquorum queue, bound to mail.deadthe parking lot, where dead letters wait with why they died
mail.eventstopic exchangemail.sent and mail.refused

The files it ends with

Its code is a module of its own, at example/rabbitmq/mailer:

example/rabbitmq/mailer/
├── main.go             humus.Main(app.Init, ...), with a grace period of 25s
├── app/                Init, the settings, BuildTelemetry and Faults
├── mail/               the request and the events, and the topology's names
├── provider/           the provider's client, over httpclient, and its Classify
├── worker/             the Processor: decode, send, decide, publish
├── internal/certfile/  the probes' certificate, read again when it's renewed
├── cmd/mailrequest/    a CLI that publishes a request, to try the mailer by hand
├── cases/              the test cases the container tests and kind share
├── functional/         the cases against kind
├── deploy/             definitions.json, the kustomization, the alerts and
│                       dashboard, and the kind overlay
└── Dockerfile          the mailer's and mailrequest's image

These pages set up the topology, then build provider, mail and worker, then app and main, then test it, deploy it and observe it, and end with the production-ready checklist.


The topology

The mailer’s exchanges, queues, bindings and policies in a RabbitMQ definitions file, importing it into the broker, and creating the logins and their permissions with rabbitmqctl.

Calling the provider

The mailer’s provider package: an httpclient client with a Classify that retries a 409, the request with its Idempotency-Key and API key, what each answer means, Retry-After, and the WireMock mappings that stand in for the provider.

Processing a request

The mailer’s Processor: decoding a request, the table that decides what becomes of it, publishing its event with confirms, an event no queue receives, and acking once its result is the broker’s.

Retries and dead letters

How the mailer parks a request that failed for now in a delay queue, counts its attempts in a header, dead-letters a request that never will succeed with why, and what the queue’s delivery limit catches.

Init

The mailer’s Init: its settings, sourced and checked; its secrets, read from files; the Publishers and the UserID a parked request carries; TrustScenario; the probes; BuildTelemetry and Faults; and main’s grace period.

Testing

How the mailer is tested: its Processor as a unit, on fakes of the provider and the Publishers; the shared cases, their Target and their barrier, run in process against containers and on kind; and the failure injection cases.

Deploying

How the mailer is deployed: its image, the kustomization, the Deployment and its disruption budget, the grace period, and the kind overlay. Then running it on kind with up.sh, and publishing requests to it with mailrequest.

Observing

What the running mailer tells you in Grafana on kind: a request’s trace in Tempo, its log records in Loki, the mailer’s, humus’s and RabbitMQ’s metrics in Mimir, the alert rules in deploy/alerts.yaml, and the dashboard in deploy/dashboard.json.

Production-ready

The checklist the mailer meets, and how it meets each item, with the page that builds it. Check your own RabbitMQ worker against it.