Billing job on Humus

The invoices

Mon, 01 Jan 0001 00:00:00 +0000

The subscriptions

postgres/schema.sql starts with the subscriptions. Your other services keep them, and the job only reads them:

CREATE TABLE IF NOT EXISTS subscriptions (
 account_id TEXT PRIMARY KEY,
 amount_cents BIGINT NOT NULL CHECK (amount_cents > 0),
 currency CHAR(3) NOT NULL CHECK (currency ~ '^[A-Z]{3}$'),
 active BOOLEAN NOT NULL
);

The invoices

One invoice per account and month. It stays pending until it is charged, then it is charged, declined or failed, each with the one column that says why:

Charging

Mon, 01 Jan 0001 00:00:00 +0000

The payments API is invented for the walkthrough, and is one endpoint: POST /v1/charges, with a bearer key, an Idempotency-Key, and the account, amount and currency as JSON.

A charge and its result

Start the payments package with what the run hands it, and what it hands back:

type Charge struct {
	ID uuid.UUID
	AccountID string
	AmountCents int64
	Currency string
}

type State int

const (
	Charged State = iota + 1
	Declined
	Failed
)

type Result struct {
	State State
	ChargeID string
	DeclineCode string
	Answer string
}

Classify

httpclient’s default Classify, plus two answers. A 409 means a charge with this key is still in flight, so it is retried, and every 5xx counts against the breaker:

The run

Mon, 01 Jan 0001 00:00:00 +0000

The run’s inputs

The billing package’s Run takes the store as an interface, so its unit tests can give it one in memory:

type Invoices interface {
	Issue(ctx context.Context, period time.Time) (issued int64, err error)
	Pending(ctx context.Context) ([]postgres.Invoice, error)
	SettleCharged(ctx context.Context, id uuid.UUID, chargeID string) (settled bool, err error)
	SettleDeclined(ctx context.Context, id uuid.UUID, declineCode string) (settled bool, err error)
	SettleFailed(ctx context.Context, id uuid.UUID, answer string) (settled bool, err error)
}

type Run struct {
	Invoices Invoices
	Payments *payments.Client
	Period time.Time
	Concurrency int
	Telemetry telemetry.SDK
}

Its telemetry

The run’s tracer, logger and three instruments, in its own scope:

Init

Mon, 01 Jan 0001 00:00:00 +0000

The settings

Everything Init reads, in one struct:

type settings struct {
	Database *pgxpool.Config
	Payments httpclient.Settings
	APIKey string
	Period time.Time
	Concurrency int
}

config.Struct sources and checks each one. A bad one makes Init fail, and the job exits 78, naming its variable:

func readSettings(ctx context.Context, s *humus.Setup) (settings, error) {
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }
	file := func(name string) config.Value[[]byte] { return env(name).Parse(readFile(s.Host.FS)) }

	return config.Struct(func(r *config.Resolver) settings {
		client := httpclient.DefaultSettings()
		client.BaseURL = r.Get(env("PAYMENTS_URL").Parse(url.Parse).Validate(checkPaymentsURL))
		client.TLS = r.Get(file("PAYMENTS_CA_FILE").Parse(certPool).Default(nil).Map(func(roots *x509.CertPool) *tls.Config {
			return &tls.Config{RootCAs: roots}
		}).Validate(httpclient.CheckTLS))
		client.Timeout = r.Get(env("PAYMENTS_TIMEOUT").Parse(time.ParseDuration).Default(client.Timeout).Validate(httpclient.CheckTimeout))
		client.AttemptTimeout = r.Get(env("PAYMENTS_ATTEMPT_TIMEOUT").Parse(time.ParseDuration).Default(client.AttemptTimeout).Validate(httpclient.CheckAttemptTimeout))
		client.MaxAttempts = r.Get(env("PAYMENTS_MAX_ATTEMPTS").Parse(strconv.Atoi).Default(client.MaxAttempts).Validate(httpclient.CheckMaxAttempts))

		return settings{
			Database: r.Get(file("DATABASE_URL_FILE").Parse(databaseURL)),
			Payments: client,
			APIKey: r.Get(file("PAYMENTS_API_KEY_FILE").Parse(apiKey)),
			Period: r.Get(env("BILLING_PERIOD").Parse(parseMonth).Default(lastMonth(time.Now())).Validate(checkEnded)),
			Concurrency: r.Get(env("BILLING_CONCURRENCY").Parse(strconv.Atoi).Default(4).Validate(checkConcurrency)),
		}
	}).Read(ctx)
}

Variable	What it is	Default
`DATABASE_URL_FILE`	a file holding the PostgreSQL URL, password included	required
`PAYMENTS_URL`	the payments API’s base URL, `https`	required
`PAYMENTS_API_KEY_FILE`	a file holding the API key	required
`PAYMENTS_CA_FILE`	the CA certificates the API is trusted through, PEM	the system’s roots
`PAYMENTS_TIMEOUT`, `PAYMENTS_ATTEMPT_TIMEOUT`, `PAYMENTS_MAX_ATTEMPTS`	`httpclient.Settings`’ fields	`httpclient.DefaultSettings()`'
`BILLING_PERIOD`	the month to issue, `YYYY-MM`, which has ended	last month, in UTC
`BILLING_CONCURRENCY`	charges in flight, 1 to 32	4

The secrets, as files

The database URL and the API key are files mounted from Secrets, read through the Host’s FS, so the tests can give the job files of their own:

Testing

Mon, 01 Jan 0001 00:00:00 +0000

A store in memory

The unit tests give Run an Invoices of their own, which keeps the subscriptions and invoices in memory:

type store struct {
	mu sync.Mutex
	subscriptions []subscription
	invoices []*invoice
	issueErr error
	settleErr error
}

Like PostgreSQL’s, it settles only an invoice still pending:

func (s *store) settle(id uuid.UUID, set func(*invoice)) (bool, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	if s.settleErr != nil {
		return false, s.settleErr
	}
	for _, inv := range s.invoices {
		if inv.ID == id && inv.state == "pending" {
			set(inv)
			return true, nil
		}
	}
	return false, nil
}

The real store is checked against the same interface:

Deploying

Mon, 01 Jan 0001 00:00:00 +0000

The image

A Go stage builds the job static, and the image copies it to /billing on a distroless base, to run as the non-root user 65532 with /billing as its entrypoint. example/job/billing/Dockerfile names the exact images and their versions.

Build it from the repository’s root, because the module replaces humus with ../../..:

docker build --file example/job/billing/Dockerfile --tag billing .

The CronJob

It runs daily at 03:00 UTC, one run at a time:

Observing

Mon, 01 Jan 0001 00:00:00 +0000

Grafana

On kind, Grafana’s admin password is in the Secret up.sh made. Forward its port, and sign in as admin at http://localhost:3000:

kubectl --namespace billing get secret grafana --output jsonpath='{.data.password}' | base64 --decode
kubectl --namespace billing port-forward service/grafana 3000

A run’s trace

Each run is a trace. Its root is the job span, or a child of the trace in TRACEPARENT when whatever started the run set one. Under it are an issue span, a charge span for each invoice, and a POST span for each attempt at the payments API. In Explore, Tempo finds the runs with:

Production-ready

Mon, 01 Jan 0001 00:00:00 +0000

Every setting is sourced and checked

config.Struct sources every setting, and each one’s check names its variable. A bad one exits 78:

Period: r.Get(env("BILLING_PERIOD").Parse(parseMonth).Default(lastMonth(time.Now())).Validate(checkEnded)),
Concurrency: r.Get(env("BILLING_CONCURRENCY").Parse(strconv.Atoi).Default(4).Validate(checkConcurrency)),

The pod failure policy fails the Job on 78, with no retry:

- action: FailJob
 onExitCodes:
 containerName: billing
 operator: In
 values: [78]

No secret is in the environment, the logs or the spans

The database URL and the API key are files from Secrets, read through the Host: