Wallet on Humus

The contract

Mon, 01 Jan 0001 00:00:00 +0000

The contract is two protobuf packages under proto/, ledger.v1 and wallet.v1, generated into ledgerpb and walletpb.

The ledger’s service

proto/ledger/v1/ledger.proto declares five methods:

service Ledger {
 rpc OpenAccount(OpenAccountRequest) returns (OpenAccountResponse);
 rpc GetAccount(GetAccountRequest) returns (GetAccountResponse);
 rpc Deposit(DepositRequest) returns (DepositResponse);
 rpc Transfer(TransferRequest) returns (TransferResponse);
 rpc WatchAccount(WatchAccountRequest) returns (stream WatchAccountResponse);
}

An account is its ID, its balance, and when the balance last changed. A transfer is its ID, the two accounts, the amount and when it was made. Amounts are whole cents, in one currency:

message Account {
 string id = 1;
 int64 balance_cents = 2;
 google.protobuf.Timestamp balance_change_time = 3;
}

message Transfer {
 string id = 1;
 string from_account_id = 2;
 string to_account_id = 3;
 int64 amount_cents = 4;
 google.protobuf.Timestamp create_time = 5;
}

A transfer’s request carries its ID, so a transfer sent twice is posted once:

The ledger's accounts

Mon, 01 Jan 0001 00:00:00 +0000

The ledger keeps its accounts and its transfers in two PostgreSQL tables, and moves money between accounts in one transaction per transfer, so money is never made or lost.

`schema.sql`

ledger/store/schema.sql makes the accounts, and the ledger’s own external account, which deposits come from:

CREATE TABLE accounts (
 id text PRIMARY KEY,
 balance_cents bigint NOT NULL DEFAULT 0,
 balance_changed_at timestamptz NOT NULL DEFAULT now(),
 CHECK (id = 'external' OR balance_cents >= 0)
);

INSERT INTO accounts (id) VALUES ('external');

The CHECK refuses a balance below zero for every account but external. external’s balance is the negative of every deposit, so the balances always sum to zero.

Serving the ledger

Mon, 01 Jan 0001 00:00:00 +0000

The ledger’s server package is ledger.v1.Ledger, on the Store.

The methods

Method	Answers
`OpenAccount`	the account, opened with a zero balance, or as it is if it was already open
`GetAccount`	the account, or `NotFound`
`Deposit`	the deposit, as a transfer from `external`. `NotFound` for no such account
`Transfer`	the transfer. `FailedPrecondition` when the source’s balance is short, `NotFound` for either account, `InvalidArgument` for an amount that isn’t positive or an account sent to itself, and `AlreadyExists` for a transfer’s ID reused for another transfer
`WatchAccount`	the account at once, then again each time its balance changes. `NotFound` for no such account

The server takes the Store as an interface, so its tests give it one of their own:

Calling the ledger

Mon, 01 Jan 0001 00:00:00 +0000

The wallet has one grpcclient Client for the ledger. Every method and readiness go through it.

The client’s settings

The wallet’s readSettings starts from grpcclient.DefaultSettings(), and sets the ledger’s address, the CA it trusts the ledger through, and the client certificate it presents:

ledger := grpcclient.DefaultSettings()
ledger.Addr = r.Get(env("LEDGER_ADDR").Validate(grpcclient.CheckAddr))
ledger.TLS = &tls.Config{
	RootCAs: r.Get(file("LEDGER_CA_FILE").Parse(certPool)),
	GetClientCertificate: r.Get(pemFile("LEDGER_TLS_PEM_FILE").Map(func(c *tls.Config) clientCertificate {
		return c.GetClientCertificate
	})),
	MinVersion: tls.VersionTLS12,
}
ledger.Timeout = r.Get(env("LEDGER_TIMEOUT").Parse(time.ParseDuration).Default(ledger.Timeout).Validate(grpcclient.CheckTimeout))
ledger.MaxAttempts = r.Get(env("LEDGER_MAX_ATTEMPTS").Parse(strconv.Atoi).Default(ledger.MaxAttempts).Validate(grpcclient.CheckMaxAttempts))
ledger.ForwardScenario = true

type (
	serverCertificate = func(*tls.ClientHelloInfo) (*tls.Certificate, error)
	clientCertificate = func(*tls.CertificateRequestInfo) (*tls.Certificate, error)
)

The client

Init builds the Client, and the wallet’s server calls the ledger through the generated client on it:

Serving the wallet

Mon, 01 Jan 0001 00:00:00 +0000

The wallet’s server package is wallet.v1.Wallet, on the ledger’s generated client.

The methods

Method	Needs	Answers
`OpenAccount`	`wallet:write`	the caller’s account, opened in the ledger
`GetAccount`	`wallet:read`	the caller’s account
`Send`	`wallet:write`	the transfer from the caller’s account, to the account it names
`WatchAccount`	`wallet:read`	the caller’s account, as the ledger’s watch sends it

The claims and the scopes

A token’s claims are its subject and its scopes:

const (
	ScopeRead = "wallet:read"
	ScopeWrite = "wallet:write"
)

type Claims struct {
	Subject string `json:"sub"`
	Scope string `json:"scope"`
}

func (c Claims) HasScope(scope string) bool {
	return slices.Contains(strings.Fields(c.Scope), scope)
}

The fetched key set

The wallet has one httpclient Client for the key set, from TOKEN_KEYS_CA_FILE and the defaults:

Init

Mon, 01 Jan 0001 00:00:00 +0000

The wallet’s settings

Everything the wallet’s Init reads, in one struct:

type settings struct {
	GRPC grpcserver.Settings
	Probes grpchealth.Settings
	Ledger grpcclient.Settings
	KeySet keySetSettings
	JWT jwt.Settings
	FaultTrustScope string
}

type keySetSettings struct {
	URL *url.URL
	Client httpclient.Settings
}

config.Struct sources and checks each one. A bad one makes Init fail, and the wallet exits 78, naming its variable:

func readSettings(ctx context.Context, s *humus.Setup) (settings, error) {
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }
	file := func(name string) config.Value[[]byte] { return env(name).Parse(readFile(s.Host.FS)) }
	pemFile := func(name string) config.Value[*tls.Config] { return env(name).Parse(pemTLS(s.Host.FS)) }

The ledger’s Client comes first, as Calling the ledger shows, then the key set’s, as Serving the wallet shows, then the tokens’ settings and the server’s:

Testing

Mon, 01 Jan 0001 00:00:00 +0000

The ledger’s unit tests

The ledger’s server takes a Store, so its tests give it a fake one, in memory, that fails when the test says so:

func (f *fakeStore) Account(_ context.Context, id string) (store.Account, error) {
	f.mu.Lock()
	defer f.mu.Unlock()
	f.reads++
	if f.err != nil {
		return store.Account{}, fmt.Errorf("reading account %s: %w", id, f.err)
	}
	a, ok := f.accounts[id]
	if !ok {
		return store.Account{}, fmt.Errorf("reading account %s: %w", id, store.ErrNotFound)
	}
	return a, nil
}

Each test serves it on the humustest Host, over TLS, and calls it with the ledger’s generated client:

Deploying

Mon, 01 Jan 0001 00:00:00 +0000

The image

One image holds the wallet and the ledger. A Go stage builds each of them static, and the image copies them to /wallet and /ledger on a distroless base, to run as the non-root user 65532. example/grpc/wallet/Dockerfile names the exact images and their versions.

Build it from the repository’s root, since the module replaces humus and humus/grpc with their directories:

docker build --file example/grpc/wallet/Dockerfile --tag wallet .

The kustomization

deploy/base holds the namespace, a ServiceAccount, a Deployment, a Service and a disruption budget for each service, and each one’s settings:

Observing

Mon, 01 Jan 0001 00:00:00 +0000

Opening Grafana

up.sh generates Grafana’s admin password into the grafana-admin Secret. Read it, then forward Grafana’s port:

kubectl --context kind-kind --namespace wallet get secret grafana-admin \
 --output jsonpath='{.data.password}' | base64 --decode
kubectl --context kind-kind --namespace wallet port-forward service/grafana 3000:3000

A send’s trace

Send from alice’s account to bob’s, as Deploying shows. Each server span is named after its method, so in Explore, with Tempo, find the sends by name:

Production-ready

Mon, 01 Jan 0001 00:00:00 +0000

The wallet and the ledger meet each item. Check your own gRPC services against them:

Item	How	Where
Every setting sourced and checked; a bad one exits 78, naming its variable	`config.Struct` and the packages’ checks, in both services	The wallet’s settings, The ledger’s settings
No secret in the environment, the logs or the spans	PostgreSQL’s password and the private keys are files, read through `s.Host.FS`. The token is metadata, which otelgrpc doesn’t record, and no handler logs it, which a test of the Apps checks	The secrets, as files, Their log records
Certificates and signing keys renewed without a restart	`internal/certfile`, for both servers’ certificates and the wallet’s client certificate. The key set is fetched again for a `kid` the wallet lacks	The certificates, read again when renewed, The fetched key set
Two replicas of each service	a PodDisruptionBudget of `minAvailable: 1`, and a rolling update of `maxUnavailable: 0`	The Deployments, The disruption budgets
Ready once it has reached its dependency, and still ready through its outages	`Connected`, and the ledger’s latch on PostgreSQL	Readiness: `Connected`, The ledger’s probes, Readiness
A rollout loses no call	`grpcserver.Main`’s 5s drain. A unary call is bounded by `LEDGER_TIMEOUT`, 10s, inside the 15s grace period. Watches end with `Unavailable` and are opened again. `terminationGracePeriodSeconds` is 30, above the drain, the grace period and the telemetry’s shutdown	The drain, Relaying the watch
Calls spread across pods	`MaxConnectionAge` on both servers	`MaxConnectionAge`
Money is never made or lost	one transaction per transfer, locks in the order of the IDs, and a `CHECK` that no account but `external` goes below zero	`schema.sql`, A transfer’s transaction
A retry never moves money twice	every transfer’s ID: the subject and the request’s ID	Why every transfer is safe to retry
The contract never breaks a caller	`buf breaking` against main, in CI	buf’s rules, `buf generate`
Least privilege	the ledger’s role has `SELECT`, `INSERT` and `UPDATE` on its two tables. Only the internal CA’s certificates call the ledger. A token can use only its subject’s account. A non-root, read-only container, with no service account token	Running it on kind, mTLS, `grpcauth`, The subject’s account, The Deployments
Observable	traces across both services, logs and metrics over OTLP, `deploy/alerts.yaml`, `deploy/dashboard.json`, and alerts on each Deployment from the Collector’s Kubernetes cluster receiver	Observing
Failures say why	statuses with `PreconditionFailure`, `BadRequest`, `ErrorInfo` and `RetryInfo` details	The statuses, with details, The ledger’s answers, as the wallet’s, A key set that can’t be fetched
Failures are rehearsed	the failure injection cases, in CI and against staging through its own `Target`	The shared cases, The failure injection cases
Safe by default	`FAULT_TRUST_SCOPE` empty, so no scenario is trusted. Server reflection off. The probes ports serve gRPC health and nothing else	`TrustScenario`, The gRPC probes