Init

Each service’s Init: its settings, sourced and checked, and every variable it reads; its secrets and certificates as files, read again when renewed; PostgreSQL through the Host; MaxConnectionAge; readiness; BuildTelemetry, Faults and TrustScenario; and main.

The wallet’s settings

Everything the wallet’s Init reads, in one struct:

type settings struct {
	GRPC            grpcserver.Settings
	Probes          grpchealth.Settings
	Ledger          grpcclient.Settings
	KeySet          keySetSettings
	JWT             jwt.Settings
	FaultTrustScope string
}

type keySetSettings struct {
	URL    *url.URL
	Client httpclient.Settings
}

config.Struct sources and checks each one. A bad one makes Init fail, and the wallet exits 78, naming its variable:

func readSettings(ctx context.Context, s *humus.Setup) (settings, error) {
	env := func(name string) config.Value[string] { return config.Env(s.Process.Environment, name) }
	file := func(name string) config.Value[[]byte] { return env(name).Parse(readFile(s.Host.FS)) }
	pemFile := func(name string) config.Value[*tls.Config] { return env(name).Parse(pemTLS(s.Host.FS)) }

The ledger’s Client comes first, as Calling the ledger shows, then the key set’s, as Serving the wallet shows, then the tokens’ settings and the server’s:

tokens := jwt.DefaultSettings()
tokens.Issuer = r.Get(env("TOKEN_ISSUER").Validate(jwt.CheckIssuer))
tokens.Audience = r.Get(env("TOKEN_AUDIENCE").Validate(jwt.CheckAudience))

return settings{
	GRPC: grpcserver.Settings{
		Addr: r.Get(env("GRPC_ADDR").Validate(grpcserver.CheckAddr)),
		TLS: &tls.Config{
			GetCertificate: r.Get(pemFile("GRPC_TLS_PEM_FILE").Map(func(c *tls.Config) serverCertificate {
				return c.GetCertificate
			})),
			MinVersion: tls.VersionTLS12,
		},
	},
	Probes: grpchealth.Settings{
		Addr: r.Get(env("PROBES_ADDR").Validate(grpchealth.CheckAddr)),
	},
	Ledger: ledger,
	KeySet: keySetSettings{
		URL:    r.Get(env("TOKEN_KEYS_URL").Parse(parseURL).Validate(checkKeySetURL)),
		Client: keys,
	},
	JWT:             tokens,
	FaultTrustScope: r.Get(env("FAULT_TRUST_SCOPE").Default("").Validate(checkScope)),
}

The key set’s URL is an absolute https URL:

func checkKeySetURL(u *url.URL) error {
	switch {
	case u.Scheme != "https" || u.Host == "":
		return errors.New("must be an absolute https URL")
	case u.User != nil:
		return errors.New("must not hold user info")
	case u.Fragment != "" || u.RawFragment != "":
		return errors.New("must not have a fragment")
	}
	return nil
}

Every variable the wallet reads

VariableWhat it isDefault
GRPC_ADDRthe address it serves onrequired
GRPC_TLS_PEM_FILEa file holding its certificate chain and its key, PEMrequired
PROBES_ADDRthe gRPC health port, in plain textrequired
LEDGER_ADDRthe ledger, host:portrequired
LEDGER_CA_FILEthe CA certificates the ledger is trusted through, PEMrequired
LEDGER_TLS_PEM_FILEa file holding its client certificate for the ledger and its key, PEMrequired
LEDGER_TIMEOUT, LEDGER_MAX_ATTEMPTSgrpcclient.Settings’ fieldsgrpcclient.DefaultSettings()'
TOKEN_KEYS_URLthe JSON Web Key Set that tokens are verified with, an absolute https URLrequired
TOKEN_KEYS_CA_FILEthe CA certificates the key set’s server is trusted through, PEMrequired
TOKEN_KEYS_TIMEOUT, TOKEN_KEYS_MAX_ATTEMPTShttpclient.Settings’ fieldshttpclient.DefaultSettings()'
TOKEN_ISSUER, TOKEN_AUDIENCEthe issuer and the audience a token must haverequired
FAULT_TRUST_SCOPEthe scope a token needs for its scenario to be trustednone
FAULT_MAX_DELAYthe longest delay a scenario injects10s
OTEL_SERVICE_NAMEthe service’s namewallet
OTEL_SDK_DISABLED, OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER, OTEL_EXPORTER_OTLP_ENDPOINTtelemetry.Settings’ fieldstelemetry.DefaultSettings()'

The ledger’s settings

The ledger’s settings, and what it sources to build them:

type settings struct {
	GRPC          grpcserver.Settings
	Probes        grpchealth.Settings
	Database      *pgxpool.Config
	WatchInterval time.Duration
}

type postgres struct {
	URL      *url.URL
	Password string
	CA       *x509.CertPool
}

type sourced struct {
	settings
	ServerTLS *tls.Config
	ClientCAs *x509.CertPool
	Postgres  postgres
}
got, err := config.Struct(func(r *config.Resolver) sourced {
	return sourced{
		settings: settings{
			GRPC: grpcserver.Settings{
				Addr:          r.Get(env("GRPC_ADDR").Validate(grpcserver.CheckAddr)),
				TrustScenario: issuedByTheCA,
			},
			Probes: grpchealth.Settings{
				Addr: r.Get(env("PROBES_ADDR").Validate(grpchealth.CheckAddr)),
			},
			WatchInterval: r.Get(env("WATCH_INTERVAL").Parse(time.ParseDuration).Default(defaultWatchInterval).Validate(positive)),
		},
		ServerTLS: r.Get(env("GRPC_TLS_PEM_FILE").Parse(pemTLS(s.Host.FS))),
		ClientCAs: r.Get(file("GRPC_CLIENT_CA_FILE").Parse(certPool)),
		Postgres: postgres{
			URL:      r.Get(env("POSTGRES_URL").Parse(parsePostgresURL).Validate(checkPostgresURL)),
			Password: r.Get(file("POSTGRES_PASSWORD_FILE").Parse(secret)),
			CA:       r.Get(file("POSTGRES_TLS_CA_FILE").Parse(certPool)),
		},
	}
}).Read(ctx)

Then it builds the server’s TLS, as Serving the ledger shows, and the pool’s config.

Every variable the ledger reads

VariableWhat it isDefault
GRPC_ADDR, GRPC_TLS_PEM_FILE, PROBES_ADDRas the wallet’srequired
GRPC_CLIENT_CA_FILEthe CA certificates its callers’ certificates chain to, PEMrequired
POSTGRES_URLPostgreSQL’s URL, without a password or TLS parametersrequired
POSTGRES_PASSWORD_FILEa file holding PostgreSQL’s passwordrequired
POSTGRES_TLS_CA_FILEthe CA certificates PostgreSQL’s certificate chains to, PEMrequired
WATCH_INTERVALhow often a watch reads its account1s
FAULT_MAX_DELAYas the wallet’s10s
OTEL_SERVICE_NAMEthe service’s nameledger
OTEL_SDK_DISABLED, OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTER, OTEL_EXPORTER_OTLP_ENDPOINTas the wallet’sas the wallet’s

PostgreSQL’s URL

The URL may hold no password and no TLS parameter, since Init sets them:

var postgresParams = []string{"password", "passfile", "service", "servicefile", "sslmode", "sslrootcert", "sslcert", "sslkey", "sslpassword"}
q := u.Query()
for _, p := range postgresParams {
	if q.Has(p) {
		errs = append(errs, fmt.Errorf("must not set %s: Init sets it", p))
	}
}
return errors.Join(errs...)

poolConfig adds the password from its file, verifies PostgreSQL’s certificate against the CA, and sets every TLS parameter, so pgx reads nothing from the real environment or ~/.pgpass:

func poolConfig(p postgres) (*pgxpool.Config, error) {
	u := *p.URL
	u.User = url.UserPassword(p.URL.User.Username(), p.Password)
	q := u.Query()
	q.Set("sslmode", "verify-full")
	for _, name := range []string{"sslrootcert", "sslcert", "sslkey"} {
		q.Set(name, "")
	}
	u.RawQuery = q.Encode()

	cfg, err := pgxpool.ParseConfig(u.String())
	if err != nil {
		return nil, &config.Error{Name: "POSTGRES_URL", Err: errors.New("must be a PostgreSQL URL that pgx accepts")}
	}
	cfg.ConnConfig.TLSConfig = &tls.Config{
		RootCAs:    p.CA,
		ServerName: u.Hostname(),
		MinVersion: tls.VersionTLS12,
	}
	cfg.ConnConfig.Fallbacks = nil
	return cfg, nil
}

Its error quotes nothing of the URL, so no password reaches a log.

The pool, through the Host

The pool dials through the Host, and leaves the host unresolved so the Host’s Dial resolves it too. OnStop closes it:

func openPostgres(ctx context.Context, s *humus.Setup, cfg *pgxpool.Config) (server.Store, func(context.Context) error, error) {
	cfg.ConnConfig.DialFunc = s.Host.Dial
	cfg.ConnConfig.LookupFunc = func(_ context.Context, host string) ([]string, error) {
		return []string{host}, nil
	}
	pool, err := pgxpool.NewWithConfig(ctx, cfg)
	if err != nil {
		return nil, nil, err
	}
	s.OnStop(func(context.Context) error {
		pool.Close()
		return nil
	})
	return store.New(pool, s.Faults), pool.Ping, nil
}

The secrets, as files

PostgreSQL’s password, the private keys and the CAs are files mounted from Secrets, each read through the Host’s FS, so the tests can give the service files of their own:

func readFile(fsys humus.FS) func(name string) ([]byte, error) {
	return func(name string) ([]byte, error) {
		f, err := fsys.Open(name)
		if err != nil {
			return nil, err
		}
		defer func() { _ = f.Close() }()
		return io.ReadAll(f)
	}
}
func secret(b []byte) (string, error) {
	s := strings.TrimRight(string(b), "\r\n")
	if s == "" {
		return "", errors.New("the file is empty")
	}
	return s, nil
}
func certPool(pem []byte) (*x509.CertPool, error) {
	roots := x509.NewCertPool()
	if !roots.AppendCertsFromPEM(pem) {
		return nil, errors.New("no PEM certificate in the file")
	}
	return roots, nil
}

The certificates, read again when renewed

Each certificate and its key are one PEM file, read by internal/certfile:

func pemTLS(fsys humus.FS) func(name string) (*tls.Config, error) {
	return func(name string) (*tls.Config, error) {
		return certfile.Config(fsys, name)
	}
}

Config reads the file once, so a bad one fails Init. Its tls.Config serves the certificate, and presents it as a client certificate:

func Config(fsys humus.FS, name string) (*tls.Config, error) {
	f := &file{fs: fsys, name: name}
	if _, err := f.certificate(); err != nil {
		return nil, err
	}
	return &tls.Config{
		GetCertificate: func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
			return f.certificate()
		},
		GetClientCertificate: func(*tls.CertificateRequestInfo) (*tls.Certificate, error) {
			return f.certificate()
		},
	}, nil
}

A handshake reads the file again, and parses it only when it changed, so a renewal needs no restart:

func (f *file) certificate() (*tls.Certificate, error) {
	data, err := readFile(f.fs, f.name)
	if err != nil {
		return nil, err
	}
	if cert := f.cached(data); cert != nil {
		return cert, nil
	}

	cert, err := tls.X509KeyPair(data, data)
	if err != nil {
		return nil, fmt.Errorf("certificate and key %s: %w", f.name, err)
	}

	f.mu.Lock()
	defer f.mu.Unlock()
	f.pem, f.cert = data, &cert
	return &cert, nil
}

MaxConnectionAge

Both servers close every connection with a GOAWAY once it is about a minute old, so each caller dials again through the Kubernetes Service and reaches whichever pod it picks:

svc, err := grpcserver.NewService(s, set.GRPC,
	unary,
	stream,
	grpc.KeepaliveParams(keepalive.ServerParameters{MaxConnectionAge: maxConnectionAge}),
)
if err != nil {
	return nil, err
}
const maxConnectionAge = time.Minute

No MaxConnectionAgeGrace is set, because a grace would end every open watch each time it ran out. A watch keeps its old connection until it ends, so watches move only at a rollout.

Readiness

Each service’s probes are grpchealth.NewService, on PROBES_ADDR. The ledger is ready once it has reached PostgreSQL, as Serving the ledger shows:

probes, err := grpchealth.NewService(s, set.Probes, health.Readiness(newReached(ping)))

The wallet is ready once its Client has connected to the ledger, as Calling the ledger shows:

probes, err := grpchealth.NewService(s, set.Probes, health.Readiness(ledger.Connected()))

Both latch.

TrustScenario

The wallet trusts a call’s failure-injection scenario when its bearer token verifies and its scopes hold FAULT_TRUST_SCOPE. With no scope set, production trusts no one:

if set.FaultTrustScope != "" {
	set.GRPC.TrustScenario = trustTokens(s.Faults, tokens.Verify, set.FaultTrustScope)
}
func trustTokens(faults fault.Injector, verify func(context.Context, string) (server.Claims, error), scope string) func(context.Context, string) bool {
	return func(ctx context.Context, _ string) bool {
		md, _ := metadata.FromIncomingContext(ctx)
		values := md.Get("authorization")
		if len(values) != 1 {
			return false
		}
		scheme, token, ok := strings.Cut(values[0], " ")
		if !ok || !strings.EqualFold(scheme, "Bearer") {
			return false
		}
		stripped, _ := faults.Strip(ctx)
		claims, err := verify(stripped, strings.TrimSpace(token))
		return err == nil && claims.HasScope(scope)
	}
}

The token is verified with the scenario stripped, since the scenario isn’t trusted yet.

The ledger trusts a caller whose client certificate the internal CA issued: the wallet, which forwards its scenarios.

func issuedByTheCA(ctx context.Context, _ string) bool {
	p, ok := peer.FromContext(ctx)
	if !ok {
		return false
	}
	info, ok := p.AuthInfo.(credentials.TLSInfo)
	return ok && len(info.State.VerifiedChains) > 0
}
TrustScenario: issuedByTheCA,

BuildTelemetry

The telemetry Builder names the service wallet, and sources the OTLP endpoint and the exporters from the standard OTEL_* variables. The ledger’s names it ledger:

func BuildTelemetry(ctx context.Context, p bedrock.Process, dial telemetry.Dialer) (telemetry.SDK, error) {
	d := telemetry.DefaultSettings()
	env := func(name string) config.Value[string] { return config.Env(p.Environment, name) }
	exporter := func(name string, def telemetry.Exporter) config.Value[telemetry.Exporter] {
		return env(name).
			Map(func(s string) telemetry.Exporter { return telemetry.Exporter(s) }).
			Default(def).
			Validate(telemetry.CheckExporter)
	}

	s, err := config.Struct(func(r *config.Resolver) telemetry.Settings {
		s := d
		s.Disabled = r.Get(env("OTEL_SDK_DISABLED").Parse(strconv.ParseBool).Default(d.Disabled))
		s.ServiceName = r.Get(env("OTEL_SERVICE_NAME").Default("wallet").Validate(telemetry.CheckServiceName))
		s.Traces = r.Get(exporter("OTEL_TRACES_EXPORTER", d.Traces))
		s.Metrics = r.Get(exporter("OTEL_METRICS_EXPORTER", d.Metrics))
		s.Logs = r.Get(exporter("OTEL_LOGS_EXPORTER", d.Logs))
		s.OTLP.Endpoint = r.Get(env("OTEL_EXPORTER_OTLP_ENDPOINT").Default(d.OTLP.Endpoint).Validate(telemetry.CheckOTLPEndpoint))
		return s
	}).Read(ctx)
	if err != nil {
		return telemetry.SDK{}, err
	}
	return telemetry.New(ctx, s, dial, p.Stdout)
}

Faults

Both services decode scenarios with jsonscenario, under the member acme.fault, and cap a delay at FAULT_MAX_DELAY:

const scenarioKey = "acme.fault"

func Faults(ctx context.Context, s *humus.Setup) (fault.Settings, error) {
	codec, err := jsonscenario.New(scenarioKey)
	if err != nil {
		return fault.Settings{}, err
	}
	d := fault.DefaultSettings()
	return config.Struct(func(r *config.Resolver) fault.Settings {
		f := d
		f.Decoder = codec
		f.MaxDelay = r.Get(config.Env(s.Process.Environment, "FAULT_MAX_DELAY").Parse(time.ParseDuration).Default(d.MaxDelay).Validate(fault.CheckMaxDelay))
		return f
	}).Read(ctx)
}

Init

The ledger’s Init passes the function that opens the Store to InitWith, which the tests call with one of their own:

func Init(ctx context.Context, s *humus.Setup) ([]humus.Service, error) {
	return InitWith(ctx, s, openPostgres)
}

type OpenStore func(ctx context.Context, s *humus.Setup, cfg *pgxpool.Config) (st server.Store, ping func(context.Context) error, err error)
set, err := readSettings(ctx, s)
if err != nil {
	return nil, err
}

st, ping, err := open(ctx, s, set.Database)
if err != nil {
	return nil, err
}
ledger, err := server.New(st, set.WatchInterval, s.Telemetry)
if err != nil {
	return nil, err
}

Then it builds grpcauth’s interceptors, the Service and the probes, and returns the Service and the probes. The wallet’s Init reads its settings, builds the ledger’s Client, the key set and the Verifier, then the same.

main

Each main passes its Init, BuildTelemetry and Faults to grpcserver.Main. The wallet’s:

func main() {
	grpcserver.Main(app.Init, humus.WithTelemetry(app.BuildTelemetry), humus.WithFaults(app.Faults))
}

The ledger’s is the same line, with its own app, example/grpc/wallet/ledger/app.