Secrets and config¶

Every workload on the dev2.mycashless.com cluster reads its secrets and runtime config from environment variables populated by a Kubernetes Secret (secrets) or ConfigMap (non-secret config). Secret payloads are not stored in git — they live in GCP Secret Manager and are projected into the cluster by the External Secrets Operator (ESO) via the ClusterSecretStore owned by services/cashless-secrets/.

Never log secrets. Never commit secrets.

Secret values stay in GCP Secret Manager and the materialised K8s Secret. Do not echo them into structured logs, traces, error messages, or git. .env files are gitignored — do not weaken that.

The flow¶

flowchart LR
    A[Engineer / Ops] -->|gcloud secrets create| B[(GCP Secret Manager<br/>mycashless-pa-development)]
    C[ClusterSecretStore<br/>gcp-secret-manager] -.->|Workload Identity| B
    D[ExternalSecret CR<br/>per-service chart] -->|secretStoreRef| C
    D -->|refreshInterval 1h| E[K8s Secret<br/>cashless namespace]
    E -->|envFrom: secretRef| F[Pod / Deployment / Job]
    F -->|Pydantic Settings validate| G([Service boots])

Engineer creates the secret payload in GCP Secret Manager (gcloud secrets create / gcloud secrets versions add).
ESO authenticates to GCP via Workload Identity — the GSA eso-secret-accessor@mycashless-pa-development.iam.gserviceaccount.com holds roles/secretmanager.secretAccessor on the project.
The per-service Helm chart ships an ExternalSecret CR pointing at the gcp-secret-manager ClusterSecretStore. ESO resolves each remoteKey and writes a K8s Secret of the chart's fullname in the cashless namespace.
The Deployment / Job mounts that Secret via envFrom: - secretRef, so the resolved values arrive in the pod as plain environment variables.
The service's Pydantic Settings validates required keys at startup — missing required fields raise before the HTTP server binds.

`services/cashless-secrets`¶

In-tree chart (parent repo, not a submodule) that ships one CR: the ClusterSecretStore named gcp-secret-manager. Every per-service ExternalSecret references that store via secretStoreRef.name: gcp-secret-manager.

File	Purpose
`charts/cashless-secrets/values.yaml`	Backend selector (`gcp` \| `aws` \| `vault`) + GCP project / cluster identity.
`charts/cashless-secrets/templates/secretstore.yaml`	Renders the `(Cluster)SecretStore` for the selected provider.
`services/cashless-secrets/README.md`	Provider details + AWS / Vault swap path.

The ArgoCD Application deploy/argocd/cashless-secrets.yaml runs at sync-wave -2 so the store exists before any service's ExternalSecret (default wave 0) tries to dereference it. cashless-migrate pins its ExternalSecret to wave -3 so the K8s Secret materialises before the migration Job fires at -1.

Adding a new secret¶

Create in GCP SM (project: mycashless-pa-development):

echo -n "<value>" | gcloud secrets create cashless-prod-<key> \
  --project=mycashless-pa-development \
  --data-file=- --replication-policy=automatic

In-place rotation: gcloud secrets versions add cashless-prod-<key> --data-file=<file>.

Declare the mapping in the consuming chart's values.yaml:

externalSecret:
  enabled: true
  dataFrom:
    - secretKey: MY_NEW_ENV_VAR        # env var the service reads
      remoteKey: cashless-prod-<key>   # GCP SM secret name

Surface the env var in the matching Settings class so it is validated at startup.
Commit to cashless-v2. ArgoCD reconciles within ~3 minutes; ESO writes the K8s Secret on next refresh.

ESO's GSA must have secretAccessor on the new key

If kubectl describe externalsecret <name> shows SecretSyncedError, the usual cause is missing IAM. Grant roles/secretmanager.secretAccessor to eso-secret-accessor@mycashless-pa-development.iam.gserviceaccount.com on the project (or on the specific secret resource).

Refresh interval¶

Every ExternalSecret ships with refreshInterval: 1h — ESO re-pulls upstream payloads hourly. A new version in GCP SM reaches pods within an hour without intervention.

To force an immediate refresh:

kubectl annotate externalsecret <name> -n <ns> \
  force-sync=$(date +%s) --overwrite

Refreshing the Secret does NOT restart pods

Env vars are projected once at container start. After a rotation that must take effect immediately: kubectl rollout restart deploy/<name> -n cashless.

Config validation¶

Every Python service validates its env at startup with Pydantic Settings. Missing required keys raise on Settings() construction, so the pod fails its readiness probe rather than booting half-configured.

Service	Settings module	Required env (excerpt)
`cashless-auth`	`auth/config.py`	`DATABASE_URL`, `JWT_SECRET_KEY`
`cashless-backend` (cm_v2)	`apiserver/cm_v2/config.py`	`DATABASE_URL`
`personalaccount-api` (pa_v4)	`pa_v4/config.py`	`DATABASE_URL`

Canonical shape (truncated from cashless-auth):

from pydantic import Field, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_file=".env", env_file_encoding="utf-8", case_sensitive=False,
    )

    # Required — Settings() raises ValidationError if unset.
    DATABASE_URL: str
    JWT_SECRET_KEY: str

    # Optional with safe defaults.
    REDIS_URL: str = "redis://localhost:6379/0"
    CORS_ORIGINS: list[str] = Field(default_factory=list)
    CORS_ALLOW_CREDENTIALS: bool = True

    @model_validator(mode="after")
    def _validate_cors(self) -> "Settings":
        if "*" in self.CORS_ORIGINS and self.CORS_ALLOW_CREDENTIALS:
            raise ValueError("CORS wildcard + credentials is browser-rejected.")
        return self

case_sensitive=False means the field JWT_SECRET_KEY resolves either from JWT_SECRET_KEY or jwt_secret_key — the same class reads from .env locally and from envFrom in-cluster.

TBD: TypeScript / frontend env validation

The Angular frontends compile per-environment config from src/environments/environment.{ts,prod,staging,alpha}.ts at build time — public client config only (API URLs, Firebase web config, Stripe publishable keys). No Zod-style runtime validation today.

Env-var conventions¶

Naming — upper snake case, no service prefix (DATABASE_URL, JWT_SECRET_KEY, STRIPE_WEBHOOK_SECRET).
GCP SM keys — prefixed cashless-prod-<service>-<purpose> (e.g. cashless-prod-cm-database-url, cashless-prod-pa-database-url, cashless-prod-models-field-encryption-key).
.env.example — only the legacy backend/personalaccount-api/.env.example exists, and it reflects the pre-v2 key set. Treat the Settings class as the source of truth and seed .env from there.
What never goes in env — PII (names, emails, phone numbers, ID numbers, CLABE accounts), business data, anything not config-shaped. Customer PII is encrypted at rest via cashless_models.encryption using MODELS_FIELD_ENCRYPTION_KEY (a Fernet key sourced from GCP SM and mounted into every service that imports cashless-models).

Cluster Secret inventory¶

Active ExternalSecret CRs and the env var each materialises. All remoteKey values resolve from project mycashless-pa-development in GCP Secret Manager. Payloads are intentionally not listed.

GCP SM key	Mounted env var	Consuming workload
`cashless-prod-database-url`	`DATABASE_URL`	`cashless-auth`
`cashless-prod-jwt-secret`	`JWT_SECRET_KEY`	`cashless-auth`
`cashless-prod-whatsapp-token`	`WHATSAPP_TOKEN`	`cashless-auth`
`cashless-prod-whatsapp-phone-number-id`	`WHATSAPP_PHONENUMBER_ID`	`cashless-auth`
`cashless-prod-cm-database-url`	`DATABASE_URL`	`cashless-backend-v2`, `cashless-migrate`
`cashless-prod-cm-server-database-url`	`CM_SERVER_DATABASE_CONNECT_URL`	`cashless-worker`
`cashless-prod-cm-local-database-url`	`CM_LOCAL_DATABASE_CONNECT_URL`	`cashless-worker`
`cashless-prod-mongo-uri`	`MONGO_URI`	`cashless-worker`
`cashless-prod-firebase-credentials`	`FIREBASE_CREDENTIALS`	`cashless-worker`
`cashless-prod-pa-database-url`	`DATABASE_URL`	`personalaccount-api-v4`
`cashless-prod-models-field-encryption-key`	`MODELS_FIELD_ENCRYPTION_KEY`	every service importing `cashless-models`
`cashless-prod-redis-password`	composed into `REDIS_URL` (shared `cashless-redis-creds` Secret)	`cashless-auth`, `cashless-backend-v2`, `personalaccount-api-v4`
`cashless-prod-db-backup-credentials` (JSON)	`DATABASE_URL` (property)	`cashless-db-backup` CronJob
`cashless-prod-cron-token`	mounted as file `token`	`cashless-cron` CronJob
`otel-collector-uptrace-dsn`	Uptrace DSN	`otel-collector`
`argocd-image-updater-github-{app-id,installation-id,private-key}`	git write-back creds	`argocd-image-updater`

TBD: Stripe / Adyen / PayPal / MercadoPago credentials

pa_v4 Settings declares STRIPE_WEBHOOK_SECRET, ADYEN_WEBHOOK_HMAC_KEY, PAYPAL_CLIENT_ID / _SECRET / _WEBHOOK_ID, MERCADO_PAGO_WEBHOOK_SECRET_KEY, and MERCADO_PAGO_ACCESS_TOKEN, but the personalaccount-api-v4 chart's externalSecret.dataFrom only wires DATABASE_URL + MODELS_FIELD_ENCRYPTION_KEY today. Per-event processor credentials flow through EventAccount.get_processor_config(...) (DB-resident, not env). Same applies to SENTRY_DSN across every chart — declared but commented out pending the SM key.

TBD: Keycloak admin credentials

Keycloak appears in the global stack defaults but is not deployed as an in-cluster Application today. No ExternalSecret references a Keycloak admin secret on cashless-v2.

Local development¶

Each Python service reads a .env file from its working directory via SettingsConfigDict(env_file=".env"). To seed it:

cd services/cashless-auth
touch .env
$EDITOR .env                # add DATABASE_URL, JWT_SECRET_KEY, ...
uv pip install -e ".[dev]"
uvicorn auth.main:app --reload

For values that must match the dev cluster (e.g. a working MODELS_FIELD_ENCRYPTION_KEY), either:

Ask a teammate who already has a working .env, or

Pull from GCP SM if you have secretmanager.secretAccessor:

gcloud secrets versions access latest \
  --project=mycashless-pa-development \
  --secret=cashless-prod-models-field-encryption-key