Observability¶

OpenTelemetry end-to-end: browsers, FastAPI services, and (where wired) background workers push OTLP into a cluster-local collector, which fans out to Uptrace (traces + metrics + logs) and Cloud Trace (traces only). kube-prometheus-stack scrapes pod-shaped metrics directly. This page goes deeper than the one-paragraph summary on the Architecture page.

Topology¶

flowchart LR
    subgraph Clients
        A1[admin-ui]
        A2[refund-ui]
        A3[marketplace-ui]
        A4[pa-ui]
    end
    subgraph Backends
        B1[cashless-auth]
        B2[cm_v2]
        B3[pa_v4]
        B4[cashless-worker]
    end
    A1 & A2 & A3 & A4 -- OTLP/HTTP :4318 --> C[otel-collector<br/>otelcol-contrib 0.110.0]
    B1 & B2 & B3 & B4 -- OTLP/gRPC :4317 --> C
    C -- OTLP/gRPC --> U[Uptrace<br/>traces + metrics + logs]
    C -- googlecloud exporter --> G[Cloud Trace]
    subgraph kps[kube-prometheus-stack]
        P[Prometheus<br/>15d retention]
        AM[AlertManager]
        GR[Grafana]
    end
    SM[ServiceMonitor + PodMonitor CRs] -. scrape .-> P
    P --> AM
    P --> GR
    C -. /metrics :8888 .-> P

The collector lives in the monitoring namespace (2 replicas) and is the only sink every signal-producer talks to. Browsers reach it over OTLP/HTTP on otelcol.observability:4318; in-cluster pods use OTLP/gRPC on :4317. The collector's own internal metrics are exposed on :8888 for Prometheus to scrape.

What each surface emits¶

Producer	Traces	Metrics	Logs
`cashless-admin-ui`	document-load, fetch, XHR, custom spans, `recordException`	— (RUM)	— (browser console)
`cashless-refund-ui`	same as admin-ui	—	—
`marketplace-user-ui`	same as admin-ui	—	—
`personalaccount-ui`	same as admin-ui	—	—
`cashless-auth`	FastAPI, httpx, SQLAlchemy, Redis (via OTel instrumentors)	OTel auto-metrics (`http_server_duration_*`)	structlog JSON to stdout
`cm_v2` (cashless-backend)	FastAPI, httpx, SQLAlchemy	OTel auto-metrics	structlog JSON to stdout
`pa_v4` (personalaccount)	FastAPI, httpx, SQLAlchemy	OTel auto-metrics	structlog JSON to stdout
`cashless-worker`	!!! note "TBD — no observability module wired yet"	—	stdlib logging
`cashless-mysql`	—	mysqld_exporter via PodMonitor	—
`cashless-redis`	—	redis-exporter via ServiceMonitor (`metrics.serviceMonitor.enabled`)	—

Browser RUM¶

Every Angular 21 frontend ships an identical core/observability/ module. The four resource attributes are the same shape, only service.name differs (cashless-admin-ui, cashless-refund-ui, cashless-marketplace-ui, cashless-pa-ui).

What it ships:

DocumentLoadInstrumentation — page-load timing (FP, FCP, resource timing).
FetchInstrumentation — every fetch() with W3C traceparent propagation to the backend so the trace continues server-side.
XMLHttpRequestInstrumentation — same, for legacy XHR paths.
OtelErrorHandler — Angular's global ErrorHandler is replaced; every unhandled error calls recordException(err) which sets the exception.type / exception.message / exception.stacktrace attributes per OTel semantic conventions and creates a synthetic parent span if none is active.
LoggerService.error() and .warn() also funnel through recordException so structured log errors reach the collector.

Sampling: 100%, head-sampled. No Sampler is configured on the WebTracerProvider, so the SDK default (AlwaysOn via the parent-based sampler) applies. Tail-sampling on the collector is explicitly deferred — see the comment block at the top of templates/configmap.yaml.

Endpoint (per environment.*.ts):

Env	`otel_exporter_otlp_endpoint`	`otel_environment`
development	`http://localhost:4318`	`development`
staging	`http://otelcol.observability:4318`	`staging`
production	`http://otelcol.observability:4318`	`production`

Backend traces¶

cashless-auth, cm_v2, and pa_v4 all ship a observability.py module that wires three things at startup: structlog (always on), Sentry (opt-in via SENTRY_DSN), and OpenTelemetry (opt-in via OTEL_EXPORTER_OTLP_ENDPOINT). The wiring is deliberately parallel across services — same resource attributes, same instrumentors, same JSON log shape.

Auto-instrumentation registered on every backend:

Instrumentor	What it captures
`FastAPIInstrumentor`	One server span per HTTP request (excl. `/health`).
`HTTPXClientInstrumentor`	Outbound HTTP from `httpx` clients (Stripe, MP, Adyen, PayPal).
`SQLAlchemyInstrumentor`	One span per SQL statement, bound to the lazy engine.
`RedisInstrumentor`	`cashless-auth` only (auth-claims cache, OTP store).

Resource attributes stamped on every span:

service.name           = cashless-auth | cm_v2 | pa_v4
service.namespace      = cashless
deployment.environment = development | staging | production

The collector additionally stamps cluster.name=mycashless-development and deployment.environment via the resource processor.

Adding a custom span — canonical pattern from the browser README (backend services do not yet have an explicit get_tracer() callsite — all spans are auto-instrumented today):

import { otelTracer } from './core/observability/instrumentation';
import { SpanStatusCode } from '@opentelemetry/api';

const span = otelTracer.startSpan('my-operation');
try {
  // ... work ...
} catch (err) {
  span.recordException(err as Error);
  span.setStatus({ code: SpanStatusCode.ERROR });
  throw err;
} finally {
  span.end();
}

For Python, use the standard OTel SDK already on each service:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("my-operation") as span:
    span.set_attribute("event_id", event_id)
    ...  # work

Logging conventions¶

All three FastAPI services use structlog with JSON output to stdout. The wiring lives in each service's observability.py::_configure_structlog. Pino is not used — that's the stack-defaults aspiration for greenfield TS services; today every backend is Python.

Every request gets a request-scoped log binding via the _request_logging_middleware in each service's main.py. The middleware reads X-Request-Id from the request (or mints a UUID4 hex), binds it to the structlog contextvars for the duration of the request, and echoes it back in the response header.

Fields every log line carries (structlog processors + middleware):

Field	Source
`timestamp`	`structlog.processors.TimeStamper(fmt="iso", utc=True)`
`level`	`structlog.processors.add_log_level`
`event`	The log message itself
`request_id`	`_request_logging_middleware` (`X-Request-Id` or UUID4)
`method`	Bound by `_request_logging_middleware`
`path`	Bound by `_request_logging_middleware`
`status_code`	On the final `request` access log line
`elapsed_ms`	On the final `request` access log line

Never log tokens or PII

Tokens, passwords, OTP codes, Stripe / Mercado Pago / Adyen / PayPal secrets, and raw email / phone / IP must not appear in log payloads. cashless-auth ships auth/utils/redaction.py — hash_email, hash_phone, hash_ip — that returns a 64-bit HMAC-SHA-256 truncated digest keyed by settings.PII_HASH_SALT. Operators correlate hashed log lines by re-hashing the DB value, not by reversing the hash. Sentry's send_default_pii=False is set on all three services but covers Sentry events only, not structlog.

Metrics¶

No backend service exposes a Prometheus /metrics endpoint today. The FastAPI instrumentor emits OTel metrics (http_server_duration_*, http_server_active_requests) that ride the same OTLP pipeline as traces and land in Uptrace. Backend metrics are queried via Uptrace's Prometheus-compatible API — see e2-traffic-monitoring.md for the exact PromQL shape and auth.

What kube-prometheus-stack actually scrapes today:

Source	How
`otel-collector` self-metrics	`:8888` (gated by `serviceMonitor.enabled` — currently off)
`cashless-mysql` (Percona)	mysqld_exporter sidecar via operator-supplied PodMonitor
`cashless-redis`	redis-exporter ServiceMonitor (`metrics.serviceMonitor.enabled: true`)
Node / kube-state-metrics	Bundled exporters in `kube-prometheus-stack`

Aspirational custom OTel metrics documented in slos.md but not yet emitted by code:

cashless_auth_token_mint_duration_seconds
cashless_pa_v4_payment_external_latency_seconds{processor=...}
cashless_cm_v2_report_query_duration_seconds{report_type=...}

These land alongside the per-service Grafana dashboards (see below).

Dashboards¶

TBD — no Grafana dashboard JSON checked into the cluster-bootstrap chart yet

services/cluster-bootstrap/ ships ESO + GHCR + repo-creds bootstrap only — no grafana/ provisioning subfolder exists today. Grafana is deployed by the upstream kube-prometheus-stack chart with defaultDashboardsEnabled: true, which bundles Kubernetes / Node / Prometheus dashboards. Custom cashless business dashboards are pending under Phase 4 (per slos.md § Current gap).

Access Grafana via port-forward — ingress is intentionally off:

kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80

Admin credentials live in the grafana-admin Secret materialised by ESO from GCP Secret Manager keys grafana-admin-user / grafana-admin-password. See deploy/argocd/values/kube-prometheus-stack.yaml.

Alerts¶

Where rules live: PrometheusRule CRs in any namespace, picked up by the operator because ruleSelectorNilUsesHelmValues: false is set on Prometheus. The label release: kube-prometheus-stack is the convention every rule must carry.

`PrometheusRule`	What it covers	Source
`cashless-mysql`	Pod down, crashloop, group-replication offline, replica lag, connection saturation, InnoDB buffer-pool thrash, backup staleness / failure	`deploy/manifests/cashless-mysql/prometheusrule.yaml`
kube-prometheus-stack defaults	Kubernetes node / pod / control-plane defaults (`defaultRules.create: true`)	upstream chart

Routing. AlertManager config is inline in deploy/argocd/values/kube-prometheus-stack.yaml: default receiver is pagerduty-default; severity=info is silenced via a null receiver. The PagerDuty integration key is currently a placeholder (<TBD-PD-INTEGRATION-KEY>) — see oncall-rotation.md for who gets paged.

Severity	Channel	Notes
`critical`	PagerDuty (default)	On-call ack ≤ 5 min for sev-1.
`warning`	PagerDuty (default)	!!! note "TBD: Slack-only routing not yet split out"
`info`	Dropped (`null` receiver)	Intentional — silences chatter.

SLOs¶

Published targets live in deploy/runbooks/slos.md.

Service	Availability (30d)	p95 latency	5xx error rate
`cashless-auth`	99.95%	200ms (`/oauth/token`, `/oauth/introspect`, `/jwks`)	< 0.1%
`pa_v4`	99.9%	500ms (payment routes) / 300ms (other)	< 0.5%
`cm_v2`	99.5%	800ms (`/v2/reports/*`) / 300ms (other)	< 1%

SLOs are aspirational pre-launch

The infrastructure to measure them (Uptrace ingestion live per service, Grafana dashboards, burn-rate alert wiring) is pending under Phase 4. Real numbers replace round-number defaults after staging soak. Multi-window multi-burn-rate alerting (Google SRE pattern, 14.4× / 6× / 3× / 1× thresholds) is specified but not yet wired into AlertManager.

Common debug recipes¶

Question	Where to look
Why is request `X` slow?	Uptrace UI → filter by `request_id` from the response `X-Request-Id` header. Drill into child spans (SQL, httpx).
Where did this 500 come from?	Uptrace UI → search `status_code=500` + `path=/v4/...` over the window. Cross-ref structlog `request_error` line by `request_id`.
Frontend exception with no clear stack	Uptrace → filter `service.name=cashless-*-ui`, `exception.type` attribute. Stacktrace is on the `exception.stacktrace` attribute.
Cross-service trace continuity	Browser propagates `traceparent` (`FetchInstrumentation.propagateTraceHeaderCorsUrls: [/.*/]`) → backend FastAPI instrumentor picks it up automatically.
MySQL is crashlooping	AlertManager → `MysqlPodRestarting`. `kubectl -n cashless logs --previous <pod>`. Runbook link on the alert.
Backup hasn't run	AlertManager → `MysqlBackupStale`. Check operator logs + `PerconaServerMySQLBackup` CR status.
Is the OTel pipeline actually receiving spans?	Grafana → otelcol self-metrics on `:8888` (`otelcol_receiver_accepted_spans`, `otelcol_exporter_sent_spans`). See `e2-traffic-monitoring.md` § Pre-flight check.
ArgoCD sync stuck on `otel-collector`	Sync policy on `otel-collector` + `kube-prometheus-stack` is manual by design — see Operations § Known cluster tunings.