Ingress¶

The whole platform sits behind one GCE Ingress on a single pre-reserved global static IP. A single Google ManagedCertificate covers every host on *.dev2.mycashless.com plus the bare dev2.mycashless.com apex. There is no per-service Ingress resource in production — each per-chart Ingress template is disabled in deploy/argocd/values/*.yaml and only exists for local / one-off installs.

Chart lives at services/cashless-ingress/charts/cashless-ingress, deployed by the cashless-ingress ArgoCD Application (deploy/argocd/cashless-ingress.yaml).

Topology¶

flowchart LR
    Internet([Internet]) --> IP["cashless-dev2-ingress<br/>(global static IP)"]
    IP --> LB["GCE Ingress<br/>cashless-dev2"]
    LB -. TLS .- CERT["ManagedCertificate<br/>cashless-dev2-cert"]
    LB -->|admin.dev2| ADMIN["Service<br/>cashless-admin-ui:80"]
    LB -->|refund.dev2| REFUND["Service<br/>cashless-refund-ui:80"]
    LB -->|marketplace.dev2| MKT["Service<br/>marketplace-user-ui:80"]
    LB -->|pa.dev2| PAUI["Service<br/>personalaccount-ui:80"]
    LB -->|auth.dev2| AUTH["Service<br/>cashless-auth:8000"]
    LB -->|api.dev2 /v2| CMV2["Service<br/>cashless-backend-v2:8090"]
    LB -->|api.dev2 /| CM["Service<br/>cashless-backend:8080"]
    LB -->|pa-api.dev2 /v4| PAV4["Service<br/>personalaccount-api-v4:8082"]
    LB -->|pa-api.dev2 /| PA["Service<br/>personalaccount-api:8081"]
    LB -->|dev2 apex| DOCS["Service<br/>cashless-docs:80"]
    ADMIN -.NEG.- ADMINPODS([admin-ui pods])
    CMV2 -.NEG.- CMPODS([cm_v2 pods])
    AUTH -.NEG.- AUTHPODS([auth pods])

Every backend Service is ClusterIP (not NodePort) and annotated cloud.google.com/neg: '{"ingress": true}'. That switches the GCE Ingress controller into container-native load balancing — backends are wired as Network Endpoint Groups bound directly to pod IPs, bypassing kube-proxy. Without that annotation the controller falls back to legacy instance-group LBs which would force every Service to be NodePort.

Host inventory¶

All backends live in the cashless namespace alongside the Ingress (GCE Ingress backends are namespace-local).

Host	Path	Backend Service	Port	Chart
`admin.dev2.mycashless.com`	`/`	`cashless-admin-ui`	80	`frontend/cashless-admin-ui`
`refund.dev2.mycashless.com`	`/`	`cashless-refund-ui`	80	`frontend/cashless-refund-ui`
`marketplace.dev2.mycashless.com`	`/`	`marketplace-user-ui`	80	`frontend/marketplace-user-ui`
`pa.dev2.mycashless.com`	`/`	`personalaccount-ui`	80	`frontend/personalaccount-ui`
`auth.dev2.mycashless.com`	`/`	`cashless-auth`	8000	`services/cashless-auth`
`api.dev2.mycashless.com`	`/v2`	`cashless-backend-v2`	8090	`backend/cashless-backend` (`cm_v2` chart)
`api.dev2.mycashless.com`	`/`	`cashless-backend`	8080	`backend/cashless-backend` (legacy chart)
`pa-api.dev2.mycashless.com`	`/v4`	`personalaccount-api-v4`	8082	`backend/personalaccount-api` (`pa_v4`)
`pa-api.dev2.mycashless.com`	`/`	`personalaccount-api`	8081	`backend/personalaccount-api` (legacy)
`dev2.mycashless.com`	`/`	`cashless-docs`	80	`services/cashless-docs`

api.dev2 and pa-api.dev2 are dual-mounted: the modernized FastAPI service answers under /v2 (or /v4 for personal account) while the legacy Flask service answers under /. GCE Ingress matches longest-prefix first, so a request to /v2/users lands on cashless-backend-v2 and /users falls through to cashless-backend.

The Ingress resource¶

# services/cashless-ingress/charts/cashless-ingress/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cashless-dev2
  namespace: cashless
  annotations:
    kubernetes.io/ingress.global-static-ip-name: cashless-dev2-ingress
    networking.gke.io/managed-certificates: cashless-dev2-cert
    kubernetes.io/ingress.class: gce

Three annotations do the work:

kubernetes.io/ingress.global-static-ip-name — pin the LB to the pre-reserved global static IP. Survives Ingress recreation.
networking.gke.io/managed-certificates — attach the Google-managed cert that covers every host below.
kubernetes.io/ingress.class: gce — bind to the external (internet-facing) GCE Ingress controller, not the in-cluster nginx controller.

Per-host fan-out is templated from values.yaml. See values.yaml.

ManagedCertificate¶

# services/cashless-ingress/charts/cashless-ingress/templates/managed-certificate.yaml
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: cashless-dev2-cert
  namespace: cashless
spec:
  domains:
    # Auto-generated from the same hosts list the Ingress consumes.
    - admin.dev2.mycashless.com
    - auth.dev2.mycashless.com
    - api.dev2.mycashless.com
    - refund.dev2.mycashless.com
    - marketplace.dev2.mycashless.com
    - pa-api.dev2.mycashless.com
    - pa.dev2.mycashless.com
    - dev2.mycashless.com

Google provisions the cert out-of-band via HTTP-01 once DNS resolves to the LB IP and the Ingress is healthy. The cert refreshes automatically on its own schedule — operator action is only required to add a new host (which means re-rendering the template with an extra entry).

Cert propagation can take 15-60 minutes

A freshly added host shows status: Provisioning until Google completes the HTTP-01 challenge. Until then curl https://<new-host> fails TLS validation. Check progress with:

kubectl -n cashless get managedcertificate cashless-dev2-cert -o yaml

The status.domainStatus[].status field flips from Provisioning to Active per-host once each challenge resolves.

The chart's ArgoCD application (deploy/argocd/cashless-ingress.yaml) explicitly ignores status mutations on the ManagedCertificate — Google's controller writes that field continuously and we don't want ArgoCD OutOfSync storms.

BackendConfig¶

Per-service BackendConfig overrides default GCE LB health-check behaviour for the backend Service it binds to. Today only one service ships a BackendConfig:

Service	Health-check path	Why
`cashless-admin-ui`	`/dashboard/`	`GET /` returns a 301 redirect; GCE marks 301 as `UNHEALTHY` → LB 502s. Probing `/dashboard/` returns 200.

Manifest:

# frontend/cashless-admin-ui/charts/cashless-admin-ui/templates/backendconfig.yaml
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: cashless-admin-ui
spec:
  healthCheck:
    type: HTTP
    requestPath: /dashboard/
    port: 80
    checkIntervalSec: 15
    timeoutSec: 5
    healthyThreshold: 1
    unhealthyThreshold: 3

The Service binds it via:

annotations:
  cloud.google.com/backend-config: '{"default": "cashless-admin-ui"}'

Every other service exposes a 2xx on / (frontends serve the SPA shell, backends serve a /health redirect or static OK) so GCE's default "GET / expects 200" probe is sufficient.

TBD — IAP, security policy, session affinity

No service today sets iap, securityPolicy, sessionAffinity, timeoutSec, or connectionDraining on its BackendConfig. Defaults apply everywhere. If we add Cloud Armor or IAP in front of a service, that goes here.

Health-check paths¶

Two distinct probes per service — Kubernetes uses the in-pod readiness / liveness paths; GCE LB uses the BackendConfig path (or falls back to /).

Service	Pod readiness/liveness	GCE LB probe
`cashless-backend-v2`	`/health/ready`, `/health`	`/` (default 200)
`cashless-backend`	TCP probe (no `/health`)	`/` (default 200)
`personalaccount-api-v4`	`/health/ready`, `/health`	`/` (default 200)
`personalaccount-api`	TCP probe	`/` (default 200)
`cashless-auth`	`/health` (both probes)	`/` (default 200)
`cashless-admin-ui`	nginx-default	`/dashboard/` (via BackendConfig)
`cashless-refund-ui`	nginx-default	`/` (default 200)
`marketplace-user-ui`	nginx-default	`/` (default 200)
`personalaccount-ui`	nginx-default	`/` (default 200)
`cashless-docs`	nginx-default	`/` (default 200)

The pod-level paths come from each chart's values.yaml (healthcheck.livenessPath / healthcheck.readinessPath for FastAPI services).

Static IP + DNS¶

The static IP is reserved out-of-band, not via Terraform or Helm:

gcloud compute addresses create cashless-dev2-ingress --global

Once reserved, look up the address with:

gcloud compute addresses describe cashless-dev2-ingress --global \
  --format='value(address)'

Every host listed in the host inventory has an A record on the mycashless.com zone pointing at that address. The bare dev2.mycashless.com apex also resolves to the same IP — its A record was added when cashless-docs joined the host fan-out.

TBD — DNS provider

Records are managed externally (Cloud DNS or registrar zone — verify against gcloud dns managed-zones list --project mycashless-pa-development). There is no Terraform / gcloud dns record-sets script checked into this repo for the dev2 zone.

Adding a new host¶

End-to-end, in order:

Pick a hostname under *.dev2.mycashless.com (e.g. reporting.dev2.mycashless.com).
Create the A record on the mycashless.com zone pointing at the cashless-dev2-ingress static IP. Wait for DNS to propagate (dig +short reporting.dev2.mycashless.com returns the LB IP).

Deploy the backend chart. The Service must be ClusterIP and carry both annotations:

annotations:
  cloud.google.com/neg: '{"ingress": true}'
  # Only if the default `GET /` probe won't return 200:
  cloud.google.com/backend-config: '{"default": "<svc-name>"}'

If the service redirects / or returns non-2xx on the root, ship a BackendConfig (mirror cashless-admin-ui).

Add the host block to services/cashless-ingress/charts/cashless-ingress/values.yaml:

hosts:
  # …existing hosts…
  - host: reporting.dev2.mycashless.com
    paths:
      - path: /
        service: cashless-reporting
        port: 80

The ManagedCertificate template iterates over the same list — no separate cert edit needed.

Push to cashless-v2. ArgoCD picks up the change, re-renders the Ingress + ManagedCertificate, and triggers Google to provision the new cert SAN.
Wait for cert propagation (10-60 min — see warning above). Until then https://<new-host> returns a TLS error; HTTP works immediately once the Ingress reconciler picks up the new rule.

Common ingress debugging¶

Symptom	Likely cause	Check
`404 Not Found` from the LB	Host not in `values.yaml`, or Ingress not yet reconciled. GCE returns its default 404 page (not the pod's).	`kubectl -n cashless get ingress cashless-dev2 -o yaml` — verify `spec.rules` includes the host.
`ERR_CERT_AUTHORITY_INVALID` in browser	`ManagedCertificate` still provisioning for that host.	`kubectl -n cashless get managedcertificate cashless-dev2-cert -o yaml` — look for `Provisioning` vs `Active` in `domainStatus`.
`502 Bad Gateway` on every request	Backend health-check failing. NEG endpoints all `UNHEALTHY`.	Check the GCE health-check log + Service `cloud.google.com/backend-config`. Most common: redirect on `/` (see admin-ui BackendConfig).
`502` intermittent	NEG endpoints stale during rollout, or pod ready but app not yet listening.	Verify pod `readinessProbe` matches the path the app actually serves with 200.
NEG has 0 endpoints	Service selector doesn't match any pod, or pods not `Ready`.	`kubectl -n cashless get endpoints <svc>` — should list pod IPs.
ManagedCertificate stuck `Provisioning`	DNS not yet pointing at the LB IP, or HTTP-01 challenge being blocked.	`dig +short <host>` must return the static IP. Cert provisioning will not start until then.
Old IP still served	An override created a second Ingress (per-chart `ingress.enabled=true`).	`kubectl -n cashless get ingress` — should only show `cashless-dev2`. Per-chart Ingresses must stay disabled in `deploy/argocd/values/*.yaml`.

For end-to-end deploy debugging (CI → GHCR → ArgoCD → rollout) see OPERATIONS.md.