Operations¶

In-cluster ops hub for the cashless platform. This page lifts the most-needed facts out of OPERATIONS.md in the parent monorepo and links back for anything multi-paragraph. Treat OPERATIONS.md as the source of truth; this page is the index.

For the full deploy-flow diagram and CI matrix see Architecture. For per-host URLs see Services.

Cluster topology¶

Item	Value
GKE cluster	`mycashless-development`
Region	`us-central1`
GCP project	`mycashless-pa-development`
Platform workloads namespace	`cashless`
ArgoCD namespace	`argocd`
Image Updater namespace	`argocd-image-updater`
External-Secrets namespace	`external-secrets`
Monitoring namespace	`monitoring` (kube-prometheus-stack, otel-collector)
Image registry	`ghcr.io/cashless-media/*` (private)
Parent repo	`Cashless-Media/platform` on `cashless-v2`
ArgoCD repo auth	GitHub App `ci-cashless` (App ID `3623051`)
ESO secret backend	GCP Secret Manager via Workload Identity

ArgoCD's project model puts every platform workload in the cashless AppProject (deploy/argocd/project.yaml). Cluster-scoped resources (child Applications, the ClusterSecretStore) live in the default project so the root App-of-Apps can manage them.

Connect to the cluster:

gcloud container clusters get-credentials mycashless-development \
  --region us-central1 --project mycashless-pa-development
kubectl -n cashless get pods

What lives where¶

Every entry in deploy/argocd/ is an Application CR. The image column is empty for charts that ship no image of their own (operators, upstream charts, CRD bundles).

Application	Chart path	Image (`ghcr.io/cashless-media/...`)
`cashless-admin-ui`	`frontend/cashless-admin-ui/charts/cashless-admin-ui`	`cashless-admin-ui`
`cashless-refund-ui`	`frontend/cashless-refund-ui/charts/cashless-refund-ui`	`cashless-refund-ui`
`marketplace-user-ui`	`frontend/marketplace-user-ui/charts/marketplace-user-ui`	`marketplace-user-ui`
`personalaccount-ui`	`frontend/personalaccount-ui/charts/personalaccount-ui`	`personalaccount-ui`
`cashless-backend-v2`	`backend/cashless-backend/charts/cashless-backend-v2`	`cashless-backend-v2`
`cashless-worker`	`backend/cashless-backend/charts/cashless-worker`	`cashless-worker`
`personalaccount-api-v4`	`backend/personalaccount-api/charts/personalaccount-api-v4`	`personalaccount-api-v4`
`cashless-auth`	`services/cashless-auth/charts/cashless-auth`	`cashless-auth`
`cashless-migrate`	`core/models/charts/cashless-migrate`	`cashless-migrate`
`cashless-cron`	`services/cashless-cron/charts/cashless-cron`	`cashless-cron`
`cashless-redis`	`services/cashless-redis/charts/cashless-redis`	—
`cashless-db-backup`	`services/cashless-db-backup/charts/cashless-db-backup`	`cashless-db-backup`
`cashless-mysql`	`percona/ps-db` (Percona upstream) + parent values	—
`percona-mysql-operator`	Percona upstream operator chart	—
`cashless-secrets`	`services/cashless-secrets/charts/cashless-secrets`	—
`cashless-ingress`	`services/cashless-ingress/charts/cashless-ingress`	—
`cashless-docs`	`services/cashless-docs/charts/cashless-docs`	`cashless-docs`
`argocd-image-updater`	`services/argocd-image-updater/charts/argocd-image-updater`	—
`kube-prometheus-stack`	upstream + `deploy/argocd/values/kube-prometheus-stack.yaml`	—
`otel-collector`	`services/otel-collector/charts/otel-collector`	upstream `otel/...:0.110.0`

The legacy single-source cashless-backend / personalaccount-api Application CRs also live in deploy/argocd/ for reference; the active surface is *-v2 and *-v4.

ArgoCD app structure¶

Every service Application is multi-source:

A $values ref source pointing at the parent monorepo (ref: values, no path).
The chart source pointing at the submodule chart path.
(Optional) Additional sources for upstream charts.

The chart source layers two value files:

valueFiles:
  - values.yaml                                       # shipped with the chart
  - $values/deploy/argocd/values/<svc>.yaml           # parent monorepo override

This keeps the auto-bumped image.tag in the parent monorepo so submodules are never touched on image updates. See deploy/argocd/cashless-admin-ui.yaml for the canonical shape.

helm.parameters does not survive $values layering

In the multi-source render, ArgoCD applies spec.sources[N].helm.parameters before the layered $values value file. A parameter-only write-back is silently overridden by image.tag in the values file. image.tag must be pinned in deploy/argocd/values/<svc>.yaml — that is the only place that survives selfHeal. The seeded helm.parameters block on each Application exists only to mark the source as Helm-typed so Image Updater doesn't classify it as type=Directory and skip it.

The root App-of-Apps adds matching entries under ignoreDifferences (/spec/sources/0..2/helm/parameters) so selfHeal does not revert Image Updater's writes while it reconciles.

ArgoCD Image Updater¶

Polls GHCR every two minutes for new ghcr.io/cashless-media/<svc>:sha-<12hex> tags. On match, writes image.tag into deploy/argocd/values/<svc>.yaml on cashless-v2 using the ci-cashless GitHub App credentials, then ArgoCD's auto-sync (~3 min) rolls the Deployment.

The annotation shape on each Application:

argocd-image-updater.argoproj.io/image-list:
  app=ghcr.io/cashless-media/<svc>
argocd-image-updater.argoproj.io/app.update-strategy: newest-build
argocd-image-updater.argoproj.io/app.allow-tags: regexp:^sha-[0-9a-f]{12}$
argocd-image-updater.argoproj.io/app.helm.image-name: image.repository
argocd-image-updater.argoproj.io/app.helm.image-tag: image.tag
argocd-image-updater.argoproj.io/write-back-method:
  git:secret:argocd-image-updater/git-creds
argocd-image-updater.argoproj.io/write-back-target:
  helmvalues:../../../../deploy/argocd/values/<svc>.yaml
argocd-image-updater.argoproj.io/git-repository:
  https://github.com/Cashless-Media/platform.git
argocd-image-updater.argoproj.io/git-branch: cashless-v2

When an app stops auto-updating, check in order:

Updater paused / errored — kubectl -n argocd-image-updater logs deploy/argocd-image-updater. Look for "denied" (registry auth) or "skipping app … of type 'Directory'" (Helm typing — confirm helm.parameters is still seeded on the Application).
Annotation drift — compare against cashless-admin-ui.yaml.
CI path filter missed the gitlink — submodule pointer-bumps need the submodule path and the gitlink entry (no wildcard) in the workflow's paths: block; <sub>/** alone won't match the pointer-bump commit.
git-creds not materialised — kubectl -n argocd-image-updater describe externalsecret. If SecretSyncedError, the GCP SM key is missing or ESO's GSA lost secretAccessor.

Background on the wrapper chart + bootstrap: services/argocd-image-updater/charts/argocd-image-updater/README.md.

Day-to-day flows¶

Each one-liner below maps to OPERATIONS.md § Useful commands.

Check ArgoCD app health¶

kubectl get app -n argocd
kubectl get app -n argocd <name> -o jsonpath='{.status.conditions}'

See the live image used by a pod¶

kubectl get pod <pod> -n cashless \
  -o jsonpath='{.spec.containers[0].image}'

Force Image Updater to re-poll¶

kubectl rollout restart deploy/argocd-image-updater \
  -n argocd-image-updater

Force External-Secrets to refresh a secret¶

kubectl annotate externalsecret <name> -n <ns> \
  force-sync=$(date +%s) --overwrite

Port-forward the ArgoCD UI¶

kubectl -n argocd port-forward svc/argocd-server 8080:443
# open https://localhost:8080  (user: admin)

Initial admin password:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

Roll back a service¶

Two equivalent paths — pick whichever is closer:

ArgoCD UI — Application → History → prior revision → Sync.
Git — git revert the Image Updater commit on deploy/argocd/values/<svc>.yaml. ArgoCD picks it up on next sync.

Full rollback procedure: OPERATIONS.md § Rolling back.

Known cluster tunings + limitations¶

Component / area	Tuning or gap
`argocd-repo-server` storage	`ephemeral-storage` raised to 8Gi request + limit. Autopilot's 1Gi default evicted the pod every ~5min while cloning ~19 submodules.
Image Updater + multi-source	Image Updater ≤ 0.16 only treats a source as Helm-typed when `helm.parameters` is present. Seeded `image.repository` / `image.tag` stay even though the values file wins at render time.
CI quality gates	`ruff`, `mypy`, `pytest`, `helm template` all run with `continue-on-error: true` pre-launch. Re-tighten by removing the flag from each reusable workflow once debt is cleared.
Observability sync policy	`kube-prometheus-stack` and `otel-collector` are pre-staged with manual sync so ops can review before they land. Flip `syncPolicy.automated` on after first apply.

Full list with rationale: OPERATIONS.md § Known cluster tunings and OPERATIONS.md § Known limitations.

First-time cluster bootstrap¶

Greenfield clusters (or destructive recreates) come up via services/cluster-bootstrap/.

Order of operations:

Push GitHub App credentials to GCP Secret Manager (services/argocd-image-updater/scripts/bootstrap-secrets.sh).
Push the GHCR pull token (services/cluster-bootstrap/scripts/bootstrap-ghcr-pat.sh).
Run the bootstrap script (services/cluster-bootstrap/scripts/bootstrap.sh).
Apply the post-bootstrap manifests in services/cluster-bootstrap/manifests/ (ESO repo creds, GHCR pull Secret).
Patch every ServiceAccount in cashless with imagePullSecrets: [{name: ghcr-pull}].

After step 5, the root App-of-Apps discovers every child Application and syncs them in sync-wave order — cashless-secrets (wave -2), cashless-migrate + argocd-image-updater (wave -1), then everything else.