ADR-013: Kubernetes Deployment — Verifying Pessimistic Locking Across Pods

Context

The project needs to prove it can be deployed on Kubernetes, and in particular verify that pessimistic locking (against overdraft, see ADR-007) works correctly when several application instances run in parallel (replicas > 1).

Options Considered

Deployment scope

• Option A (chosen) — deploy only the Java app to K8s; Postgres/Oracle/Kafka/Redis stay in docker-compose. Reason: most real systems (banks included) use managed services for the data layer rather than running complex StatefulSets themselves; every skill to demonstrate (deployment, probes, service, scaling, verifying distributed locking) is fully covered with just the app in K8s.
• Option B — StatefulSets for the whole stack (DB/Kafka/Redis included) in K8s. Not chosen because it adds operational complexity that this ADR's goal does not need.

Local K8s tooling

• Docker Desktop Kubernetes (chosen) instead of minikube/kind — it shares the Docker engine/image cache with docker build, so there is no separate step to load images into the cluster.

Decision

Deployment + Service + ConfigMap + Secret + Namespace for the Java app (namespace core-banking, 3 replicas, image core-banking-system:local with imagePullPolicy: Never because it is built locally without a registry).

Before deploying, a REST controller (AccountController) had to be added — the codebase previously had no HTTP endpoint at all, and AccountService was only called in-process from tests. There is no way to verify pessimistic locking across real Pods without an HTTP entry point (several Pods means several separate JVMs/processes, which cannot be called in-process like the existing tests).

Actuator (spring-boot-starter-actuator) was added for liveness/readiness probes. The startupProbe matters because the app needs time to start (connecting to DB/Kafka/Redis) before it can take traffic — it prevents the liveness probe from killing a Pod mid-startup.

Real technical issues encountered

1. Kafka advertised listener: localhost is only correct when the app runs on the host, not inside a container/Pod (inside a container, localhost is the container itself, not the host). Fixed by adding a second listener (PLAINTEXT_HOST, advertised via host.docker.internal, separate port 9094) alongside the existing one (PLAINTEXT, port 9092, still used by processes running directly on the host such as mvn spring-boot:run/the test suite).
2. The Dockerfile uses the official Maven image with a pinned version instead of ./mvnw — because maven-wrapper.jar is gitignored, avoiding a dependency on a file that is missing in a clean git clone.

Experimental verification (the most important part of this ADR)

Scale to 3 replicas and fire 5 concurrent withdraw requests (funds for exactly 1 to succeed) through the Service — verified with two separate layers of evidence:

• (a) Correct business result: 1 request succeeds, 4 are rejected with HTTP 409 (IllegalStateException → CONFLICT), the final balance is exact, never negative, never double-debited.
• (b) Requests really spread across ALL 3 different Pods — confirmed with kubectl logs --prefix, logs tagged with the specific Pod name (AccountController reads HOSTNAME — the environment variable K8s sets to the Pod name — to log clearly which Pod handled which request). This includes a case where one Pod received 2 requests at once and serialized them correctly. This is evidence that the lock lives in the DB (independent of which Pod/JVM calls it), not that K8s happened to route every request to one place.

Consequences

Positive:

• Experimental evidence (not just theoretical reasoning) that the overdraft invariant holds at the scale of a distributed, multi-instance system.

Known limitation (not a hidden bug):

• OutboxEventPublisher has no distributed lock when several Pods run @Scheduled at the same time — the 3 PoC consumers (Notification/Fraud Detection/Reporting) may log duplicates if several Pods pick up the same unpublished OutboxEvent. AuditComplianceConsumer is unaffected because it is already idempotent via the processed_events table (see ADR-009).

Related

• ADR-007 — the pessimistic locking mechanism re-verified at multi-Pod scale in this ADR.
• ADR-009 — OutboxEventPublisher and the idempotent consumer, related to the known limitation with several Pods.