Phase 1 — Core Banking (Sub-project B)

Status: [Done]

Full goals for this phase: Roadmap — Phase 1.

Completed work

• Initial dev environment (Java, Maven, Docker Postgres, Flyway) — see Environment setup below.
• Maven build succeeds (BUILD SUCCESS).
• Flyway migrations run successfully, creating the accounts and ledger_entries tables.
• Entity layer: Account, LedgerEntry mapped to the V1 migration schema — see Entity layer.
• LedgerService.recordTransaction() — validates Debit/Credit balance before saving, atomically in one transaction.
• Unit test LedgerServiceTest (3 test cases) — BUILD SUCCESS, Tests run: 3, Failures: 0, Errors: 0.
• AccountService.getBalance() — balance derived from the ledger instead of stored — see AccountService: deriving the balance from the ledger.
• Unit test AccountServiceTest (1 test case) — BUILD SUCCESS, Tests run: 4, Failures: 0, Errors: 0.
• AccountService.withdraw() — pessimistic locking (SELECT ... FOR UPDATE) prevents races on concurrent withdrawals — see Pessimistic locking for withdrawals.
• Race condition unit test withdraw_shouldPreventOverdraft_whenConcurrentRequests — BUILD SUCCESS, Tests run: 5, Failures: 0, Errors: 0.
• (2026-09-16) DDD refactor — Money value object, Transaction aggregate root, and ADR-008 — see DDD refactor.
• (2026-09-17) Outbox Pattern + Kafka event publishing — Transaction raises domain events, 4 independent consumers (Audit/Compliance, Fraud Detection, Notification, Reporting), and ADR-009 — see Outbox Pattern & Kafka Event Publishing.
• (2026-09-17) Redis cache for AccountService.getBalance(), invalidation via AFTER_COMMIT, and ADR-010 — see Redis cache for account balance.
• (2026-09-17) Oracle dual-profile support (45/45 tests passing on both databases) and the fixed Money scale — ADR-011, ADR-012 — see Oracle dual profile & Money scale fix.
• (2026-09-17) REST API + Kubernetes deployment, pessimistic locking verified across 3 real Pods — ADR-013 — see Kubernetes deployment.

Environment setup

The problems met while setting up the dev environment for the first time, in chronological order.

1. Java version mismatch

• Problem: the pom.xml generated by Spring Initializr required Java 21 by default, but the dev machine only had JDK 17.
• Cause: the wrong target Java version was selected when creating the project on Spring Initializr.
• Fix: lowered java.version in pom.xml to 17.

2. Corrupted local Maven cache

• Problem: the build failed with zip file is empty for objenesis-3.3.jar in ~/.m2/repository.
• Cause: the artifact had been partially downloaded/corrupted in the local Maven cache in an earlier run.
• Fix: deleted the objenesis cache folder in ~/.m2/repository so Maven would download it again.

3. Missing DataSource configuration

• Problem: after fixing the Java version, Spring Boot failed to start with Failed to determine a suitable driver class.
• Cause: application.properties had no datasource configuration, so Spring Boot could not tell which JDBC driver to use.
• Fix: added spring.datasource.url, spring.datasource.username, spring.datasource.password pointing to PostgreSQL.

4. Docker port conflict

• Problem: the project's Postgres container would not start / port conflict.
• Cause: the dev machine already had another Postgres container (used for work, taking the default port 5432), and Windows also had PostgreSQL 17 installed as a service — both competing for ports 5432/5433.
• Fix: moved the project's Postgres container to its own port 5434 in docker-compose.yml.

5. Spring Boot 4.x breaking change with Flyway

• Problem: flyway-core was on the classpath but Flyway did not run migrations on startup.
• Cause: Spring Boot 4.x moved Flyway auto-configuration into a separate module; flyway-core on the classpath is no longer enough to trigger it as in earlier Spring Boot versions.
• Fix: replaced flyway-core with spring-boot-starter-flyway.

6. Password authentication failed because of an old Docker volume

• Problem: after the port change in step 4, Postgres still reported password authentication failed even though the password in docker-compose.yml was correct.
• Cause: the old Docker volume had been initialized with the password from a previous run; Postgres only applies the password environment variable when the volume is first initialized, not for an existing volume.
• Fix: ran docker compose down -v to delete the old volume and start a clean container.

7. Invalid "Asia/Saigon" timezone

• Problem: Postgres rejected the connection with FATAL: invalid value for parameter "TimeZone": "Asia/Saigon".
• Cause: the JDBC driver sends the Windows system timezone name (the legacy Asia/Saigon) to the server, and the tzdata bundled in the postgres:16 image does not recognize that name.
• Fix: downgraded the Postgres image to postgres:15, whose tzdata recognizes Asia/Saigon.

Result

After fixing all 7 problems: the Maven build succeeded (BUILD SUCCESS), and Flyway migrations ran and created the accounts and ledger_entries tables.

Entity layer

• Account entity (package account/): id (UUID), accountNumber, accountType, currency, status, createdAt — mapped to the accounts table from migration V1.
• LedgerEntry entity (package ledger/): id, accountId, transactionId, entryType (enum DEBIT/CREDIT), amount (BigDecimal), createdAt — mapped to the ledger_entries table.
• An EntryType enum instead of a raw String for entryType — validated at compile time, matching the CHECK constraint in the migration SQL. @Enumerated(EnumType.STRING) stores the name, not the ordinal.
• BigDecimal for amount, not double/float — avoids floating-point rounding errors in a financial system.

LedgerService: Debit/Credit balance validation

• recordTransaction(List<LedgerEntry>) validates total Debit = total Credit before saving and throws IllegalStateException if they differ.
• @Transactional ensures the whole list of LedgerEntry is saved atomically (all or nothing).
• transactionId is generated inside the service (not accepted from outside) — so every entry in one call belongs to the same logical transaction.
• BigDecimal.compareTo() instead of equals() when comparing Debit/Credit totals — equals() treats 100.0 and 100.00 as different because of scale, compareTo() compares the numeric value.

Unit tests and the bugs they found

LedgerServiceTest has 2 test cases: a balanced transaction must succeed, an unbalanced one must throw. Running them surfaced 2 errors, both caused by invalid test data rather than business logic:

1. Foreign key constraint violation: the first version used random UUIDs as accountId without creating a real Account first — Postgres rejected it because ledger_entries.account_id has REFERENCES accounts(id). This shows the database-level integrity constraint works as designed. Fixed with a helper that creates a real Account before testing LedgerEntry.
2. Value too long for varchar(20): after fixing error 1, the helper built accountNumber from a full UUID (36 characters) while account_number is limited to VARCHAR(20). Fixed by truncating the UUID to its first 8 characters for test account numbers.

Final result: Tests run: 3, Failures: 0, Errors: 0 — BUILD SUCCESS.

AccountService: deriving the balance from the ledger

• Added 2 queries to LedgerEntryRepository: sumDebitByAccountId, sumCreditByAccountId — JPQL with COALESCE(SUM(...), 0) to avoid NULL for accounts with no transactions.
• AccountService.getBalance(accountId) = total Credit - total Debit, the correct accounting rule for customer accounts (liabilities): increases are credits, decreases are debits.
• AccountServiceTest: creates 2 accounts, posts one transaction through LedgerService (100.00 CREDIT to the account, 100.00 DEBIT to the counterparty), and checks that getBalance() returns exactly 100.00.
• Result: Tests run: 4, Failures: 0, Errors: 0 — BUILD SUCCESS. The whole Account/Ledger/Balance foundation of Phase 1 works correctly.
• Wrote an ADR for the design decision: immutable ledger, balance derived rather than stored (see ADR-006).

Pessimistic locking for withdrawals

• Read first: optimistic vs. pessimistic locking in Spring Data JPA.
• Added AccountRepository.findByIdForUpdate() with @Lock(LockModeType.PESSIMISTIC_WRITE) — generates SELECT ... FOR UPDATE (Postgres specifically shows FOR NO KEY UPDATE).
• Added AccountService.withdraw(accountId, counterpartyAccountId, amount): locks the account, checks for sufficient funds, posts a LedgerEntry pair (DEBIT the account, CREDIT the counterparty) through LedgerService.
• Test withdraw_shouldPreventOverdraft_whenConcurrentRequests: 2 threads withdraw 80.00 at the same time from an account holding 100.00, using ExecutorService + CountDownLatch + AtomicInteger to count successes. Confirms exactly 1 of the 2 threads succeeds.
• Result: Tests run: 5, Failures: 0, Errors: 0 — BUILD SUCCESS. Hibernate logs confirm the lock works (the 2 threads are serialized, never running in parallel on the same account).
• Wrote ADR-007 recording why pessimistic rather than optimistic locking was chosen (see ADRs).

DDD refactor: Money value object and Transaction aggregate root

(2026-09-16) Refactored the domain layer towards DDD, in 3 consecutive steps on the same day.

Step 1 — Money value object

• Added Money (package shared/money) — immutable, wraps BigDecimal amount + Currency; add()/subtract() throw CurrencyMismatchException when the two Money values have different currencies.
• LedgerEntry.amount and Account.currency moved from raw BigDecimal/String to Money/Currency.
• LedgerService.validateBalanced() now sums through Money — a transaction mixing currencies across entries fails immediately (via CurrencyMismatchException) instead of silently adding up wrong numbers.
• Migration V3__add_currency_to_ledger_entries.sql adds ledger_entries.currency, backfilled from accounts.currency.
• Tests: MoneyTest (85 lines), updated AccountServiceTest, LedgerServiceTest.

Step 2 — Transaction aggregate root

• Added the Transaction entity (package ledger/) as the aggregate root for a group of LedgerEntry: Transaction.record(entries, createdBy) validates Debit = Credit in the constructor itself; Transaction.reverse(reversedBy) creates a symmetric reversal. Transaction.id is generated (UUID.randomUUID()) in the constructor rather than by the DB via @GeneratedValue, following the DDD principle that an aggregate is responsible for creating itself in a valid state.
• LedgerService became thinner and no longer validates balance itself — that responsibility moved entirely into Transaction.
• Migration V4__create_transactions_table.sql creates the transactions table, backfilled from existing ledger_entries.
• Bug found while debugging: after implementation, Transaction was saved, but the cascade to LedgerEntry did not run — producing empty, incorrect records with no exception thrown (a "silently wrong" bug). Cause: Spring Data JPA assumes an entity already exists in the DB if its ID is non-null when save() is called — since Transaction.id is assigned manually in the constructor, JPA treated it as an update (merge()) instead of a new insert (persist()), so the PERSIST cascade did not fire properly.
• Fix: Transaction implements Persistable<UUID> and defines its own isNew() (via a @Transient boolean isNew field reset to false in @PostLoad/@PostPersist) to tell JPA exactly when the entity is new, whether or not the ID is set. After the fix: 21/21 tests pass and the cascade works.
• Tests: TransactionTest (86 lines).
• The full design decision (options A/B, rationale, the 2 technical issues — MIN(uuid) on Postgres and the Persistable<UUID> pitfall) is recorded in ADR-008.

Step 3 — Test coverage for the bug found

• While building the Transaction aggregate, a bug was found: createdBy was not set correctly on some paths — previously only caught by the DB's NOT NULL constraint in integration tests, with no unit-test assertion catching it earlier.
• Added explicit createdBy assertions on both happy paths (Transaction.record() and Transaction.reverse()) in TransactionTest, so a similar regression fails immediately in unit tests without waiting for the DB.

Outbox Pattern & Kafka Event Publishing

(2026-09-17) Other systems need to be notified when a Transaction is posted or reversed, for 4 business purposes: Audit/Compliance, Notification, Fraud Detection, Reporting. Implemented in 3 steps.

Step 1 — Outbox Pattern

• Transaction (aggregate root) raises domain events (TransactionPostedEvent, TransactionReversedEvent) in record()/reverse() and exposes pullDomainEvents() — consistent with the DDD principle applied in ADR-008.
• LedgerService pulls the events after a successful save(), maps them to OutboxEvent, and stores them in outbox_events in the same DB transaction as Transaction/LedgerEntry — avoiding the dual-write risk (DB write succeeds but the Kafka send fails, or vice versa).
• OutboxEventPublisher (@Scheduled, every 5s by default) reads unpublished events, sends them to the transaction-posted-topic topic (key = transactionId to keep ordering), then sets published_at.
• Self-contained JSON payload: eventId (separate from transactionId, used for idempotency), transactionId, createdBy, occurredAt, entries[], and eventType embedded in the payload (not in a Kafka header).
• Migration V5__create_outbox_events_table.sql.

Step 2 — Audit Compliance Consumer (idempotent)

• AuditComplianceConsumer listens to transaction-posted-topic (groupId audit-compliance-group) and writes to compliance_records — one row per LedgerEntry (not grouped by transaction), because real banking compliance/audit needs to know which specific accounts were involved (debit/credit), not only the transaction total.
• Idempotent via the processed_events table (keyed by eventId) — mandatory because Kafka only guarantees at-least-once delivery and messages may be redelivered.
• Migrations V6__create_processed_events_table.sql, V7__create_compliance_records_table.sql.
• Tests: AuditComplianceConsumerIntegrationTest, TransactionToComplianceEndToEndTest (real end-to-end, Kafka not mocked).

Step 3 — 3 PoC consumers

• FraudDetectionConsumer, NotificationConsumer, ReportingConsumer — each with its own groupId (fraud-detection-group, notification-group, reporting-group) on the same topic, only logging to prove the publish/subscribe architecture works for several independent consumers — no real business logic (no actual fraud rules, notifications or reports).

Technical issues encountered

1. Jackson 3, not Jackson 2. Spring Boot 4.1.1 uses tools.jackson.* — JacksonJsonSerializer must be used instead of JsonSerializer, plus a separate spring-boot-starter-json dependency.
2. Serious trap: Spring Boot loads only the first application.properties it finds and does not merge main + test — a separate application-test.properties overrides the whole DB/Flyway configuration instead of adding to it. Fixed with inline @SpringBootTest(properties = ...).
3. Most important finding: raw spring-kafka is not enough for Spring Boot 4 to auto-configure the KafkaTemplate bean — spring-boot-starter-kafka is needed. The bug stayed hidden through Step 1 because OutboxEventPublisher was always disabled in tests (outbox.publisher.enabled=false), and only surfaced with a real, unmocked end-to-end test.
4. Testcontainers 2.x renamed the artifact testcontainers-kafka (different prefix than 1.x) — causing a confusing "version is missing" error at first.

The full design decision (Context, Options Considered A/B, Decision, Consequences with the 4 issues above) is recorded in ADR-009.

Redis cache for account balance

(2026-09-17) AccountService.getBalance() computes SUM() over every LedgerEntry on each call — inefficient for accounts with many transactions. Added a Redis cache-aside in 4 steps.

Step 1 — Redis setup

• Added a redis service (redis:7-alpine) to docker-compose.yml. Port 6379 was taken by another Redis container on the machine → host port changed to 6380 (same reason Postgres had moved to 5434).
• Added the spring-boot-starter-data-redis dependency. Configured spring.data.redis.host/port and app.cache.balance.ttl-seconds (default 3600) in application.properties.

Step 2 — Cache-aside

• AccountBalanceCache (using StringRedisTemplate) caches only the amount as a string (BigDecimal.toPlainString()), not the currency, which is read from Account and never changes.
• AccountService.getBalance() reads the cache first; on a miss it computes from the DB and writes the result back (cache-aside).
• No cache in withdraw(): a separate private computeBalanceFromDb() computes straight from the DB for the sufficient-funds check in withdraw() — this step runs inside the pessimistic lock, and reading the cache there could return an old balance and defeat the lock (see ADR-007).

Step 3 — AFTER_COMMIT invalidation

• AccountBalanceCacheEvictionListener listens to TransactionPostedEvent/TransactionReversedEvent — the same domain events used for Outbox/Kafka in ADR-009 (LedgerService publishes them both as Outbox content and as an in-process Spring ApplicationEvent), no new event class.
• @TransactionalEventListener(phase = AFTER_COMMIT) instead of evicting directly in the same @Transactional — if the eviction happens before commit, a concurrent read may query the DB (still seeing the old balance because the write has not committed) and write the old balance back into the cache right after the eviction, where it stays until the TTL expires. Evicting AFTER_COMMIT narrows that race window considerably.

Step 4 — Tests

• AccountBalanceCacheTest (unit, 78 lines), AccountServiceCacheTest (unit, 95 lines), AccountBalanceCacheIntegrationTest (144 lines, real Redis via Testcontainers).
• Testcontainers 2.x has no separate testcontainers-redis module (unlike Kafka's testcontainers-kafka) → GenericContainer with the redis:7-alpine image is used directly.
• The existing withdraw() regression and concurrency tests still pass — confirming the cache does not affect the overdraft invariant. A dedicated integration test confirms the balance is never stale after a new transaction.

The full design decision (Context, 2 options considered, Decision, Consequences with the 2 issues above) is recorded in ADR-010.

Oracle dual profile & Money scale fix

(2026-09-17) The goal was to prove the system works with an enterprise database (Oracle is common in Vietnamese banks) without affecting the 45 tests passing on PostgreSQL. Done in 4 steps, with 1 real bug that needed its own fix.

Step 1 — Oracle setup

• Added an oracle service (gvenzl/oracle-free:23-slim-faststart) to docker-compose.yml, host port 1522 (container port 1521 — same reason as the other port changes, avoiding default ports on the dev machine). The Oracle container starts noticeably slower than Postgres/Kafka/Redis even with the slim-faststart image.
• Added the ojdbc11 (JDBC driver) and flyway-database-oracle dependencies.
• application-oracle.properties (profile oracle, enabled with --spring.profiles.active=oracle) overrides the datasource/driver and points spring.flyway.locations=classpath:db/migration/oracle. It runs alongside the default (Postgres) profile rather than replacing it.
• Wrote a new README.md documenting the port mapping, how to run the app/tests per profile, and the migration layout.

Step 2 — Split migrations by vendor & rewrite 7 files for Oracle

• Split db/migration/ into db/migration/postgresql/ (the existing V1-V7, moved unchanged) and db/migration/oracle/ (V1-V7 rewritten for the Oracle dialect).
• 4 main dialect differences:
  1. UUID → RAW(16) for every id/foreign key column.
  2. UPDATE ... FROM (Postgres-only syntax) → a correlated subquery in V3, → MERGE INTO in V4 (backfilling reversal_of_transaction_id).
  3. JSONB → JSON (Oracle 23ai's native JSON type).
  4. Partial index (CREATE INDEX ... WHERE published_at IS NULL) → an equivalent function-based index (CASE WHEN published_at IS NULL THEN created_at END), since Oracle does not support partial indexes.
• Side finding: columnDefinition = "jsonb" on OutboxEvent.java did not need changing for Oracle, because ddl-auto=validate does not use columnDefinition for actual validation — @JdbcTypeCode(SqlTypes.JSON) adapts to the active dialect.

Step 3 — Running the tests on both profiles, finding the Money bug

• Running the existing 45 tests on Oracle (./mvnw test -Dspring.profiles.active=oracle) showed that Oracle NUMBER does not keep a fixed scale on read (unlike Postgres NUMERIC, which keeps the declared scale — standard Oracle behaviour, not a database bug).
• Trying to fix it with Currency.getDefaultFractionDigits() (ISO 4217) revealed a deeper problem: under ISO 4217 VND has 0 decimal places, but the whole domain model (from the Audit Trail to Redis) had implicitly assumed 2 decimal places for VND — following ISO 4217 strictly would throw for VND values with a fractional part that already existed in the tests/data of the 4 previous features.
• Decision: fix scale = 2 for every currency instead of strict ISO 4217. Money.java changed: a dedicated constructor enforces setScale(2, RoundingMode.UNNECESSARY), and every way of creating Money (of, zero, add, subtract) goes through it, so all are normalized automatically.
• The full design decision (2 options considered, rationale, recorded risk for real multi-currency later) is in ADR-012.

Step 4 — Result

• 45/45 tests pass on both databases (Postgres and Oracle), with no profile-specific test changes — the database differences are handled entirely in configuration/migrations and do not leak into the tests.
• The full Oracle dual-profile decision (options considered, 5 technical issues while rewriting the migrations) is in ADR-011.

Kubernetes deployment

(2026-09-17) The goal was to prove the system can be deployed on Kubernetes, and especially to verify pessimistic locking holds when several application instances run in parallel. Done in 4 steps.

Step 1 — REST controller

• The codebase previously had no HTTP endpoint — AccountService was only called in-process from tests. There is no way to verify pessimistic locking across real Pods (several separate JVMs/processes) without an HTTP entry point.
• Added AccountController: POST /accounts, GET /accounts/{id}/balance, POST /accounts/{id}/deposit, POST /accounts/{id}/withdraw. Added AccountService.deposit() (posts a CREDIT/DEBIT pair through LedgerService, symmetric to withdraw() but without a lock since it checks no balance invariant).
• Request/response logging in withdraw() tagged with the Pod name (read from the HOSTNAME environment variable — K8s sets it to the Pod name) — the evidence for Step 4.
• Added spring-boot-starter-actuator and enabled management.health.livenessstate/readinessstate for K8s probes.

Step 2 — Dockerfile

• Multi-stage build: the build stage uses the official Maven image with a pinned version (not ./mvnw, because maven-wrapper.jar is gitignored — avoiding a dependency on a file missing from a clean git clone), the runtime stage is just eclipse-temurin:17-jre-alpine.
• Kafka advertised listener fix: localhost is only correct when the app runs on the host, not in a container (inside a container, localhost is the container itself). Added a second listener (PLAINTEXT_HOST, advertised via host.docker.internal, port 9094) alongside the original (PLAINTEXT, port 9092, still used by the test suite running directly on the host).

Step 3 — Deploying 1 replica

• Wrote k8s/namespace.yaml, k8s/configmap.yaml (pointing to host.docker.internal for Postgres/Kafka/Redis, which still run outside the cluster via docker-compose), k8s/secret.yaml (DB credentials, separate from the ConfigMap), k8s/deployment.yaml, k8s/service.yaml (NodePort, reachable directly at localhost:30080 from the host on Docker Desktop K8s, no kubectl port-forward needed).
• Docker Desktop Kubernetes (not minikube/kind) — shares the Docker engine/image cache with docker build, imagePullPolicy: Never.
• startupProbe/livenessProbe/readinessProbe point to /actuator/health/* — the startupProbe gives the app enough time to connect to DB/Kafka/Redis before liveness starts counting, so it is not killed while starting up.

Step 4 — Scaling to 3 replicas & verifying pessimistic locking

• Scaled to replicas: 3 and fired 5 concurrent withdraw requests (funds for exactly 1 to succeed) through the Service. Verified with two separate layers of evidence:
  1. Correct business result: 1 request succeeded, 4 were rejected with HTTP 409, the final balance was exact, never negative, never double-debited.
  2. Requests really spread across all 3 different Pods — confirmed with kubectl logs --prefix, logs tagged with specific Pod names, including a case where one Pod received 2 requests at once and serialized them correctly. Evidence that the lock lives in the DB (independent of which Pod calls it), not that K8s happened to route everything to one place.
• Known limitation: OutboxEventPublisher has no distributed lock with several Pods — the 3 PoC consumers may log duplicates if several Pods pick up the same unpublished OutboxEvent; AuditComplianceConsumer is unaffected because it is idempotent.
• The full design decision (options considered, 2 technical issues, experimental verification details) is in ADR-013.

Challenges & solutions

The real technical problems of each step are recorded in the sections above. A summary in symptom → cause → fix → lesson form: Core Banking — Challenges & Lessons Learned.

Concepts learned

• Double-entry bookkeeping — a customer deposit account is a liability of the bank: increases are credits, decreases are debits; total Debit always equals total Credit.
• Check-then-act vs. lost update — optimistic locking (@Version) only catches lost updates on the entity itself; a "check, then act" race needs a pessimistic lock.
• Aggregate root & invariants — business invariants live in the aggregate; services only orchestrate.
• JPA persist vs. merge — an entity with a pre-assigned ID needs Persistable.isNew() to be inserted correctly.
• Dual write & Outbox Pattern — writing to the DB and sending a message cannot be atomic when done separately; the Outbox turns it into one DB transaction.
• At-least-once delivery & idempotent consumers — a message may arrive several times, so consumers must deduplicate.
• Cache-aside & invalidation after commit — evicting before commit opens a race that writes stale data back into the cache.
• BigDecimal scale — equals() compares scale too, compareTo() only compares value; different databases preserve scale differently.