BankOrg Docs

In progress

BankOrg Data Platform

Scope and Roadmap

What the project covers, what is built so far and what is planned.

Scope

The project focuses on dbt and Snowflake, and aims to show senior-level data engineering patterns rather than breadth. There is no user interface for now. The pipeline runs in the cloud once a day, with simpler orchestration than Airflow.

The MVP covers four Core Banking entities:

  • CUSTOMERS and ACCOUNTS: snapshot feeds.
  • TRANSACTIONS: CDC, planned with Snowflake Streams and Tasks.
  • BRANCHES: reference table.

A customer belongs to one home branch, while its accounts can belong to different branches. Raw CSV files land in Google Cloud Storage and load into RAW with COPY INTO on a daily schedule. Data generation and data processing are two separate daily jobs, with a readiness marker file signalling that a batch is complete.

Built so far

  • The daily feed job that generates a business day and uploads it to GCS. A Cloudflare Worker starts it and the dbt workflow every day.
  • RAW: the Snowflake load tasks, the generic load procedure and the monitor that retries failed loads.
  • Staging: four dbt views with tests and unit tests, built every day by the dbt job once each RAW load is done, followed by the seed, the snapshots, the silver models and the gold marts in the same workflow.
  • Silver: three SCD Type 2 snapshots, the dimensions dim_branch, dim_customer and dim_account, and the incremental fact_transaction at the grain of a posting leg, and the two balance facts, built with dbt build.
  • Gold: the two balance facts and four marts in PRD_GOLD, daily_account_balance, daily_transaction_summary, branch_performance and customer_360, rebuilt by the daily dbt job, and the Dashboard that reads them.

Planned

  • Fault injection: faults are injected into daily runs to demonstrate the patterns. The first demo runs the same batch twice and skips it on success, using a strict hash fingerprint across layers.
  • Backfill safety: the query ID and time of each job are kept, so an earlier table version could be restored with Time Travel before a backfill. This is a design-level pattern, with no automated restore.
  • Security: a masking policy and a branch-based row access policy.