Documentation
BankOrg Data Platform
A banking data platform on Snowflake: how it is set up, modeled and engineered.
Daily Load Grid
Load status of each RAW dataset, one column per snapshot day, last 30 days
OK rate
—
Last OK load
—
Retried days
—
Avg duration
—
- OK
- OK after retry
- Error
- Skipped
- Running
- No run logged
- No data
What this is
BankOrg is a banking data platform, built as a portfolio project on synthetic data only: 50 branches, customers and their accounts, in VND, with double-entry transactions. It uses Google Cloud Storage, Snowflake and dbt, and applies data engineering design patterns, idempotency first.
Source code
The code is in the GitHub repository n2tuan/bankorg.
Layers
Data moves from Google Cloud Storage into Snowflake RAW, then through STAGING, SILVER and GOLD. dbt transforms the data between layers inside Snowflake.
| Layer | Where | What it holds |
|---|---|---|
| GCS | bankorg-prd-raw-landing | Daily CSV feeds of branches, customers, accounts and transactions, one folder per dataset and snapshot_date, each with a _SUCCESS.json marker. |
| RAW | PRD_RAW.CORE_BANKING | The four landing tables, loaded by Snowflake tasks. Append-only. |
| Staging | PRD_STAGING.CORE_BANKING | dbt views: latest snapshot, typed, cleaned, deduplicated. |
| Silver | PRD_SILVER.SNAPSHOTS, PRD_SILVER.CORE | SCD Type 2 snapshots, the dimensions, the transaction fact and the two account balance facts. |
| Gold | PRD_GOLD | Four business marts, one schema per domain: daily_account_balance, daily_transaction_summary, branch_performance and customer_360. |
Daily run
The pipeline runs once a day. A Cloudflare Worker starts the two GitHub workflows; the times are Vietnam time.
- 00:30: the
daily-feedworkflow generates yesterday's data and uploads it to GCS. - 05:00: the Snowflake load tasks load it into RAW.
- 05:30: the
daily-dbt-stagingworkflow builds staging, then the snapshots, the silver models and the four gold marts, and runs their tests.
Where to look
The Dashboard shows figures from the gold tables. The Architecture page describes the layers, the daily flow and the design decisions, and Scope and Roadmap the scope.