BankOrg Docs

In progress

Documentation

BankOrg Data Platform

A banking data platform on Snowflake: how it is set up, modeled and engineered.

Daily Load Grid

Load status of each RAW dataset, one column per snapshot day, last 30 days

OK rate

—

Last OK load

—

Retried days

—

Avg duration

—

Loading
branches
customers
accounts
transactions
  • OK
  • OK after retry
  • Error
  • Skipped
  • Running
  • No run logged
  • No data

What this is

BankOrg is a banking data platform, built as a portfolio project on synthetic data only: 50 branches, customers and their accounts, in VND, with double-entry transactions. It uses Google Cloud Storage, Snowflake and dbt, and applies data engineering design patterns, idempotency first.

Source code

The code is in the GitHub repository n2tuan/bankorg.

Layers

Data moves from Google Cloud Storage into Snowflake RAW, then through STAGING, SILVER and GOLD. dbt transforms the data between layers inside Snowflake.

LayerWhereWhat it holds
GCSbankorg-prd-raw-landingDaily CSV feeds of branches, customers, accounts and transactions, one folder per dataset and snapshot_date, each with a _SUCCESS.json marker.
RAWPRD_RAW.CORE_BANKINGThe four landing tables, loaded by Snowflake tasks. Append-only.
StagingPRD_STAGING.CORE_BANKINGdbt views: latest snapshot, typed, cleaned, deduplicated.
SilverPRD_SILVER.SNAPSHOTS, PRD_SILVER.CORESCD Type 2 snapshots, the dimensions, the transaction fact and the two account balance facts.
GoldPRD_GOLDFour business marts, one schema per domain: daily_account_balance, daily_transaction_summary, branch_performance and customer_360.

Daily run

The pipeline runs once a day. A Cloudflare Worker starts the two GitHub workflows; the times are Vietnam time.

  • 00:30: the daily-feed workflow generates yesterday's data and uploads it to GCS.
  • 05:00: the Snowflake load tasks load it into RAW.
  • 05:30: the daily-dbt-staging workflow builds staging, then the snapshots, the silver models and the four gold marts, and runs their tests.

Where to look

The Dashboard shows figures from the gold tables. The Architecture page describes the layers, the daily flow and the design decisions, and Scope and Roadmap the scope.

BankOrg data pipeline

click a component · drag to pan · Ctrl/⌘ + scroll to zoom