BankOrg Docs

In progress

BankOrg Data Platform / Source Data / Upload Data to GCS

Initial Data

Upload the initial files of 2026-10-18 and their marker files to the GCS bucket.

The initial files are uploaded to the bucket bankorg-prd-raw-landing, in the folder core_banking/ that the stage GCS_CORE_BANKING_STG points at. Each dataset has its own folder for the snapshot date 2026-10-18, with the CSV and a marker file _SUCCESS.json.

Upload with the script

The script scripts/upload_to_gcs.sh uploads the four datasets of the snapshot date 2026-10-18 from data/initial/.

Terminal
scripts/upload_to_gcs.sh --date 2026-10-18 --source initial

For each dataset the script does these steps in order:

  • Finds the CSV and the marker in the dataset folder.
  • Checks that the CSV header equals the source columns of the RAW table.
  • Compares the number of data rows with the row count in the marker.
  • Uploads the CSV, then the marker.

All checks run before the first upload. The date folders were empty, so the default --overwrite N had nothing to skip.

Output of the upload script.

Bucket layout

Output of gcloud storage ls -l on the four folders.

Marker files

transactions/snapshot_date=2026-10-18/_SUCCESS.json
{
  "dataset": "transactions",
  "snapshot_date": "2026-10-18",
  "based_on": null,
  "file": "transactions_original_4410.csv",
  "rows": 4410
}

The load procedure LOAD_BRANCHES checks that the marker of its snapshot date exists before running COPY INTO, and the load reads only *.csv files, so the marker is never loaded as data.

Upload of 2026-10-01

The output above is from the first upload, for 2026-10-18. On 2026-10-23 the initial load was regenerated with the snapshot date 2026-10-01 (5,074 transaction rows) and uploaded again after the bucket was emptied:

Terminal
scripts/upload_to_gcs.sh --date 2026-10-01 --source initial