BankOrg Data Platform / Source Data / Upload Data to GCS
Initial Data
Upload the initial files of 2026-10-18 and their marker files to the GCS bucket.
The initial files are uploaded to the bucket bankorg-prd-raw-landing, in the folder core_banking/ that the stage GCS_CORE_BANKING_STG points at. Each dataset has its own folder for the snapshot date 2026-10-18, with the CSV and a marker file _SUCCESS.json.
Upload with the script
The script scripts/upload_to_gcs.sh uploads the four datasets of the snapshot date 2026-10-18 from data/initial/.
scripts/upload_to_gcs.sh --date 2026-10-18 --source initialFor each dataset the script does these steps in order:
- Finds the CSV and the marker in the dataset folder.
- Checks that the CSV header equals the source columns of the RAW table.
- Compares the number of data rows with the row count in the marker.
- Uploads the CSV, then the marker.
All checks run before the first upload. The date folders were empty, so the default --overwrite N had nothing to skip.
Bucket layout
Marker files
{
"dataset": "transactions",
"snapshot_date": "2026-10-18",
"based_on": null,
"file": "transactions_original_4410.csv",
"rows": 4410
}The load procedure LOAD_BRANCHES checks that the marker of its snapshot date exists before running COPY INTO, and the load reads only *.csv files, so the marker is never loaded as data.
Upload of 2026-10-01
The output above is from the first upload, for 2026-10-18. On 2026-10-23 the initial load was regenerated with the snapshot date 2026-10-01 (5,074 transaction rows) and uploaded again after the bucket was emptied:
scripts/upload_to_gcs.sh --date 2026-10-01 --source initial