BankOrg Data Platform / Source Data / Daily Feed Job
Job and GCP Access
The GitHub Actions job that generates a business date at 00:30 Vietnam time and uploads it to GCS, started by a Cloudflare Worker, with its GCP access.
The daily feed job generates one business date of the core banking feeds and uploads it to GCS. It is a GitHub Actions workflow in the n2tuan/bankorg repository that a Cloudflare Worker starts at 00:30 Vietnam time (17:30 UTC). The load tasks in Snowflake run at 05:00 and load the files it uploaded.
What a run does
The workflow checks out the repository, logs in to GCP and calls scripts/run_daily.sh.
| Step | What happens |
|---|---|
| 1. Date | The snapshot date is the argument, or yesterday in Vietnam time when none is given. |
| 2. Pull the state | The generator state is copied from gs://bankorg-prd-raw-landing/_generator_state to data/generated/_state. When nothing is saved yet, the generator starts from data/initial. |
| 3. Generate | scripts/generate_daily_data.py --date <date> writes the four CSV files and their _SUCCESS.json markers. |
| 4. Upload | scripts/upload_to_gcs.sh --date <date> uploads each CSV first and its marker last. A dataset whose marker is already in the bucket is skipped. |
| 5. Push the state | The state of the day is copied back to gs://bankorg-prd-raw-landing/_generator_state. |
The generator builds each date from the state of the day before, and that state is not in git. It is kept in GCS under _generator_state, outside core_banking/, so the load never reads it. Running the job again for a date that is already published changes nothing: the generator gives the same files and the upload skips each dataset that has its marker.
#!/usr/bin/env bash
# Generate one day of core-banking feeds and upload it to GCS (run by the GitHub Actions schedule).
#
# Usage:
# scripts/run_daily.sh [YYYY-MM-DD]
#
# date snapshot date; default = yesterday in Vietnam time (what the 05:00 load tasks load)
#
# The generator evolves the state saved for the previous day, and that state is not in git, so it
# is kept in GCS: pulled before the run and pushed back after the upload.
#
# Env: STATE_URI (default gs://bankorg-prd-raw-landing/_generator_state), outside core_banking/ so
# the load never sees it. Needs gcloud logged in.
set -euo pipefail
STATE_URI="${STATE_URI:-gs://bankorg-prd-raw-landing/_generator_state}"
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
STATE_DIR="$REPO/data/generated/_state"
DATE="${1:-$(python3 -c "from datetime import datetime, timedelta; from zoneinfo import ZoneInfo; print((datetime.now(ZoneInfo('Asia/Ho_Chi_Minh')) - timedelta(days=1)).date())")}"
[[ "$DATE" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || { echo "bad date: $DATE" >&2; exit 1; }
echo "run_daily: snapshot date $DATE"
# 1) Pull the generator state (nothing there yet on the first run: the generator then starts from data/initial)
mkdir -p "$STATE_DIR"
if gcloud storage ls "$STATE_URI/" >/dev/null 2>&1; then
gcloud storage rsync --recursive "$STATE_URI" "$STATE_DIR"
else
echo "run_daily: no saved state at $STATE_URI, starting from data/initial"
fi
# 2) Generate the day
python3 "$REPO/scripts/generate_daily_data.py" --date "$DATE"
# 3) Upload the CSVs, then the markers (skips a dataset already published)
"$REPO/scripts/upload_to_gcs.sh" --date "$DATE"
# 4) Push the state back
gcloud storage rsync --recursive "$STATE_DIR" "$STATE_URI"
echo "run_daily: done $DATE"GCP access
The workflow logs in to GCP with Workload Identity Federation, so no key file is stored in GitHub. The objects created in project bankorg:
| Object | Name | Access |
|---|---|---|
| Service account | gha-daily-feed@bankorg.iam.gserviceaccount.com | roles/storage.objectAdmin on gs://bankorg-prd-raw-landing |
| Workload identity pool | github | Project bankorg, location global |
| Provider | bankorg-repo | OIDC issuer https://token.actions.githubusercontent.com, only the repository n2tuan/bankorg |
What happens on every run, from the GitHub token to the upload, is shown on GCP Access Diagram.
Commands run, in this order:
gcloud iam service-accounts create gha-daily-feed --project=bankorg --display-name="GitHub Actions daily feed"gcloud storage buckets add-iam-policy-binding gs://bankorg-prd-raw-landing --member="serviceAccount:gha-daily-feed@bankorg.iam.gserviceaccount.com" --role="roles/storage.objectAdmin"gcloud services enable iamcredentials.googleapis.com sts.googleapis.com --project=bankorggcloud iam workload-identity-pools create github --project=bankorg --location=global --display-name="GitHub Actions"gcloud iam workload-identity-pools providers create-oidc bankorg-repo --project=bankorg --location=global --workload-identity-pool=github --issuer-uri="https://token.actions.githubusercontent.com" --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository" --attribute-condition="assertion.repository == 'n2tuan/bankorg'"gcloud iam service-accounts add-iam-policy-binding gha-daily-feed@bankorg.iam.gserviceaccount.com --project=bankorg --role="roles/iam.workloadIdentityUser" --member="principalSet://iam.googleapis.com/projects/$(gcloud projects describe bankorg --format='value(projectNumber)')/locations/global/workloadIdentityPools/github/attribute.repository/n2tuan/bankorg"gcloud iam workload-identity-pools providers describe bankorg-repo --project=bankorg --location=global --workload-identity-pool=github --format='value(name)'
# projects/590845415587/locations/global/workloadIdentityPools/github/providers/bankorg-repoWorkflow
name: Daily feed
# Generate one day of core-banking data and upload it to GCS.
# Started by the Cloudflare Worker bankorg-workflow-trigger at 17:30 UTC = 00:30 Vietnam time, before the
# 05:00 load tasks (cloudflare/workflow-trigger). The date defaults to yesterday in Vietnam time.
on:
workflow_dispatch:
inputs:
date:
description: "Snapshot date YYYY-MM-DD (empty = yesterday in Vietnam time)"
required: false
permissions:
contents: read
id-token: write
jobs:
daily-feed:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: projects/590845415587/locations/global/workloadIdentityPools/github/providers/bankorg-repo
service_account: gha-daily-feed@bankorg.iam.gserviceaccount.com
- uses: google-github-actions/setup-gcloud@v2
- name: Generate and upload
env:
SNAPSHOT_DATE: ${{ inputs.date }}
run: scripts/run_daily.sh "$SNAPSHOT_DATE"A run can also be started by hand from the Actions tab, with a date.
Trigger
The workflow has no schedule: of its own. A Cloudflare Worker, bankorg-workflow-trigger, starts it through the GitHub workflow_dispatch API with a Cron Trigger at 30 17 * * * (UTC). A second cron, 30 22 * * *, starts daily-dbt-staging.yml. The date input is left empty, so each run uses yesterday in Vietnam time. The Worker is in the cloudflare/workflow-trigger folder of the repository.
Commands run, in this order:
cd cloudflare/workflow-trigger
npx wrangler login
npx wrangler deploy
npx wrangler secret put GITHUB_TOKEN
# https://bankorg-workflow-trigger.tuannn-data.workers.dev
# schedule: 30 17 * * *
# schedule: 30 22 * * *The secret is a fine-grained GitHub token for n2tuan/bankorg only, with the permission Actions: read and write. A new cron trigger can take up to 15 minutes to take effect after a deploy.
Check on 2026-10-23
The dbt workflow did not start from the Worker at first. Workers Logs were off, so Workers Logs were switched on in wrangler.toml and the Worker was deployed again:
[observability]
enabled = trueA temporary cron was added a few minutes ahead of the clock for each test and removed afterwards. The first one, at 05:50 UTC (12:50 Vietnam time), logged dispatch failed 401 Bad credentials: the GITHUB_TOKEN secret of the Worker was not accepted by GitHub. A fine-grained token bankorg-workflow-trigger was created for n2tuan/bankorg only, with Actions: read and write, expiring on 2027-01-21, and set as the secret with npx wrangler secret put GITHUB_TOKEN. The cron at 06:03 UTC did not fire, because the new schedule had not taken effect yet; the cron at 06:30 UTC (13:30 Vietnam time) dispatched the workflow:
2026-10-23 12:50:46 daily-dbt-staging.yml: dispatch failed 401 { "message": "Bad credentials", ... }
2026-10-23 13:30:13 daily-dbt-staging.yml: dispatchedThe temporary cron was then removed and the Worker deployed again with the two daily crons, 30 17 * * * and 30 22 * * *. The dbt workflow is named Daily dbt in GitHub.
First run
The workflow was started by hand for 2026-10-19, with no saved state, so the generator built the date on the initial files of 2026-10-18. After the run, the bucket has the uploaded date and the saved state:
gcloud storage ls gs://bankorg-prd-raw-landing/core_banking/branches/ gs://bankorg-prd-raw-landing/_generator_state/
# gs://bankorg-prd-raw-landing/core_banking/branches/snapshot_date=2026-10-18/
# gs://bankorg-prd-raw-landing/core_banking/branches/snapshot_date=2026-10-19/
# gs://bankorg-prd-raw-landing/_generator_state/dt=2026-10-19/Reset on 2026-10-23
The transactions layout gained LEG_NO and DIRECTION after the first runs, so the state saved in GCS and the files already loaded had the old 13 columns, and the run of 2026-10-22 failed on CHANGE_SEQ. The generator now ignores a saved state whose transactions header is outdated and starts again from data/initial. Everything was then rebuilt from the beginning, with the initial snapshot moved to 2026-10-01 so that the daily dates are 2026-10-02 to 2026-10-22:
- The four load tasks and the monitor task were suspended. The four RAW tables,
OPS.LOAD_LOGandOPS.MONITOR_LOGwere truncated, and the three snapshot tables inPRD_SILVER.SNAPSHOTSwere dropped. core_banking/and_generator_state/in the bucket were deleted.- The initial files were uploaded for 2026-10-01 (
scripts/upload_to_gcs.sh --date 2026-10-01 --source initial), thenscripts/run_daily.shran for each date from 2026-10-02 to 2026-10-22 in order. OPS.LOAD_DATASETloaded the 22 dates of the four datasets, and the tasks were resumed.dbt build --full-refresh --target prodrebuilt staging, the snapshots and the marts: 8 models, 95 tests, 3 snapshots and 12 unit tests, 118 of 118 passed.