BankOrg Docs

In progress

BankOrg Data Platform / Source Data / Daily Feed Job

Job and GCP Access

The GitHub Actions job that generates a business date at 00:30 Vietnam time and uploads it to GCS, started by a Cloudflare Worker, with its GCP access.

The daily feed job generates one business date of the core banking feeds and uploads it to GCS. It is a GitHub Actions workflow in the n2tuan/bankorg repository that a Cloudflare Worker starts at 00:30 Vietnam time (17:30 UTC). The load tasks in Snowflake run at 05:00 and load the files it uploaded.

What a run does

The workflow checks out the repository, logs in to GCP and calls scripts/run_daily.sh.

StepWhat happens
1. DateThe snapshot date is the argument, or yesterday in Vietnam time when none is given.
2. Pull the stateThe generator state is copied from gs://bankorg-prd-raw-landing/_generator_state to data/generated/_state. When nothing is saved yet, the generator starts from data/initial.
3. Generatescripts/generate_daily_data.py --date <date> writes the four CSV files and their _SUCCESS.json markers.
4. Uploadscripts/upload_to_gcs.sh --date <date> uploads each CSV first and its marker last. A dataset whose marker is already in the bucket is skipped.
5. Push the stateThe state of the day is copied back to gs://bankorg-prd-raw-landing/_generator_state.

The generator builds each date from the state of the day before, and that state is not in git. It is kept in GCS under _generator_state, outside core_banking/, so the load never reads it. Running the job again for a date that is already published changes nothing: the generator gives the same files and the upload skips each dataset that has its marker.

scripts/run_daily.sh
#!/usr/bin/env bash
# Generate one day of core-banking feeds and upload it to GCS (run by the GitHub Actions schedule).
#
# Usage:
#   scripts/run_daily.sh [YYYY-MM-DD]
#
#   date  snapshot date; default = yesterday in Vietnam time (what the 05:00 load tasks load)
#
# The generator evolves the state saved for the previous day, and that state is not in git, so it
# is kept in GCS: pulled before the run and pushed back after the upload.
#
# Env: STATE_URI (default gs://bankorg-prd-raw-landing/_generator_state), outside core_banking/ so
#      the load never sees it. Needs gcloud logged in.

set -euo pipefail

STATE_URI="${STATE_URI:-gs://bankorg-prd-raw-landing/_generator_state}"
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
STATE_DIR="$REPO/data/generated/_state"

DATE="${1:-$(python3 -c "from datetime import datetime, timedelta; from zoneinfo import ZoneInfo; print((datetime.now(ZoneInfo('Asia/Ho_Chi_Minh')) - timedelta(days=1)).date())")}"
[[ "$DATE" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]] || { echo "bad date: $DATE" >&2; exit 1; }
echo "run_daily: snapshot date $DATE"

# 1) Pull the generator state (nothing there yet on the first run: the generator then starts from data/initial)
mkdir -p "$STATE_DIR"
if gcloud storage ls "$STATE_URI/" >/dev/null 2>&1; then
  gcloud storage rsync --recursive "$STATE_URI" "$STATE_DIR"
else
  echo "run_daily: no saved state at $STATE_URI, starting from data/initial"
fi

# 2) Generate the day
python3 "$REPO/scripts/generate_daily_data.py" --date "$DATE"

# 3) Upload the CSVs, then the markers (skips a dataset already published)
"$REPO/scripts/upload_to_gcs.sh" --date "$DATE"

# 4) Push the state back
gcloud storage rsync --recursive "$STATE_DIR" "$STATE_URI"
echo "run_daily: done $DATE"

GCP access

The workflow logs in to GCP with Workload Identity Federation, so no key file is stored in GitHub. The objects created in project bankorg:

ObjectNameAccess
Service accountgha-daily-feed@bankorg.iam.gserviceaccount.comroles/storage.objectAdmin on gs://bankorg-prd-raw-landing
Workload identity poolgithubProject bankorg, location global
Providerbankorg-repoOIDC issuer https://token.actions.githubusercontent.com, only the repository n2tuan/bankorg

What happens on every run, from the GitHub token to the upload, is shown on GCP Access Diagram.

Commands run, in this order:

Terminal
gcloud iam service-accounts create gha-daily-feed --project=bankorg --display-name="GitHub Actions daily feed"
Terminal
gcloud storage buckets add-iam-policy-binding gs://bankorg-prd-raw-landing --member="serviceAccount:gha-daily-feed@bankorg.iam.gserviceaccount.com" --role="roles/storage.objectAdmin"
Terminal
gcloud services enable iamcredentials.googleapis.com sts.googleapis.com --project=bankorg
Terminal
gcloud iam workload-identity-pools create github --project=bankorg --location=global --display-name="GitHub Actions"
Terminal
gcloud iam workload-identity-pools providers create-oidc bankorg-repo --project=bankorg --location=global --workload-identity-pool=github --issuer-uri="https://token.actions.githubusercontent.com" --attribute-mapping="google.subject=assertion.sub,attribute.repository=assertion.repository" --attribute-condition="assertion.repository == 'n2tuan/bankorg'"
Terminal
gcloud iam service-accounts add-iam-policy-binding gha-daily-feed@bankorg.iam.gserviceaccount.com --project=bankorg --role="roles/iam.workloadIdentityUser" --member="principalSet://iam.googleapis.com/projects/$(gcloud projects describe bankorg --format='value(projectNumber)')/locations/global/workloadIdentityPools/github/attribute.repository/n2tuan/bankorg"
Terminal
gcloud iam workload-identity-pools providers describe bankorg-repo --project=bankorg --location=global --workload-identity-pool=github --format='value(name)'
# projects/590845415587/locations/global/workloadIdentityPools/github/providers/bankorg-repo

Workflow

.github/workflows/daily-feed.yml
name: Daily feed

# Generate one day of core-banking data and upload it to GCS.
# Started by the Cloudflare Worker bankorg-workflow-trigger at 17:30 UTC = 00:30 Vietnam time, before the
# 05:00 load tasks (cloudflare/workflow-trigger). The date defaults to yesterday in Vietnam time.
on:
  workflow_dispatch:
    inputs:
      date:
        description: "Snapshot date YYYY-MM-DD (empty = yesterday in Vietnam time)"
        required: false

permissions:
  contents: read
  id-token: write

jobs:
  daily-feed:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4

      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: projects/590845415587/locations/global/workloadIdentityPools/github/providers/bankorg-repo
          service_account: gha-daily-feed@bankorg.iam.gserviceaccount.com

      - uses: google-github-actions/setup-gcloud@v2

      - name: Generate and upload
        env:
          SNAPSHOT_DATE: ${{ inputs.date }}
        run: scripts/run_daily.sh "$SNAPSHOT_DATE"

A run can also be started by hand from the Actions tab, with a date.

Trigger

The workflow has no schedule: of its own. A Cloudflare Worker, bankorg-workflow-trigger, starts it through the GitHub workflow_dispatch API with a Cron Trigger at 30 17 * * * (UTC). A second cron, 30 22 * * *, starts daily-dbt-staging.yml. The date input is left empty, so each run uses yesterday in Vietnam time. The Worker is in the cloudflare/workflow-trigger folder of the repository.

Commands run, in this order:

Terminal
cd cloudflare/workflow-trigger
npx wrangler login
npx wrangler deploy
npx wrangler secret put GITHUB_TOKEN
# https://bankorg-workflow-trigger.tuannn-data.workers.dev
# schedule: 30 17 * * *
# schedule: 30 22 * * *

The secret is a fine-grained GitHub token for n2tuan/bankorg only, with the permission Actions: read and write. A new cron trigger can take up to 15 minutes to take effect after a deploy.

Check on 2026-10-23

The dbt workflow did not start from the Worker at first. Workers Logs were off, so Workers Logs were switched on in wrangler.toml and the Worker was deployed again:

cloudflare/workflow-trigger/wrangler.toml
[observability]
enabled = true

A temporary cron was added a few minutes ahead of the clock for each test and removed afterwards. The first one, at 05:50 UTC (12:50 Vietnam time), logged dispatch failed 401 Bad credentials: the GITHUB_TOKEN secret of the Worker was not accepted by GitHub. A fine-grained token bankorg-workflow-trigger was created for n2tuan/bankorg only, with Actions: read and write, expiring on 2027-01-21, and set as the secret with npx wrangler secret put GITHUB_TOKEN. The cron at 06:03 UTC did not fire, because the new schedule had not taken effect yet; the cron at 06:30 UTC (13:30 Vietnam time) dispatched the workflow:

Workers Logs
2026-10-23 12:50:46  daily-dbt-staging.yml: dispatch failed 401 { "message": "Bad credentials", ... }
2026-10-23 13:30:13  daily-dbt-staging.yml: dispatched

The temporary cron was then removed and the Worker deployed again with the two daily crons, 30 17 * * * and 30 22 * * *. The dbt workflow is named Daily dbt in GitHub.

First run

The workflow was started by hand for 2026-10-19, with no saved state, so the generator built the date on the initial files of 2026-10-18. After the run, the bucket has the uploaded date and the saved state:

Terminal
gcloud storage ls gs://bankorg-prd-raw-landing/core_banking/branches/ gs://bankorg-prd-raw-landing/_generator_state/
# gs://bankorg-prd-raw-landing/core_banking/branches/snapshot_date=2026-10-18/
# gs://bankorg-prd-raw-landing/core_banking/branches/snapshot_date=2026-10-19/
# gs://bankorg-prd-raw-landing/_generator_state/dt=2026-10-19/

Reset on 2026-10-23

The transactions layout gained LEG_NO and DIRECTION after the first runs, so the state saved in GCS and the files already loaded had the old 13 columns, and the run of 2026-10-22 failed on CHANGE_SEQ. The generator now ignores a saved state whose transactions header is outdated and starts again from data/initial. Everything was then rebuilt from the beginning, with the initial snapshot moved to 2026-10-01 so that the daily dates are 2026-10-02 to 2026-10-22:

  • The four load tasks and the monitor task were suspended. The four RAW tables, OPS.LOAD_LOG and OPS.MONITOR_LOG were truncated, and the three snapshot tables in PRD_SILVER.SNAPSHOTS were dropped.
  • core_banking/ and _generator_state/ in the bucket were deleted.
  • The initial files were uploaded for 2026-10-01 (scripts/upload_to_gcs.sh --date 2026-10-01 --source initial), then scripts/run_daily.sh ran for each date from 2026-10-02 to 2026-10-22 in order.
  • OPS.LOAD_DATASET loaded the 22 dates of the four datasets, and the tasks were resumed.
  • dbt build --full-refresh --target prod rebuilt staging, the snapshots and the marts: 8 models, 95 tests, 3 snapshots and 12 unit tests, 118 of 118 passed.