Skip to content

fix(cli): require a loadable bank for each run locale before any model call - #64

Merged
3mei merged 4 commits into
mainfrom
yev/fix/locale-bank-preflight
Oct 19, 2026
Merged

3mei merged 4 commits into
mainfrom
yev/fix/locale-bank-preflight

Conversation

@3mei

@3mei 3mei commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

What changed and why

The asset pre-flight checked only that a locale's folder exists. A folder can exist and still fail every row.

  • The pre-flight loads each locale-partitioned bank (financial_services, sov_ai_facts, sov_ai_dynamic) for the run's locales, after the India-variant mapping, with the loader the probe itself uses. A refusal names the locale, the reason (no bank, or the loader's error) and the locales that can run.
  • Every path runs it. simulate --dry-run also checks a panel's locales; --materialize-inputs and --inputs run it, the latter checking each probe against the locales its own rows use, under the asset root they run on; notebook 01 calls it before its run loop.
  • Docs. architecture/probes.md describes the check, and notebook 01's comment no longer lists hi_Deva_IN as a finance locale.

How it was verified

  • make check-all: lint clean, 333 files already formatted.
  • make test: 4087 passed, 2 failed. The two failures are in tests/test_replay_reasoning_config.py, which read a models.local.toml when the checkout has one; they pass without it.
  • make smoke, make check-license-headers, make check-doc-links, make check-notebooks-clean and make check-dist pass.
  • New tests use a fixture asset tree (a copy of the en_US finance bank, plus an fr_FR copy without tasks), so they don't depend on which banks ship. They cover a bank that doesn't load, the locales that can run, the dry runs with --locale and --panel, --materialize-inputs, and --inputs checked probe by probe. Each fails without the change.
  • On the shipped banks: simulate --dry-run --locale hi_Deva_IN --probe-mix financial_services=1 exits 2 with "Locales that can run: en_IN, en_US"; ta_Taml_IN runs through en_IN; notebook 01's default settings pass.
  • No make health-check: no change to models, providers, prompts, scorers or the trajectory store.

3mei added 4 commits October 18, 2026 22:02
…ectory

A locale folder can exist and still abort every row, as a finance bank without tasks does. The pre-flight now loads each locale-partitioned probe's bank for the run's locales, after the India-variant mapping, and a refusal names the locales that can run.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
simulate --dry-run checks a panel's locales, and --materialize-inputs and --inputs run the asset pre-flight; stored rows are checked probe by probe, against the asset root they run on.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei requested a review from a team as a code owner October 19, 2026 02:03
@3mei
3mei merged commit 2b2220e into main Oct 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant