Repository navigation
fix(cli): require a loadable bank for each run locale before any model call - #64
Merged
Merged
Conversation
…ectory A locale folder can exist and still abort every row, as a finance bank without tasks does. The pre-flight now loads each locale-partitioned probe's bank for the run's locales, after the India-variant mapping, and a refusal names the locales that can run. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
simulate --dry-run checks a panel's locales, and --materialize-inputs and --inputs run the asset pre-flight; stored rows are checked probe by probe, against the asset root they run on. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
The asset pre-flight checked only that a locale's folder exists. A folder can exist and still fail every row.
financial_services,sov_ai_facts,sov_ai_dynamic) for the run's locales, after the India-variant mapping, with the loader the probe itself uses. A refusal names the locale, the reason (no bank, or the loader's error) and the locales that can run.simulate --dry-runalso checks a panel's locales;--materialize-inputsand--inputsrun it, the latter checking each probe against the locales its own rows use, under the asset root they run on; notebook 01 calls it before its run loop.architecture/probes.mddescribes the check, and notebook 01's comment no longer listshi_Deva_INas a finance locale.How it was verified
make check-all: lint clean, 333 files already formatted.make test: 4087 passed, 2 failed. The two failures are intests/test_replay_reasoning_config.py, which read amodels.local.tomlwhen the checkout has one; they pass without it.make smoke,make check-license-headers,make check-doc-links,make check-notebooks-cleanandmake check-distpass.en_USfinance bank, plus anfr_FRcopy without tasks), so they don't depend on which banks ship. They cover a bank that doesn't load, the locales that can run, the dry runs with--localeand--panel,--materialize-inputs, and--inputschecked probe by probe. Each fails without the change.simulate --dry-run --locale hi_Deva_IN --probe-mix financial_services=1exits 2 with "Locales that can run: en_IN, en_US";ta_Taml_INruns throughen_IN; notebook 01's default settings pass.make health-check: no change to models, providers, prompts, scorers or the trajectory store.