[andrewklatzke]

Summary

• A wildcard handler can take a providers list, so a LangChain adapter used only for Bedrock accepts config.provider.name == "Bedrock" and no other provider.
• Exact [provider, mode] still wins. Among scoped wildcards that contain the name, the shortest list wins, then the earlier registration. An unscoped * is the fallback.
• Registry identity is provides_for plus the provider set, so two scoped wildcards of the same mode can be registered together. Judges, graph selection, and the evaluations runner use the same order.

Test plan

• uv run pytest on the client, langchain-messages, and langchain-agents suites
• ruff and mypy on the touched packages
• Integration success and failure for agent, graph, streaming, graph-streaming, judge, langchain, langchain-messages, langchain-agents, and native-graph-langchain

Made with Cursor

Changelog

This PR changes core routing and the evaluations runner. 1.0 is not published, so both lines are breaking and land in that cut:

BEGIN_COMMIT_OVERRIDE
feat(AIC-3439)!: let wildcard handlers accept only named providers
feat(evaluations)!: route generation and judges to a scoped wildcard handler only for its named providers, and accept two scoped wildcards of one mode when their lists differ
END_COMMIT_OVERRIDE

[andrewklatzke]

Human message:

I think this is a "nice to have" not necessarily "must do it". I'm thinking of situations where, for example, a user might want to use the Google ADK and scope it only to gemini (create_google_adk_handler(providers=['Gemini'])). Another example might be folks wanting to use langchain but only for Bedrock so they don't have to deal with boto.

You can technically achieve this now by properly including the right subset of handlers, but this makes it more obvious/colocates it with the code if you want to scope them down.

tldr; if we think this is confusing or don't want it, we can close it

[cursor]

Stale Bugbot comment from a previous run.

[jeffdupont]

The routing code is close, and it matches js #111 on almost every probe case. This PR also fixes a Python bug. On main, run_judges used the parent handler for any judge whose provider no handler covered, because the no-match branch fell through to judge_handler = handler (main judges.py:110-147). An OpenAI parent would run an Anthropic judge. build_judge_tasks already skipped that case. Two things need doing before this merges: a rebase that fixes two tests, and agreement with #118 on how the evaluations runner picks handlers.

At ee8f6f0c I get 1427 passed and 11 skipped. mypy (51 files), ruff and ruff format --check are clean.

1. Two new tests fail once this is merged with today's main. Git merges it without conflicts. But test_scoped_wildcard_does_not_run_a_judge_outside_its_list and test_two_scoped_judge_handlers_divide_providers (test_evaluations_run.py:2100 and :2133) call init_evaluations(api_token=...) and pass project_key to run(). Since this branch was cut, main has renamed the argument to api_key (7a238e5) and moved project_key onto init_evaluations (I found #127 / be1e64a with git log -S, but didn't read the commit). After I fixed both calls, the merged tree gives 1505 passed and 11 skipped, with mypy (52 files) and ruff clean. The green CI ran on the old base.
2. #118 conflicts with this, in the code and in what it does. Merging this into #118 (4cd1e0ee) conflicts in evaluations/runner.py and packages/client/README.md. #118 replaces _find_judge_handler and _covers_provider with a single _select_handler, and its rules contradict this PR in two ways. First, "a single handler always runs, whatever it declares", so a lone Bedrock-scoped handler would run an Anthropic judge. Second, two wildcard candidates raise "can't disambiguate" instead of the shortest list winning. Whichever PR lands second has to rebuild the scoped order inside _select_handler. I'd land this one first and have #118 call select_mode_handler. #39 §3.13 already says that a judge outside every scoped list raises EvaluationsError naming the provider.
3. Bugbot's untyped-parent finding is real, but I'd keep the new behaviour. I reproduced it. With run_judges(handler=untyped, handlers=[untyped]), the judge ran on main and is skipped now. graph() picks a lone untyped handler through the non-strict fallback, so that node's judges are now dropped without a log line. But main's build_judge_tasks already skipped the same case (0 tasks in my probe), and JS skips it too, because parentCovers is false without providesFor (judges.ts:25-28). This same continue is also what stops an OpenAI parent from running an Anthropic judge. So the change makes Python consistent with itself and with JS. I'd ask for two things: log when a judge is skipped because no handler fits (neither SDK says anything today), and add a changelog line. The evaluations runner still treats an untyped generation handler as covering every judge (runner.py:179-180). That's a separate path and fine as it is, but #39 should say so.
4. Python replaces a re-registered handler in place, and JS moves it to the end. registry.py:74 keeps the slot, while js registry.ts:29 filters and appends. Ties go to the earlier registration, so the SDKs now choose differently. As a repro, register ['Bedrock', 'Gemini'], then ['Bedrock', 'Anthropic'], then ['Bedrock', 'Gemini'] again. Python then runs the Gemini-list handler for Bedrock, and JS runs the Anthropic-list one. compose differs the same way. We need one rule in #39 §3.6. I'd keep Python's.
5. select_handler now drops matches that aren't ProviderHandler instances (utils.py:601). On main, the exact and wildcard steps returned any object with provides_for. Now isinstance(chosen, ProviderHandler) throws the match away, and config() raises Handler for provider OpenAI with mode messages not found even though a matching handler is registered. I probed this with a duck-typed ('*', 'messages') object. If only ProviderHandler is supported, the error message is misleading. Otherwise, drop the check and use a cast for mypy.
6. Repeated names count toward list length (utils.py:97). frozenset makes ['Bedrock', 'Bedrock'] the same identity as ['Bedrock'], but best_wildcard ranks it by len(tuple(allow)), so it ties with ['Bedrock', 'Anthropic']. Deduping in normalize_providers fixes this. JS has the same issue.
7. The docstring and the code disagree on whitespace. utils.py:26-28 says surrounding whitespace is rejected rather than trimmed, but [' Bedrock'] is accepted (and never matches). Either reject it or fix the docstring. JS needs the same change.

I tried 23 mutations, and 17 of them fail a test. The 6 that get through are in the inline comments: unscoped beating scoped, the strict coverage check, _provider_matches, the runner's _covers_provider and its shortest-list sort, and accepting a bare string.

Smaller items

• ProviderHandler.__init__ takes providers positionally and doesn't validate or copy it (types.py:169, :175). ProviderHandler(fn, pf, providers=[]) builds a handler that matches nothing. Please run it through normalize_providers, or make it keyword-only. This signature freezes at 1.0 too.
• Exact handlers that have a list keep a separate identity (registry.py:20). The list is ignored for routing on an exact handler, so two ('OpenAI', 'messages') handlers should collide, as #39 says. JS does the same.
• The duplicate warning prints a frozenset repr: Handler for (('*', 'messages'), frozenset({'Gemini', 'Bedrock'})) already registered. The name order changes from run to run. Sorting the names for the message would fix that.
• Public surface. providers is a keyword-only argument on create_handler, create_langchain_messages_handler and create_langchain_agents_handler, and an attribute on ProviderHandler, with the same name and meaning as JS. No new names reach __all__ (normalize_providers, wildcard_covers, best_wildcard, select_mode_handler and covers_provider_name aren't exported), so py #144's ROOT_SURFACE snapshot isn't affected.
• #107 overlap. Merging this into #107 (c76acd58) conflicts in utils.py, where model_parameters and the new helpers are both additions, and in the collections.abc import line of both LangChain handlers (Mapping vs Sequence). Keep both sides. I don't see a clash in behaviour: #107 is about parameter forwarding, not routing.
• graph_judge passes handlers here (graph.py:1016-1024 and :1238-1250), which is right. JS doesn't, so I've asked for that on js #111.

Parity with js-ai-sdk #111

The two SDKs agree on selection order, ties, case, a None / "" provider, empty and blank lists, both strict error strings (Handler for provider OpenAI not found and ... with mode agent not found), and set identity. They differ in four places:

• re-registration position (4)
• the graph-level judge (JS is wrong)
• copying the caller's list (Python copies, JS doesn't)
• ['a,b'] vs ['a', 'b'], which collide in JS and stay separate here

[jeffdupont]

Pushed 0d0e893, which merges origin/main and resolves the conflicts with #152:

• The scoped order is in utils.select_mode_handler, built on main's handler_key. select_handler calls it, so config(), graph(), online judges and the evaluations runner all route the same way: exact provider, then the shortest scoped list that contains the provider, then unscoped *.
• The runner's own selection code is gone, and its scoped tests are ported to handlers=[...].
• The registry and the evals duplicate check share utils.registration_key.
• run_judge now logs the same warning as run_judges when it skips a judge.
• normalize_providers moved to types.py, so there is no inline import in ProviderHandler.__init__.
• I ran make api-report: client, langchain-agents and langchain-messages changed.
• I added a BEGIN_COMMIT_OVERRIDE to the description with a separate feat(evaluations): line.

This needs a human reviewer.

Both items are addressed in 0d0e893 (merge with main; scoped order moved into select_mode_handler). Dismissing my own stale request; this needs a second approver since I pushed code.

[jeffdupont]

I merged main and moved the scoped order into select_mode_handler. Since I pushed those changes, I'm dismissing my 10-08 change request. @andrewklatzke, please check the merge; this needs a second approver.

[jeffdupont]

Approving 0d0e893. The change is additive (keyword-only providers on create_handler and both LangChain factories), the API reports are updated, and it matches launchdarkly/ai-sdks-monorepo#39. Note: the branch now conflicts with main, so it needs a main merge before it can land.

[jeffdupont]

My approval was at 0d0e893. Main has moved since, and the branch now conflicts in five files: api-report/launchdarkly_ai_server.txt, evaluations/module.py, types.py, and the langchain-agents and langchain-messages handler.py. Please merge main, regenerate the API report and push. I'll re-check the merge resolution only.

[cursor]

Stale Bugbot comment from a previous run.

[andrewklatzke]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit fa001bd. Configure here.}