Repository navigation
fix(local): retry a remote endpoint's refused JSON schema unconstrained - #2446
officialasishkumar wants to merge 1 commit into
Conversation
Only the llama.cpp provider fell back when a json_schema request came back 400. OpenAICompatibleChatProvider, which every remote OpenAI-compatible connection uses, raised the refusal. DeepSeek's API answers such a request with "This response_format type is unavailable now", so a DeepSeek connection failed the quiz, flashcards, simple web page and comic plan before any token, though its model was fine. The fallback moves from LlamaCppProvider into the client both of them send through. A 400 to a request carrying response_format is sent once more without it. Anything else is not retried: any other status, or a 400 to a request that carried no schema. The refusal comes before the stream starts, so nothing is yielded twice, and the caller's parse_json() reads the unconstrained answer as it already does for an endpoint that ignores the schema. The llama.cpp path is unchanged: the same retry, now in the client, still carries the sampling and the thinking-off fields. LlamaCppProvider's logger was only used by the fallback and goes with it. selection.md and runtime.md say where the fallback now lives.
|
@officialasishkumar is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
What
When an OpenAI-compatible endpoint refuses a
json_schemarequest with a 400, the request is now sent once more without the schema. Before this, onlyLlamaCppProviderdid that.OpenAICompatibleChatProvider, which every remote OpenAI-compatible connection uses, passed the 400 back to the caller.LlamaCppProvider.chat_deltas()intoOpenAICompatibleChatProvider.chat_deltas(), which both send through.response_formatis sent again with the same body minusresponse_format. Nothing else is retried: no other status, and no 400 to a request without a schema.parse_json(), as it already is for an endpoint that ignoresresponse_format.LlamaCppProvidernow runs in the client on the same body, so the publisher's sampling and the thinking-off fields still go with it.test_constrained_output.py's llama.cpp cases pass unchanged.LlamaCppProvider'sloggingimport andloggerwere only used by the fallback, so they go with it.Docs:
selection.md(Constrained decoding) andruntime.mdsay the fallback lives in the client and name the two known refusals.Why
DeepSeek is a ready connection in the remote catalog (
api.deepseek.com). Its API answers ajson_schemarequest with a 400, "This response_format type is unavailable now" (reported for example in openai-agents-python#2032 and on Discourse Meta).connection_generator()builds anOpenAICompatibleChatProviderfor it. So with a DeepSeek model selected, the quiz, flashcards, simple web page and comic plan fail before the model writes anything, though the model itself is fine. #2445 sends a schema from the image, infographic and podcast as well, which would put every JSON format in Studio on this path.The change was part of #2149. When closing it, the maintainers said "moving the 400 fallback into the OpenAI-compatible client were right when we reviewed them" and asked for a fresh PR. This is that part on its own, because it is a provider fix and not a Studio format. It merges cleanly with #2445 in either order.
I have not run it against DeepSeek itself (I have no key). The new tests use a mock endpoint that sends DeepSeek's refusal body.
How to test
New:
tests/unit/llm/providers/openai_compatible/test_structured_output.py, against a mock remote endpoint:The two retry tests fail with the client change reverted.
Summary by CodeRabbit