Skip to content

fix(local): retry a remote endpoint's refused JSON schema unconstrained - #2446

Open
officialasishkumar wants to merge 1 commit into
MODSetter:devfrom
officialasishkumar:fix/remote-schema-refusal-falls-back
Open

officialasishkumar wants to merge 1 commit into
MODSetter:devfrom
officialasishkumar:fix/remote-schema-refusal-falls-back

Conversation

@officialasishkumar

@officialasishkumar officialasishkumar commented Nov 19, 2026 •

Copy link
Copy Markdown

What

When an OpenAI-compatible endpoint refuses a json_schema request with a 400, the request is now sent once more without the schema. Before this, only LlamaCppProvider did that. OpenAICompatibleChatProvider, which every remote OpenAI-compatible connection uses, passed the 400 back to the caller.

  • The fallback moves from LlamaCppProvider.chat_deltas() into OpenAICompatibleChatProvider.chat_deltas(), which both send through.
  • A 400 to a request carrying response_format is sent again with the same body minus response_format. Nothing else is retried: no other status, and no 400 to a request without a schema.
  • The refusal is raised when the status is checked, before the stream starts, so no token is sent twice. The unconstrained answer is read by the caller's parse_json(), as it already is for an endpoint that ignores response_format.
  • The local path is unchanged. The retry that used to live in LlamaCppProvider now runs in the client on the same body, so the publisher's sampling and the thinking-off fields still go with it. test_constrained_output.py's llama.cpp cases pass unchanged.
  • LlamaCppProvider's logging import and logger were only used by the fallback, so they go with it.

Docs: selection.md (Constrained decoding) and runtime.md say the fallback lives in the client and name the two known refusals.

Why

DeepSeek is a ready connection in the remote catalog (api.deepseek.com). Its API answers a json_schema request with a 400, "This response_format type is unavailable now" (reported for example in openai-agents-python#2032 and on Discourse Meta). connection_generator() builds an OpenAICompatibleChatProvider for it. So with a DeepSeek model selected, the quiz, flashcards, simple web page and comic plan fail before the model writes anything, though the model itself is fine. #2445 sends a schema from the image, infographic and podcast as well, which would put every JSON format in Studio on this path.

The change was part of #2149. When closing it, the maintainers said "moving the 400 fallback into the OpenAI-compatible client were right when we reviewed them" and asked for a fresh PR. This is that part on its own, because it is a provider fix and not a Studio format. It merges cleanly with #2445 in either order.

I have not run it against DeepSeek itself (I have no key). The new tests use a mock endpoint that sends DeepSeek's refusal body.

How to test

cd surfsense_local/backend
uv run pytest -m unit                                            # 5744 passed, 19 skipped
uv run pytest -m integration tests/integration/llm tests/integration/chat   # 371 passed, 66 skipped
uv run ruff check .
cd ../.. && python scripts/check_docs.py

New: tests/unit/llm/providers/openai_compatible/test_structured_output.py, against a mock remote endpoint:

  • A refused schema is asked for again without it, and the retry's body is otherwise identical.
  • An endpoint that takes the schema is asked once.
  • A 400 without a schema is not retried.
  • A 503 with a schema is not retried.
  • A 400 the unconstrained request also gets fails after exactly one retry.

The two retry tests fail with the client change reverted.

Summary by CodeRabbit

  • Bug Fixes
    • OpenAI-compatible chat connections now retry a JSON-schema request once without the schema if the endpoint rejects it with a 400 error. This applies to both local and remote connections; other errors continue to be reported.
  • Documentation
    • Clarified how schema fallback works and added an example of an endpoint refusing a schema.

Only the llama.cpp provider fell back when a json_schema request came
back 400. OpenAICompatibleChatProvider, which every remote
OpenAI-compatible connection uses, raised the refusal. DeepSeek's API
answers such a request with "This response_format type is unavailable
now", so a DeepSeek connection failed the quiz, flashcards, simple web
page and comic plan before any token, though its model was fine.

The fallback moves from LlamaCppProvider into the client both of them
send through. A 400 to a request carrying response_format is sent once
more without it. Anything else is not retried: any other status, or a
400 to a request that carried no schema. The refusal comes before the
stream starts, so nothing is yielded twice, and the caller's
parse_json() reads the unconstrained answer as it already does for an
endpoint that ignores the schema. The llama.cpp path is unchanged: the
same retry, now in the client, still carries the sampling and the
thinking-off fields.

LlamaCppProvider's logger was only used by the fallback and goes with
it. selection.md and runtime.md say where the fallback now lives.
@vercel

vercel Bot commented Nov 19, 2026

Copy link
Copy Markdown

@officialasishkumar is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel.

A member of the Team first needs to authorize it.

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The shared OpenAI-compatible provider retries a JSON Schema request once without its schema after an HTTP 400. The llama.cpp provider no longer performs its own retry. Unit tests and documentation cover the fallback behavior.

Changes: Structured-output fallback

Layer / File(s) Summary
Provider retry behavior
surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py, surfsense_local/backend/modules/llm/providers/llamacpp/provider.py
The shared provider removes response_format and retries once after an HTTP 400 on a schema request. The llama.cpp provider no longer catches status errors to retry without the schema. Deadline handling moves to _deadlined.
Fallback tests and documentation
surfsense_local/backend/tests/unit/llm/providers/openai_compatible/test_structured_output.py, docs/architecture/local-models/runtime.md, docs/architecture/local-models/selection.md
Tests cover the retry and non-retry cases. Documentation describes the fallback for local and remote OpenAI-compatible connections.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant ChatClient
  participant OpenAICompatibleChatProvider
  participant OpenAICompatibleEndpoint
  ChatClient->>OpenAICompatibleChatProvider: Send schema request
  OpenAICompatibleChatProvider->>OpenAICompatibleEndpoint: Stream with response_format
  OpenAICompatibleEndpoint-->>OpenAICompatibleChatProvider: HTTP 400
  OpenAICompatibleChatProvider->>OpenAICompatibleEndpoint: Retry without response_format
  OpenAICompatibleEndpoint-->>OpenAICompatibleChatProvider: Stream response
  OpenAICompatibleChatProvider-->>ChatClient: Yield response
Loading
Merge Risk: Minimal · up to 88664

The shared OpenAI-compatible client now retries a rejected JSON-schema request once without the schema, for both local and remote connections. The retry is narrow and covered by unit tests. No merge-blocking risk is identified.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the main change: the local OpenAI-compatible path now retries a remote endpoint's refused JSON schema once without constraints. The changeset matches this. The summaries show the HTTP …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the main change: the local OpenAI-compatible path now retries a remote endpoint's refused JSON schema once without constraints. The changeset matches this. The summaries show the HTTP 400 fallback moved into OpenAICompatibleChatProvider.chat_deltas, the llama.cpp recovery path removed, and unit tests added for the retry conditions. The title's wording is slightly narrow because the retry applies to both local and remote OpenAI-compatible connections, and the title says 'remote endpoint'. It still reflects the core behavior and is specific enough for a teammate scanning history.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 7e03b16 and 8866458.

Files selected for processing: 5

  • docs/architecture/local-models/runtime.md
  • docs/architecture/local-models/selection.md
  • surfsense_local/backend/modules/llm/providers/llamacpp/provider.py
  • surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py
  • surfsense_local/backend/tests/unit/llm/providers/openai_compatible/test_structured_output.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant