[al3xanndru]

Problem

https://uipath.atlassian.net/browse/PC-5280

Debug and playground runs that simulate tools through the legacy toolsToSimulate path can fault with:

UiPathMockResponseGenerationError: Tool call arguments missing 'response' key: {}

The agent's own LLM calls succeed. Only the tool simulation fails, and only when the simulated output is large, for example when the tool's output schema has an open-ended list or anyOf: [{}, null] field.

Root cause

Two gaps in uipath.eval.mocks:

1. The mocker drops the agent's max_tokens. build_mocking_context falls back to the agent's model as ModelSettings(model=agent_model), which carries the model name only. The eval runtime does the same with ModelSettings(model=get_agent_model(...)). So the simulation call goes out at the UiPathLlmChatService.chat_completions default of max_tokens=4096, even when the agent is configured for more (16384 in the reported case). In the reported failure, the logged request body shows max_tokens: 4096, and both simulation calls used exactly 4096 completion tokens.
2. Truncation is never detected. With an OpenAI model, the response_format reply is cut off and fails json.loads. The forced submit_tool_response fallback is cut off the same way, and the gateway returns its arguments as {}. extract_response never looks at finish_reason, so it reports a missing key instead of a length cut-off.

Change

• eval/helpers.py: new get_agent_max_tokens(schema), the counterpart of get_agent_model. It reads schema.metadata["settings"]["max_tokens"] (the low-code runtime dumps AgentSettings by field name) or maxTokens. Anything that isn't a positive int is ignored.
• eval/mocks/_mock_runtime.py: build_mocking_context, build_mocking_context_from_dict and load_simulation_config take an optional agent_max_tokens. When the simulation falls back to the agent's model, it uses ModelSettings(model=agent_model, max_tokens=agent_max_tokens). The new parameter is keyword-compatible, so existing callers are unaffected.
• _cli/cli_debug.py, _cli/cli_run.py, eval/runtime/runtime.py: pass the agent's max tokens wherever the agent's model is already passed.
• eval/mocks/_structured_output.py:
  • When the forced tool call has no response key and finish_reason == "length", extract_response raises a clear error: "LLM response was truncated at the max_tokens limit (finish_reason='length') ... increase the model's max tokens".
  • The response_format path logs a warning when its output was truncated, before it falls back to tools.
• Version bump: 2.14.44 → 2.14.45 (pyproject.toml, uv.lock).

Deliberately left out:

• Other agent settings. The agent's temperature (and its other settings) are still not forwarded. Forwarding temperature to models that reject it is what caused PC-4769.
• Explicit simulation models. An explicit SimulationConfig.model does not inherit the agent's max_tokens. That model may have a lower output cap than the agent's model, and sending the agent's cap to it could turn today's successful runs into 400s.

Tests

• tests/cli/eval/mocks/test_mock_runtime_model_settings.py (new):
  • the agent-model fallback carries max_tokens, and it reaches simulation_completion_kwargs;
  • max_tokens is omitted when the agent has none;
  • an explicit simulation model does not take the agent's cap.
• tests/cli/eval/mocks/test_structured_output.py:
  • a truncated tool call with {} arguments raises the truncation error;
  • complete arguments with finish_reason == "length" are still returned;
  • end to end: a truncated response_format reply followed by a truncated tool call raises the truncation error, and both calls carry max_tokens.
• tests/cli/eval/test_eval_runtime_metadata.py: get_agent_max_tokens handles snake_case, camelCase, a missing value, and unusable values.

Results: see the summary below the fold. (TEST_RESULTS)

Risks / open questions

• Larger simulation responses. Simulation calls can now use up to the agent's configured max_tokens, which costs more tokens and time on runs whose mock output really is large. That is the agent author's configured budget.
• A remaining 4096 cap. Runs that set an explicit simulation model, and coded agents (no model in their schema), still go out at the 4096 default. If those need a larger cap, the next step would be a mocker default max_tokens.
• Error classification. The truncation error is still wrapped as UiPathMockResponseGenerationError, which the agents runtime maps to Unknown. Classifying it correctly is a separate change in uipath-agents-python's exception mapper.
• Rollout. The fix reaches production only after uipath-agents-python picks up the new uipath version.

---

Drafted by the wabbit-dri draft-fixes loop for PC-5280 (draft #1), for a human to review: not ready to merge as is.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate failed

Failed conditions
85.7% Coverage on New Code (required ≥ 90%)

See analysis details on SonarQube Cloud