Skip to content

fix(endpoint): raise JobError when a job fails - #699

Open
deanq wants to merge 1 commit into
mainfrom
deanq/sls-1003-run-sync-failed-error
Open

deanq wants to merge 1 commit into
mainfrom
deanq/sls-1003-run-sync-failed-error

Conversation

@deanq

@deanq deanq commented Nov 7, 2026

Copy link
Copy Markdown
Member

What

Endpoint.run_sync and Job.output returned None when a job ended FAILED, TIMED_OUT or CANCELLED, and the job's error was dropped. A failed job looked the same as a handler that returned None:

endpoint.run_sync({"input": {...}})   # handler raised ValueError
# before: None
# after:  JobError: job <id> FAILED: {"error_type": "<class 'ValueError'>", "error_message": "...", ...}

Changes:

  • New runpod.error.JobError(RunPodError) with job_id, status and error.
  • Endpoint.run_sync, Job.output and the asyncio Job.output raise it for any final state other than COMPLETED. The asyncio docstring already said output() raises when the job fails.
  • COMPLETED jobs return their output as before, including None.
  • CANCELLED is now handled the same as the other final states in run_sync, which previously only checked COMPLETED, FAILED and TIMED_OUT.

Behavior change

Code that checked result is None to detect a failure now gets a JobError instead. Raising was chosen over returning the error so a failure can't be silently treated as a result. It matches runpod.apps, which raises RemoteExecutionError for failed jobs.

Test plan

  • New tests/test_endpoint/test_failed_jobs.py:
    • run_sync raises with status and error for FAILED, TIMED_OUT and CANCELLED
    • run_sync raises when a polled job fails
    • a COMPLETED job with no output still returns None
    • Job.output raises on every call after a failure
    • a failure without an error field still names the status
    • asyncio Job.output raises on failure
  • The raise tests failed before the change
  • make quality-check (2781 passed)
  • Live check against a deployed queue endpoint whose handler raises: on unfixed code run_sync returned None; with this change run_sync and run().output() raise JobError with status FAILED and the handler's error message. A successful call still returns its output.

Endpoint.run_sync and Job.output returned None for a job that ended
FAILED, TIMED_OUT or CANCELLED, and dropped the job's error. A failed
job looked the same as a handler that returned None.

- New runpod.error.JobError carries job_id, status and error.
- Endpoint.run_sync, Job.output and the asyncio Job.output raise it
  for any final state other than COMPLETED.
- COMPLETED jobs return their output as before, including None.

SLS-1003

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant