Skip to content

fix(apps): accept datacenter ids outside the built-in list - #690

Merged
deanq merged 2 commits into
mainfrom
deanq/sls-989-datacenter-passthrough
Nov 7, 2026
Merged

deanq merged 2 commits into
mainfrom
deanq/sls-989-datacenter-passthrough

Conversation

@deanq

@deanq deanq commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Summary

runpod.apps resources now accept datacenter ids that are not in the SDK's built-in DataCenter enum. Unrecognized ids are passed to the API unchanged, with a warning logged once per id.

Why

The DataCenter enum is a hardcoded snapshot and goes stale whenever new datacenters come online. Today normalize_datacenter() raises InvalidResourceError for anything outside it, so a user whose workers can only run in a datacenter the SDK does not list has no way to name it. Without an explicit datacenter the SDK falls back to its built-in default lists, and on such an account the endpoint is created but workers are never scheduled (jobs stay IN_QUEUE). The backend validates datacenter ids anyway, so the client-side check only blocks valid configurations.

An explicit datacenter= already bypasses the CPU/GPU default location lists in the deploy and dev payloads, so accepting the id is enough for it to reach locations. The default lists are unchanged.

Changes

  • spec.normalize_datacenter: known values and string variants (case, underscores, whitespace) normalize exactly as before. Unrecognized non-empty ids pass through (whitespace-stripped) with a single warning per id. Blank and non-string values still raise InvalidResourceError.
  • datacenter.py: adds datacenter_id() (lenient normalization) and DataCenter.is_known().
  • placement.candidates: unrecognized pins stay candidates. Stock is only queried for built-in datacenters, so the SDK cannot rule them out; the backend decides.
  • placement.solve_placement: an existing network volume in an unrecognized datacenter no longer raises a bare ValueError. It resolves normally when resources are pinned there; otherwise it surfaces the usual PlacementError.
  • volume.VolumeResolver: network volume datacenter pins accept unrecognized ids. Network volume support is still checked against the live datacenter catalog.

Rollout note

Workers re-run ResourceSpec.validate when they import the user module, and they use the SDK version baked into the worker image. Until a release containing this change reaches the worker images, workers running an older SDK still reject an unrecognized datacenter id at import time, even though the client accepted it and the endpoint was created with it. Adding this version to the endpoint's dependencies does not help, because runpod is already imported on the worker before dependencies are installed.

Until then, pin the datacenter on the client only:

datacenter = "XX-TEST-1" if runpod.is_local() else None

Test plan

  • Known values (enum, case/underscore/whitespace variants) normalize unchanged and do not warn
  • Unrecognized id passes through with exactly one warning, including across repeated revalidation
  • Blank, whitespace-only, None and non-string values are rejected
  • Deploy payload locations contains the unrecognized id for GPU, cpu3 and cpu5 resources
  • Placement and network volume resolution handle unrecognized pins and existing volumes
  • make quality-check: 2773 passed, 94.35% coverage

The SDK's DataCenter enum is a hardcoded snapshot and goes stale as new
datacenters come online. Rejecting ids outside it left users unable to
name the datacenter their workers can actually run in. The backend
validates datacenter ids, so the SDK now passes unrecognized ids through
unchanged and logs a warning once per id.

- normalize_datacenter keeps known values and string variants as
  before; unrecognized non-empty ids pass through with a warning; blank
  and non-string values still raise InvalidResourceError
- placement keeps unrecognized pins as candidates (stock is only
  queried for built-in datacenters) and no longer crashes on an
  existing volume in an unrecognized datacenter
- network volume pins accept unrecognized ids; the datacenter catalog
  still decides network volume support
Comment thread runpod/apps/spec.py Resolved
Reword the warning for an unrecognized datacenter id to say it does not
look valid but will be used anyway, and link the data center list in
the API reference so users can check the id.
@deanq
deanq merged commit dbc251a into main Nov 7, 2026
12 of 17 checks passed
@deanq
deanq deleted the deanq/sls-989-datacenter-passthrough branch November 7, 2026 04:31
@promptless

promptless Bot commented Nov 7, 2026

Copy link
Copy Markdown

Promptless documentation updates

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants