Skip to content

ci: Test OpenMP offloading on Intel GPUs, with the Devito fixes it needs - #3095

Open
ggorman wants to merge 7 commits into
mainfrom
intel-gpu-sycl-fixes
Open

ggorman wants to merge 7 commits into
mainfrom
intel-gpu-sycl-fixes

Conversation

@ggorman

@ggorman ggorman commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

Fixes for Devito on Intel GPUs, found while bringing DevitoPRO's SYCL backend back to a green test suite on a Data Center GPU Max 1100 (devitocodespro/devitopro#252), plus a CI job for OpenMP offloading on Intel GPUs (#2370). The commits are kept separate.

1. compiler: Substitute the object of a method Call in Uxreplace

Uxreplace.visit_Call substituted into a Call's arguments and retobj, never into its callee. For a call through an object (q.memcpy(...), built from a CallFromPointer), the object q is part of the callee. Call.base and the constructor arguments kept it, but Call.name (a string, which code generation and signatures use) was never rebuilt. reuse_efuncs relies on Uxreplace to abstract every symbol of a Callable, so two Callables that differed only in that object kept distinct signatures and were both emitted.

No OSS backend emits such Calls; DevitoPRO's SYCL backend does. With two streamed Functions it emitted two copies of each streaming thread function, identical apart from the queue. CUDA, HIP and OpenMP pass the stream as an argument and were never affected.

The fix makes Call.callee expose the callee object, and has Uxreplace rebuild the name only when the callee actually changes. Subclasses that fix their own callee (e.g. a parallel_for on a handler that is never in the mapper) are therefore never rebuilt with name.

Test: tests/test_iet.py::test_reuse_efuncs_call_through_object, modelled on test_reuse_efuncs_natural_numbering. It is IET-level because no OSS backend emits such a Call. It fails on main and passes with the fix.

2. arch: Query the memory of Intel GPUs through xpu-smi

The Intel branch of get_gpu_info installed mem.* callbacks that always returned None. Device.memavail() was therefore None on Intel GPUs, and nbytes_avail_mapper raised TypeError: unsupported operand type(s) for -=: 'NoneType' and 'Integer'. In DevitoPRO this broke every snapshotting Operator on an Intel GPU.

The callbacks now read memory_physical_size_byte and memory_free_size_byte from xpu-smi discovery -d <id> -j, and still return None if xpu-smi is missing or fails. They take a physical device ID, as with nvidia-smi. To map MPI ranks correctly:

  • ZE_AFFINITY_MASK is now one of the visible-devices variables.
  • xpu-smi runs without the mask, which would otherwise renumber the devices. Without this, ranks 1-3 of a 4-rank run on one GPU got None.

The old comment about "weird errors in docker" did not reproduce with the new images below, run unprivileged.

Tests: TestGPUInfo::test_memavail and TestDeviceID::test_visible_devices[{ZE_AFFINITY_MASK: 1}]. Both fail on main and pass with the fix on a Max 1100.

3. Intel GPU CI (#2370)

  • docker: Take the Intel GPU runtime from the LTS 2523 channel. The old graphics/ubuntu jammy unified channel stops at compute-runtime 23.17, with which oneAPI 2026 finds no Level Zero device on a Max 1100, and OpenMP offload images fail to load. The images now take compute-runtime 25.18 / IGC 2.11 / Level Zero 1.24 from gpu/ubuntu jammy/lts/2523, with the renamed packages. A new gpu-icx target sets DEVITO_PLATFORM=intelgpuX.

  • compiler: Stop passing -fopenmp-target-simd for Intel GPUs. With oneAPI 2026.0 every Operator fails at image load:

    • IGC 1.0.15468 (LTS 2350) reports "SLM variable usage in indirectly called functions is not supported";
    • IGC 2.11 (LTS 2523, the newest for Ubuntu 22.04 on PVC) crashes.

    Without the flag, the job's test files pass.

  • ci: Run the GPU tests on Intel GPUs with OpenMP offloading. The new pytest-gpu-omp-intel matrix entry runs the same test files and examples as the AMD OpenMP job, on self-hosted INTELGPU runners. The runner picks the device via ZE_AFFINITY_MASK.

  • ci: Build the Intel base images again, including gpu-icx. This undoes c57bf01 ("docker: Disable oneapi updates"). That commit stopped rebuilding the Intel bases, which froze them at a runtime (23.17) that cannot drive current Intel GPUs with the current compiler. The new job needs the gpu-icx image, and the runtime fix above only reaches cpu-icx/cpu-sycl/gpu-sycl through a rebuild. If you prefer to keep automatic rebuilds off, this commit can be dropped and the images pushed once by hand from docker/Dockerfile.intel.

  • docker: Let Intel MPI spawn processes in the Intel images. Intel MPI 2021.19 uses the shm fabric alone on one node, and shm refuses MPI_Comm_spawn. DevitoPRO's decoupler tests hung in the rebuilt images until I_MPI_FABRICS=shm:ofi was set.

Results (ese-ponte: 2x Data Center GPU Max 1100, oneAPI 2026.0.0)

  • Job rehearsal in an image built from the new gpu-icx target, using the job's own commands:
    • pytest step: 259 passed, 5 skipped (MPI tests included);
    • examples: 46 passed;
    • mpiexec -n 2 examples: 46 passed.
  • CPU, this branch, full suite (-m "not parallel"): C 3706 passed, CXX 3706 passed, openmp 3709 passed, 0 failed (gcc).
  • pre-commit run on the touched files (isort, ruff, flake8, typos, actionlint, hadolint): passes.

Before the Intel job can run

  • The Intel base images must be rebuilt and published (see above).
  • A self-hosted runner labelled INTELGPU must be registered. None exists yet. Both are deferred by the owner, so the new job will queue until then.

Fixes #2370

Related: devitocodespro/devitopro#252 (pins this branch).

🤖 Generated with Claude Code

A Call through an object, e.g. `q.memcpy(...)`, keeps its callee only as a
string, so Uxreplace could not rename `q`. reuse_efuncs then saw two
Callables differing only in that object as distinct, and kept both.

Keep the callee object on the Call and let Uxreplace substitute into it.
The Intel GPU branch of get_gpu_info installed memory callbacks that always
returned None, so Device.memavail() was None on Intel GPUs and anything
budgeting device memory (e.g. DevitoPRO's layered snapshotting) could not
work. Read `memory_physical_size_byte` and `memory_free_size_byte` from
`xpu-smi discovery -d <id> -j`, and keep returning None if xpu-smi is
unavailable or fails.

The callbacks take a physical device ID, as with nvidia-smi, so treat
ZE_AFFINITY_MASK as a visible-devices variable when mapping an MPI rank to
its device, and run xpu-smi without it, since xpu-smi otherwise numbers
only the devices the mask leaves visible.
The `graphics/ubuntu jammy unified` channel stops at compute-runtime 23.17.
With it, the oneAPI 2026 compilers find no Level Zero device on a Data
Center GPU Max 1100 and OpenMP offload images fail to load. Install the
Max-series user-space stack (compute-runtime 25.18, IGC 2.11, Level Zero
loader 1.21+) from `gpu/ubuntu jammy/lts/2523`, under its current package
names, and add a `gpu-icx` target for OpenMP offloading to Intel GPUs.
With oneAPI 2026.0 on a Data Center GPU Max 1100 the explicit-SIMD device
compilation fails at load time for every Devito Operator: IGC 1.0.15468
(LTS 2350) rejects the image ("SLM variable usage in indirectly called
functions is not supported") and IGC 2.11 (LTS 2523, the newest for
Ubuntu 22.04) crashes. Without the flag, the GPU test files pass.
Add a pytest-gpu-omp-intel job, on self-hosted runners labelled INTELGPU,
running the same test files and examples as the AMD OpenMP job. The runner
selects its device through ZE_AFFINITY_MASK.
Undo the switch-off of c57bf01 so that the LTS 2523 runtime reaches
devitocodes/bases:{cpu-icx,cpu-sycl,gpu-sycl}, and build the new gpu-icx
image the Intel GPU job uses.
Intel MPI 2021.19 picks the shm fabric alone on a single node, and shm refuses MPI_Comm_spawn, so the DevitoPRO decoupler tests hung in the rebuilt images. Use shm:ofi.
@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.41463% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.43%. Comparing base (61a2745) to head (141bd8d).

Files with missing lines Patch % Lines
devito/arch/archinfo.py 6.66% 14 Missing ⚠️
devito/ir/iet/visitors.py 85.71% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3095      +/-   ##
==========================================
+ Coverage   84.40%   84.43%   +0.02%     
==========================================
  Files         260      260              
  Lines       58153    58189      +36     
  Branches     4948     4950       +2     
==========================================
+ Hits        49086    49129      +43     
+ Misses       8245     8237       -8     
- Partials      822      823       +1     
Flag Coverage Δ
pytest-gpu-aomp-amdgpuX 68.45% <32.14%> (+<0.01%) ⬆️
pytest-gpu-gcc- 79.08% <48.78%> (-0.02%) ⬇️
pytest-gpu-icx- 79.01% <51.21%> (-0.02%) ⬇️
pytest-gpu-nvc-nvidiaX 69.16% <32.14%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mloubout mloubout left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall yes, just one main comment

Comment thread devito/ir/iet/nodes.py
self.base = None
# The called object: a str or, for a call through an object such as
# `obj.method(...)`, a CallFromPointer
self.callee = name

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand why this needs to be separate from name

Comment thread devito/ir/iet/visitors.py
else:
arguments.append(uxreplace(i, self.mapper))
kwargs = {'arguments': arguments}
if o.base is not None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks like a rebuild bug being bypassed here

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test intel GPU

2 participants