feat(aarch64): add GDB debugging for Linux KVM by cshung · Pull Request #2016 · hyperlight-dev/hyperlight

cshung · 2026-11-03T00:34:05Z

Summary

ARM64 Linux/KVM guests support GDB register and memory access, two hardware breakpoints, BRK software breakpoints, and hardware single-step.

KVM reports debug stops through KVM_EXIT_DEBUG. Guest address translation uses the existing page-table reader. Debugger code writes synchronize the instruction cache. Debug state survives vCPU reset, and register access respects the active stack pointer.

Depends on #1907. The two KVM commits sit above its updated head, 00b72128. Shared snapshot, cancellation, and crash-detach recovery fixes belong to that foundation.

KVM-only comparison.

Validation

Windows ARM64: all six GDB client tests and 14 backend/recovery tests pass after rebase. GDB-enabled Clippy passes.
The full standard test suite, build, formatting, and standard Clippy pass locally. The combined source tree is byte-identical before and after the stack rebase.
Earlier Raspberry Pi 4 validation: 16 targeted native unit tests and six GDB integration tests passed. Linux runtime tests were not repeated for this update.

Pi validation uses a separate, uncommitted ARMv8.0 allocator fix and guests built with RUSTFLAGS="-C target-feature=-lse". Pi compatibility changes are outside these commits.

Open review finding

Debugger memory reads and writes still need guest-page-sized splitting on ARM64 Linux hosts with 16 KiB or 64 KiB pages.

Adds GDB stub support for guests running on the ARM64 WHP backend. * Split the x86_64 debug code out of `hyperlight_vm/x86_64.rs` into a shared `hyperlight_vm/debug.rs`. * Rename `gdb/x86_64_target.rs` to `gdb/target.rs` and add per-architecture register handling in `gdb/arch/`. * Give `DebuggableVm` default methods so backends without hardware breakpoints or single step report `Unsupported`. * Implement `DebuggableVm` for the aarch64 WHP VM, using the batched register helpers. * Flush the host instruction cache after guest memory writes so software breakpoints take effect. * Document ARM64 debugging in `docs/how-to-debug-a-hyperlight-guest.md` and `CHANGELOG.md`. On Windows ARM64 (WHP): * `just build`, `just clippy`, `just guests`, `just test` and `just fmt-check` pass. --------- Signed-off-by: cshung <3410332+cshung@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

## Changes * Run the full disable path when the debugger detaches after a crash, so software breakpoints and pending steps are cleaned up. * Cancel pending software steps for any cancellation source, not only debugger interrupts. * Reject `map_region` requests that overlap the hidden cache sync page at GPA 0x1000, and add a test. * Restore the `≡` and `×` characters in `hyperlight_vm/x86_64.rs` comments. ## Verification On Windows ARM64 (WHP): * `just clippy`, `just fmt-check` and the `hypervisor::` and `map_region` lib tests with the `gdb` feature pass. --------- Signed-off-by: cshung <3410332+cshung@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Guest synchronous exception vectors forward debug stops through HVC. Hardware single-step supports branches, returns, and self-loops. Two hardware breakpoint slots and BRK software breakpoints are available. Interrupted register context and breakpoint configuration survive cache synchronization and vCPU reset. Asynchronous vectors retain their normal exception paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

Copilot

🟡 Changes recommended

Unresolved critical flaws allow guest modification of executable trampoline code and debugger memory corruption, while breakpoint cleanup and snapshot-ordering issues can leave sandboxes unrecoverable.

5 open issues

What changed in this PR

Adds ARM64 guest debugging on Linux/KVM, alongside the shared debugger infrastructure and Windows/WHP support from #1907.

Changes:

Supports ARM64 register and memory access, software and hardware breakpoints, and hardware single-step.
Adds instruction-cache synchronization, debug-state preservation across reset, and snapshot safeguards.
Adds debugger tests and documents platform capabilities.

File	Description
`src/hyperlight_host/src/sandbox/uninitialized_evolve.rs`	Initializes ARM64 debug vectors during VM setup.
`src/hyperlight_host/src/sandbox/snapshot/memory/backing.rs`	Exposes backing ranges for cache synchronization.
`src/hyperlight_host/src/sandbox/initialized.rs`	Guards snapshots and restores against active software breakpoints.
`src/hyperlight_host/src/mem/mgr.rs`	Exposes checked host memory ranges to the debugger.
`src/hyperlight_host/src/mem/exe.rs`	Carries debug-vector addresses in load metadata.
`src/hyperlight_host/src/mem/elf.rs`	Finds and relocates the debug-vector symbol.
`src/hyperlight_host/src/hypervisor/virtual_machine/whp/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/whp/aarch64.rs`	Implements WHP debugging, cache synchronization, and tests.
`src/hyperlight_host/src/hypervisor/virtual_machine/mshv/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/mod.rs`	Adds ARM64 debug exits and initialization errors.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Adapts address translation to the shared interface.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/aarch64.rs`	Implements KVM debugging and active-stack-pointer handling.
`src/hyperlight_host/src/hypervisor/regs/aarch64/whp_reg.rs`	Defines WHP debug register identifiers.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Uses shared debugger handling and breakpoint storage.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/mod.rs`	Integrates ARM64 stops, stepping, and mapping safeguards.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/debug.rs`	Provides shared requests and software-breakpoint lifecycle handling.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Connects ARM64 VM setup and execution to GDB.
`src/hyperlight_host/src/hypervisor/gdb/target.rs`	Adds ARM64 register conversion to the shared target.
`src/hyperlight_host/src/hypervisor/gdb/mod.rs`	Adds page-table translation and cache-flush support.
`src/hyperlight_host/src/hypervisor/gdb/event_loop.rs`	Reports ARM64 initial stops to GDB.
`src/hyperlight_host/src/hypervisor/gdb/arch/x86_64.rs`	Houses x86 breakpoint constants and stop classification.
`src/hyperlight_host/src/hypervisor/gdb/arch/aarch64.rs`	Defines ARM64 software breakpoints and alignment checks.
`src/hyperlight_host/src/hypervisor/gdb/arch.rs`	Selects architecture-specific debugger helpers.
`src/hyperlight_host/examples/guest-debugging/main.rs`	Adds ARM64 breakpoint, stepping, and code-write integration tests.
`src/hyperlight_host/build.rs`	Enables GDB for supported ARM64 platforms.
`src/hyperlight_guest_bin/src/arch/aarch64/exception/entry.rs`	Exports debug vectors and forwards synchronous exceptions.
`src/hyperlight_common/src/arch/aarch64/exn.rs`	Defines the shared debug-exception HVC immediate.
`docs/how-to-debug-a-hyperlight-guest.md`	Documents ARM64 capabilities and guest-symbol requirements.
`CHANGELOG.md`	Records Linux/KVM and Windows/WHP ARM64 debugging support.

_{Review effort: Balanced}

Poisoned sandbox restore discards software breakpoint state. Crash detach disables backend debugging even when breakpoint memory is unavailable. Snapshot checkpoints reject installed software breakpoints before replacing memory. The WHP cache-sync page permits guest reads and execution. Exception-vector tests use a separate writable stack. Cache flushing resolves layered snapshot memory through the guest memory view. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

ARM64 KVM debugging uses KVM_SET_GUEST_DEBUG for hardware single-step and two hardware breakpoints. Software breakpoints use BRK. Guest virtual addresses use the existing page-table reader. Debugger code writes synchronize the host instruction cache. Register access respects the active stack pointer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

ARM64 KVM debug jobs run the GDB suite before the general Rust tests. Install GDB when the runner lacks it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

hyperlight-gh-bot · 2026-11-04T00:19:59Z

Benchmark Results

Measured commit: a26c2d9dc6e6
Baseline commit: fa13116cd998

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`389.55 µs` (➖ 1.00x faster)
`byte_chunks`		`754.59 ns` (➖ 1.01x faster)
`vec_bytes`		`585.94 ns` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`142.19 ns` (➖ 1.06x faster)
`262144`	`531.51 ns` (➖ 1.09x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`16.07 ms` (➖ 1.05x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.72 ns` (➖ 1.46x faster)	`7.71 ns` (➖ 1.00x slower)	`8.00 ns` (➖ 1.02x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`102.17 µs` (➖ 1.03x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`1.94 µs` (➖ 1.10x faster)	`2.03 µs` (➖ 1.06x slower)
`262144`	`7.49 µs` (➖ 1.05x slower)	`7.16 µs` (➖ 1.02x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.02 µs` (➖ 1.10x faster)	`1.04 µs` (➖ 1.13x faster)
`65536`	`6.17 µs` (➖ 1.04x faster)	`6.19 µs` (➖ 1.03x faster)
`262144`	`27.73 µs` (➖ 1.04x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`673.68 µs` (➖ 1.01x faster)
`byte_chunks`		`671.33 ns` (➖ 1.08x faster)
`vec_bytes`		`515.30 ns` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`162.86 ns` (➖ 1.12x faster)
`262144`	`533.71 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`15.17 ms` (➖ 1.02x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.02 ns` (➖ 1.02x faster)	`7.00 ns` (➖ 1.02x faster)	`6.98 ns` (➖ 1.03x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`50.62 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.17 µs` (➖ 1.02x faster)	`2.20 µs` (➖ 1.00x slower)
`262144`	`7.81 µs` (➖ 1.01x slower)	`7.81 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`711.82 ns` (➖ 1.03x faster)	`717.13 ns` (➖ 1.04x faster)
`65536`	`7.06 µs` (➖ 1.04x faster)	`7.08 µs` (➖ 1.04x faster)
`262144`	`29.68 µs` (➖ 1.05x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`317.31 µs` (➖ 1.01x slower)
`byte_chunks`		`909.01 ns` (➖ 1.00x faster)
`vec_bytes`		`732.07 ns` (➖ 1.03x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`182.34 ns` (➖ 1.09x faster)
`262144`	`678.13 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`64.36 ms` (➖ 1.02x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.87 ns` (➖ 1.01x faster)	`9.87 ns` (➖ 1.00x slower)	`9.90 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`109.04 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.46 µs` (➖ 1.03x faster)	`2.61 µs` (➖ 1.02x faster)
`262144`	`9.43 µs` (➖ 1.05x slower)	`9.30 µs` (➖ 1.06x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.26 µs` (➖ 1.04x faster)	`1.26 µs` (➖ 1.02x faster)
`65536`	`8.08 µs` (➖ 1.02x faster)	`8.51 µs` (➖ 1.02x faster)
`262144`	`35.28 µs` (➖ 1.01x slower)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`777.20 µs` (➖ 1.11x slower)
`byte_chunks`		`924.46 ns` (➖ 1.03x slower)
`vec_bytes`		`647.72 ns` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`270.97 ns` (➖ 1.03x slower)
`262144`	`657.20 ns` (➖ 1.02x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`74.44 ms` (➖ 1.00x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.66 ns` (➖ 1.00x faster)	`8.72 ns` (➖ 1.00x slower)	`8.67 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`54.83 µs` (➖ 1.02x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.27 µs` (➖ 1.01x slower)	`2.29 µs` (➖ 1.01x slower)
`262144`	`7.97 µs` (➖ 1.00x faster)	`7.98 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`884.65 ns` (➖ 1.04x faster)	`891.22 ns` (➖ 1.04x faster)
`65536`	`7.44 µs` (➖ 1.00x faster)	`7.53 µs` (➖ 1.00x faster)
`262144`	`40.12 µs` (➖ 1.03x slower)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`2.22 ms` (➖ 1.15x slower)
`byte_chunks`		`1.23 µs` (➖ 1.02x faster)
`vec_bytes`		`779.70 ns` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`230.01 ns` (➖ 1.00x faster)
`262144`	`790.25 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`11.79 ms` (➖ 1.02x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.09 ns` (➖ 1.01x slower)	`10.03 ns` (➖ 1.00x slower)	`9.93 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`839.08 µs` (➖ 1.19x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.28 µs` (➖ 1.13x faster)	`2.31 µs` (➖ 1.02x faster)
`262144`	`9.17 µs` (➖ 1.10x faster)	`8.99 µs` (➖ 1.03x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.31 µs` (➖ 1.01x faster)	`1.33 µs` (➖ 1.05x slower)
`65536`	`9.06 µs` (➖ 1.04x faster)	`8.95 µs` (➖ 1.01x faster)
`262144`	`39.05 µs` (➖ 1.21x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`2.89 ms` (➖ 1.01x faster)
`byte_chunks`		`1.18 µs` (➖ 1.00x slower)
`vec_bytes`		`750.95 ns` (➖ 1.01x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`221.17 ns` (➖ 1.05x slower)
`262144`	`753.34 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`11.91 ms` (➖ 1.03x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.07 ns` (➖ 1.01x slower)	`9.75 ns` (➖ 1.01x slower)	`9.86 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`568.51 µs` (➖ 1.17x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.44 µs` (➖ 1.03x slower)	`2.28 µs` (➖ 1.02x faster)
`262144`	`7.65 µs` (➖ 1.00x faster)	`7.67 µs` (➖ 1.02x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.12 µs` (➖ 1.07x slower)	`1.18 µs` (➖ 1.08x slower)
`65536`	`8.43 µs` (➖ 1.02x slower)	`8.43 µs` (➖ 1.02x slower)
`262144`	`41.47 µs` (➖ 1.32x faster)

_{Reported by cargo ci bench-report --candidate run:41740336200 --config-file bench_report.toml.}

cshung and others added 3 commits November 2, 2026 08:09

cshung added kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. ready-for-review PR is ready for (re-)review labels Nov 3, 2026

Copilot AI balanced review requested due to automatic review settings November 3, 2026 00:34

cshung requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners November 3, 2026 00:34

cshung marked this pull request as draft November 3, 2026 00:36

cshung removed the ready-for-review PR is ready for (re-)review label Nov 3, 2026

Copilot started reviewing on behalf of cshung November 3, 2026 00:37

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

cshung force-pushed the cshung/kvm-aarch64-gdb branch from a60be9b to 55da94f Compare November 3, 2026 01:58

This comment has been minimized.

Sign in to view

cshung and others added 3 commits November 3, 2026 11:24

ci(aarch64): run KVM GDB tests

2418379

ARM64 KVM debug jobs run the GDB suite before the general Rust tests. Install GDB when the runner lacks it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

cshung force-pushed the cshung/kvm-aarch64-gdb branch from 55da94f to 2418379 Compare November 3, 2026 19:28

This comment has been minimized.

Sign in to view

fix(gdb): split memory access at guest page boundaries

a26c2d9

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: cshung <3410332+cshung@users.noreply.github.com>

Conversation

cshung commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Open review finding

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Nov 4, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cshung commented Nov 3, 2026 •

edited

Loading