feat!: replace shared-memory init data with virtqueue buffers by andreiltd · Pull Request #2030 · hyperlight-dev/hyperlight

andreiltd · 2026-11-06T09:03:05Z

Remove the peb backed init-data region and APIs. Use retained virtqueue ByteChunks for startup data. After removing the init section PEB no longer contains any data so it can be removed as well.

Copilot

🔵 Needs a closer look

Guest entry ABI and persisted snapshot layout changes across x86_64 and aarch64 require validation of rebuilt guests and initialization, restore, fork, and OCI round trips on supported hypervisors before approval.

No open issues 🎉

What changed in this PR

This PR routes guest startup data through existing virtqueue buffers and removes the PEB-backed initialization region. This is a breaking change requiring rebuilt guests and regenerated snapshots.

Changes:

Remove initialization-data APIs and simplify host and guest startup arguments.
Update snapshots to ABI 10 and config schema v6, with legacy formats rejected.
Add retained-buffer snapshot tests and document startup-data migration.

File	Description
`src/tests/rust_guests/simpleguest/src/main.rs`	Remove the legacy user-memory test function.
`src/tests/rust_guests/simpleguest/src/bin/simpleguest-initial-data.rs`	Add a guest that retains startup buffers.
`src/tests/rust_guests/dummyguest/src/main.rs`	Validate the simplified entry arguments.
`src/hyperlight_host/tests/snapshot_goldens/goldens_version.rs`	Set snapshot goldens to v10.0.
`src/hyperlight_host/tests/integration_test.rs`	Remove assertions for deleted region variants.
`src/hyperlight_host/src/sandbox/uninitialized.rs`	Remove blob wrappers and accept `GuestBinary` directly.
`src/hyperlight_host/src/sandbox/uninitialized_evolve.rs`	Initialize guests without a PEB address.
`src/hyperlight_host/src/sandbox/snapshot/tripwires.rs`	Pin the updated snapshot contract.
`src/hyperlight_host/src/sandbox/snapshot/mod.rs`	Build snapshots without PEB or init-data regions.
`src/hyperlight_host/src/sandbox/snapshot/file/mod.rs`	Load schema v6 and reject older schemas.
`src/hyperlight_host/src/sandbox/snapshot/file/media_types.rs`	Define config v6 and ABI 10.
`src/hyperlight_host/src/sandbox/snapshot/file/config.rs`	Remove init-data fields from serialized configuration.
`src/hyperlight_host/src/sandbox/snapshot/file_tests.rs`	Test retained startup buffers and format compatibility.
`src/hyperlight_host/src/sandbox/initialized.rs`	Update initialization and snapshot tests.
`src/hyperlight_host/src/sandbox/builder.rs`	Remove the initialization-data builder option.
`src/hyperlight_host/src/mem/virtq/tests.rs`	Update the memory-layout fixture.
`src/hyperlight_host/src/mem/mgr.rs`	Update snapshot and layout test constructors.
`src/hyperlight_host/src/mem/memory_region.rs`	Remove PEB and init-data region definitions.
`src/hyperlight_host/src/mem/layout.rs`	Simplify initial memory to guest code and update layout tests.
`src/hyperlight_host/src/mem/exe.rs`	Update tests for `Snapshot::from_binary`.
`src/hyperlight_host/src/hypervisor/mod.rs`	Update the guest initialization test.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/x86_64.rs`	Adjust x86-64 startup registers.
`src/hyperlight_host/src/hypervisor/hyperlight_vm/aarch64.rs`	Adjust AArch64 startup registers.
`src/hyperlight_guest/src/lib.rs`	Remove the guest-handle module export.
`src/hyperlight_guest/src/guest_handle/handle.rs`	Delete the PEB-backed guest handle.
`src/hyperlight_guest_bin/src/lib.rs`	Remove PEB initialization and simplify random seeding.
`src/hyperlight_guest_bin/src/host_comm.rs`	Remove the legacy user-memory reader.
`src/hyperlight_guest_bin/src/arch/amd64/init.rs`	Update entry arguments and stack-pivot register.
`src/hyperlight_guest_bin/src/arch/aarch64/mod.rs`	Update entry arguments and stack-pivot register.
`src/hyperlight_common/src/mem.rs`	Delete shared PEB structures.
`src/hyperlight_common/src/lib.rs`	Remove the deleted memory module export.
`docs/snapshot-versioning.md`	Document ABI 10 and config v6 compatibility.
`docs/snapshot-oci-format.md`	Reference config schema v6.
`docs/how-to-build-a-hyperlight-guest-binary.md`	Explain startup-buffer retention and pool sizing.
`CHANGELOG.md`	Record breaking APIs and migration requirements.

_{Review effort: Balanced}

ludfjig

LGTM, with 1 nit: build-rust-guests-non-pie runs cargo build without --bin, so it also builds this binary. Only simpleguest is copied to non_pie. Adding --bin simpleguest to that recipe avoids an extra build in each guest CI job.

Remove the peb backed init-data region and APIs. Use retained virtqueue ByteChunks for startup data. After removing the init section PEB no longer contains any data so it can be removed as well. Signed-off-by: Tomasz Andrzejak <andreiltd@gmail.com>

hyperlight-gh-bot · 2026-11-07T01:06:23Z

Benchmark Results

Measured commit: 9b9389c7e5bb
Baseline commit: 6c8cf87bad1f

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`771.71 ns` (➖ 1.00x slower)
`vec_bytes`	`602.98 ns` (➖ 1.04x slower)
		`375.23 µs` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`522.76 ns` (➖ 1.00x slower)
`65536`	`141.21 ns` (➖ 1.01x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`13.49 ms` (➖ 1.02x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`7.68 ns` (➖ 1.12x faster)	`7.70 ns` (➖ 1.00x slower)	`7.68 ns` (➖ 1.00x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`99.06 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.27 µs` (➖ 1.01x faster)	`7.33 µs` (➖ 1.01x slower)
`65536`	`1.92 µs` (➖ 1.08x faster)	`2.06 µs` (➖ 1.06x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`26.85 µs` (➖ 1.03x faster)
`65536`	`6.45 µs` (➖ 1.03x slower)	`6.28 µs` (➖ 1.01x faster)
`8192`	`1.06 µs` (➖ 1.03x slower)	`1.06 µs` (➖ 1.02x slower)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`688.85 ns` (➖ 1.00x slower)
`vec_bytes`	`527.54 ns` (➖ 1.01x slower)
		`685.36 µs` (➖ 1.04x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`505.89 ns` (➖ 1.00x slower)
`65536`	`158.78 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.29 ms` (➖ 1.07x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`6.92 ns` (➖ 1.01x slower)	`6.86 ns` (➖ 1.00x faster)	`6.89 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`50.02 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.71 µs` (➖ 1.01x slower)	`7.60 µs` (➖ 1.00x faster)
`65536`	`2.13 µs` (➖ 1.00x slower)	`2.14 µs` (➖ 1.03x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`29.43 µs` (➖ 1.03x faster)
`65536`	`7.15 µs` (➖ 1.01x faster)	`7.16 µs` (➖ 1.02x faster)
`8192`	`716.25 ns` (➖ 1.00x faster)	`737.65 ns` (➖ 1.01x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`929.71 ns` (➖ 1.03x slower)
`vec_bytes`	`704.62 ns` (➖ 1.01x slower)
		`323.46 µs` (➖ 1.00x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`669.99 ns` (➖ 1.00x slower)
`65536`	`181.48 ns` (➖ 1.02x faster)

sandboxes

	`create_initialized_and_drop`
`medium`	`70.60 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`9.87 ns` (➖ 1.01x faster)	`9.86 ns` (➖ 1.00x faster)	`9.89 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`109.38 µs` (➖ 1.01x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`9.13 µs` (➖ 1.00x slower)	`9.07 µs` (➖ 1.04x faster)
`65536`	`2.45 µs` (➖ 1.00x faster)	`2.63 µs` (➖ 1.03x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`34.61 µs` (➖ 1.03x faster)
`65536`	`8.05 µs` (➖ 1.04x faster)	`7.95 µs` (➖ 1.04x faster)
`8192`	`1.25 µs` (➖ 1.08x faster)	`1.30 µs` (➖ 1.07x faster)

mshv3 / intel (Linux) (❌ 1.59x)

Top regressions

payload_allocation/slot_pool_segmented/262144 — ❌ 1.59x slower

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`922.88 ns` (➖ 1.02x slower)
`vec_bytes`	`660.49 ns` (➖ 1.02x slower)
		`627.22 µs` (➖ 1.21x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`1.04 µs` (❌ 1.59x slower)
`65536`	`266.56 ns` (➖ 1.10x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`74.60 ms` (➖ 1.06x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`8.69 ns` (➖ 1.14x faster)	`8.68 ns` (➖ 1.00x slower)	`9.48 ns` (➖ 1.09x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`52.80 µs` (➖ 1.01x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.89 µs` (➖ 1.01x faster)	`7.91 µs` (➖ 1.05x faster)
`65536`	`2.28 µs` (➖ 1.01x faster)	`2.28 µs` (➖ 1.01x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`39.17 µs` (➖ 1.00x faster)
`65536`	`7.46 µs` (➖ 1.00x faster)	`7.52 µs` (➖ 1.00x slower)
`8192`	`891.47 ns` (➖ 1.02x faster)	`899.25 ns` (➖ 1.03x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.16 µs` (➖ 1.01x faster)
`vec_bytes`	`782.24 ns` (➖ 1.00x slower)
		`1.99 ms` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`262144`	`794.75 ns` (➖ 1.01x slower)
`65536`	`238.48 ns` (➖ 1.03x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.38 ms` (➖ 1.06x faster)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`10.01 ns` (➖ 1.00x faster)	`10.05 ns` (➖ 1.01x slower)	`11.11 ns` (➖ 1.12x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`787.25 µs` (➖ 1.02x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`9.25 µs` (➖ 1.09x faster)	`9.23 µs` (➖ 1.01x faster)
`65536`	`2.38 µs` (➖ 1.05x faster)	`2.34 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`40.30 µs` (➖ 1.09x faster)
`65536`	`8.96 µs` (➖ 1.02x slower)	`8.90 µs` (➖ 1.01x slower)
`8192`	`1.29 µs` (➖ 1.06x faster)	`1.33 µs` (➖ 1.01x slower)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`encode_control`	`decode_vec_bytes_copy`
`byte_chunks`	`1.17 µs` (➖ 1.03x faster)
`vec_bytes`	`781.17 ns` (➖ 1.04x slower)
		`2.91 ms` (➖ 1.03x faster)

payload_allocation

	`slot_pool_segmented`
`262144`	`738.16 ns` (➖ 1.02x faster)
`65536`	`209.16 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.06 ms` (➖ 1.04x slower)

slot_pool

	`alloc_dealloc_128`	`alloc_dealloc_4096`	`alloc_dealloc_1500`
	`9.73 ns` (➖ 1.01x slower)	`10.05 ns` (➖ 1.02x slower)	`9.66 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`678.74 µs` (➖ 1.00x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`7.47 µs` (➖ 1.02x faster)	`7.73 µs` (➖ 1.03x slower)
`65536`	`2.46 µs` (➖ 1.03x slower)	`2.32 µs` (➖ 1.01x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`262144`	`42.22 µs` (➖ 1.21x faster)
`65536`	`8.66 µs` (➖ 1.05x slower)	`8.19 µs` (➖ 1.02x faster)
`8192`	`1.08 µs` (➖ 1.01x slower)	`1.22 µs` (➖ 1.16x slower)

_{Reported by cargo ci bench-report --candidate run:42301418743 --config-file bench_report.toml.}

andreiltd requested review from danbugs, dblnz, devigned, jprendes, jsturtevant, ludfjig, simongdavies, squillace and syntactically as code owners November 6, 2026 09:03

Copilot AI balanced review requested due to automatic review settings November 6, 2026 09:03

Copilot started reviewing on behalf of andreiltd November 6, 2026 09:03

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

andreiltd force-pushed the remove-init-data branch from 11f0a16 to e9a13c4 Compare November 6, 2026 09:07

This comment has been minimized.

Sign in to view

andreiltd added kind/refactor For PRs that restructure or remove code without adding new functionality. regen-goldens Regenerate snapshot golden fixtures labels Nov 6, 2026

andreiltd mentioned this pull request Nov 6, 2026

Move the executable heap flag into the PEB and make it a runtime setting #2026

Open

ludfjig previously approved these changes Nov 6, 2026

View reviewed changes

andreiltd dismissed ludfjig’s stale review via 05b0780 November 6, 2026 19:53

andreiltd force-pushed the remove-init-data branch from e9a13c4 to 05b0780 Compare November 6, 2026 19:53

This comment has been minimized.

Sign in to view

Merge branch 'main' into remove-init-data

9b9389c

This comment has been minimized.

Sign in to view

Conversation

andreiltd commented Nov 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

This comment has been minimized.

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

hyperlight-gh-bot Bot commented Nov 7, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Top regressions

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants