Skip to content

Fix pointer overflows when splitting holes near the end of the address space - #93

Open
AlixANNERAUD wants to merge 3 commits into
rust-osdev:mainfrom
AlixANNERAUD:fix-split-pointer-overflow
Open

AlixANNERAUD wants to merge 3 commits into
rust-osdev:mainfrom
AlixANNERAUD:fix-split-pointer-overflow

Conversation

@AlixANNERAUD

@AlixANNERAUD AlixANNERAUD commented Nov 17, 2026 •

Copy link
Copy Markdown

Problem

Cursor::split_current computes the placement of an allocation inside a hole with wrapping pointer arithmetic and then compares the results against the end of the hole. If the hole is close to the end of the address space, these calculations can wrap around, and a hole that cannot serve the request is accepted. There are three places where this happens:

  1. Aligned start address: align_up(new_start, required_align) wraps around to 0 when the next aligned address is behind the end of the address space.
  2. Allocation end: aligned_addr.wrapping_add(required_size) wraps around when the aligned address is valid, but the allocation does not fit below the end of the address space.
  3. Back padding end: back_padding_start.wrapping_add(hole_layout.size()) wraps around when the hole ends less than one hole node below the end of the address space.

In all three cases the wrapped address compares as <= hole_end, so the split is carried out. Hole nodes are then written to addresses outside of the heap and the hole list is corrupted. This is reachable through the safe Heap::allocate_first_fit (and GlobalAlloc::alloc) with a valid Layout.

It requires a heap that is located near the end of the address space (closer than the requested alignment or size), so most users are not affected. Heaps in the higher half of the address space combined with large alignments are the realistic case.

Reproduction

I reproduced it on wasm32-unknown-unknown (linear memory grown to 4 GiB, run with Node.js), because that allows placing a heap at the end of the address space:

Scenario Before After
4096 byte heap at 0xF000_0000, allocate Layout(16, align 1 GiB) (case 1) Release: returns Err, but writes a hole node to address 0x10. A following 8192 byte allocation returns 0x10. Debug: panics with a subtraction overflow. Err, no write, following allocation fails
1 MiB heap ending at 0xFFFF_FFF8, allocate 16 bytes, then Layout(600 KiB, align 512 KiB) (case 2) Returns 0xFFF8_0000 (the allocation wraps around the address space) and writes a hole node to address 0x16000, in debug and release Err, no write

Fix

  • The first commit moves the placement calculation out of split_current into a function that only does address calculations (split_hole). It is a pure move without functional changes. This makes it possible to test the calculation for holes at arbitrary addresses, without needing memory there.
  • The second commit replaces the wrapping operations with checked ones and treats the hole as unsuitable if a calculation overflows.

Testing

  • New unit tests for the three cases above, using a hole address just below the end of the address space, plus one test for a regular split at such an address. The three overflow tests fail without the fix.
  • cargo test on nightly with --all-features, on stable with and without use_spin, and cargo miri test --all-features with -Zmiri-strict-provenance all pass.
  • The two wasm scenarios above, before and after the change.

Pure refactoring, no functional change. Computing the split without
access to the cursor makes it testable for holes at arbitrary addresses.
The address calculations for splitting a hole used wrapping pointer
arithmetic and compared the results against the end of the hole. For
holes close to the end of the address space, the calculated addresses
could wrap around, so that unsuitable holes were considered a fit:

- the aligned start address wrapped around to zero for large alignments
- the end address of the allocation wrapped around
- the end address of the back padding hole node wrapped around

As a result, hole nodes were written to addresses outside of the heap
and the hole list was corrupted. Use checked arithmetic instead and
treat such holes as unsuitable.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant