Skip to content

Add support for native macOS #12

Closed

Description

@zfarrell

Native macOS is not supported today. This ticket records what I measured, and what I think it means. It is not a plan.

Test system: aarch64-apple-darwin, rustc 1.94.1 stable.

Evidence: the build

cargo check --workspace --all-targets fails in one crate only.

The benchmark crate uses perf-event2 (benchmark/Cargo.toml:44). This library needs perf_event_open. Only Linux has this system call. The build gives 6 errors.

Two other libraries in the same crate do build on macOS. I tested usdt 0.6 and pprof 0.15 in a separate crate. Both compile.

The dev-tools crate also fails, but it fails for the generated assets/tailwind.css file. This happens on Linux too. It is not related to the platform.

Evidence: the tests

Crate Pass Fail
liquid-cache 218 18
liquid-cache-common 8 0
liquid-cache-datafusion 41 21
liquid-cache-datafusion-client 28 0
liquid-cache-datafusion-local 31 1
liquid-cache-datafusion-server 3 12
doc-tests — 6

57 of the 58 failures give the same message:

InvalidArgument("direct_io not supported on target_os")

t4::MountOptions::default() sets direct_io: true (t4-0.1.7 src/store.rs:28-34). t4 gives an error for this option on all systems that are not Linux (src/store.rs:60-67).

Two of these failures are not test-only:

  • src/core/src/cache/builders.rs:140 — LiquidCacheBuilder::build() stops the program. This is the README example. examples/core.rs:10 also uses this path.
  • src/datafusion-server/src/service.rs:61 — the Flight server stops at start.

There are 15 t4::mount( calls in 10 files.

src/datafusion-local/src/lib.rs:192-197 does not use the default. It sets direct_io: cfg!(target_os = "linux"). Its 31 tests pass. This shows the buffered I/O path of t4 operates correctly on macOS. It writes to disk, reads from disk, and removes entries.

Evidence: one different failure

liquid-cache-datafusion-local tests::test_provide_schema2 fails for a different reason. It compares usage.memory_bytes with a recorded value. macOS gives a value 935 bytes lower at all three points (1000915 → 999980, 1036304 → 1035369).

The value is the same in each run. The temporary directory name is not the cause. I ran the test again with TMPDIR=/tmp and got the same value.

Six other snapshots record an exact usage.memory_bytes. All six pass on macOS. Four of them write to disk and read from disk:

Snapshot memory_bytes disk_bytes
referer_filtering 917947 877216
url_selection_and_ordering 189553 706252
os_selection 954618 164536
url_prefix_filtering 912473 139416

Hypotheses

The 935-byte difference relates to the data, not to the platform. test_provide_schema2 is the only test that reads dev/test_parquet/openobserve.parquet (tests/mod.rs:32). All other tests read nano_hits.parquet.

I removed three other possible causes:

  • Not the data type sizes. Both processors use the same sizes.
  • Not the memory allocator. Arrow makes buffer sizes a multiple of 64. 935 is not a multiple of 64. A counter that uses len() is more probable, for example src/core/src/liquid_array/raw/fsst_buffer.rs:134-136.
  • Not the I/O mode. The four snapshots above use buffered I/O on macOS and direct I/O on Linux. They give the same values.

A change to buffered I/O has a cost that is not visible. The operating system then keeps a second copy of the data in its own cache. The admission gate assumes that counted bytes are bytes in memory (src/datafusion/src/optimizers/mod.rs:58). This is not correct with buffered I/O. A developer can measure incorrect behavior on a Mac.

A quiet fallback is a risk. README.md:96 tells the user that direct I/O makes LiquidCache look slower in development. On macOS the opposite is true.

Three code paths are different on macOS. Two are not tested today.

  • t4 has two I/O methods (t4/src/io/io_worker.rs:11-15). Linux uses io_uring.rs. All other systems use generic.rs. generic.rs has unsafe code (unsafe impl Send, from_raw_parts_mut). No job in CI runs it.
  • src/datafusion/src/utils.rs:75-79 uses a BMI2 function on x86_64. This is unsafe code on the filter path. The two tests that compare it with the standard function are #[cfg(target_arch = "x86_64")] (utils.rs:316, :359). They do not run on ARM.

t4::mount() is the origin of the problem. Its default option set cannot mount on the platform that compiled it. A correction in t4 (fcntl(F_NOCACHE) is the macOS equivalent of O_DIRECT) would remove the problem for all 15 call sites and keep the single-copy property. t4 is a different repository.

Not a problem

t4::PAGE_SIZE is a constant 4096 (t4/src/lib.rs:28). It does not read the system page size. The 16 KB pages of Apple Silicon do not make store files incompatible.

Open questions

  • Do we support macOS for development only, or also for the published crates?
  • If a fallback to buffered I/O is acceptable, is it quiet or does it give a warning?
  • Is a Linux virtual machine a better answer for developers who need correct performance data?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions