Skip to content

Sail: OOM on t3a.small machine. #593

Closed

Description

@alexey-milovidov
No description provided.

Activity

  1. shehabgamin commented on Nov 12, 2025

    @shehabgamin
    Contributor

    Sail 0.4.1 likely resolves this:
    #679

  2. rschu1ze commented on Nov 13, 2025

    @rschu1ze
    Member

    Running sail 0.41 (added here) on t3a.small fails with

    14221312K ........ ........ ........ ........ 98% 54.0M 2s
    14254080K ........ ........ ........ ........ 98%  130M 2s
    14286848K ........ ........ ........ ........ 99% 69.1M 1s
    14319616K ........ ........ ........ ........ 99%  115M 1s
    14352384K ........ ........ ........ ........ 99% 50.2M 1s
    14385152K ........ ........ ........ ........ 99% 48.1M 0s
    14417920K ........ .......                   100%  257M=2m43s
    
    2025-11-13 14:52:31 (86.6 MB/s) - ‘hits.parquet’ saved [14779976446/14779976446]
    
    Traceback (most recent call last):
      File "/ClickBench/sail/./query.py", line 3, in <module>
        from pysail.spark import SparkConnectServer
    ModuleNotFoundError: No module named 'pysail'
    

    This funny error happens because pysail cannot be build earlier:

      note: This error originates from a subprocess, and is likely not a problem with pip.
      full command: /ClickBench/sail/myenv/bin/python3 /ClickBench/sail/myenv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmplwdqskbg
      cwd: /tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35
      Building wheel for pysail (pyproject.toml): finished with status 'error'
      ERROR: Failed building wheel for pysail
    Failed to build pysail
    ERROR: Could not build wheels for pysail, which is required to install pyproject.toml-based projects
    

    ... which happens because parquet cannot be build successfully:

    
         Compiling brotli v8.0.2
         Compiling seq-macro v0.3.6
         Compiling parquet v56.2.0
         Compiling hashbrown v0.14.5
         Compiling bigdecimal v0.4.9
         Compiling xz2 v0.1.7
         Compiling bon-macros v3.8.1
         Compiling libbz2-rs-sys v0.2.2
         Compiling bzip2 v0.6.1
         Compiling bon v3.8.1
         Compiling serde_bytes v0.11.19
         Compiling anyhow v1.0.100
         Compiling apache-avro v0.20.0
         Compiling sqlparser v0.58.0
      error: could not compile `parquet` (lib)
    
      Caused by:
        process didn't exit successfully: `/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name parquet --edition=2021 /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/parquet-56.2.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C codegen-units=1 --cfg 'feature="arrow"' --cfg 'feature="arrow-array"' --cfg 'feature="arrow-buffer"' --cfg 'feature="arrow-cast"' --cfg 'feature="arrow-data"' --cfg 'feature="arrow-ipc"' --cfg 'feature="arrow-schema"' --cfg 'feature="arrow-select"' --cfg 'feature="async"' --cfg 'feature="base64"' --cfg 'feature="brotli"' --cfg 'feature="default"' --cfg 'feature="encryption"' --cfg 'feature="flate2"' --cfg 'feature="flate2-zlib-rs"' --cfg 'feature="futures"' --cfg 'feature="lz4"' --cfg 'feature="lz4_flex"' --cfg 'feature="object_store"' --cfg 'feature="simdutf8"' --cfg 'feature="snap"' --cfg 'feature="tokio"' --cfg 'feature="zstd"' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values("arrow", "arrow-array", "arrow-buffer", "arrow-cast", "arrow-csv", "arrow-data", "arrow-ipc", "arrow-schema", "arrow-select", "arrow_canonical_extension_types", "async", "base64", "brotli", "clap", "cli", "crc", "default", "encryption", "experimental", "flate2", "flate2-rust_backened", "flate2-zlib-rs", "futures", "json", "lz4", "lz4_flex", "object_store", "parquet-variant", "parquet-variant-compute", "parquet-variant-json", "serde", "serde_json", "simdutf8", "snap", "test_common", "tokio", "variant_experimental", "zstd"))' -C metadata=2a6924a297c3344e -C extra-filename=-ebce14d2fb8b0b34 --out-dir /tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps -C strip=symbols -L dependency=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps --extern ahash=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libahash-984de0cf1c7a72a5.rmeta --extern arrow_array=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libarrow_array-cb35d26e5ec65e4d.rmeta --extern arrow_buffer=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libarrow_buffer-81ecfcc714266320.rmeta --extern arrow_cast=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libarrow_cast-d27e4c7ec3a5eda4.rmeta --extern arrow_data=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libarrow_data-e153fe864d924307.rmeta --extern arrow_ipc=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libarrow_ipc-a60244de29e639d0.rmeta --extern arrow_schema=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libarrow_schema-29028523f6dd6e6f.rmeta --extern arrow_select=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libarrow_select-b8c73e7e6471d83d.rmeta --extern base64=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libbase64-e0364e688f623ddb.rmeta --extern brotli=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libbrotli-5239773707001201.rmeta --extern bytes=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libbytes-710cbcdac1d754cf.rmeta --extern chrono=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libchrono-dbc7802aa2d70874.rmeta --extern flate2=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libflate2-2fb994863dfe1a66.rmeta --extern futures=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libfutures-11f31fe12b36ce36.rmeta --extern half=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libhalf-3f696188e4bc6ff8.rmeta --extern hashbrown=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libhashbrown-97fa42b468f3aed9.rmeta --extern lz4_flex=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/liblz4_flex-3d09acc2d7e0f233.rmeta --extern num=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libnum-fe556af4dd806228.rmeta --extern num_bigint=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libnum_bigint-7c844b215c97b926.rmeta --extern object_store=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libobject_store-aa65b4e918b1ec3b.rmeta --extern paste=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libpaste-18dd9b821b35dfb0.so --extern ring=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libring-51953c953e996da6.rmeta --extern seq_macro=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libseq_macro-ec6cdea0f97d0c95.so --extern simdutf8=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libsimdutf8-97fa5fbb520a5ac5.rmeta --extern snap=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libsnap-29bbf04151ce02e8.rmeta --extern thrift=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libthrift-a1c90bb637d50563.rmeta --extern tokio=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libtokio-012e2e3af3f2920d.rmeta --extern twox_hash=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libtwox_hash-facd70d793927392.rmeta --extern zstd=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/deps/libzstd-46a1020d4b2a4816.rmeta --cap-lints allow -C target-cpu=native -L native=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/build/zstd-sys-eb5c5be095bfc64e/out -L native=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/build/aws-lc-sys-51b83bbf600986c8/out -L native=/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/target/release/build/ring-dd7990a9f0021ca3/out` (signal: 9, SIGKILL: kill)
      warning: build failed, waiting for other jobs to finish...
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_BUILD_EXTENSION_MODULE="1" PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="/ClickBench/sail/myenv/bin/python3" PYTHON_SYS_EXECUTABLE="/ClickBench/sail/myenv/bin/python3" "cargo" "rustc" "--profile" "release" "--features" "pyo3/extension-module" "--features" "pyo3/abi3-py38" "--features" "pyo3/generate-import-lib" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/pip-install-lrtuoi63/pysail_b4807127b28440b597dd4db3ec033f35/crates/sail-python/Cargo.toml" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/ClickBench/sail/myenv/bin/python3', '--compatibility', 'off'] returned non-zero exit status 1
      error: subprocess-exited-with-error
    

    which is probably because 2 cores and 2 GB DRAM are too little resources.

  3. shehabgamin commented on Nov 13, 2025

    @shehabgamin
    Contributor

    @rschu1ze Oh interesting, I guess building from source becomes problematic on instances with small resources

    env RUSTFLAGS="-C target-cpu=native" pip install --no-cache-dir "pysail==0.4.1" -v --no-binary pysail
    

    Although, when optimizing for performance, building Sail from source is the preferred installation method.

  4. alexey-milovidov commented on Nov 13, 2025

    @alexey-milovidov
    MemberAuthor

    Use the default installation method, recommended in Sail's documentation: https://docs.lakesail.com/sail/latest/introduction/getting-started/

  5. alexey-milovidov commented on Nov 13, 2025

    @alexey-milovidov
    MemberAuthor

    Otherwise, the system will fall into the "tuned" category.

  6. shehabgamin commented on Nov 13, 2025

    @shehabgamin
    Contributor

    @alexey-milovidov For a bit of context, I'm one of the creators of Sail. The Python package is intended as a simple way for people to get started quickly, but for production environments we generally recommend building Sail from source.

    For the ClickBench script, I chose to build Sail from source via the Python package simply because it kept the benchmark setup straightforward. If I were aiming to squeeze out maximum performance, I would skip PySail entirely.

    Using:

    env RUSTFLAGS="-C target-cpu=native" pip install --no-cache-dir "pysail==0.4.1" -v --no-binary pysail
    

    is essentially the same process used to build any Rust-based engine (for example, DataFusion’s ClickBench script).

    Because of this, I’m not sure this should fall into the "tuned" category. Otherwise, by the same logic, every Rust-based engine would also be considered "tuned"?

    Happy to adjust if you think there’s a better way to keep things consistent across engines.

  7. rschu1ze commented on Nov 14, 2025

    @rschu1ze
    Member

    Imho Joe Average user shouldn't be bothered with compiling any software from scratch. Compiling software is fine for development purposes. In the context of a benchmark, it opens the door for more specific optimizations, e.g. compiling specifically for the target machine (think march=native for GCC) which would be an unfair advantage over other databases (I am not saying that you are doing this EDIT: You are doing this.).

    I would prefer a plain and simple binary install as described in the Sail docs as well.

  8. alexey-milovidov commented on Nov 14, 2025

    @alexey-milovidov
    MemberAuthor

    It will be okay if the official docs use it as the first (preferred) installation method; otherwise, it is a little bit strange.

  9. shehabgamin commented on Nov 14, 2025

    @shehabgamin
    Contributor

    @rschu1ze I see, that reasoning makes sense to me.

    With that in mind, this:

    env RUSTFLAGS="-C target-cpu=native" pip install --no-cache-dir "pysail==0.4.1" -v --no-binary pysail
    

    Can change to this:

    pip install "pysail==0.4.2"
    

    A new version of Sail came out, so we can bump the version too.

    I'm just getting offline now, but i'm happy to whip up a PR when I'm back online if needed.

  10. shehabgamin commented on Nov 14, 2025

    @shehabgamin
    Contributor

    It will be okay if the official docs use it as the first (preferred) installation method; otherwise, it is a little bit strange.

    Installation from source is the first sub-section in the docs. Imo first installation method should generally be geared towards quickly getting started and then immediately after it makes sense to talk about building for production.

    https://docs.lakesail.com/sail/latest/introduction/installation/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions