Skip to content

Maybe the max buffer capacity should be customized #3046

Description

@cichengzi

Feature Description (功能描述)

In some extreme big data query tasks, if the amount of data I need to export is very large, MAX_BUFFER_CAPACITY will limit the export. This parameter is located in BytesBuffer under org.apache.hugegraph.backend.serializer in hugegraph-core. It defaults to 128MB and does not support customization. It is desirable to enable customization through configuration files.

Activity

  1. imbajin commented on Jun 2, 2026

    @imbajin
    Member

    @dosu could u analysis & answer the question?

  2. added theissue type on Jun 2, 2026
  3. dosubot commented on Jun 2, 2026

    @dosubot

    This is a valid feature request. Here's the analysis:

    Current Behavior:
    MAX_BUFFER_CAPACITY is hardcoded as 128 * 1024 * 1024 (128 MB) in BytesBuffer.java. It acts as a hard ceiling in two places:

    1. Buffer creation — throws IllegalArgumentException if the requested capacity exceeds 128 MB [1]
    2. Dynamic resizing via require() — fails when the expanded capacity would exceed 128 MB [2]

    This means any single serialization/deserialization operation that produces data larger than 128 MB will fail, which can block large data exports.

    Developer Intent:
    The code already contains // TODO: support user-defined configuration comments for related constants (ID_LEN_MAX, BYTES_LEN_MAX) [3], indicating the team recognized the need but hasn't implemented it yet.

    Feasibility:
    HugeGraph's CoreOptions class already defines several similar configurable memory limits [4] — e.g., MAX_MEMORY_CAPACITY (1 GB default), TASK_INPUT_SIZE_LIMIT (16 MB), and TASK_RESULT_SIZE_LIMIT (16 MB). Adding a new option for MAX_BUFFER_CAPACITY would follow the same pattern:

    public static final ConfigOption<Integer> SERIALIZER_BUFFER_MAX_CAPACITY =
        new ConfigOption<>(
            "serializer.buffer_max_capacity",
            "Maximum capacity of the serialization buffer in bytes.",
            rangeInt(64 * 1024 * 1024, 512 * 1024 * 1024), // example: 64MB–512MB
            128 * 1024 * 1024 // default 128MB
        );

    Implementation steps would include:

    1. Add a ConfigOption in CoreOptions
    2. Modify BytesBuffer to read the configured value (likely cached at initialization for performance in hot paths like require())
    3. Update references in LZ4 compression utilities that also use this constant

    This seems like a reasonable enhancement given the existing pattern and the TODO comments already in the code.

    To reply, just mention @dosu.


    Share context across your team and agents. Try Dosu.

    Leave Feedback Learn about hugegraph with Dosu Add Dosu to your team

  4. LegendPei commented on Jun 4, 2026

    @LegendPei
    Contributor

    @imbajin I would like to try it,I will address this issue through the plan outlined by dosubot

  5. imbajin commented on Jun 4, 2026

    @imbajin
    Member

    @LegendPei Thanks, I assigned this issue to you.

    I agree this is a reasonable scoped improvement, but let's keep it simple and avoid over-designing it. Please focus on making the single BytesBuffer max capacity configurable while keeping the current default behavior.

    Suggested scope:

    • Add a CoreOptions config such as serializer.buffer_max_capacity, defaulting to the current 128MB.
    • Use the configured value in BytesBuffer capacity checks and related LZ4Util logic.
    • Keep a reasonable upper bound and make error messages show the effective limit.
    • Add focused unit tests and update the sample config/docs if needed.

    Note: this option should be treated as the max capacity of one serialization buffer, not as a general solution for huge exports. Large exports should still rely on paging/streaming when possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions