You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Maybe the max buffer capacity should be customized #3046
In some extreme big data query tasks, if the amount of data I need to export is very large, MAX_BUFFER_CAPACITY will limit the export. This parameter is located in BytesBuffer under org.apache.hugegraph.backend.serializer in hugegraph-core. It defaults to 128MB and does not support customization. It is desirable to enable customization through configuration files.
This is a valid feature request. Here's the analysis:
Current Behavior: MAX_BUFFER_CAPACITY is hardcoded as 128 * 1024 * 1024 (128 MB) in BytesBuffer.java. It acts as a hard ceiling in two places:
Buffer creation — throws IllegalArgumentException if the requested capacity exceeds 128 MB [1]
Dynamic resizing via require() — fails when the expanded capacity would exceed 128 MB [2]
This means any single serialization/deserialization operation that produces data larger than 128 MB will fail, which can block large data exports.
Developer Intent:
The code already contains // TODO: support user-defined configuration comments for related constants (ID_LEN_MAX, BYTES_LEN_MAX) [3], indicating the team recognized the need but hasn't implemented it yet.
Feasibility:
HugeGraph's CoreOptions class already defines several similar configurable memory limits [4] — e.g., MAX_MEMORY_CAPACITY (1 GB default), TASK_INPUT_SIZE_LIMIT (16 MB), and TASK_RESULT_SIZE_LIMIT (16 MB). Adding a new option for MAX_BUFFER_CAPACITY would follow the same pattern:
I agree this is a reasonable scoped improvement, but let's keep it simple and avoid over-designing it. Please focus on making the single BytesBuffer max capacity configurable while keeping the current default behavior.
Suggested scope:
Add a CoreOptions config such as serializer.buffer_max_capacity, defaulting to the current 128MB.
Use the configured value in BytesBuffer capacity checks and related LZ4Util logic.
Keep a reasonable upper bound and make error messages show the effective limit.
Add focused unit tests and update the sample config/docs if needed.
Note: this option should be treated as the max capacity of one serialization buffer, not as a general solution for huge exports. Large exports should still rely on paging/streaming when possible.
Activity
imbajin commentedon Jun 2, 2026
@dosu could u analysis & answer the question?
dosubot commentedon Jun 2, 2026
This is a valid feature request. Here's the analysis:
Current Behavior:
MAX_BUFFER_CAPACITYis hardcoded as128 * 1024 * 1024(128 MB) inBytesBuffer.java. It acts as a hard ceiling in two places:IllegalArgumentExceptionif the requested capacity exceeds 128 MB [1]require()— fails when the expanded capacity would exceed 128 MB [2]This means any single serialization/deserialization operation that produces data larger than 128 MB will fail, which can block large data exports.
Developer Intent:
The code already contains
// TODO: support user-defined configurationcomments for related constants (ID_LEN_MAX,BYTES_LEN_MAX) [3], indicating the team recognized the need but hasn't implemented it yet.Feasibility:
HugeGraph's
CoreOptionsclass already defines several similar configurable memory limits [4] — e.g.,MAX_MEMORY_CAPACITY(1 GB default),TASK_INPUT_SIZE_LIMIT(16 MB), andTASK_RESULT_SIZE_LIMIT(16 MB). Adding a new option forMAX_BUFFER_CAPACITYwould follow the same pattern:Implementation steps would include:
ConfigOptioninCoreOptionsBytesBufferto read the configured value (likely cached at initialization for performance in hot paths likerequire())This seems like a reasonable enhancement given the existing pattern and the TODO comments already in the code.
To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
LegendPei commentedon Jun 4, 2026
@imbajin I would like to try it,I will address this issue through the plan outlined by dosubot
imbajin commentedon Jun 4, 2026
@LegendPei Thanks, I assigned this issue to you.
I agree this is a reasonable scoped improvement, but let's keep it simple and avoid over-designing it. Please focus on making the single
BytesBuffermax capacity configurable while keeping the current default behavior.Suggested scope:
CoreOptionsconfig such asserializer.buffer_max_capacity, defaulting to the current 128MB.BytesBuffercapacity checks and relatedLZ4Utillogic.Note: this option should be treated as the max capacity of one serialization buffer, not as a general solution for huge exports. Large exports should still rely on paging/streaming when possible.