@Rocky how to change streaming_load_max_mb from 100m --> 200m?

To change the maximum size for streaming loads from 100MB to 200MB, you are likely looking for the BE parameter

streaming_load_max_batch_size_mb

(which defaults to 100MB, specifically for JSON files) rather than

streaming_load_max_mb

(which defaults to 100GB in recent versions). You can update this configuration using one of the following methods: Method 1: Dynamic Update (No restart required) You can use a

curl

command to update the configuration on each Backend (BE) node immediately. Replace

<be_host>

and

<be_http_port>

(default 8040) with your actual values:

bash
curl -XPOST http://<be_host>:<be_http_port>/api/update_config?streaming_load_max_batch_size_mb=200

Note: If you have multiple BEs, you must run this on each one. Method 2: Permanent Configuration (Requires restart) To ensure the change persists after a restart, add or update the parameter in the

be.conf

file of each BE: 1. Open

be.conf

(usually located in the

conf/

directory of your BE installation). 2. Add or modify the following line:

conf
   streaming_load_max_batch_size_mb = 200

3. Restart the BE service. ────────── Key Distinctions *

streaming_load_max_batch_size_mb

: Defaults to 100MB. It controls the maximum size of a single JSON file/batch. *

streaming_load_max_mb

: Defaults to 100GB (v3.0+) or 10GB (older). It controls the overall maximum size of a file that can be streamed. * JSON Workaround: If you are hitting the 100MB limit while loading JSON, you can also add

"ignore_json_size:true"

to your HTTP request header to bypass the check without changing system configurations. References • administration/management/BE_configuration.md • faq/Deploy_faq.md

@Rocky http://10.0.27.71:8040/api/update_config?streaming_load_max_batch_size_mb=200 Not Found

The "Not Found" error occurs because the

/api/update_config

endpoint only accepts

POST

requests. If you pasted that link into a web browser or used a standard

curl

without

-XPOST

, the server will return a 404 or 405 error because it does not support

GET

requests for configuration changes. To fix this, use the following

curl

command in your terminal:

bash
curl -XPOST "<http://10.0.27.71:8040/api/update_config?streaming_load_max_batch_size_mb=200>"

Important Checks: 1. POST Method: Ensure the

-XPOST

flag is included. 2. Quotes: Wrap the URL in double quotes if your shell has trouble with the

?

or

=

characters. 3. Authentication: If your cluster has HTTP authentication enabled, you must provide your credentials:

bash
    curl -u admin:password -XPOST "<http://10.0.27.71:8040/api/update_config?streaming_load_max_batch_size_mb=200>"

4. Verify the change: You can verify if the change took effect by running:

bash
    curl <http://10.0.27.71:8040/varz> | grep streaming_load_max_batch_size_mb

Note: This command only updates the configuration on the specific Backend (BE) at

10.0.27.71

. If you have multiple BEs, you must run it for each one or update

be.conf

and restart to make it permanent cluster-wide. References • administration/management/BE_configuration.md • administration/http_interface.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md

@Rocky what recommend value for this mb number?

There is no single "perfect" value, but here are the recommendations based on how StarRocks handles memory: 1. The "Safe" Range: 200MB – 500MB If your JSON files frequently hover just above 100MB, increasing this to 200MB or 500MB is generally safe for most production environments. It provides flexibility without significantly risking the stability of the Backend (BE) nodes. 2. When to use

ignore_json_size: true

(Recommended over config changes) Instead of increasing the global limit, the recommended "best practice" for large JSON files is to keep the default config and add a header to your load command:

bash
-H "ignore_json_size: true"

This allows a specific large file to pass through without changing the safety limit for all other concurrent loads. 3. Considerations for Higher Values If you are considering values higher than 500MB (e.g., 1GB+), keep these trade-offs in mind: * Memory Usage: JSON parsing is memory-intensive. StarRocks must load the JSON into memory to parse it. If you have many concurrent stream loads, setting this value too high can lead to BE OOM (Out of Memory) crashes. * Performance: Smaller batches (under 500MB) are typically more efficient because they allow StarRocks to parallelize the work better across the cluster. * Hard Limit: Regardless of this setting, a single JSON object (one row) cannot exceed 4GB. Summary Recommendation * For minor adjustments: Set to 200MB (as you planned). * For large files (1GB+): Keep the default 100MB and use the

-H "ignore_json_size: true"

header. * Best for Stability: Split files into 100MB-200MB chunks rather than increasing the limit to several GBs. References • loading/StreamLoad.md • administration/management/BE_configuration.md References • administration/management/BE_configuration.md • loading/loading_introduction/loading_considerations.md • loading/StreamLoad.md