@Rocky please help Cancelled: transmit chunk rpc failed [dest_instance_id=01a056bf-33ed-778b-816b-f7c46cc1e87f] [dest=10.213.50.5:8060] detail:brpc failed, error=The server is overcrowded, error_text=[E1011]The server is overcrowded @10.213.50.5:8060 [R1][E1011]The server is overcrowded @10.213.50.5:8060 [R2][E1011]The server is overcrowded @10.213.50.5:8060 [R3][E1011]The server is overcrowded @10.213.50.58060 be10.213.48.80 BE:10002

The error

[E1011]The server is overcrowded

is a specific bRPC error indicating that the destination node's network buffer is full, or the server is unable to process incoming requests fast enough to clear its transmission queue. This typically happens during heavy data shuffle operations or when transferring large data blocks (such as large Bitmap or HLL types). Recommended Solutions 1. Increase bRPC Socket Buffer The most common fix is to increase the

brpc_socket_max_unwritten_bytes

parameter in the

be.conf

(or

cn.conf

if using Compute Nodes) of all nodes in your cluster. This parameter limits the amount of unwritten data allowed in a single bRPC socket. * Parameter:

brpc_socket_max_unwritten_bytes

* Default: Often 1 GB (1073741824) or 64 MB depending on the version. * Recommended Value: 4 GB (

4294967296

). * Action: Add or update this line in your

be.conf

and restart the BE:

conf
    brpc_socket_max_unwritten_bytes = 4294967296

2. Adjust Connection Settings If the cluster experiences high network or RPC latency even when resources are normal, you can try changing the bRPC connection type: * Parameter:

brpc_connection_type

* Action: Change to

pooled

in

be.conf

and restart.

conf
    brpc_connection_type = pooled

3. Manage Query Concurrency If the error occurs during high-concurrency periods: * Reduce `pipeline_dop`: For the specific query causing the issue, try reducing the degree of parallelism (DOP) to lower the RPC pressure:

sql
    SET pipeline_dop = 2; -- or lower than current

* Check Resource Pressure: Verify if the target BE (at

10.213.50.5

in your log) is experiencing 100% CPU usage or high memory pressure, which can slow down the bRPC thread's ability to clear the queue. Why this happens The

E1011

error is triggered when the size of "unwritten bytes" on a socket exceeds the threshold. This occurs when: * The sender is pushing data faster than the receiver can consume it. * The query involves very large individual rows or complex data structures (like Bitmaps) that exceed the default buffer during shuffle operations. * The BE threads are overloaded and cannot keep up with the network I/O. If the issue persists after increasing

brpc_socket_max_unwritten_bytes

, check the BE logs (

<http://be.INFO|be.INFO>

) on the destination node (

10.213.50.5

) for any signs of long GC pauses or CPU saturation. References • knowledge/usage/best-practice.md • faq/loading/Loading_faq.md • faq/loading/Routine_load_faq.md • faq/Deploy_faq.md • faq/loading/Broker_load_faq.md

@Rocky do i need to increase: brpc_max_body_size

Generally, yes, you should increase

brpc_max_body_size

alongside

brpc_socket_max_unwritten_bytes

to ensure they are aligned. Why you should increase both The "The server is overcrowded" (E1011) error specifically triggers when the socket buffer (

brpc_socket_max_unwritten_bytes

) is full. However: 1.

brpc_socket_max_unwritten_bytes

limits the total amount of data waiting to be sent on a socket. 2.

brpc_max_body_size

limits the size of a single RPC message body. If you increase the socket buffer to 4 GB but leave the body size at its default (usually 2 GB), a single large data chunk (e.g., a very large Bitmap or HLL) that exceeds 2 GB will still fail, though it might throw a different error like "Request body is too large." Recommended Configuration To avoid both "overcrowded" and "body too large" errors during heavy data transfers, update both in your `be.conf`:

conf
# Fixes E1011 "The server is overcrowded"
brpc_socket_max_unwritten_bytes = 4294967296

# Ensures single large messages are allowed (align with the above)
brpc_max_body_size = 4294967296

Note: * Both parameters require a restart of the BE/CN node to take effect. * The value

4294967296

corresponds to 4 GB. * If your machine has limited memory, be aware that increasing these values allows for more memory usage per connection during spikes in network traffic. References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md

@Rocky can i set them at session level to test once

No, you cannot set

brpc_socket_max_unwritten_bytes

or

brpc_max_body_size

at the session level. These are BE configuration parameters that govern the underlying bRPC network layer for the entire process, not just a single query. According to the StarRocks documentation: * Is mutable: No – This means these parameters are not hot-reloadable. You cannot change them while the process is running, even globally. * Requires Restart: To apply changes to these values, you must modify the

be.conf

(or

cn.conf

) on each node and restart the BE/CN service. How to test safely Since a restart is required, if you want to minimize impact while testing: 1. Test on one node first: If your cluster has multiple BEs, you can update and restart just one BE to see if the "overcrowded" errors associated with that specific IP disappear. 2. Monitor Memory: Increasing these values allows the BE to buffer more data in memory during network spikes. Keep an eye on the BE process memory usage after the restart, especially during heavy load or large shuffles. Summary of required steps: 1. Edit

conf/be.conf

. 2. Add/Update:

conf
    brpc_socket_max_unwritten_bytes = 4294967296
    brpc_max_body_size = 4294967296

3. Restart the BE:

bash
    bin/stop_be.sh
    bin/start_be.sh --daemon

References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md

@Rocky but my mv query failing

The "The server is overcrowded" (E1011) error during a Materialized View (MV) operation typically happens because the data shuffle is too intense for the default network buffers. While you cannot change the bRPC buffer settings without a restart, you can use session variables to "throttle" the query and reduce the pressure on the network, which often allows the query to complete. 1. If you are QUERYING the Materialized View If you are running a

SELECT

against the MV and it fails, try these session-level settings before running your query:

sql
-- 1. Reduce the degree of parallelism (DOP). 
-- This reduces the number of concurrent network 'chunks' being sent.
SET pipeline_dop = 2; 

-- 2. Increase memory limit to ensure data is handled properly.
SET query_mem_limit = 8589934592; -- 8GB (adjust as needed)

-- 3. Enable spill to disk (if using v3.0+) to avoid huge memory/network spikes.
SET enable_spill = true;

-- Run your query now
SELECT * FROM your_mv ...;

2. If the Materialized View REFRESH is failing If the error occurs during the automatic or manual refresh of the MV (the background task that updates the data), you can apply these settings to the MV's own properties so they apply to every refresh:

sql
ALTER MATERIALIZED VIEW your_mv_name SET (
    "session.pipeline_dop" = "2",
    "session.enable_spill" = "true",
    "session.query_mem_limit" = "8589934592"
);

Why this helps *

pipeline_dop

: By default, StarRocks tries to use all available cores. If you have 32 cores, it might try to open 32+ connections per fragment. Reducing this to

2

or

4

significantly lowers the number of simultaneous bRPC requests, preventing the "overcrowded" buffer state. *

enable_spill

: This prevents the query from trying to keep everything in memory and flushing it all at once over the network, which can cause traffic bursts that trigger the E1011 error. Summary recommendation If these session settings work, it confirms that your network buffers (

brpc_socket_max_unwritten_bytes

) are indeed too small for your data volume. You should still plan to update your

be.conf

and restart when possible for a permanent fix, as throttling with

pipeline_dop

will make your queries/refreshes run slower. References • knowledge/usage/best-practice.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • administration/management/resource_management/Query_management.md • sql-reference/System_variable.md • administration/management/monitor_manage_big_queries.md • using_starrocks/Cost_based_optimizer.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/resource_management/query_queues.md • using_starrocks/async_mv/materialized_view_task_run_details.md • faq/Sql_faq.md • administration/management/resource_management/Memory_management.md References