http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I am ingesting 1 TB of data in parq...
# questions-and-troubleshooting
y
@Rocky I am ingesting 1 TB of data in parquet files on S3 using a PIPE. Looking at the pipe I get some errors:
Copy code
{
    "errorMessage": "execution failed: Memory of process exceed limit. Pipeline Backend: kube-starrocks-cn-1.kube-starrocks-cn-search.data-infra.svc.cluster.local, fragment: 019db962-a007-7642-8574-f73a949b405b Used: 7015933520, Limit: 6957847018. Mem usage has exceed the limit of BE: BE:10002",
    "errorTime": "2026-04-23 16:09:04"
}
{
    "errorMessage": "execution failed: Memory of process exceed limit. Pipeline Backend: kube-starrocks-cn-1.kube-starrocks-cn-search.data-infra.svc.cluster.local, fragment: 019db962-a007-7642-8574-f73a949b405b Used: 7015933520, Limit: 6957847018. Mem usage has exceed the limit of BE: BE:10002",
    "errorTime": "2026-04-23 16:09:04"
}
{
    "errorMessage": "task failed with unknown status",
    "errorTime": "2026-04-23 16:29:27"
}
{
    "errorMessage": "task failed with unknown status",
    "errorTime": "2026-04-23 17:20:03"
}
{
    "errorMessage": "task failed with unknown status",
    "errorTime": "2026-04-23 17:41:05"
}
{
    "errorMessage": "execution failed: Memory limit exceeded: Memory of process exceed limit. try consume:640 Backend: kube-starrocks-cn-2.kube-starrocks-cn-search.data-infra.svc.cluster.local, Used: 4442505304, Limit: 6957847018. Mem usage has exceed the limit of BE: BE:10003",
    "errorTime": "2026-04-23 17:59:31"
}
{
    "errorMessage": "execution failed: Memory of process exceed limit. Pipeline Backend: kube-starrocks-cn-2.kube-starrocks-cn-search.data-infra.svc.cluster.local, fragment: 019db9e1-49bb-7a24-9017-d2a0cdacca94 Used: 6967340384, Limit: 6957847018. Mem usage has exceed the limit of BE: BE:10003",
    "errorTime": "2026-04-23 18:30:35"
}
I have 3 CN nodes with 8 CPUs and 32 GB RAM The parameters set at the moment are:
Copy code
connector_scan_use_query_mem_ratio	0.3
load_mem_limit	0
query_mem_limit	0
query_queue_mem_used_pct_limit	0.0
scan_use_query_mem_ratio	0.3
disable_spill_to_local_disk	false
enable_spill	true
enable_spill_to_remote_storage	false
spill_enable_direct_io	false
spill_encode_level	7
spill_mode	auto
spill_partitionwise_agg	false
spill_revocable_max_bytes	0
spill_storage_volume
What do I have to do ? Also, how can I speed this up, as it is loading at the moment 1 file at a time
r
Hello! I'm Rocky, and I'd be happy to help you with your StarRocks PIPE ingestion. Based on the errors and configurations you provided, you are hitting the load memory limit of your Compute Nodes (CN), and your ingestion is likely slow because the Pipe is processing files in very small batches, not utilizing the full parallelism of your 3-node cluster. 1. Fixing the "Memory of process exceed limit" Error The error message shows a limit of approximately 6.48 GiB (6,957,847,018 bytes). On a node with 32 GB RAM, this limit is being calculated based on the default 
load_process_max_memory_limit_percent
parameter. The Calculation: By default, StarRocks reserves memory as follows: * Total Process Limit (
mem_limit
): 90% of total RAM $\approx$ 28.8 GB. * Load Memory Limit: 30% of the process limit (default). * Result: $32\text{ GB} \times 0.9 \times 0.9 \times 0.3 \approx \mathbf{7.77\text{ GB}}$. The 6.9 GB you see is very close to this default threshold. When multiple threads try to parse Parquet footers or buffer data for writing, they exceed this 30% cap. What to do: You should increase the percentage of memory allocated to loading processes. Since you are performing a large 1 TB ingestion, you can safely give more room to the loader. 1. Modify CN Configuration: In your 
cn.conf
(or via 
ADMIN SET CONFIG
if supported by your deployment), increase the load memory ratio:
Copy code
properties
    # Increase to 60% or higher if this node is primarily for loading
    load_process_max_memory_limit_percent = 60
2. Check Kubernetes Limits: Ensure your Kubernetes Pod has a memory limit of at least 32 GB. StarRocks detects the container limit. If the Pod limit is set to 8 GB, StarRocks will use 8 GB as the base for all its calculations. 3. Session Variable: You can also try setting 
load_mem_limit
for the session that created the pipe (though for Pipe, the background tasks usually follow system defaults).
Copy code
sql
    SET GLOBAL load_mem_limit = 17179869184; -- 16GB
────────── 2. Speeding up the Ingestion You mentioned it is loading 1 file at a time. This is because the StarRocks Pipe splits the job into tasks based on 
BATCH_FILES
and 
BATCH_SIZE
properties. * 
BATCH_FILES
: The number of files in one micro-batch task. * 
BATCH_SIZE
: The cumulative size of files in one micro-batch task. If these are too small, the FE (Frontend) creates many tiny tasks and schedules them. While the 
INSERT INTO SELECT FROM FILES
statement inside a task is parallelized across CNs, having only one file per task means only a few threads can work on it at once. How to optimize: When creating or altering your pipe, increase the batch size so that each task is large enough to be distributed across all 8 CPUs on all 3 CNs. Example optimized Pipe creation:
Copy code
sql
CREATE PIPE my_s3_pipe
PROPERTIES (
    "AUTO_INGEST" = "TRUE",
    "BATCH_FILES" = "100",  -- Increase from default to group more files
    "BATCH_SIZE" = "4GB"    -- Increase so each task has enough data to saturate the cluster
)
AS 
INSERT INTO my_table
SELECT * FROM FILES (
    "path" = "<s3://your-bucket/data/*.parquet>",
    "format" = "parquet",
    ...
);
3. Summary of Recommendations 1. Increase Memory Ratio: Set 
load_process_max_memory_limit_percent = 60
in 
cn.conf
to stop the "Memory limit exceeded" errors. 2. Increase Batching: Use 
ALTER PIPE my_pipe SET ("BATCH_FILES" = "100", "BATCH_SIZE" = "4GB")
to ensure each task is large enough for parallel execution. 3. Check Parallelism: Ensure 
pipeline_dop
is set to 
0
(adaptive) or 
8
(to match your 8 CPUs). 4. Verify Pod Resources: Double-check that your K8s 
resources.limits.memory
is actually 32Gi. If the process is getting "unknown status" or killed, it might be the Kubernetes OOM Killer hitting the pod before StarRocks hits its internal limit.
y
@Rocky What is the memory and CPU resource recoomendation for a cluster with 2 FEs, 3 CN nodes, about 2 terrabytes of data. The queries do not have to be real time
r
For a cluster with 2 TB of data and the requirement that queries do not have to be real-time, you have some flexibility. However, your current errors indicate that your ingestion process is starving for memory. Here is the recommended resource allocation for your specific scale: 1. Compute Nodes (CN) - The Workers Since you are using a PIPE to ingest 1 TB at a time into a 2 TB total dataset, your CN nodes are doing the heavy lifting (parsing Parquet, compressing data, and managing the local cache). * Recommended: 16 Cores, 64 GB RAM per node. * Why? StarRocks is a vectorized engine designed to use a lot of memory for performance. The default "production" recommendation is 16/64. With only 32 GB, you are hitting the "load memory limit" (which defaults to 30% or 90% of the node depending on version/config) very quickly when processing large Parquet files. * Minimum (If staying on 8 Cores/32 GB): * You must increase 
load_process_max_memory_limit_percent
to at least 
60%
in 
cn.conf
. * You must ensure your Kubernetes Memory Limit is at least 32 GB. If the Pod limit is lower than the physical RAM, StarRocks might be killed by the K8s OOM Killer before it even reaches its internal limits. * Storage (Cache): For a 2 TB dataset, provide at least 500 GB - 1 TB of high-speed SSD per CN node for the "Data Cache." Even if queries aren't real-time, having enough cache to hold your "hot" working set will prevent the cluster from constantly pulling data from S3, which saves costs and speeds up ingestion/queries. 2. Frontend Nodes (FE) - The Managers FEs handle metadata and query planning. They are generally not resource-intensive unless you have thousands of tables or very high concurrency. * Recommended: 8 Cores, 16 GB - 32 GB RAM. * HA Note: You currently have 2 FEs. For production stability, it is strongly recommended to use 3 FEs (1 Leader, 2 Followers). A 2-node FE setup does not provide true High Availability because if one node fails, the remaining node cannot form a quorum to elect a new leader. * JVM Heap: Ensure the FE JVM heap is set to at least 8 GB or 16 GB (via 
JAVA_OPTS
in 
fe.conf
). 3. Resource Summary Table Component | CPU Cores | RAM | Local Storage (SSD) | Count FE | 8 | 16 GB | 100 GB (Metadata) | 3 (for HA) CN | 16 | 64 GB | 500 GB+ (Data Cache) | 3 Why your 1 TB ingestion is struggling: 1. Memory Overhead: Parquet is a columnar format. To write it efficiently, StarRocks must buffer data in memory to build "segments" before flushing to S3. With 32 GB total and only ~9 GB allocated to loading (by default), a single large task can easily exceed this. 2. CPU Parallelism: With 8 CPUs per node (24 total), if you ingest "1 file at a time," you are likely using only 1 or 2 cores. To speed this up, you need to increase the 
BATCH_SIZE
of your Pipe so it sends enough data to keep all 24 cores busy. Summary Recommendation: Upgrade your CN nodes to 64 GB RAM and 16 Cores. This is the single most effective way to stop the memory errors and allow the PIPE to process larger batches of files in-of-files simultaneously. References • deployment/plan_cluster.md • introduction/Architecture.md • knowledge/usage/best-practice.md • deployment/deploy_manually.md
y
@Rocky I see now in my k8s config: fe nodes: 2 CPUs, 8 Gi cn nodes: 2 CPUs, 8 Gi Based on that can I do 4 CPU, 16 Gi for each fe and CN node ?
r
While moving to 4 CPUs and 16 GiB is a step in the right direction, it is important to understand where this stands relative to StarRocks' requirements for a 1 TB ingestion and 2 TB total data volume. 1. Is 4 CPU / 16 GiB enough? * For FE nodes: Yes, this is workable. The official recommendation for FE is 8 CPUs and 16 GiB. Moving to 4 CPUs / 16 GiB will make the FE much more stable. * For CN nodes: It is the absolute bare minimum. The official production recommendation is 16 CPUs and 64 GiB. While 16 GiB is the "minimum" often cited for BE/CN nodes, it is very tight for ingesting 1 TB of data. 2. Why your current setup is failing At 8 GiB (current), StarRocks only allows about 2.1 GiB for loading by default ($8\text{ GB} \times 0.9 \times 0.3$). At 16 GiB (proposed), it will only allow about 4.3 GiB for loading by default. A 1 TB ingestion involves: * Reading large Parquet footers. * Buffering data in memory before flushing to S3. * Managing a local data cache. Crucial Step: If you move to 16 GiB, you must increase the memory ratio for loading. In your 
cn.conf
(or via Helm values):
Copy code
properties
load_process_max_memory_limit_percent = 70
This will give the loading process ~10 GB of your 16 GB, which is much more reasonable. 3. Addressing the "Unknown Status" Error The "task failed with unknown status" in your logs usually means the Kubernetes OOM Killer killed the pod before StarRocks could report a specific error. * Check your K8s manifest: Ensure 
resources.limits.memory
is exactly the same as 
resources.requests.memory
. * StarRocks `mem_limit`: StarRocks tries to manage its own memory within 90% of the pod's limit. If you set the pod to 16 GiB, StarRocks will try to stay under 14.4 GiB. 4. How to speed up the Pipe The reason it's loading "1 file at a time" is that the Pipe's default batching is too conservative for a 1 TB load. You can force it to be more aggressive: 1. Alter your Pipe:
Copy code
sql
    ALTER PIPE my_s3_pipe SET (
        "BATCH_FILES" = "50",
        "BATCH_SIZE" = "2GB"
    );
2. Increase Parallelism: Ensure your session/global variables allow the cluster to use all cores:
Copy code
sql
    SET GLOBAL pipeline_dop = 4; -- Match your new 4 CPU count
5. Final Recommendations for 2 TB of Data If you can afford it, 8 CPU and 32 GiB per CN node is the "sweet spot" for a cluster of this size. If you must stay at 4 CPU / 16 GiB: * CN Nodes: Use 3 nodes as you have, but increase 
load_process_max_memory_limit_percent
. * FE Nodes: You currently have 2 FEs. This is risky because if one fails, the cluster loses "quorum" and stops. Move to 3 FEs for High Availability. * Data Cache: Ensure your CN nodes have high-speed SSDs attached for the 
starlet_cache
. For 2 TB of data, even 200–500 GB of cache will significantly speed up queries by preventing constant S3 re-reads. Summary: 4 CPU / 16 GiB is a much better "floor" than what you have now, but you must tune the memory settings and pipe properties to make it work for a 1 TB ingestion. References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • loading/loading_introduction/loading_considerations.md • deployment/plan_cluster.md
Open in Slack
PreviousNext
Powered by