The relationship between Flink connector parallelism and the number of Compute Nodes (CNs) in a StarRocks shared-data cluster is one of performance scaling rather than a hard mechanical limit. They do not strictly "limit" each other's configuration, but they do limit each other's effective utilization. 1. Does Flink Parallelism limit CN usage? Technically No, but practically Yes for the ingestion entry point. * Coordination Layer: Each Flink sink subtask (defined by

sink.parallelism

) initiates a separate Stream Load. The StarRocks Frontend (FE) redirects each request to a CN to act as a Coordinator. If your parallelism is 1, only one CN will act as the coordinator to receive and parse the data from Flink. * Execution Layer: Even with a single coordinator, StarRocks can involve multiple CNs in the background to handle the write plan (distributing data to the CNs responsible for specific tablets). However, the single coordinator node will become a bottleneck for network I/O and data parsing if the data volume is high. * Merge Commit: If you use the Merge Commit feature (v3.4+), you can have high Flink parallelism (e.g., 20) sending data to many CNs, which then merge these requests into a single transaction. This allows you to use many CNs to "receive" data without overwhelming StarRocks with too many small transactions. 2. Does the number of CNs limit Flink Parallelism? No. * You can set Flink's

sink.parallelism

to be much higher than the number of CNs. For example, if you have 3 CNs and set parallelism to 30, each CN will simply handle 10 concurrent Stream Load coordination tasks. * Warning: If Flink parallelism is significantly higher than the number of CNs, you may see high CPU usage or "Too many connections" errors on the CNs, as each Flink subtask maintains its own HTTP connection and buffer. Summary Table Scenario | Result High Flink Parallelism / Low CN Count | CNs may become CPU/Network bottlenecks. Ingestion might fail due to resource exhaustion on the CN side. Low Flink Parallelism / High CN Count | Only a few CNs act as "entry points" (coordinators). You are likely under-utilizing the cluster's potential for ingestion throughput. Optimal Setup | Set Flink parallelism high enough to distribute the load across all CNs (usually

Parallelism >= Number of CNs

), and use Merge Commit to keep the transaction count low. Best Practices * Match Parallelism to Resources: A common starting point is setting

sink.parallelism

equal to the number of CNs or a small multiple of it. * Scaling Ingestion: If you add more CNs to your cluster, you should generally increase your Flink sink parallelism to take advantage of the new compute power for ingestion. * Monitor Bottlenecks: Use StarRocks metrics like

merge_commit_pending_total

and Flink operator CPU metrics to see which side needs more parallelism. References • loading/Flink-connector-starrocks.md • loading/StreamLoad.md • introduction/Architecture.md References • quick_start/shared-data.md • loading/StreamLoad.md • introduction/Architecture.md

To absorb 2 million events per second (EPS) in a StarRocks shared-data cluster using Merge Commit, the number of Compute Nodes (CNs) required depends heavily on your event size and schema complexity. Based on StarRocks benchmarks and architectural limits, here is the estimation for a cluster using recommended node sizing (typically 16-core / 64GB RAM or 32-core / 128GB RAM). 1. Throughput Calculation First, convert EPS to data volume. 2 million EPS results in the following data rates: * Small Events (250 bytes): ~500 MB/s * Medium Events (500 bytes): ~1 GB/s * Large Events (1 KB): ~2 GB/s 2. Estimated Node Requirements In a shared-data (Lake) architecture, StarRocks CNs must handle data parsing, encoding, and local I/O before flushing to object storage (S3/HDFS). * Benchmark Reference: In StarRocks 3.x, a single 16-core CN typically achieves between 50 MB/s and 100 MB/s of ingestion throughput for complex real-time schemas (like Primary Key tables) and higher for simpler Duplicate tables. * Merge Commit Advantage: Merge Commit removes the "Transaction Bottleneck" (FE lock contention) and the "Small File Bottleneck" (compaction overhead), allowing CNs to focus almost entirely on raw data processing. Data Volume | Throughput | Estimated CNs (16-core nodes) | Estimated CNs (32-core nodes) 500 MB/s (2M eps @ 250B) | ~5-10 MB/s per core | 8 - 12 CNs | 4 - 6 CNs 1.0 GB/s (2M eps @ 500B) | ~5-10 MB/s per core | 16 - 20 CNs | 8 - 10 CNs 2.0 GB/s (2M eps @ 1KB) | ~5-10 MB/s per core | 32 - 40 CNs | 16 - 20 CNs 3. Critical Configuration for 2M EPS To reach this scale, simply adding CNs is not enough; you must tune the Flink-to-StarRocks pipeline: * Flink Sink Parallelism: At 2M EPS, you will likely need 64 to 128+ parallelism in your Flink Sink. This ensures enough concurrent "streams" are feeding the CNs to saturate their CPU/Network. * Merge Commit Interval: Set

sink.properties.merge_commit_interval_ms

to 5000ms - 10000ms. This allows thousands of small Flink flushes to be merged into a single large StarRocks transaction. * Bucket Count: Ensure the target table has enough buckets (tablets). For a 20-node CN cluster, you should have at least 160–320 tablets (8–16 tablets per CN) to ensure even CPU distribution during the write. * FE Configuration: For 2M EPS, ensure the FE metadata is not a bottleneck by setting:

admin set frontend config("lake_enable_batch_publish_version" = "true");

Summary Recommendation For a standard log-type event (~500 bytes), you should plan for a 10-node cluster (32-core nodes) or a 20-node cluster (16-core nodes) to handle 2 million EPS with enough headroom for background compaction and query traffic. References • knowledge/usage/load/shared_data_batch_ingestion.md • loading/Flink-connector-starrocks.md • deployment/shared_data/feature-support-shared-data.md • loading/StreamLoad.md References • faq/shared_data_faq.md • knowledge/usage/load/shared_data_batch_ingestion.md • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/hdfs.md