@Rocky I have a flink job that is writing data to starrocks using the flink starrocks connector

1.2.16

. We are also loading a lot of data at the moment using stream load separately. Compaction is high, in the 100s. The flink job keeps failing because of the compaction rate being too high, despite me increasing the threads available to compactions and the max compaction tasks that can be scheduled. What can i change to improve the situation. For clarity, it seems the flink job fails due to a thread socket not receiving a response from starrocks and blocks the flink job from completing its checkpoints

The high compaction score (in the 100s) and subsequent Flink checkpoint failures are direct consequences of StarRocks' Ingestion Slowdown mechanism. When the compaction score exceeds a threshold (typically 100), StarRocks intentionally delays transaction commits to allow compaction to catch up. This delay causes the Flink sink to block while waiting for a response, eventually leading to socket timeouts or Flink checkpoint failures. Since you have already increased compaction threads, here are the recommended changes to address the root cause (excessive small versions) and the symptoms (Flink timeouts): 1. Reduce Load Frequency (Flink Connector Side) The most common cause of high compaction scores is flushing too frequently. You should increase the buffering on the Flink side to create larger Rowsets. Update these parameters in your Flink Sink

WITH

clause: *

sink.buffer-flush.interval-ms

: Increase this (e.g., to

10000

or

30000

). If you are flushing every 1–2 seconds, you are creating thousands of small versions per hour. *

sink.buffer-flush.max-rows

: Increase this (e.g.,

500000

) to ensure you only flush when a significant amount of data is ready. *

sink.buffer-flush.max-bytes

: Increase this (e.g.,

104857600

for 100MB) to favor fewer, larger files. 2. Enable Merge Commit (If on StarRocks v3.4+) If your StarRocks version is 3.4 or later, you can use the Merge Commit feature. This allows multiple Flink subtasks and even multiple Flink jobs to merge their data into a single transaction/version in StarRocks. *

sink.properties.enable_merge_commit

: Set to

true

. *

sink.properties.merge_commit_interval_ms

: Set to

10000

(10 seconds). 3. Adjust Flink Connector Timeouts Since the Flink job fails because it doesn't receive a response during the ingestion slowdown, you should increase the socket timeout to prevent the job from crashing while StarRocks is under pressure. *

sink.socket.timeout-ms

: Increase this from the default (or

-1

for no timeout) to a high value like

60000

(60s) or more to give StarRocks time to respond during high pressure. *

sink.connect.timeout-ms

: Ensure this is at least

30000

. 4. Optimize StarRocks BE Compaction Parameters Beyond just increasing threads, you should tune how aggressively the system picks up small versions for cumulative compaction: *

max_cumulative_compaction_num_singleton_deltas

: (Default 1000). Consider lowering this to

100

or

200

to trigger cumulative compaction earlier. *

max_compaction_concurrency

: If your CPU and Disk I/O have headroom, ensure this is set high enough to utilize the threads you added. *

cumulative_compaction_policy

: If you are using the

size_based

policy (default in newer versions), ensure it is configured to merge small files aggressively. 5. Check Ingestion Slowdown Thresholds If you are using a Shared-data (Cloud-native) cluster, the system specifically triggers slowdowns based on FE configurations. You can check these using `ADMIN SHOW FRONTEND CONFIG`: * `lake_ingest_slowdown_threshold`: Default is 100. If your hardware is powerful, you can temporarily increase this to 200 to stop the Flink job from blocking, but this is a "band-aid" and may eventually lead to

lake_compaction_score_upper_bound

(default 2000) where imports are rejected entirely. Summary of Priority Actions: 1. Increase

sink.buffer-flush.interval-ms

in Flink to at least 10–20 seconds. 2. Increase

sink.socket.timeout-ms

to prevent Flink job failure during StarRocks backpressure. 3. Check for "Too many versions" in the StarRocks BE logs to see if a specific tablet is the bottleneck. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md

The separate stream load is due to a data migration, the batch sizes there are 20k, should they be higher? For the flink job you're suggesting max rows as 500k, which makes me think the other job is too small @Rocky

Yes, 20k rows per batch is significantly too small for a data migration and is the primary driver of your high compaction score. When you load data in small batches frequently, StarRocks creates a new "version" (rowset) for every single commit. Compaction is the process of merging these versions. If you are committing 20k rows every few seconds across multiple threads, you are creating versions faster than the system can merge them. Why 20k is too small * Version Exhaustion: StarRocks (especially in shared-data/lake mode) triggers a "slowdown" once the compaction score hits 100. At 20k rows per batch, a migration of just 2 million rows would create 100 versions—hitting that threshold immediately. * Write Amplification: Small files are inefficient to merge. The system spends more IOPS managing metadata and small file headers than actual data. * Flink Interference: Your migration is "stealing" the version quota from your Flink job. Since Flink relies on checkpoints (which are also commits), it gets stuck behind the migration's high-frequency commits. Recommended Batch Sizes For a migration, you should aim for data volume (MB) rather than just row counts. * Target Size: Aim for 100MB to 1GB per Stream Load. * Row Count Equivalent: Depending on your schema, this usually translates to 500,000 to 2,000,000 rows per batch. * Parallelism: Instead of many small sequential loads, use a few parallel loads with large files (e.g., 3–5 threads, each loading 500MB files). How to fix the current situation 1. Scale up the Migration Batches Increase your migration batch size to at least 500k rows or 200MB per request. This will reduce the number of versions created by a factor of 25x compared to your current 20k batches, giving the compaction threads room to breathe. 2. Align Flink with StarRocks Pressure Since your Flink job is failing due to socket timeouts during the "slowdown": * Increase `sink.socket.timeout-ms`: Set this to

60000

(60s) or higher in your Flink Sink properties. This prevents the Flink job from failing when StarRocks intentionally delays a response to throttle ingestion. * Increase `sink.buffer-flush.max-rows`: As previously noted, keep this at

500000

or higher to ensure Flink isn't also contributing to the small-version problem. 3. Use "Merge Commit" (If on v3.4+) If your StarRocks version allows it, enable Merge Commit for the Flink job. This allows StarRocks to take multiple small flushes from Flink and group them into a single version on the server side, which drastically lowers the compaction score.

sql
'sink.properties.enable_merge_commit' = 'true',
'sink.properties.merge_commit_interval_ms' = '10000'

4. Temporary "Band-aid" for Migration If you must finish the migration quickly and have the CPU/IO overhead, you can temporarily raise the threshold where StarRocks starts slowing down ingestion: * Shared-data (Lake):

ADMIN SET FRONTEND CONFIG ("lake_ingest_slowdown_threshold" = "200");

* Shared-nothing: Increase

cumulative_compaction_check_interval_seconds

(lower it to check more often) or increase

max_cumulative_compaction_num_singleton_deltas

to allow more small files before they become a "critical" score. Summary: Stop the migration, re-batch your data into files of ~500MB or 1M+ rows, and restart. Your compaction score will drop naturally as the system merges the existing small versions. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md