@Rocky I am having issues where pulling disparate parts of a table from GCS takes a really long time (> 30s). usually when data is scattered. Analysis was said that parallelism was kept low. How do I increase parallelism so it pulls many chunks at once?

To increase the parallelism for scans from object storage like GCS, especially when data is scattered (e.g., many small files or filtered ranges across many row groups), you should tune the following variables. 1. Increase Query Parallelism (

pipeline_dop

) In modern StarRocks versions (v2.3+), query parallelism is controlled by the Pipeline engine. By default, this is set to

0

, which means StarRocks adaptively chooses the degree of parallelism (DOP). If your analysis shows parallelism is too low, you can set it manually. * Variable:

pipeline_dop

* Recommendation: Set it to half the number of physical CPU cores on your BE/CN nodes, or higher if you have the resources. * SQL:

SET pipeline_dop = 16;

(Adjust based on your core count) 2. Increase I/O Concurrency per Operator Even if you have a high DOP, each scan operator might be limited in how many concurrent I/O requests it can send to GCS. * Variable:

connector_io_tasks_per_scan_operator

* Description: Controls the number of concurrent I/O tasks issued to remote storage per scan operator. * Recommendation: Increase this if you see high latency from GCS and many idle CPU cycles. * SQL:

sql
    SET enable_connector_adaptive_io_tasks = false; -- Disable adaptive adjustment if it's too conservative
    SET connector_io_tasks_per_scan_operator = 32; -- Default is 16; try 32 or 64

3. Adjust Remote Scanner Limits (BE Configuration) There are backend (BE) configurations that cap the total number of concurrent remote scanners. If your query involves many files/splits, you might be hitting these limits. * Parameter:

max_hdfs_scanner_num

(Used for all remote connectors including GCS/S3) * Default: 50 * Note: This is a BE config. You can check it via

SHOW CONF

and set it via

ADMIN SET FRONTEND CONFIG

(if applicable) or in

be.conf

. * Parameter:

pipeline_connector_scan_thread_num_per_cpu

* Description: Number of threads per CPU core dedicated to scanning remote files. Increasing this can help if the system is I/O bound. 4. Enable or Tune Data Cache If you are pulling "disparate parts" of a table frequently, enabling the Data Cache can drastically reduce the need to go back to GCS. * Check: Ensure

datacache.enable

=

true

in your table properties (for cloud-native tables) or that the session variable

enable_scan_datacache

is on for external catalogs. * The first pull will still be slow, but subsequent "scattered" reads will be served from local NVMe/SSD cache. How to verify if parallelism is working: Check the Query Profile for the following metrics in the ConnectorScanNode: 1.

MorselsCount

: If this is very low (e.g., 1 or 2), the query is not split into enough parallel tasks. This usually happens if there are too few files or the files are not splittable. 2.

PeakIOTasks

: This shows the maximum number of concurrent I/O requests actually sent. If this is much lower than

connector_io_tasks_per_scan_operator

, the bottleneck might be the number of splits (morsels) or GCS throughput limits. 3.

IOTaskWaitTime

: High wait time indicates that scanners are waiting for available threads or I/O slots. Summary Recommendation:

sql
-- Try these session variables first
SET pipeline_dop = 16; 
SET enable_connector_adaptive_io_tasks = false;
SET connector_io_tasks_per_scan_operator = 64;

References • sql-reference/System_variable.md • knowledge/usage/best-practice.md • administration/management/BE_configuration.md • loading/Kafka-connector-starrocks.md • best_practices/query_tuning/query_profile_operator_metrics.md

@Rocky how much of this applies to native tables on gcs

For StarRocks native tables (specifically Cloud-native tables in shared-data mode) on GCS, the parallelism and performance tuning differ slightly from external tables (like Hive/Iceberg). Here is how the settings apply specifically to native tables on GCS: 1. The Most Important:

pipeline_dop

This is still your primary lever. In shared-data mode, StarRocks splits data into morsels.

pipeline_dop

determines how many pipeline instances (threads) work on these morsels in parallel on each node. * Action: If your analysis says parallelism is low, manually increase this. * SQL:

SET pipeline_dop = 16;

(or higher, depending on your CN node CPU cores). 2. Parallel Tablet Scanning (

enable_lake_tablet_internal_parallel

) In shared-data mode, a "tablet" is a logical unit of data. By default, older versions might scan one tablet with one thread. Starting from v3.3.0, you can enable internal parallelism for these tablets. * Variable:

enable_lake_tablet_internal_parallel

* Effect: Allows StarRocks to split a single cloud-native tablet into multiple morsels to be processed in parallel by different threads. This is crucial if you have a few large tablets. * SQL:

SET enable_lake_tablet_internal_parallel = true;

3. I/O Concurrency (

io_tasks_per_scan_operator

) While

connector_io_tasks_per_scan_operator

is for external tables, the native scan operator uses a similar mechanism to handle remote I/O to GCS. * Variable:

io_tasks_per_scan_operator

(Note: No "connector" prefix). * Default: Usually 4. * Action: Increase this to allow more concurrent GCS requests per scan thread. * SQL:

SET io_tasks_per_scan_operator = 16;

4. Check Your Tablet Count (Bucket Count) Native tables are distributed into buckets (tablets). * The Issue: If your table was created with a very low bucket count (e.g.,

DISTRIBUTED BY HASH(id) BUCKETS 1

), StarRocks cannot easily parallelize the scan across many nodes or threads because the data is physically tied to a single unit. * Check: Run

SHOW CREATE TABLE <table_name>

and check the

DISTRIBUTED BY

clause. If the bucket count is too low for your cluster size, you may need to re-create the table with more buckets. 5. Leverage Data Cache (The "30s" problem) If pulling "disparate parts" is taking >30s, you are likely suffering from GCS object metadata latency (listing/opening many small parts). The shared-data architecture is designed to solve this with the Data Cache. * Property: Ensure your table has

"datacache.enable" = "true"

. * Effect: After the first slow pull, the "disparate parts" will be stored on the local NVMe/SSD of your CN nodes. Subsequent queries will run at local disk speeds (milliseconds) instead of GCS speeds. Summary for Cloud-Native Tables:

sql
-- Increase thread count
SET pipeline_dop = 16; 

-- Enable splitting tablets into multiple parallel tasks (v3.3+)
SET enable_lake_tablet_internal_parallel = true; 

-- Increase concurrent I/O requests per thread
SET io_tasks_per_scan_operator = 32; 

-- Check if your data is actually being cached
-- (Look for 'CachedPagesNum' in the Query Profile)

Note: If you are on a version earlier than 3.0, you are likely using "External Tables" via a GCS Connector rather than "Native Tables," in which case the

connector_

variables from the previous response apply. Native tables on GCS require the Shared-data (3.0+) architecture. References • sql-reference/System_variable.md • knowledge/usage/best-practice.md • best_practices/query_tuning/query_profile_operator_metrics.md