Hi! In our evaluation of starrocks we created a narrow table for our scalar (mostly floats) telemetry values, this is a pretty big table (billions of rows) but very small in size. I was using a PK (not a UNIQUE KEY for example) as doing upserts would be a great benefit for us. However, the table started to be very slow. I had enabled

enable_range_distribution

but then realized that that only triggers by size, and seemingly after 2GB min, and my table was very slow even at 1.6gb. I ended up setting a weekly partition. Does this seem correct? Am I missing something? are there plans to enable auto distribution on row count besides size?

I'm not sure what you're referring to! Could you provide more context or let me know which StarRocks feature, query, or issue you'd like my opinion on? I'm here to help with any technical questions you have.

@Rocky I'll repeat the question: Hi! In our evaluation of starrocks we created a narrow table for our scalar (mostly floats) telemetry values, this is a pretty big table (billions of rows) but very small in size. I was using a PK (not a UNIQUE KEY for example) as doing upserts would be a great benefit for us. However, the table started to be very slow. I had enabled

enable_range_distribution

but then realized that that only triggers by size, and seemingly after 2GB min, and my table was very slow even at 1.6gb. I ended up setting a weekly partition. Does this seem correct? Am I missing something? are there plans to enable auto distribution on row count besides size?

Your situation is a classic "narrow table" challenge in StarRocks. You are correct in your observation: StarRocks' automatic distribution and bucketing logic is primarily driven by data size (bytes) rather than row count. Here is a breakdown of why you are seeing this behavior and why your partition strategy is the right move. 1. The "Hidden" Cost: Primary Key Index While your raw data is only 1.6GB, the Primary Key Index is likely much larger. * How it works: StarRocks PK tables maintain an index to map primary keys to row locations. A rough formula for the index size is:

(KeySize + 8 bytes) * RowCount

. * The Problem: With billions of rows, even a simple 8-byte key results in an index that can be tens of gigabytes. If the system only sees 1.6GB of data, it might only create 1 or 2 buckets (tablets). This forces a single tablet to manage hundreds of millions (or billions) of index entries, leading to high memory pressure, slow lookups, and heavy compaction overhead. 2. Why

enable_range_distribution

didn't help

enable_range_distribution

(introduced in v4.1) allows for Range-based Distribution and automatic tablet splitting/merging. However: * It is currently triggered by the

tablet_reshard_target_size

(default is 10GB). * Since your table is only 1.6GB, the system "thinks" the tablets are small and healthy, even though the row count is causing a performance bottleneck. 3. Is your weekly partition correct? Yes, this is the recommended approach. By partitioning (e.g., weekly or daily), you are: * Pruning data: Queries with time filters can skip most of the table. * Splitting the Index: Each partition maintains its own PK index. This keeps the index per partition/tablet at a manageable size, which is critical for performance. * Parallelism: It allows the system to use more threads and nodes for both ingestion and queries. 4. Are there plans for row-count based distribution? Current development (v4.x) is heavily focused on the Shared-Data architecture and the Range-based Distribution mentioned above. While the primary trigger is still size, the goal is "No-Bucket" design where the system manages scaling automatically. * Currently: There isn't a specific configuration to trigger auto-bucketing based on row count. * Workaround: For tables with very small rows (scalars/telemetry), the best practice is to manually set a higher bucket count or use fine-grained partitioning as you have done. Summary Checklist for your Evaluation: * Manual Bucketing: If you don't want to rely solely on partitioning, you can manually set a higher number of buckets at table creation:

DISTRIBUTED BY HASH(pk_col) BUCKETS 32

(or more, depending on your BE count). * Persistent Index: Ensure

enable_persistent_index

is

true

(it is by default). This allows the PK index to reside on SSD instead of consuming all your RAM. * Columnar vs. Row Store: If you are doing frequent point lookups on these billions of rows, look into the Row-Column Hybrid storage (available in newer versions) to further accelerate those specific queries. Does your weekly partitioning solve the "slow" behavior you were seeing, or are you still hitting specific queries still struggling? References • best_practices/primarykey_table.md • knowledge/usage/best-practice.md • table_design/table_types/primary_key_table.md