http://starrocks.io logo
Join Slack
Powered by
Hey folks :wave: — sharing evaluation feedback fr...
# questions-and-troubleshooting
d
Hey folks 👋 — sharing evaluation feedback from a week-long shared-data POC on StarRocks 4.0.8. We were planning a production deployment at 700 TB / 8 TB-per-day and hit enough blockers that we're reassessing the path forward. _TL;DR: • Under sustained load, one hot partition reached MAX_CS 148,471 and the cluster entered a stuck state that ~8 hours of tuning couldn't recover • Hit what looks like the compaction-MemTracker leak (stuck TXNs with 
CommitTime=NULL
for 1h+, 
compaction_mem_bytes=50GB
on a CN with no active work) • One CN crashed mid-compaction; FE didn't re-dispatch for ~2 hours • 
max_running_txn_num_per_db
exhaustion from one bad partition stalled ingestion on 19 other tables in the same DB • Couldn't find a Docker image for 4.0.9 (the version reportedly fixing the MemTracker leak) — 4.1.0 has an image but isn't prominent on GitHub releases_ We know this is OSS and we haven't paid — just sharing because the experience surprised us and might be useful signal. Full details, configs tried, TXN IDs, and errors in thread. Happy to share logs / redo the test on a recommended version. And open to conversations with CelerData if paid support is the right path for us.
👍 1
_Workload / hardware • Target: 700 TB / 8 TB per day / ~20 tables, NDJSON Stream Load, 64–100 MB batches, up to 90 concurrent streams • POC: 1× FE (r8.xlarge), 4× CN (is4gen.4xlarge — 16 vCPU / 96 GiB / 2× 1875 GB NVMe) • Prod planned: 3× FE + 9× is4gen.8xlarge • Storage: shared-data on S3, StarRocks 4.0.8 via Docker • Hot table: 
DUPLICATE KEY
, daily partitioned, 24 buckets, 
replication_num=1
. The problem partition was 1.7 TB / 2.96 B rows._
What broke
Observations under load
Hot daily partition reached MAX_CS = 148,471
Multiple compaction TXNs got stuck with 
CommitTime=NULL
for 1h+ (e.g. TXN 3169499 ran 03:46 → 06:30 with zero progress before we cancelled it)
On one CN, 
compaction_mem_bytes
stayed at ~50 GB after we cancelled stuck tasks — only 
docker restart
cleared it
One CN crashed mid-compaction: 
fail to call compact, error=Host is down, [E112]Not connected to <ip>:8060 yet
(TXN 3305859). After the CN came back, the FE took ~2 hours to re-dispatch compaction on the affected partition. During that window backlog kept growing.
max_running_txn_num_per_db
default 200 was hit immediately due to slow commits on the hot partition → rejections cascaded to the 19 other tables in the same DB.
In the final attempt — loader paused, 3 of 4 CNs healthy, one partition to drain, 
lake_compaction_max_tasks=64
— 
DATA_VERSION
did not advance for 76 minutes. Compaction was effectively dead despite tasks visible in 
be_cloud_native_compactions
.
What we tried
Tuning attempted (via ADMIN SET + /api/update_config):
FE:
max_running_txn_num_per_db = 2000
lake_compaction_max_tasks = 64
lake_compaction_interval_ms_on_success = 1000
lake_compaction_score_upper_bound = 100000 (later reset)
lake_ingest_slowdown_threshold = 10000 (later reset)
publish_version_max_threads = 1024
lake_batch_publish_max_version_num = 100
lake_compaction_disable_tables = <19 healthy table IDs>
CN:
compact_threads = 12
cumulative_compaction_num_threads_per_disk = 4
base/update_compaction_num_threads_per_disk = 2
max_cumulative_compaction_num_singleton_deltas = 100
Plus:
ALTER TABLE ... COMPACT <partition> (force-dispatch)
CANCEL COMPACTION WHERE TXN_ID = ... (on stuck tasks)
docker restart starrocks-cn (to clear leaked mem)
Paused loaders to the hot table (impossible in prod — but we did it in POC)
None of this got 
DATA_VERSION
to move past the stuck point.
Questions we'd love answers to
Is the compaction MemTracker leak definitively fixed in 4.0.9? (PRs #71126, #68877 look related)
Why the ~2-hour gap before FE re-dispatched compaction after a CN crash? Is there a config knob, or is this a scheduler bug?
Is per-table or per-partition txn isolation on the roadmap? DB-wide 
max_running_txn_num_per_db
has a large blast radius in shared-data.
Is there documented partitioning guidance by daily ingest volume for shared-data tables? Our 24 buckets / 70 GB-per-tablet design clearly didn't scale.
Is 
CANCEL COMPACTION
safe in 4.0.x, or is CN restart still the recommended recovery path?
What's the recommended loader backpressure signal (HTTP response / header / retry-after) for a production loader that can't be manually paused?
Version / artifact confusion
Small operational friction worth mentioning: we couldn't identify a clear "this is the current production-recommended version" for shared-data. • 4.0.9 has a git tag but we couldn't find 
starrocks/fe-ubuntu:4.0.9
or 
starrocks/cn-ubuntu:4.0.9
on Docker Hub (happy to be corrected if we missed it) • 4.1.0 has Docker images but doesn't appear prominently on GitHub releases or the docs versions dropdown at the time we checked
A "current recommended version for shared-data production" marker on the docs or releases page would have saved us several hours of evaluation time.
Offer
Happy to share: • Full FE / CN configs and table DDL • 
SHOW PROC '/compactions'
dumps showing the stuck TXNs • Log bundles from the stuck period • Re-run the workload on whichever version you recommend, once we know which container image to pull
Not expecting any SLA here — we know this is OSS. Just offering because if any of this is useful for a regression test or docs improvement, we'd rather be helpful than silent. And if the right answer is "you should talk to CelerData for support at this scale," we're open to that too.
s
@Rocky and Dhruv can you please let me know The DR strategy for the restoration
r
Disaster Recovery (DR) in StarRocks depends on whether you are using the Shared-nothing (Standard/On-premise) or Shared-data (Cloud-native) architecture. Here is the breakdown of DR strategies and restoration procedures: 1. Shared-nothing Architecture (Standard) For traditional clusters where data is stored on local BE disks, the primary DR strategy involves standard backup and restore operations to remote object storage or HDFS. * Backup & Restore: * Process: You create a repository (S3, GCS, Azure, HDFS) and use the 
BACKUP
command to create a snapshot of your databases or tables. * Restoration: Use the 
RESTORE
command to recover data into the same or a new cluster. * Syntax Example:
Copy code
sql
        -- Create a repository
        CREATE REPOSITORY my_repo WITH BROKER ON LOCATION "<s3a://bucket/path>" PROPERTIES (...);
        -- Backup
        BACKUP SNAPSHOT db_name.snapshot_1 TO my_repo;
        -- Restore
        RESTORE SNAPSHOT db_name.snapshot_1 FROM my_repo;
* Cross-Cluster Replication (CCR): * StarRocks provides a Cross-cluster Data Migration Tool (
starrocks-cluster-sync
). * This tool continuously checks for data lag and synchronizes data from a primary cluster to a secondary (DR) cluster. * It supports databases, internal tables, and view schemas. 2. Shared-data Architecture (Cloud-native) In shared-data clusters, data is already in object storage, so DR focuses on Metadata (FE state) protection. * Cluster Snapshot (v3.4.2+): * This is the fundamental DR strategy for shared-data. It captures both data and metadata in object storage. * Workflow: The system automatically maintains a cluster snapshot. In case of failure, you can restore a new cluster using the 
cluster_snapshot.yaml
configuration and starting the FE with the 
--cluster_snapshot
flag. * Restoration: You can restore to a new cluster in the same or a different region (provided the object storage is accessible or replicated). * Second-level Recovery (
ADMIN REPAIR TABLE
): * If you encounter accidental deletions or metadata corruption, you can use 
ADMIN REPAIR TABLE
to roll back the metadata to a healthy historical version. This is an ultra-fast, metadata-only operation that doesn't involve data movement. 3. Recommended DR Strategy Summary Feature | Architecture | Use Case Backup/Restore | Shared-nothing | Long-term archival and manual recovery. Cluster Sync Tool | Shared-nothing | Active-Standby setup for near real-time DR. Cluster Snapshot | Shared-data | Full cluster recovery for cloud-native deployments. Metadata Rollback | Shared-data | Fast recovery from operational errors or corruption. Pro-tip: For high-criticality systems, many users combine Cloud Provider Cross-Region Replication (e.g., S3 Cross-Region Replication) with StarRocks Cluster Snapshots to ensure the data itself is physically present in a secondary region. References • administration/data_migration_tool.md • administration/cluster_snapshot.md • administration/management/FE_configuration.md
Open in Slack
PreviousNext
Powered by