http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> we continue to get bitten by ```SQL...
# questions-and-troubleshooting
g
@Rocky we continue to get bitten by
Copy code
SQL Error [1064] [42000]: load_segments failed tablet:28769437 rowset:35 segid:0: starlet err Object https://<mystorage_account>.<http://dfs.core.windows.net/<myname>/starrocks/<uuid>/db68230/28693266/28769428/data/0000000001bbf2d2_c45b7e4b-c0ac-4d03-9b03-9e7f4c761ddd.dat|dfs.core.windows.net/<myname>/starrocks/<uuid>/db68230/28693266/28769428/data/0000000001bbf2d2_c45b7e4b-c0ac-4d03-9b03-9e7f4c761ddd.dat> does not exist: 
Common causes for this error are:
1. A schema change has been made to the table
2. The current query is a slow query
For the first case, you can retry the current query.
For the second case, see "admin show frontend config like 'lake_autovacuum_grace_period_minutes'" for reference.: BE:1199579
This is for a primary key table that we had 4 separate pipes writing to at the same time, could that be a cause? I have since changed 
lake_autovacuum_grace_period_minutes
to 240 but I'm not optimistic that will help.
r
This error in a Shared-Data (Cloud-native) StarRocks cluster indicates that the Storage Engine (Starlet) is attempting to read a data segment from Azure Blob Storage that has already been deleted by the background AutoVacuum process. While you have increased 
lake_autovacuum_grace_period_minutes
to 240, there are several reasons why high-concurrency writes (4 separate pipes) to a Primary Key (PK) table can still trigger this: 1. The Interaction of PK Tables and Concurrent Writes Primary Key tables use a Delete+Insert strategy. Every batch of data from your 4 pipes creates a new version of the table metadata. * High Version Churn: 4 pipes writing simultaneously create a very high volume of small rowsets and versions. * Aggressive Compaction: StarRocks will aggressively compact these small rowsets into larger ones to maintain query performance. * Vacuuming: Once a compaction task completes, the old (original) segments become "obsolete." The AutoVacuum process then deletes these files after the grace period expires. If a query or a concurrent load job (like your pipes) is still referencing an older version's segments when the vacuum kicks in, you get this error. 2. Why it happens during "Loading" (Pipes) You might wonder why a load fails with "Object does not exist." For PK tables, loading is not just a "write" operation: * Index Lookups: To perform the "Delete+Insert" logic, the BE must check the Primary Key index to see if the incoming keys already exist. * Persistent Index: If you have Persistent Index enabled (standard for large PK tables in shared-data), the index itself is stored in segments on Azure. If the load job is slow or there is a massive backlog, it might try to load an index segment that was just vacuumed. * Partial Updates: If your pipes are performing Partial Updates, StarRocks must read the existing row segments to merge them. If those segments are vacuumed mid-process, the load fails. 3. Recommended Troubleshooting Steps A. Verify the Configuration Persistence If you set 
lake_autovacuum_grace_period_minutes
using 
ADMIN SET FRONTEND CONFIG
, it is not persistent across FE restarts. * Run 
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum_grace_period_minutes';
to ensure it is actually 240. * Ensure it is added to your 
fe.conf
file so it remains active after a restart. B. Check for Compaction Backlog High-concurrency writes often lead to a "Compaction Score" explosion. * Check the compaction score: 
SHOW PROC '/compactions';
* If the score is very high (e.g., > 100), the system is struggling to keep up with the 4 pipes. This causes "version exhaustion" where the gap between the version the pipe started with and the current version becomes too large. * Solution: Reduce the frequency of pipe commits (larger batches) or increase compaction resources (
compact_threads
in 
be.conf
). C. Monitor Tablet "Apply" Latency In PK tables, data isn't visible until the "Apply" phase is finished. If the Apply phase is lagging due to high CPU/IO on the BEs, the load job stays "open" longer, making it more vulnerable to the vacuum grace period. * Check BE logs (
<http://be.INFO|be.INFO>
) for 
_apply_rowset_commit error
or 
load primary index failed
. D. Azure-Specific Considerations Since you are using 
<http://dfs.core.windows.net|dfs.core.windows.net>
(ADLS Gen2): * Ensure there are no Azure Storage limits being hit (Throttling). StarRocks logs would show 503 or 429 errors if this were the case, but it's worth checking the Azure Monitor for "Ingress/Egress" and "Transactions." * If you are performing a Schema Change (as the error suggests), suspend the pipes during the operation. Schema changes in PK tables require a full rewrite of the data, which creates a massive conflict with concurrent high-frequency writes. Summary Checklist
1. Check StarRocks Version: If you are on an early 3.1.x version, there were known bugs with vacuuming aggressive deletions; consider upgrading to the latest 3.2.x or 3.3.x. 2. Batches over Frequency: If possible, consolidate your 4 pipes or increase the batch size/interval to reduce the number of versions created per minute. 3. Persistent Index: If you have it enabled, check if the BEs have enough disk space for the local index cache, as frequent cache misses will force more reads from Azure. References • faq/loading/Loading_faq.md • administration/management/monitoring/metrics.md • quick_start/shared-data.md • administration/Meta_recovery.md
g
@Rocky we aren't running into this issue from long queries and we don't have any errors loading data with the pipes, the error comes after when we try to query the table that was loaded
r
Hello! This is a classic "version churn" issue that occurs in StarRocks Shared-Data (Cloud-native) clusters when high-concurrency writes (like your 4 pipes) are targeted at a Primary Key (PK) table. Yes, having 4 separate pipes writing simultaneously to the same PK table is almost certainly the cause. Here is why this happens and how to address it beyond just the grace period. 1. The "Version Churn" Race Condition In a shared-data PK table, every commit from your pipes creates a new data version. Because PK tables use a Delete+Insert strategy: * Aggressive Compaction: StarRocks must constantly compact these small rowsets from your 4 pipes to maintain query performance and update the Primary Key Index. * The Race: When you start a query, it "pins" a specific version of the table metadata. If your 4 pipes are committing frequently (e.g., every few seconds), the table version advances very rapidly. * The Vacuum: Even if your query isn't "long" in minutes, if the system creates hundreds of versions in a few minutes, the AutoVacuum process will eventually identify the older versions as obsolete. If the AutoVacuum deletes the underlying 
.dat
files from Azure before your query has finished reading them, you get the "Object does not exist" error. 2. Why 
lake_autovacuum_grace_period_minutes
might not be enough While you increased it to 240, there are two common reasons it might still fail: * Persistence: If you set this via 
ADMIN SET FRONTEND CONFIG
, the change is not persistent across FE restarts. You must add it to your 
fe.conf
file. Verify the current value with:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum_grace_period_minutes';
* Compaction Backlog: If the 4 pipes are overwhelming the BEs, the Compaction Score will skyrocket. When the score is too high, the "Apply" process (which makes data visible in PK tables) can become unstable or lead to metadata inconsistencies where the FE points to segments that were "merged away" but not yet fully synchronized. 3. Troubleshooting & Recommended Actions A. Check Compaction Health Run this command to see if your table is struggling to keep up with the pipes:
Copy code
sql
SHOW PROC '/compactions';
* Look for the Compaction Score. If it is consistently
100
, your 4 pipes are writing faster than the system can compact. This causes an explosion of metadata versions. * Fix: Increase compaction threads in `be.conf`: 
compact_threads = 8
(or higher depending on your CPU). B. Consolidate Your Loads 4 pipes writing to one table creates significant overhead. In StarRocks, larger batches are always better than frequent small commits. * Try to increase the 
batch_size
or the time interval of your pipes (e.g., if using Routine Load, increase 
desired_max_interval_ms
). * Consolidating the 4 pipes into 1 or 2 streams with larger batches will drastically reduce the "version churn." C. Persistent Index Check In shared-data PK tables, the Primary Key Index can be stored in Azure (
enable_persistent_index = true
). * Check if your BE logs (
<http://be.INFO|be.INFO>
) show 
load primary index failed
. If the index segment itself is being vacuumed prematurely due to a bug in older versions (pre-3.1.5), this error can occur regardless of the data grace period. * Recommendation: Ensure you are on at least v3.1.10, v3.2.5, or v3.3.x, as many stability fixes for high-concurrency PK tables were introduced in these versions. D. Azure-Specific Latency While ADLS Gen2 is strongly consistent, high-frequency "Delete" and "Read" operations on the same objects can occasionally hit Azure storage throttling (503/429 errors). Check your Azure Storage account metrics for "Throttling" or "Client Errors" during the period the query fails. Summary Checklist 1. Verify 
lake_autovacuum_grace_period_minutes
is actually 240 via 
ADMIN SHOW FRONTEND CONFIG
.
2. Check `SHOW PROC '/compactions'`—if the score is high, your ingestion frequency is the problem. 3. Slow down the commit frequency of your pipes to reduce the number of versions created per minute. 4. If the error persists even with low write frequency, consider upgrading to the latest maintenance release of your StarRocks version. References: • StarRocks Shared-Data FAQ • Primary Key Table Best Practices References • faq/shared_data_faq.md • deployment/shared_data/feature-support-shared-data.md • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md
g
@Rocky so we piped a bunch of data into a pk table and things were fine for a bit but then later querying the table we ran into the same error I posted to start this thread. I suspect this was caused by compaction and perhaps my cluster attempted to write a new 
.dat
file, updated the ref, but the new 
.dat
creation failed causing the ref to be bad? I need help troubleshooting, we are hosed at the moment
r
This is a critical situation in a Shared-Data (Cloud-native) cluster. When you see "Object does not exist" on the current visible version of a Primary Key (PK) table, it means the metadata (FE) and the physical storage (Azure ADLS) have fallen out of sync. Your suspicion about compaction is likely correct. In shared-data mode, compaction follows a transaction: Write -> Commit -> Publish. If the FE "Publishes" a new version but the BE (CN) failed to actually persist the 
.dat
file to Azure (or if Azure had a transient failure during the write), the table becomes "hosed." Here is how to troubleshoot and attempt to recover: 1. Identify the specific file status First, find out exactly where StarRocks thinks this file should be and if it actually exists in your storage account. * Run: 
SHOW TABLET 28769437;
* In the output, look for the 
DbName
, 
TableName
, and 
StoragePath
. * Log into your Azure Portal or use 
az storage blob exists
to check if the specific file mentioned in your error (
0000000001bbf2d2_... .dat
) actually exists in that path. * If the file exists: The issue is likely a BE/CN Cache problem. The BE thinks it doesn't exist or is getting a 403/404 erroneously. * If the file is truly missing: Your theory is confirmed; a compaction or "Apply" task failed to write but reported success. 2. Check for Persistent Index issues PK tables in shared-data mode use a Persistent Index stored in object storage. If the index segment is missing, the whole tablet becomes unreadable. * Check if you have persistent index enabled: 
SHOW CREATE TABLE <your_table>;
* Look for 
"enable_persistent_index" = "true"
. * Troubleshooting Step: Try temporarily disabling the persistent index to see if you can query the table again. This forces the BE to rebuild the index in memory.
Copy code
sql
    ALTER TABLE <your_table> SET ("enable_persistent_index" = "false");
Wait a minute and retry the query. 3. Inspect Tablet Metadata You can see what the FE thinks is the "correct" set of files for that tablet: 1. Get the Backend ID: 
SHOW BACKENDS;
2. Run this "admin execute" command (replace 
<be_id>
, 
<tablet_id>
, and 
<version>
from your error/show tablet output):
Copy code
sql
    -- Get the visible version from SHOW PARTITIONS FROM <table_name>
    ADMIN EXECUTE ON <be_id>
    'System.print(StorageEngine.get_lake_tablet_metadata_json(28769437, <visible_version>))';
Look at the 
rowsets
array. If the 
rowset_id
from your error (
35
) is in there but the file is missing from Azure, the metadata is corrupted. 4. Recovery Options Since you are currently "hosed," here are your recovery paths: * Path A: Fix the Compaction/Apply Backlog If 
SHOW PROC '/compactions'
shows a very high score (>100) or stuck tasks, the system is failing to "Apply" the rowsets from your 4 pipes. • Increase BE threads: 
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "16");
(in 
fe.conf
too). • Increase BE apply threads: 
update_apply_threads = 16
in 
be.conf
. * Path B: Selective Partition Truncate If the error only happens when querying certain time ranges, only those partitions are "hosed." • 
TRUNCATE PARTITION <p_name>;
and then reload just that data. This is often faster than dropping the whole table. * Path C: The "Nuclear" Option (Drop and Reload) If the metadata is pointing to missing files in the latest visible version, StarRocks currently does not have a "Rollback Version" command. You must: 1. Stop the pipes. 2. Drop the table. 3. Wait for the vacuum or manually clean the Azure path to ensure no "ghost" files remain. 4. Recreate and reload. 5. Why did this happen? (Azure Specifics) Since you are on Azure ADLS Gen2 (
<http://dfs.core.windows.net|dfs.core.windows.net>
):
* Throttling: Check Azure Monitor for 503 (Server Busy) errors. High-concurrency writes from 4 pipes plus aggressive compaction can hit the IOPS limit of a single Azure Storage Account. If a "Commit" to the filesystem fails but the BE doesn't catch the exception correctly, you get this metadata drift. * Version: If you are on an early 3.x version, please upgrade to 3.1.10+, 3.2.5+, or 3.3.x. There were significant fixes for "Object does not exist" errors in PK tables specifically related to how compaction handles Azure storage responses. Immediate Recommendation: Check 
SHOW PROC '/compactions'
and 
SHOW TABLET
. If the compaction score is high, stop 2 of your 4 pipes immediately to let the cluster "catch up" and stabilize its metadata. References • faq/shared_data_faq.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CHECK_TABLET.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky Hmm the PVC for my fe logs is filled up, could that impact anything here?
r
Yes, a full FE log PVC can absolutely impact this issue, and in a Shared-Data (cloud-native) cluster, it is a likely contributor to the metadata drift you are seeing. In StarRocks, the FE (Frontend) is not just a query planner; in Shared-Data mode, it is the authoritative coordinator for all storage metadata. Here is how a full disk on the FE breaks your PK table: 1. Metadata Write-Ahead Log (BDBJE) Blockage Most StarRocks deployments (especially Kubernetes/Helm) store the 
log/
directory (system logs) and the 
meta/
directory (metadata edit logs) on the same volume. * The Conflict: StarRocks uses BDBJE (Oracle Berkeley DB Java Edition) for metadata consensus. BDBJE requires constant disk writes for its own internal replication and checkpointing. * The Impact: If the logs (
fe.log
, 
fe.audit.log
) fill up the PVC, BDBJE will fail to write. When BDBJE can't write, the FE cannot commit any metadata changes. This includes new data versions from your pipes or successful compaction tasks. 2. Broken Compaction/Publish Cycle In Shared-Data mode, the compaction process follows this chain: 
CN writes new segment to Azure
-> 
FE Commits Txn
-> 
FE Publishes new version
. * If your pipes are hitting the cluster hard (4 pipes), the FE is trying to write thousands of "edit logs" to its disk to record these new versions. * If the disk is full exactly when the FE is "Publishing" a compaction result, you can end up with a Metadata-to-Storage Mismatch. The FE might have partially updated its memory but failed to persist the state, or the BE/CN might have deleted old files (Vacuum) before the FE successfully recorded the new version. This results in the 
Object does not exist
error. 3. Heartbeat & Coordination Failures The FE manages the "Tablet Meta" for the cluster. It sends heartbeats to the BEs to tell them which version of a tablet is "Visible." * If the FE is struggling with disk I/O (because the disk is at 100%), heartbeats can time out. * When heartbeats fail, BEs may use "stale" metadata. A BE might try to read an old file that the FE has already marked for deletion (Vacuum), or it might not yet know about the new file Azure just received. Immediate Troubleshooting Steps 1. Clear the Logs Immediately: * Exec into the FE pod and check 
/opt/starrocks/fe/log
. * Remove or truncate 
fe.audit.log
and old rolled-over 
fe.log.X
files. _ _Tip:* Check if 
fe.audit.log
is massive; this is common with 4 high-frequency pipes. You may need to tune 
audit_log_roll_num
or use a sidecar to ship logs out. 2. Separate Meta from Logs (Long-term): * In your Kubernetes 
values.yaml
or Deployment, ensure the 
meta
directory and 
log
directory are on separate PVCs or at least that the log volume has a strict rotation policy that keeps it well under the PVC limit. 3. Check FE Health: * Run 
SHOW FRONTENDS;
and check the 
Alive
and 
Join
columns. If they are 
false
or flickering, the disk-full issue has broken the Raft consensus. * Check 
fe.warn.log
for 
BDBJE InsufficientLogException
or 
No space left on device
. 4. Verify the Tablet again: * Once the disk is cleared, run 
ADMIN REPAIR TABLE <table_name>;
. While this is primarily for shared-nothing, in shared-data it can help the FE re-sync its internal state. Summary: Your "hosed" state is likely because the FE successfully told the BE to compact/write data, but then the FE crashed or failed to save that metadata change to its own disk, leaving the FE pointing to files that were either never finalized or already deleted. Clear that disk space immediately before you continue loading. References • administration/management/FE_configuration.md • administration/management/logs.md • faq/shared_data_faq.md
g
@Rocky note my PVCs for 
starrocks-meta
and 
starrocks-log
are different and the 
starrocks-meta
has plenty of space
r
If your meta PVC has space and is separate from the log PVC, the core metadata database (BDBJE) is likely safe. However, a full log PVC is still a "silent killer" for a high-throughput StarRocks cluster, especially in Shared-Data mode. Here is why your "hosed" state and the 
Object does not exist
error are likely still linked to that full log disk: 1. Log4j Blocking (Process Stalling) StarRocks uses Log4j for its logging framework. In most Java configurations, if the disk is 100% full and the logger cannot write a line, the thread attempting to log will block until space becomes available. * The Impact: Every time a pipe commits or a compaction finishes, the FE tries to log the transaction details (Audit log, INFO log). If those threads block, the Metadata Publish step (which makes the new data visible) hangs. * Result: A CN node might have finished writing the data to Azure, but the FE is stuck mid-transaction. This creates a "split" where the physical data exists but the FE hasn't finished the bookkeeping, leading to inconsistencies in which versions are "visible." 2. Audit Log Dependency With 4 pipes running, your FE is generating thousands of audit log entries. * The 
fe.audit.log
tracks every single DML statement (the 
INSERT
from your pipes). * If the FE cannot write to the audit log, it can lead to internal RPC timeouts. If an 
INTERNAL_ERROR
or 
TIMEOUT
occurs during the "Publish" phase because of log I/O blocking, the FE might not correctly update the tablet metadata in its memory, even if the BDBJE (meta) part succeeded. 3. "Object Does Not Exist" - The Metadata Drift In shared-data mode, the FE maintains a mapping of 
Tablet -> Version -> File List
. * If the FE process was unstable or partially blocked by log I/O when it was trying to handle a compaction task, it might have marked older files for Vacuum (deletion) but failed to correctly register the new compacted files. _ When the Vacuum thread (which is a background task) eventually runs, it sees "unreferenced" files in Azure and deletes them. If your query then tries to access what the FE _thinks* is the current version, but that version was corrupted during the disk-full event, you get the 
Object does not exist
error. 4. Troubleshoot and Clear the "Hosed" State Now that you've confirmed the volumes are separate, do this immediately: 1. Truncate the Audit Log:
Copy code
bash
    # From inside the FE pod
    > /opt/starrocks/fe/log/fe.audit.log
2. Restart the Leader FE: Even if it's "Alive," it might be in a zombie state due to blocked threads. Restarting it will force it to reload the metadata from the 
meta
PVC and clear any blocked I/O threads. 3. Check Tablet Health: Run this to see if the FE can even talk to the metadata for that specific tablet:
Copy code
sql
    SHOW TABLET 28769437;
If it returns an error or empty result, the metadata for that tablet was corrupted during the disk-full event. 4. Force a Metadata Image: Once you have cleared space and restarted, run:
Copy code
sql
    ALTER SYSTEM CREATE IMAGE;
This forces the FE to write a clean "snapshot" of the current metadata to the 
meta
PVC, which can help stabilize the cluster. Long-term Fix: * Audit Log Rotation: In your 
fe.conf
, check 
sys_log_roll_num
and 
audit_log_roll_num
. If they are set too high for your PVC size, decrease them (e.g., set 
audit_log_roll_num = 5
). * Automated Cleanup: Ensure your Helm chart or PVC management has a sidecar or a cronjob to truncate logs if they exceed 80% capacity. If the table is still unreadable after a restart and log cleanup: You likely have a "Version Mismatch" in Azure. Check the BE logs (
<http://be.INFO|be.INFO>
) for the string 
failed to get manifest
. If you see that, the tablet's metadata file on Azure is missing or corrupted, and you will likely need to 
TRUNCATE
that partition or table and reload. References
g
@Rocky can i put 
audit_log_delete_age
in the helm values?
r
Yes, you can absolutely put 
audit_log_delete_age
 in your Helm values. In the standard StarRocks Helm chart (using the StarRocks Operator), you should place this under the 
starrocksFESpec.config
 section. This is a multi-line string where you can pass any 
fe.conf
parameters. How to add it to 
values.yaml
Copy code
yaml
starrocks:
  starrocksFESpec:
    config: |
      # Retain audit logs for only 7 days instead of the default 30
      audit_log_delete_age = 7d
      
      # You can also limit the total number of files to prevent PVC overflow
      audit_log_roll_num = 10
      
      # Optional: enable compression for rotated logs (available in 3.2.12+)
      audit_log_enable_compress = true
Why this is helpful for your situation: * Default is 30 days: With 4 high-frequency pipes, 30 days of audit logs can easily grow to hundreds of gigabytes. * Safety net: Since your log PVC is currently filling up, reducing this value to 
7d
or even 
3d
will trigger the FE to purge old files during the next check cycle. * Mutable vs. Immutable: Note that 
audit_log_delete_age
is generally not mutable via 
ADMIN SET CONFIG
, so applying it via Helm will trigger a rolling restart of your FE pods, which is actually recommended anyway to clear those blocked threads we discussed earlier. Related Parameters to Consider: If you are still hitting disk limits, you should also look at these in your 
config
block: 1. 
audit_log_roll_num
: Limits the number of log segments kept. If you set this to 
10
, the system will only keep the 10 most recent segments regardless of age. 2. 
log_cleaner_audit_log_min_retention_days
: This is a "hard floor" for the auto-cleaner. 3. 
sys_log_delete_age
: Similar to audit logs but for the main 
fe.log
. Recommendation: After you update your Helm values and the FEs restart, double-check that the space is being reclaimed by running 
df -h
inside the FE pod. If the logs don't delete immediately, you can manually delete the oldest 
fe.audit.log.YYYYMMDD
files to get the get the cluster back to a healthy state. References • administration/management/FE_configuration.md
g
@Rocky you put Long-term Fix: * Audit Log Rotation: In your 
fe.conf
, check 
sys_log_roll_num
and 
audit_log_roll_num
. If they are set too high for your PVC size, decrease them (e.g., set 
audit_log_roll_num = 5
). * Automated Cleanup: Ensure your Helm chart or PVC management has a sidecar or a cronjob to truncate logs if they exceed 80% capacity. is that an either or? do I need both?
r
It is not an "either/or" in terms of functionality, but it is an "either/or" in terms of what you need to do today to get stable. Here is the breakdown of why you might want one or both: 1. The Internal Config (Primary Solution) Do you need this? YES. This is the "standard" way StarRocks manages itself. By setting 
audit_log_delete_age
and 
audit_log_roll_num
, you are telling the StarRocks process itself to keep its house clean. * Pros: Easy to set in Helm; no extra Kubernetes objects to manage. * Cons: StarRocks usually checks these limits on a cycle (e.g., once a day or when a log file rolls over). If your 4 pipes generate 50GB of logs in 2 hours but your PVC is only 20GB, the internal "cleaner" might not trigger fast enough to save you. 2. The Automated Cleanup/Sidecar (Safety Net) Do you need this? ONLY if your logs grow faster than your PVC can handle between cycles. An external cronjob or sidecar (like a simple script that runs 
find /log -type f -mtime +3 -delete
) acts as a "hard" enforcement. * Pros: It doesn't rely on the StarRocks JVM. Even if the FE is "hosed" or hanging, a sidecar can still clear disk space to allow the FE to recover. * Cons: Adds complexity to your Helm chart/deployment. ────────── Recommendation for Your Setup: Since you are running 4 pipes (which is high-intensity for audit logs), you should prioritize the Internal Config first, but tune it aggressively. Start with these in your Helm `values.yaml`:
Copy code
yaml
starrocksFESpec:
  config: |
    # Reduce retention from 30 days to 3
    audit_log_delete_age = 3d
    # Keep fewer rolled files (default is usually 90)
    audit_log_roll_num = 10
    # Roll the log every 512MB instead of 1GB to trigger the cleaner more often
    log_roll_size_mb = 512
When to add the "Automated Cleanup" (Sidecar): * If you find that even with the settings above, your disk hits 100% before the files are deleted. * If you are in a production environment where "Disk Full" is a critical P0 incident and you want a script to 
truncate
the audit log if the PVC hits 90% capacity. Summary: Apply the Internal Config via Helm now. It is usually sufficient for 95% of users. Only look into sidecars/cronjobs if your log volume is so extreme that it outpaces the FE's internal cleanup frequency of deletion. References • administration/management/FE_configuration.md • administration/management/logs.md • deployment/sr_operator.md
g
@Rocky I guess it wasn't the full logs dir, getting
Copy code
[PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 30724399 of txnIds [31059316]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[[30724408]], error msg: prepare_primary_index: load primary index failed: Not found: load_segments failed tablet:30724408 rowset:3 segid:0: starlet err Object https://
<mystorageaccount>.<http://dfs.core.windows.net/<myblob>/starrocks/99279b65-e878-4e12-ae2d-52298f81f77c/db68230/28693266/30724399/data/0000000001d97e57_251e94a6-483e-4b20-a175-f835c977c649.dat|dfs.core.windows.net/<myblob>/starrocks/99279b65-e878-4e12-ae2d-52298f81f77c/db68230/28693266/30724399/data/0000000001d97e57_251e94a6-483e-4b20-a175-f835c977c649.dat> does not exist
be/src/storage/lake/rowset.cpp:366 load_segments(&segments, false), host: kube-starrocks-cn-1.kube-starrocks-cn-search.kube-starrocks.svc.cluster.local
now
r
This is a much more serious error than just a full log disk. The error 
Object ... does not exist
during the 
PublishVersion
phase indicates Metadata Inconsistency or Data Loss in your Azure Blob Storage. Specifically, your CN (Compute Node) is trying to load a data segment (
.dat
file) to build/update the Primary Key Index, but the file is physically missing from the storage bucket. Because this is a Primary Key table, StarRocks cannot proceed with the transaction without that file (it needs to check for duplicate keys or apply updates). Why did this happen? 1. Vacuum Race Condition: If your 
lake_autovacuum_grace_period_minutes
(FE config) is set too low (default is 30 mins), and you are running very high-frequency pipes, the StarRocks "AutoVacuum" process might have deleted a file that the FE still thinks is active. 2. Incomplete Upload during Disk Full: Earlier, when your log PVC was 100% full, the CN node might have encountered I/O errors. It is possible it "committed" the metadata to the FE but failed to actually persist the 
.dat
file to Azure, or the upload was interrupted but not correctly rolled back. 3. External Deletion: Ensure no Azure lifecycle policies (TTL) or manual deletions were performed on that bucket path. Step 1: Verify the File Existence First, check your Azure portal or use the CLI to see if the file actually exists at that path: 
.../starrocks/99279b65-e878-4e12-ae2d-52298f81f77c/db68230/28693266/30724399/data/0000000001d97e57_251e94a6-483e-4b20-a175-f835c977c649.dat
* If the file is THERE: This is likely a permission issue or a networking/endpoint issue between the CN and Azure. * If the file is MISSING: The tablet is corrupted. Step 2: Check Tablet Metadata Run the following to see what the FE thinks about this tablet:
Copy code
sql
SHOW TABLET 30724408;
Check the 
IsSync
and 
Version
columns. If 
IsSync
is 
false
, the metadata is already aware of a discrepancy. Step 3: Identify the Affected Partition The error mentions partition 
30724399
. You can find the human-readable partition name with:
Copy code
sql
SELECT partition_name, table_name, database_name 
FROM information_schema.partitions_meta 
WHERE partition_id = 30724399;
Step 4: How to fix the "Hosed" state Since the FE is stuck in a loop trying to publish a transaction that will never succeed (because the file is gone), you have a few options: Option A: Increase Vacuum Grace Period (If the file is "about" to be deleted) If this is a race condition, try setting this in the FE (via Helm or SQL):
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
This won't bring back a deleted file, but it prevents the issue from spreading. Option B: Truncate the Partition (Fastest Fix) If you can afford to re-ingest the data for that specific partition, truncating it will wipe the corrupted metadata and physical files, allowing the pipes to start fresh:
Copy code
sql
ALTER TABLE <table_name> TRUNCATE PARTITION <partition_name>;
This is often the only way to clear the 
PublishVersion
block if the physical data is gone. Option C: Drop and Recreate the Primary Index (Advanced) If the table is Cloud-Native Primary Key, you can try dropping the index for that tablet to force a rebuild:
Copy code
sql
ALTER TABLE <table_name> DROP PERSISTENT INDEX ON TABLETS(30724408);
Recommendation 1. Check Azure immediately to confirm if the file is missing. 2. If the file is missing, Truncate the affected partition. 3. Restart your CN nodes once the log space is cleared to ensure they aren't holding stale connections or blocked I/O handles to Azure. 4. Increase 
lake_autovacuum_grace_period_minutes
to 
60
or 
120
in your 
fe.conf
(via Helm) to provide more buffer for your 4 pipes. References • knowledge/usage/best-practice.md
g
@Rocky Maybe I should remove the cache PVC from my CNs and restart them?
r
Removing the cache PVC is a possible step, but it might not be the "magic fix" you're hoping for because the error specifically states that the file is missing from Azure Blob Storage, not just your local cache. However, in a Primary Key table (Cloud-Native), the CN nodes store a Persistent Index on that same PVC. If that local index is out of sync with what’s actually in Azure, clearing it can force the CN to rebuild its state. Here is the strategic way to handle this: 1. The "Why" of the Cache PVC In your architecture (StarRocks Lake / Shared-Data), the CN's local disk (PVC) stores two critical things: * Data Cache (
/datacache
): Temp copies of remote 
.dat
files to speed up queries. * Persistent Index (
/persist
): The metadata that keeps track of where every Primary Key is located. 2. Should you remove the PVC? Don't remove the PVC mapping in Helm/K8s entirely. If you do, StarRocks will either fail to start (because it expects a 
storage_root_path
) or it will use the Pod's ephemeral storage, which will fill up and crash your CNs instantly. 3. What you should do instead: "The Deep Reset" Instead of removing the PVC, you can wipe the contents of the storage directory. This forces the CN to start with a blank slate and rebuild its index from the metadata stored in the FE. Steps to perform a safe reset: 1. Check Azure first: (As mentioned before) If the file 
.../0000000001d97e57_251e94a6-483e-4b20-a175-f835c977c649.dat
is truly gone from your Azure container, resetting the CN will not fix this. You must truncate the partition. 2. If you suspect local corruption: * Scale your CN deployment to 0. * Manually delete the contents of the PVC (specifically the 
starlet_cache
and 
persist
folders). * Scale the CN back to 1. * The CN will pull the necessary metadata from the FE and try to re-index the data it finds in Azure. 4. When to actually disable the cache If your PVC is too small and constantly causing "Disk Full" errors that lead to these corruptions, you can disable the Data Cache while keeping the Persistent Index (which is much smaller). In your Helm `values.yaml`:
Copy code
yaml
starrocksCNFESpec:
  config: |
    # Disable the data cache (saves disk space)
    datacache_enable = false
    # Keep the storage_root_path for Persistent Indexes!
    storage_root_path = /opt/starrocks/cn/storage
Summary Recommendation * Immediate Fix: Run 
SELECT partition_name FROM information_schema.partitions_meta WHERE partition_id = 30724399;
and then 
ALTER TABLE <name> TRUNCATE PARTITION <name>;
. This is the only way to stop the "Publish Version" error if the file is missing from Azure. * Sanity Check: Clear the 
/persist
directory on the CN pods only if you think the FE and CN are "disagreeing" about what files exist. * Long-term: Increase your 
lake_autovacuum_grace_period_minutes
to 
60
or 
120
to prevent the system from deleting files too aggressively while your high-frequency pipes are still working on them. References • using_starrocks/caching/block_cache.md • table_design/table_types/primary_key_table.md • deployment/shared_data/s3.md • data_source/data_cache_troubleshooting.md References • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md • deployment/downgrade.md
g
@Rocky this continues to be an issue. Should 
lake_autovacuum_grace_period_minutes
be set for as long as it takes the pipes to run?
r
Yes, in high-intensity ingestion scenarios (like running 4 active pipes), the default 
lake_autovacuum_grace_period_minutes = 30
is often too short and is the most likely cause of your "Object does not exist" errors. Why it should be longer In StarRocks Lake (Shared-data) mode, the process works like this: 1. Ingestion: Your pipes write new data segments to Azure. 2. Compaction: StarRocks merges many small segments into larger ones to keep queries fast. Once merged, the old segments are marked as "stale." 3. Vacuum: The AutoVacuum process physically deletes those "stale" files from Azure after the 
lake_autovacuum_grace_period_minutes
has passed. The Problem: If a CN node is still busy building a Primary Key Index or a user is running a long query that references those "stale" segments, and the 30-minute timer runs out, AutoVacuum deletes the file. The next time the CN tries to read that segment to finish its job, it finds nothing. Should it be as long as the "Pipe"? Not necessarily the entire life of the pipe (if the pipe runs for days), but it must be longer than the following two things: 1. Maximum Query Time: The longest time any single SQL query takes to run. 2. Maximum Publish/Index Time: The time it takes for a CN node to "catch up" and build the Primary Index for a high-volume partition. If your CNs were slowed down by the previous "Disk Full" issue or high CPU, this process could easily exceed 30 minutes. Recommendation For your environment, I recommend increasing this to at least 2–4 hours (120 or 240 minutes). This provides a much safer buffer for compaction and indexing to complete without the vacuum "cleaning up" from under them. Apply this via SQL (dynamic):
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
And update your Helm 
values.yaml
to make it permanent:
Copy code
yaml
starrocksFESpec:
  config: |
    lake_autovacuum_grace_period_minutes = 120
One Final Check: Azure Lifecycle Policies Check if your Azure Storage Account has its own "Lifecycle Management" policies. If you have a rule in Azure that deletes blobs older than $X$ days or hours, it will conflict with StarRocks and cause this exact error. StarRocks must be the only entity managing the deletion of files in its data directory. Summary * Is it the Pipe duration? No, it's the "Transaction + Indexing + Query" duration. * Is 30 mins enough? Clearly not for your current load. * Is there a penalty? The only penalty for increasing this is that your Azure Storage usage will be slightly higher because "deleted" files stick around a few hours longer before being purged. Given being purged. In deleted for good wiped. Given storage is cheap compared to cluster stability, 120–240 minutes is the production sweet spot. References • administration/Meta_recovery.md • faq/loading/Loading_faq.md • administration/management/monitoring/metrics.md • data_source/datalake_faq.md
g
@Rocky how can I find out what 
*Maximum Publish/Index Time:* The time it takes for a CN node to "catch up" and build the Primary Index for a high-volume partition. If your CNs were slowed down by the previous "Disk Full" issue or high CPU, this process could easily exceed 30 minutes.
is?
r
To find the "Maximum Publish/Index Time" and determine the correct buffer for 
lake_autovacuum_grace_period_minutes
, you can look at specific FE Metrics and FE/CN Logs. 1. Prometheus / Grafana Metrics (Best Method) If you have Prometheus/Grafana set up, the Leader FE exposes the exact latency of the publish phase. Look for these metrics: * 
starrocks_fe_txn_publish_latency_ms
: The total time from when a transaction was committed to when it was finished (visible). * 
starrocks_fe_txn_publish_execute_latency_ms
: This is the most important one for you. It measures the active execution time on the CN node. For Primary Key tables, this includes the time to: 1. Download metadata from Azure. 2. Load or rebuild the Primary Key Index (this is what failed in your error log). 3. Apply updates/deletes. Action: Check the 99th Percentile (P99) or the Max value of 
starrocks_fe_txn_publish_execute_latency_ms
. If this value is near or exceeds 1,800,000 ms (30 minutes), your 
lake_autovacuum_grace_period_minutes
is too low. 2. FE Log Analysis You can see real-time "Publish" durations in your 
fe.log
. Search for the 
PublishVersionDaemon
or the specific transaction IDs mentioned in your error. Command to run on your FE node:
Copy code
bash
grep "finish publish version" fe.log | tail -n 20
Example Log Entry: 
... finish publish version for txn 31059316, cost 45000ms ...
If you see transactions taking 10-20 minutes during peak load, and then suddenly failing with "Object not found" after 30 minutes, it confirms the Vacuum is deleting files while the CN is still working on that transaction. 3. Checking for Queueing (Bottlenecks) If your 
starrocks_fe_txn_publish_schedule_latency_ms
is high, it means the FE is waiting to send tasks to the CN because the CN is overwhelmed. On the CN node (be.conf), you can check/increase this concurrency: * `transaction_publish_version_worker_count`: (Default is CPU cores). If this is set too low, transactions will wait in a queue, eating into your 30-minute grace period. Summary Strategy 1. Find the peak latency: Use the metrics above to find your "worst-case" publish time. 2. Set the buffer: Set 
lake_autovacuum_grace_period_minutes
to at least 2x or 3x that maximum time. _ _Example:* If your max publish time is 20 minutes, set the grace period to 60 minutes. _ _Example:* Since you have 4 pipes and are hitting errors, I strongly recommend starting with 120 minutes (2 hours). How to apply the change:
Copy code
sql
-- Change it immediately without restart
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
Why this fixes your specific error: The error 
load primary index failed: Not found
happened because the CN was told to publish a version, it went to Azure to grab the data segments to build the index, but the AutoVacuum process had already "cleaned up" those segments because they were more than 30 minutes old (likely from a previous failed or slow attempt). Increasing the grace period keeps those files alive long enough for the CN to finish its work. References • loading/loading_introduction/troubleshooting_loading.md • best_practices/primarykey_table.md • administration/management/monitoring/metrics.md • faq/operation_maintenance_faq.md
g
@Rocky can i see 1. Prometheus / Grafana Metrics (Best Method) If you have Prometheus/Grafana set up, the Leader FE exposes the exact latency of the publish phase. Look for these metrics: * 
starrocks_fe_txn_publish_latency_ms
: The total time from when a transaction was committed to when it was finished (visible). * 
starrocks_fe_txn_publish_execute_latency_ms
: This is the most important one for you. It measures the active execution time on the CN node. For Primary Key tables, this includes the without prometheus/grafana set up?
r
Yes, you can absolutely see these metrics without Prometheus or Grafana. StarRocks provides several built-in ways to access this data directly. 1. The Raw Metrics Endpoint (Recommended) This is exactly where Prometheus gets its data. You can access it using 
curl
or just by typing the URL into your browser. Command:
Copy code
bash
curl -s http://<FE_LEADER_IP>:8030/metrics | grep starrocks_fe_txn_publish
_(Replace 
<FE_LEADER_IP>
with the IP of your Leader FE node. The default port is 8030)._ What to look for: You will see output like this:
Copy code
text
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.99",type="all",is_leader="true"} 45000.0
starrocks_fe_txn_publish_execute_latency_ms_sum{type="all",is_leader="true"} 1.25e+08
starrocks_fe_txn_publish_execute_latency_ms_count{type="all",is_leader="true"} 5420
* 
quantile="0.99"
: This is your P99 latency. In this example, 99% of transactions finished the "execute" phase in under 45,000ms (45 seconds). * If you see values like 
1800000.0
(30 mins), it confirms your CNs are struggling to finish before the default vacuum window closes. ────────── 2. SQL: Inspecting Specific Transactions If you have a specific Transaction ID from an error log, you can see its exact "Publish" duration using SQL. Command:
Copy code
sql
SHOW TRANSACTION WHERE ID = 31059316;
How to calculate the latency: In the result, look at these two timestamps: * 
CommitTime
: When the data was safely written to Azure. * 
FinishTime
: When the transaction became visible to users (after indexing/publishing). Latency = 
FinishTime
minus 
CommitTime
. If the gap between these two is consistently hitting 30 minutes, you've found your bottleneck. ────────── 3. StarRocks Web UI StarRocks has a basic built-in dashboard that requires no setup: 1. Open 
http://<FE_IP>:8030
in your browser. 2. Log in (default is 
root
with no password). 3. Click on the "System" or "Metrics" tab. _ _Note: While the 
/metrics
endpoint is the most precise, the Web UI often provides a "Table" view of common health stats.* 4. FE Log "Slow Publish" Warnings The FE will automatically log transactions that take a long time to publish. Check your 
fe.warn.log
or 
fe.log
for lines like: 
... finish publish version for txn XXX, cost YYY ms ...
Summary for your Troubleshooting: Run the 
curl
command in Step 1. If the 
quantile="0.99"
value for 
starrocks_fe_txn_publish_execute_latency_ms
is high (e.g., > 15 minutes), it is a clear signal that you must increase 
lake_autovacuum_grace_period_minutes
to a much higher value (like 120 or 240) to prevent the vacuum from deleting files before the CN can finish indexing them. References • administration/management/monitoring/metrics.md • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • administration/http_interface.md • administration/management/proc_profile.md References • administration/management/monitoring/metrics.md • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/resource_management/query_queues.md
g
@Rocky the curl only shows 0's for everything, is that because i'm hitting the load balancer ip?
r
Yes, your suspicion is exactly correct. In StarRocks, transaction and publishing metrics are only collected and reported by the Leader FE node. When you hit a Load Balancer, your request is likely being routed to a Follower or Observer FE. These nodes do not manage the transaction lifecycle, so they report 
0
for those specific metrics. How to get the correct data: 1. Find the Leader FE IP Run this SQL command in your StarRocks console:
Copy code
sql
SHOW FRONTENDS;
Look for the row where 
IsMaster
 (or 
IsLeader
in newer versions) is 
true
. Note the 
Host
(IP address) of that specific node. 2. Curl the Leader IP directly Run the curl command again, but replace the Load Balancer IP with the specific IP of the Leader FE:
Copy code
bash
curl -s http://<LEADER_FE_IP>:8030/metrics | grep starrocks_fe_txn_publish
What to look for in the output: When you hit the correct node, the metrics will include a label 
is_leader="true"
. * If you see `is_leader="false"`: You are hitting a follower; the values will be 
0
. * If you see `is_leader="true"`: These are the real numbers for your cluster. Key Metrics to check on the Leader: * `starrocks_fe_txn_publish_execute_latency_ms{quantile="0.99",...}`: If this value is high (e.g., 
600000
for 10 mins), it means your CNs are taking a long time to finish the data "visibility" phase. * `starrocks_fe_txn_publish_latency_ms_count`: This tells you how many transactions have actually been recorded. If this is 
0
even on the leader, it means no transactions have completed since the last FE restart. Quick Alternative (SQL only) If you can't easily curl the Leader IP due to network restrictions, you can check the "Slowest" transactions directly via SQL to see if they are approaching that 30-minute mark:
Copy code
sql
-- This shows you the last 10 transactions
-- Look for the difference between 'CommitTime' and 'FinishTime'
SHOW TRANSACTION FROM <your_database_name> ORDER BY TransactionId DESC LIMIT 10;
If the duration between Commit and Finish is consistently long, it confirms that your 
lake_autovacuum_grace_period_minutes
needs to be increased to be increased. References • administration/management/monitoring/metrics.md • release_notes/release-3.5.md • administration/management/BE_configuration.md
g
@Rocky mine are all super fast
Copy code
# HELP starrocks_fe_txn_publish_execute_latency_ms
# TYPE starrocks_fe_txn_publish_execute_latency_ms summary
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.75", is_leader="true", type="all"} 153.0
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.95", is_leader="true", type="all"} 197.0
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.98", is_leader="true", type="all"} 219.0
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.99", is_leader="true", type="all"} 234.0
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.999", is_leader="true", type="all"} 353.0
starrocks_fe_txn_publish_execute_latency_ms_sum{is_leader="true", type="all"} 1.537227008160427E7
starrocks_fe_txn_publish_execute_latency_ms_count{is_leader="true", type="all"} 116464
seems like that shouldn't be the issue
r
Since your current metrics show very fast times (P99 = 234ms), it means your cluster is currently healthy and "caught up." The scenario described in the quote happens when the system is under extreme stress or recovering from an incident (like your previous Disk Full or High CPU event). Here is how you can find out what that "Maximum Publish/Index Time" actually is or was during the problematic period: 1. Check the CN (Compute Node) Logs While FE metrics give you the "average" or "percentile" view, the CN logs show the gritty details of individual partitions that are struggling. Connect to your CN/BE nodes and search for the Apply cost. This is the specific step where the Primary Index is built or updated. Run this command on your CN nodes:
Copy code
bash
grep "apply rowset cost" <http://be.INFO|be.INFO> | tail -n 20
Look for output like this: 
... apply rowset cost: 120000ms, tablet: 12345, version: 100 ...
* If you see 
cost
values in the thousands of milliseconds (seconds), it's normal. * If you see values approaching 1,800,000ms (30 minutes), that is exactly the "Index Time" the quote is warning you about. 2. Check "Load Primary Index" specific logs When a CN first accesses a Primary Key partition (or after a restart), it must "load" the index. If the data volume is huge, this is the "Catch Up" phase. Search for the load/recovery logs:
Copy code
bash
grep "load primary index" <http://be.INFO|be.INFO>
# OR
grep "recover primary index" <http://be.INFO|be.INFO>
If these logs show durations longer than 30 minutes, or if they are followed by an error like 
Not found
, it confirms that the files were deleted by the Vacuum process while the CN was still trying to read them to build the index. 3. Metric: 
update_rowset_commit_apply_duration_us
 This is a CN-level metric (per node) that measures the total time spent on the "Apply" task for Primary Key tables. Run this on each CN node:
Copy code
bash
curl -s http://<CN_IP>:8040/metrics | grep update_rowset_commit_apply_duration_us
* Divide the result by 1,000,000 to get seconds. * This is a cumulative counter. To find the "Maximum" time for a single transaction, you would typically monitor the increase in this metric over time via Prometheus/Grafana. Since you don't have that set up, the Logs (Method 1 & 2) are your best source of truth for historical "spikes." 4. Why your current FE metrics are "Super Fast" The reason your 
starrocks_fe_txn_publish_execute_latency_ms
is only 234ms right now is likely because: 1. Warm Cache: The Primary Indexes for your active partitions are already built and sitting in the CN's memory. Updating an index that is already "warm" is near-instant. 2. No Backlog: Your CNs are not currently fighting for CPU/IO, so they process the "Apply" tasks immediately. Summary: When should you worry? The "30-minute danger zone" only applies when: * You are loading massive amounts of data into a new or cold partition. * The CN just restarted and has to rebuild indexes for everything. * The CN is CPU-saturated, causing the "Apply" thread to move slowly. Recommendation: Even if things are fast now, if you ever saw that "Not Found" error, it means you hit the limit in the past. To prevent it from happening during the next "Disk Full" or high-load event, you should still increase the grace period:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
References • administration/management/monitoring/metrics.md • loading/loading_introduction/troubleshooting_loading.md • administration/management/BE_configuration.md • best_practices/primarykey_table.md References
g
@Rocky i see this often in the CN logs 
primary_index_load_latency_us
is this a good metric?
r
Yes, 
primary_index_load_latency_us
is an excellent diagnostic metric, especially for Cloud Native (Shared-Data) clusters using Primary Key tables. While your FE metrics (P99 = 234ms) look great now, this specific CN metric explains what happens during those "bad" periods when things feel slow or you encounter "Not Found" errors. What this metric tells you In a Shared-Data cluster, the Primary Index (the map that tells StarRocks where each key is located) is stored in object storage (Azure Blob). * When it's low: The index is already in the CN's local disk cache or RAM. Updates are near-instant. * When it's high: The CN had to go all the way to Azure Blob to fetch the index files before it could even begin processing your data. How to interpret the values The metric is in microseconds (us). To make it readable: * 1,000 us = 1ms (Fast/Normal) * 1,000,000 us = 1 second (Slightly "Cold" index) * 60,000,000 us = 1 minute (Significant bottleneck) Why you see it "often" If you see this metric popping up frequently with high values, it means your Index Cache is "thrashing." This happens if: 1. Memory/Disk Pressure: Your CN nodes don't have enough local disk space (the 
starrocks_path/storage
directory) or RAM to keep the indexes of all your active partitions. It's constantly deleting old indexes to make room for new ones. 2. Wide Ingestion: You are loading data into a very large number of partitions simultaneously. Each partition requires its own index to be loaded. 3. Cold Starts: After a CN restart or a long period of inactivity, the first load will always be slow as it pulls from Azure. The connection to your "Vacuum" issue There is a direct chain of cause-and-effect here: 1. Event: You start a big load after an error/restart. 2. Bottleneck: 
primary_index_load_latency_us
spikes because the CN is fetching indexes from Azure. 3. Result: The "Apply" phase (data visibility) slows down. 4. The Crash: If this process takes longer than the Vacuum Grace Period (default 30 mins), the Vacuum job might delete the very files the CN is trying to load, leading to a "failed to load primary index: Not Found" error. Recommendation * Monitor for Spikes: If you see this value exceeding 10–20 seconds (`10,000,000`–`20,000,000` us), it’s a sign that your CNs are struggling with "Index Cold Starts." * Optimization: Ensure your CN nodes have high-speed local SSDs for the 
storage
path. This is where the Persistent Index is cached to avoid going back to Azure. * Setting: If you use Persistent Index (highly recommended for large PK tables), check your 
be.conf
for 
persistent_index_read_buffer_size
. Summary: If 
primary_index_load_latency_us
is high, your "Publish" phase will be slow. If you can't make the storage faster, you must increase 
lake_autovacuum_grace_period_minutes
to give the CN enough time to finish these slow loads during these "cold" starts. References • administration/management/monitoring/metrics.md • loading/loading_introduction/troubleshooting_loading.md • best_practices/primarykey_table.md
g
@Rocky is this log telling me anything good?
Copy code
I20260508 17:26:25.633194 140109558076992 lake_service.cpp:331] Published txns=31058816. tablets=30730274,30730276 cost=1927735us, trace: {"child_traces":[["PublishTablet",{"base_version":9,"deletes":0,"do_update_latency_us":539,"get_column_values_latency_us":59223,"get_each_segment_iterator_with_delvec_us":101876,"handle_index_op_latency_us":31,"lake_persistent_index_insert_us":38312,"load_segment_us":108952,"new_del":2,"pindex_memtable_erase_us":0,"pindex_memtable_get_us":25,"pindex_memtable_insert_us":34076,"pindex_memtable_upsert_us":48,"primary_index_commit_latency_us":2,"primary_index_load_latency_us":978713,"queuing_latency_us":42,"rebuild_build_values_cost_us":179,"rebuild_get_next_cost_us":830824,"rebuild_index_num_rows":104613,"rebuild_index_segment_cnt":8,"rebuild_index_segment_cost_us":876388,"rebuild_pk_encode_cost_us":4240,"rewrite_segment_latency_us":768564,"rowsetid":9,"segment_io_local_disk_us":0,"segment_io_remote_us":758780,"tablet_id":30730274,"total_del":2,"total_num_rows":104613,"total_segment_cnt":8,"update_index_latency_us":1497,"upsert_rows":13,"upserts":1}],["PublishTablet",{"base_version":9,"deletes":0,"do_update_latency_us":449,"get_column_values_latency_us":81295,"get_each_segment_iterator_with_delvec_us":218224,"handle_index_op_latency_us":32,"lake_persistent_index_insert_us":36305,"load_segment_us":127910,"new_del":3,"pindex_memtable_erase_us":0,"pindex_memtable_get_us":27,"pindex_memtable_insert_us":32096,"pindex_memtable_upsert_us":44,"primary_index_commit_latency_us":1,"primary_index_load_latency_us":1038581,"queuing_latency_us":47,"rebuild_build_values_cost_us":205,"rebuild_get_next_cost_us":776143,"rebuild_index_num_rows":104618,"rebuild_index_segment_cnt":8,"rebuild_index_segment_cost_us":819921,"rebuild_pk_encode_cost_us":4235,"rewrite_segment_latency_us":746344,"rowsetid":9,"segment_io_local_disk_us":0,"segment_io_remote_us":695346,"tablet_id":30730276,"total_del":3,"total_num_rows":104618,"total_segment_cnt":8,"update_index_latency_us":1341,"upsert_rows":13,"upserts":1}]]}
r
This log is extremely good for diagnostic purposes, but it reveals some inefficiency in how the cluster is currently handling data. It perfectly captures why you might have been seeing issues earlier. Here is the breakdown of what those numbers actually mean: 1. The "Cold Start" Penalty * 
primary_index_load_latency_us: 978,713
(~1 second) * 
segment_io_remote_us: 758,780
(~0.76 seconds) Out of the 1.9 seconds total cost, about 1.7 seconds was spent just "getting ready"—fetching the index and data from Azure Blob storage. The Verdict: Your CN (Compute Node) is currently "Cold." It doesn't have the Primary Index or the data segments for these tablets in its local disk cache. Every time a load happens, it has to download them from the cloud. For a load of only 13 rows (
upsert_rows: 13
), a 2-second wait is a lot of overhead. 2. Why this matters for your "Vacuum" problem This log proves the "30-minute danger" theory. Look at these two fields: * 
rebuild_index_num_rows: 104,613
 * 
rebuild_index_segment_cost_us: 876,388
 Right now, your tablet is small (~100k rows), so rebuilding the index takes less than 1 second. However, if this tablet grew to 100 million rows, and your network to Azure slowed down, that 
rebuild_index_segment_cost_us
could easily climb from 1 second to 20 or 30 minutes. If that happens, and the Vacuum job deletes the old files while the CN is still in that 
rebuild
phase, the load will fail with the "Not Found" error you saw. 3. Local Cache is not being used * 
segment_io_local_disk_us: 0
 This confirms that zero data was read from the CN's local SSD. This usually means: 1. The CN was recently restarted. 2. Or, the CN's local storage is too small, and it's constantly "evicting" (deleting) cached data to make room for new things. 3. Or, you are loading into so many different tables/partitions that the cache can't keep up. Is this "Good"? * Yes, because the transaction finished successfully and the "Apply" logic is working as intended. * No, because the ratio of "Work" (13 rows) to "Wait" (1.9 seconds) is very poor. Recommendations based on this log: 1. Increase Cache: If you see 
segment_io_local_disk_us: 0
constantly in your logs, consider increasing the local disk space allocated to your CN nodes. This will make future "Publish" tasks near-instant (milliseconds instead of seconds). 2. Batch Your Loads: Loading 13 rows at a time in a Cloud Native (Shared-Data) architecture is expensive. If possible, group your data so you are loading thousands of rows per transaction. 3. Confirm the Fix: Since you now know it takes ~1 second to load 100k rows, you should definitely keep that 
lake_autovacuum_grace_period_minutes
set to at least 60 or 120 to give the system a "safety buffer" for when your tables grow much larger. References • administration/management/monitoring/metrics.md • loading/loading_introduction/troubleshooting_loading.md
g
@Rocky why did you say this is ~1 sec 
primary_index_load_latency_us: 978,713
(~1 second), is that not miliseconds?
r
That is a great catch. To be precise: 
_us
stands for microseconds, not milliseconds. Here is the conversion for the number in your log: * Metric: 
primary_index_load_latency_us: 978,713
* In Milliseconds: 978.7 ms * In Seconds: ~0.97 seconds So, while it is technically "milliseconds" (978 of them), in the world of database performance, once a single sub-step of a transaction hits the 1,000,000 us mark, we generally start calling it 1 second. Why the distinction matters: If that number were 978,713 milliseconds, that would be 16 minutes, which would be a massive system failure. Because it is 978,713 microseconds: 1. It is fast: Under 1 second to load an index from Azure is actually decent performance. 2. It is the "smoking gun": Even though 0.9 seconds is "fast" for a human, notice that the Total Cost of your transaction was 
1,927,735us
(~1.9 seconds). * This means 50% of the entire transaction time was spent just loading the index (
978ms
out of 
1927ms
). * The actual work of writing your 13 rows took almost no time at all. Summary You are correct—it is 978 milliseconds. I rounded up to "*~1 second*" to highlight that your CN is spending a significant chunk of its time (nearly a full second) waiting on Azure Blob storage to send the index files before it can finish the load. If you see that number jump from 978,713 (0.9s) to 90,000,000 (90s), that's when you'll know your object storage or network is having a is having a bad day! References • administration/management/monitoring/metrics.md • loading/loading_introduction/troubleshooting_loading.md • administration/management/BE_configuration.md • best_practices/primarykey_table.md References • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • faq/Others.md • administration/management/logs.md
g
@Rocky should I grep all of my cn logs to see if any got high enough to be over the 30 minute mark?
r
Yes, absolutely. Grepping your CN logs for extreme latency values is the single best way to prove if your "Not Found" errors were caused by the Vacuum job outrunning the CN's ability to load data. If you hit the 30-minute mark, the CN is still trying to "Apply" the transaction while the Vacuum job (on the FE) thinks the old data is no longer needed and deletes it. The Math for your Grep Since the metric is in microseconds (us), you are looking for 1.8 billion. * 30 minutes = 1,800 seconds * 1,800 seconds = 1,800,000,000 us How to Grep You want to look in your 
<http://cn.INFO|cn.INFO>
(or 
<http://be.INFO|be.INFO>
) logs. Since the numbers are huge, you can search for strings that have 10 digits (which starts at 1 billion). Try this command on your CN nodes:
Copy code
bash
grep "primary_index_load_latency_us" <http://cn.INFO|cn.INFO> | awk -F'primary_index_load_latency_us":' '{print $2}' | awk -F',' '{if($1 > 1000000000) print "WARNING: Extremely high latency: " $1 " us"}'
Note: This script looks for anything over 1 billion us (~16.6 minutes). If you see results here, you are dangerously close to the default 30-minute vacuum window. What else to look for in that same log line While 
primary_index_load_latency_us
is the "loading" part, there are two other fields in that same JSON blob that can eat up your 30-minute window: 1. 
rebuild_index_segment_cost_us
: If your table is huge and the CN has to rebuild the index from scratch, this can take a long time. 2. 
segment_io_remote_us
: This is the time spent actually downloading the data segments from Azure. The Golden Rule: If the sum of these latency metrics in a single log line ever approaches 1,800,000,000, you have found your root cause. Recommendation If your grep returns several lines where the latency is over 500,000,000 (about 8 minutes), your system is "jittery." You should: 1. Increase the safety margin: Set 
lake_autovacuum_grace_period_minutes = 120
(2 hours) in your FE config. This gives the CN plenty of "breathing room" to finish these slow loads without the files being deleted from under it. 2. Check CN Disk Space: Ensure your CNs have enough local cache space so they don't have to go to Azure for the index so often. References • administration/management/monitoring/metrics.md • faq/shared_data_faq.md • administration/management/FE_configuration.md • administration/management/logs.md References • administration/management/monitoring/metrics.md
g
@Rocky this is how we have our CNs configured in helm
Copy code
storageSize: 1Ti
      # the number of persistent volumes for data.
      # if storageCount == 1
      #   the storageMountPath field is used to specify the mount path of the persistent volume. If storageMountPath is empty,
      #   the storageMountPath will be set to /opt/starrocks/cn/storage.
      #   If storageMountPath is not /opt/starrocks/cn/storage, you must add in config the following configuration: storage_root_path = xxx.
      # if storageCount > 1
      #   the storageMountPath field is used to specify the prefix of mount path of the persistent volume. For example, if the
      #   storageMountPath is /opt/starrocks/cn/storage, the real mount path will be /opt/starrocks/cn/storage0, /opt/starrocks/cn/storage1, ...
      #   You must add in config the following configuration: storage_root_path = /opt/starrocks/cn/storage0;/opt/starrocks/cn/storage1;...
      storageCount: 1 #DEPLOY-VALUE
      # see the comment of storageCount for the usage of storageMountPath.
      storageMountPath: "/opt/starrocks-storage" #DEPLOY-VALUE
      # If not set will use the value of the storageClassName field.
      logStorageClassName: ""
      # the storage size of persistent volume for log, and the mount path is /opt/starrocks/cn/log.
      # If you set it to 0Gi, the related PVC will not be created, and the log will not be persisted.
      logStorageSize: 100Gi
      # If logMountPath is empty, the logMountPath will be set to /opt/starrocks/cn/log.
      # If logMountPath is not /opt/starrocks/cn/log, you must add in config the following configuration: sys_log_dir = xxx.
      logMountPath: "/opt/starrocks-log" #DEPLOY-VALUE
r
The configuration you shared defines the infrastructure (the 1Ti persistent volume and its mount point), but it does not necessarily tell the StarRocks process to use it for caching. Based on your previous log showing 
segment_io_local_disk_us: 0
, your CN nodes are likely ignoring that 1Ti of storage for data caching. Here is what you need to check in your Helm 
values.yaml
to fix this: 1. The 
storage_root_path
Requirement As noted in your Helm comments:
_If storageMountPath is not /opt/starrocks/cn/storage, you must add in config the following configuration: storage_root_path = xxx._
Since you set your path to 
/opt/starrocks-storage
, you must ensure that your 
cn.conf
(under the 
config:
section of your Helm chart) includes this line:
Copy code
yaml
config: |
  storage_root_path = /opt/starrocks-storage
If this is missing, the CN defaults to looking at 
/opt/starrocks/cn/storage
. If that directory is on the container's root filesystem (and not the 1Ti PV), it will fill up instantly, and StarRocks will stop caching data, resulting in the 
0
local disk I/O you saw. 2. Configure Cache Size By default, StarRocks might not use the full 1Ti for the data cache unless you tell it to. In your 
config:
section, add these parameters to maximize the 1Ti disk:
Copy code
yaml
config: |
  storage_root_path = /opt/starrocks-storage
  datacache_enable = true
  # Set this to about 80-90% of your 1Ti to leave room for logs/metadata
  datacache_disk_size = 900G
3. Verify the Mount Inside your CN pod, run this command to make sure the StarRocks cache directory actually exists on the mounted drive:
Copy code
bash
ls -lh /opt/starrocks-storage/starlet_cache
If that directory is empty or doesn't exist, your CN is "running cold" and fetching everything from Azure every single time. Why this is the "Fix" for your 30-minute issue: * Current State: Every load has to download the Primary Index from Azure (
primary_index_load_latency_us: 978,713
). * Corrected State: Once the 1Ti cache is working, the CN will find the index on the local SSD. That 
978ms
will drop to under 10ms. * Result: Even if your tables grow 100x larger, the "Apply" phase will happen so fast that the Vacuum job will never have a chance to delete the files while the CN is still using them. Summary: Your Helm storage setup is great (1Ti is plenty), but you likely need to update your StarRocks config (the 
cn.conf
part of the Helm chart) to actually point the engine at that storage. References • using_starrocks/caching/block_cache.md • deployment/shared_data/azure.md • administration/management/BE_configuration.md • quick_start/helm.md References • using_starrocks/caching/block_cache.md • deployment/shared_data/azure.md • administration/management/BE_configuration.md • quick_start/helm.md References • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md • deployment/shared_data/azure.md • deployment/shared_data/gcs.md • using_starrocks/caching/block_cache.md • quick_start/helm.md
g
@Rocky i don't have datacache_enabled = true in my values.yaml but it seems to be working, here is my 
starrocks-storage
Copy code
starrocks@kube-starrocks-cn-0:/opt/starrocks-storage$ du -hc --max-depth=1
800G    ./datacache
412K    ./tmp
16K     ./lost+found
56M     ./meta
4.5M    ./error_log
4.0K    ./data
4.0K    ./persistent
800G    .
800G    total
r
This is excellent news—it means your CNs are correctly utilizing the 1Ti persistent volume you provisioned, even without you explicitly setting 
datacache_enable = true
. Here is why it is working and what your 
du
output tells us: 1. It is "On" by Default In recent versions of StarRocks (Shared-Data / Cloud Native mode), the Data Cache is enabled by default. * 
starlet_use_star_cache
: This parameter defaults to 
true
starting from v3.2.3. * 
datacache_enable
: This parameter (which unifies the cache logic) defaults to 
true
starting from v3.3.0. Since you didn't explicitly turn it off, StarRocks saw your 
storage_root_path
and immediately started using it to accelerate queries and loads. 2. Why is it exactly 800GB? The 
800G ./datacache
in your output is no coincidence. * The default setting for 
starlet_star_cache_disk_size_percent
 is 80. * Since you provisioned a 1Ti (1,024 GB) disk, StarRocks calculated 
1024 * 0.8 = 819 GB
and capped the cache right around that 800GB mark. This is perfect—it leaves ~200GB of "breathing room" for your logs and metadata (the 
./meta
and 
./error_log
directories you see). 3. Re-evaluating your earlier "0 us" log In our previous conversation, we saw a log line with 
segment_io_local_disk_us: 0
. Now that we know your cache is 80% full (800GB), we can interpret that differently: * The Cache was full, but that specific file was a "Miss": If the CN hasn't needed that specific Primary Index file recently, or if it was evicted to make room for other data, the CN had to go to Azure. That’s why you saw 
978ms
for the remote load and 
0
for the local disk. * Index vs. Data: Sometimes "Data Cache" metrics and "Index Load" metrics are reported in separate buckets. Even if your data segments are hitting the local disk, the Primary Index must be loaded into memory. If it’s not in the local persistent cache, it has to be pulled from Azure first. What this means for your 30-minute issue Since your cache is clearly working and large (800GB), you are in a very good position. However, if you are still getting "Not Found" errors: 1. Check if the cache is "thrashing": If you have 10Ti of data but only 800GB of cache, the CN might be constantly deleting old index files to make room for new ones. 2. Increase the Grace Period: Even with a good cache, a "Cold Start" (like after a CN restart or a very large bulk load) will still require a slow download from Azure. To be safe, I still recommend increasing the 
lake_autovacuum_grace_period_minutes
to 120 (2 hours) on your FE. Summary of your setup: * Storage Path: 
/opt/starrocks-storage
(Correctly mapped) * Cache Status: Active (800GB used) * Strategy: You don't need to change your 
values.yaml
to enable the cache, but you might want to increase the 
grace_period
in your FE config to handle those occasional "cold" loads that take ~1 second per file. References • administration/management/BE_configuration.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md • administration/management/FE_configuration.md References • administration/management/BE_configuration.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md • administration/management/FE_configuration.md References
g
@Rocky 1. Check if the cache is "thrashing": If you have 10Ti of data but only 800GB of cache, the CN might be constantly deleting old index files to make room for new ones. how?
r
To check if your cache is "thrashing" (meaning StarRocks is constantly deleting old data to make room for new data because your active dataset is larger than the 800GB cache), you cannot rely on 
du -h
because the cache files stay on disk even after the data inside is marked for eviction. Instead, you should use the internal Data Cache API or Query Profiles. 1. Use the Data Cache API (Best Method) Run this command from a machine that can reach your CN nodes (use the CN's IP and its heartbeat/HTTP port, usually 
8040
):
Copy code
bash
curl http://<CN_IP>:8040/api/datacache/app_stat
What to look for: * 
block_cache_hit_rate_last_minute
: If this is consistently low (e.g., below 0.20 or 20%) while you are running your daily workloads, your CN is failing to find data in the cache and is "thrashing." * 
block_cache_miss_bytes_last_minute
: If this number is very high (gigabytes per minute) while your 
hit_rate
is low, it confirms the CN is fetching almost everything from Azure. 2. Check the "Remove" Metric To see exactly how much data is being kicked out of the cache, use the more detailed stat endpoint:
Copy code
bash
curl http://<CN_IP>:8040/api/datacache/stat
Look for 
block_cache_remove_bytes
. * If this number is growing rapidly while you are loading data, it means for every 1GB you download from Azure, you are deleting 1GB of older data from your local SSD to make room. This is the definition of thrashing. 3. Check a Query Profile If you have a specific query or load that is slow, run it and then check its profile:
Copy code
sql
-- Run your query, then:
get profile for <query_id>;
Look for these two lines in the 
Table
or 
Scan
node: * 
DataCacheReadBytes
: Data pulled from your local 1Ti SSD (Fast). * 
DataCacheWriteBytes
: Data pulled from Azure and then written to the cache because it wasn't there (Slow). If 
DataCacheWriteBytes
is much larger than 
DataCacheReadBytes
on a query you run frequently, your cache is too small for your working set. How to fix Thrashing: If you confirm the cache is thrashing, you have three options: 1. Increase Disk Size: Change your Helm 
storageSize
from 
1Ti
to 
2Ti
or more. 2. Increase Cache Percentage: If you have 1Ti of disk and want to use more than 800GB for cache, set 
starlet_star_cache_disk_size_percent = 90
in your 
cn.conf
. 3. Horizontal Scaling: Add more CN nodes. Since each CN has its own 1Ti cache, adding nodes increases the total "cluster-wide" cache capacity. References • data_source/data_cache_observe.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • data_source/data_cache.md References • data_source/data_cache_observe.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • data_source/data_cache.md References • data_source/data_cache_observe.md
g
@Rocky ok I definitely need to get prometheus/grafana going to make this easier but for now i'm planning to: • set 
lake_autovacuum_grace_period_minutes = 420
r
Setting 
lake_autovacuum_grace_period_minutes = 420
(7 hours) is a solid, "safety first" move. Since your CN is currently taking a long time to load indices from Azure, this 7-hour window guarantees that even the slowest load or query will never have its files deleted from under it by the system. Here is what you need to know about this change: 1. How to apply it Since this is a mutable FE configuration, you don't need to restart your cluster. You can run this command from any SQL client (like MySQL or DBeaver) connected to the FE:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "420");
2. The "Side Effect": Object Storage Cost The only real downside is storage overhead in Azure. * In Shared-Data mode, when a "Compaction" happens, StarRocks creates a new version of the data. * With a 30-minute grace period (default), the old version is deleted after 30 minutes. * With 420 minutes, those old files stick around for 7 hours. * Result: If you have a high volume of writes/updates, you will see your Azure Blob Storage usage "puff up" during the day as it holds onto 7 hours of historical data. Since object storage is cheap, this is usually a very worthwhile trade-off for stability. 3. Why 420 minutes is a good choice for you Given that we saw your primary index loads taking ~1 second per file and your loads can last 30+ minutes, the default 30-minute window was likely "too tight." * If a load starts at 12:00 and hits a specific file version, but Compaction runs at 12:05... * Under the old settings, that file would be deleted at 12:35. * If your load is still running at 12:36, it crashes with a "Not Found" error. * With 420 minutes, that file is safe until 19:05. Pro-Tip: Monitor Azure Storage After applying this, keep an eye on your Azure Storage account. If it grows much more than expected, you can eventually dial this back to 120 or 180 minutes once you have your Prometheus/Grafana dashboards showing that your "hit rates" are high and your "load latencies" have dropped. Summary: This is a safe and effective way to stop the "Not Found" errors immediately while you work on setting up your monitoring. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • deployment/shared_data/azure.md References • administration/management/FE_configuration.md • faq/shared_data_faq.md
g
@Rocky outside of that i'm going to drop the offending partitions and re-pipe, should that clear anything else out that may jam me up? meaning nothing sticks around in cache that could still reference non-existant 
.dat
files
r
Dropping the partitions and re-piping is the correct way to handle this at a logical level. It effectively "resets" the metadata in the FE so it no longer looks for the specific versions of the 
.dat
files that were giving you "Not Found" errors. To answer your specific concerns about "jamming": 1. Will old cache blocks "jam" the new ones? No. In StarRocks' Shared-Data architecture, the Data Cache is keyed by a unique File ID and Offset. * When you re-pipe data into new partitions, StarRocks creates entirely new files in Azure with new, unique File IDs. * The CN will see these as brand-new requests. It will not look at the 800GB of "old" cached data because the File IDs won't match. * The old data will simply sit in the cache as "cold" data and will be the first to be evicted (overwritten) as your new data starts filling up the 1Ti disk. 2. What about the "Persistent" directory? Your 
du
output showed 
./persistent
was only 
4.0K
. This means you likely aren't using the Persistent Primary Index (which usually takes up significant space for PK tables). _ If you _were* using it, dropping the partition is supposed to clean up the associated index files eventually. * Given it's currently empty/tiny, you don't need to worry about stale index files "jamming" your primary key lookups. 3. The "Not Found" Error Loop If you want to be 100% certain that no stale metadata or "negative cache" (info about missing files) is sticking around on a specific CN node, you can perform a manual "Emergency Clear": 1. Connect to the CN node. 2. Delete the contents of the datacache: 
rm -rf /opt/starrocks-storage/datacache/*
3. Restart the CN pod/process. Note: This will make the CN "cold," meaning the first few hours of queries will be slow as it re-downloads everything from Azure. Recommended Workflow for your "Re-pipe": 1. Set the Grace Period first: 
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "420");
. This ensures that while you are re-piping, if any background compaction happens, the "just-written" files aren't deleted too quickly. 2. Drop the offending partitions. 3. Optional: If you are still paranoid about the CN state, clear the 
/datacache/*
folder on the CNs and restart them. 4. Start your Pipe/Load. Summary Dropping the partition is a "clean" operation. Because the new data will have new filenames in Azure, there is zero risk of the CN accidentally pulling old, corrupted, or "non-existent" data from the cache and serving it as part of the new partition. The system will treat it as a fresh start. References • data_source/data_cache_troubleshooting.md • using_starrocks/caching/block_cache.md • administration/management/FE_configuration.md • deployment/shared_data/feature-support-shared-data.md References • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md • deployment/shared_data/feature-support-shared-data.md
g
@Rocky you said 
Your du output showed ./persistent was only 4.0K. This means you likely aren't using the *Persistent Primary Index* (which usually takes up significant space for PK tables).
, this is likely because we have these two configs in our DDL
Copy code
"enable_persistent_index" = "true",	-- Allow the BE/CN to store entire indexes local in RAM
"persistent_index_type" = "CLOUD_NATIVE",	-- Specify the index persistence type based on storage location
r
You are exactly right. Because you are using 
persistent_index_type = "CLOUD_NATIVE"
, the behavior of your system is different from a standard local setup, and it explains why your 
./persistent
directory is empty. 1. Why 
./persistent
is 4.0K In StarRocks, there are three ways to handle the Primary Key index: * In-Memory (
false
): Index lives entirely in RAM. Fast but memory-intensive. * Local Persistent (
LOCAL
): Index is stored on the CN/BE's local disk in the 
./persistent
folder. * Cloud Native (
CLOUD_NATIVE
): This is the "Shared-Data" way. The primary index itself is treated like data—it is stored in Azure Blob Storage as files. The local disk is used only as a cache for these index files. This is why your 
./persistent
folder is empty. Your index files are actually sitting in Azure, and the CN downloads them into the same 1Ti Data Cache where your regular data lives. 2. How this relates to your "Thrashing" This confirms that your "thrashing" is likely even worse than we thought: * Data vs. Index Competition: Your 800GB cache is being shared by both your actual data and your primary key index files. * The "Not Found" Loop: Because your indices are stored in Azure, they are also subject to the 
lake_autovacuum_grace_period
. If a compaction happens and the old index file version is deleted from Azure before the CN finishes a load/query, you get the "Not Found" error. * Index Loading Latency: Every time you have a "cache miss" on an index file, the CN has to fetch it from Azure before it can even figure out where the data is. This is why you saw those ~1s delays for index loads in your logs. 3. Will dropping partitions clear the indices? Yes. When you drop a partition in a 
CLOUD_NATIVE
setup: 1. StarRocks marks the data files AND the associated index files in Azure for deletion. 2. The 
lake_autovacuum
will eventually delete those index files from Azure (after your new 420-minute grace period). 3. The CN will stop trying to cache those specific index versions. Summary of your "Re-pipe" Plan Your plan is solid. By using 
CLOUD_NATIVE
, you've actually made the "re-pipe" cleaner because: * You don't have to worry about stale index files taking up local disk space. * The "Reset" is truly global (metadata in FE + files in Azure). * Increasing the 
lake_autovacuum_grace_period_minutes
to 420 will protect your Index Files just as much as your Data Files, which is critical for Primary Key tables. Rocky's Advice: Since your indices are now competing for space in that 1Ti cache, keep a close eye on the 
block_cache_hit_rate
using the API I mentioned earlier. If it stays low, you'll definitely want to increase that SSD size or add more CN nodes to increase the aggregate cache pool of cache space. References • best_practices/primarykey_table.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • table_design/table_types/primary_key_table.md • table_design/indexes/inverted_index.md
g
@Rocky and reason to set that timeout to something higher than 7 hours?
r
While 7 hours (420 minutes) is a massive improvement over the default 30 minutes, there are several scenarios where you might want to push it even higher (e.g., 24 hours / 1440 minutes). Here is why you would consider a longer window: 1. The "Cloud Native Index" Safety Net Because you are using 
persistent_index_type = "CLOUD_NATIVE"
, your Primary Key indices are stored as files in Azure. * If a CN node has a specific index version in its local cache and that file is deleted from Azure because the 7-hour window expired, the CN might fail when it tries to refresh or verify that index. * In high-load scenarios where the CN is struggling to keep up with compaction metadata, having a 24-hour buffer ensures that even if a node is "lagging" behind the FE's current version, the files it is looking for are guaranteed to exist. 2. Operational "Overnight" Protection If a massive load or a complex query fails at 2:00 AM due to a "Not Found" error, and your grace period is only 7 hours, the evidence (the files themselves) will be permanently deleted by 9:00 AM. * Setting it to 1440 minutes (24 hours) ensures that if something goes wrong overnight, you can still investigate the state of the storage the next morning. * It also protects against "zombie" load jobs that might be stuck or retrying for long periods. 3. Handling Long-Running Maintenance If you need to perform maintenance on your CN nodes (e.g., a rolling restart or resizing the disk) and it takes longer than 7 hours, you run the risk of the cluster "waking up" to a storage environment that has been aggressively cleaned. A 24-hour window gives you much more breathing room for infrastructure changes. 4. Storage vs. Stability Trade-off The only reason not to set it to 24 hours or even 48 hours is Azure Blob Storage cost. * The Math: If you ingest 100GB of data per hour and your compaction ratio is 1:1, a 7-hour window keeps ~700GB of "garbage" files. A 24-hour window keeps ~2.4TB. * The Decision: Given that Azure storage is very inexpensive ($0.015 - $0.02 per GB/month), the cost of storing an extra few TBs is usually negligible compared to the cost of production downtime or failed ETL pipelines. Rocky’s Recommendation If your Azure budget allows for a few extra TBs of temporary storage, I actually recommend 1440 (24 hours) for the first week of your "re-pipe" project.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "1440");
Once your system is stable and your Prometheus/Grafana dashboards show that your "Not Found" errors have disappeared, you can safely dial it back down to 420 or 180 minutes to save on storage costs. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • administration/management/compaction.md • best_practices/primarykey_table.md References • administration/management/FE_configuration.md • faq/shared_data_faq.md • best_practices/primarykey_table.md • administration/management/compaction.md
g
@Rocky nothing alarming yet
Copy code
# HELP starrocks_fe_publish_version_daemon_loop_total counter of publish version daemon loop runs
# TYPE starrocks_fe_publish_version_daemon_loop_total counter
starrocks_fe_publish_version_daemon_loop_total 1737904
# HELP starrocks_fe_txn_publish_latency_ms
# TYPE starrocks_fe_txn_publish_latency_ms summary
starrocks_fe_txn_publish_latency_ms{quantile="0.75", is_leader="true", type="all"} 260.0
starrocks_fe_txn_publish_latency_ms{quantile="0.95", is_leader="true", type="all"} 370.0
starrocks_fe_txn_publish_latency_ms{quantile="0.98", is_leader="true", type="all"} 478.0
starrocks_fe_txn_publish_latency_ms{quantile="0.99", is_leader="true", type="all"} 547.0
starrocks_fe_txn_publish_latency_ms{quantile="0.999", is_leader="true", type="all"} 3442.0
starrocks_fe_txn_publish_latency_ms_sum{is_leader="true", type="all"} 3.905821545986903E7
starrocks_fe_txn_publish_latency_ms_count{is_leader="true", type="all"} 182182
# HELP starrocks_fe_txn_publish_schedule_latency_ms
# TYPE starrocks_fe_txn_publish_schedule_latency_ms summary
starrocks_fe_txn_publish_schedule_latency_ms{quantile="0.75", is_leader="true", type="all"} 98.0
starrocks_fe_txn_publish_schedule_latency_ms{quantile="0.95", is_leader="true", type="all"} 170.0
starrocks_fe_txn_publish_schedule_latency_ms{quantile="0.98", is_leader="true", type="all"} 208.0
starrocks_fe_txn_publish_schedule_latency_ms{quantile="0.99", is_leader="true", type="all"} 239.0
starrocks_fe_txn_publish_schedule_latency_ms{quantile="0.999", is_leader="true", type="all"} 842.0
starrocks_fe_txn_publish_schedule_latency_ms_sum{is_leader="true", type="all"} 1.0840252426712029E7
starrocks_fe_txn_publish_schedule_latency_ms_count{is_leader="true", type="all"} 182174
# HELP starrocks_fe_txn_publish_execute_latency_ms
# TYPE starrocks_fe_txn_publish_execute_latency_ms summary
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.75", is_leader="true", type="all"} 167.0
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.95", is_leader="true", type="all"} 234.0
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.98", is_leader="true", type="all"} 286.0
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.99", is_leader="true", type="all"} 543.0
starrocks_fe_txn_publish_execute_latency_ms{quantile="0.999", is_leader="true", type="all"} 5284.0
starrocks_fe_txn_publish_execute_latency_ms_sum{is_leader="true", type="all"} 2.8686551461715374E7
starrocks_fe_txn_publish_execute_latency_ms_count{is_leader="true", type="all"} 182182
# HELP starrocks_fe_txn_publish_can_finish_latency_ms
# TYPE starrocks_fe_txn_publish_can_finish_latency_ms summary
starrocks_fe_txn_publish_can_finish_latency_ms{quantile="0.75", is_leader="true", type="all"} 0.0
starrocks_fe_txn_publish_can_finish_latency_ms{quantile="0.95", is_leader="true", type="all"} 0.0
starrocks_fe_txn_publish_can_finish_latency_ms{quantile="0.98", is_leader="true", type="all"} 0.0
starrocks_fe_txn_publish_can_finish_latency_ms{quantile="0.99", is_leader="true", type="all"} 0.0
starrocks_fe_txn_publish_can_finish_latency_ms{quantile="0.999", is_leader="true", type="all"} 0.0
starrocks_fe_txn_publish_can_finish_latency_ms_sum{is_leader="true", type="all"} 0.0
starrocks_fe_txn_publish_can_finish_latency_ms_count{is_leader="true", type="all"} 182182
# HELP starrocks_fe_txn_publish_ack_latency_ms
# TYPE starrocks_fe_txn_publish_ack_latency_ms summary
starrocks_fe_txn_publish_ack_latency_ms{quantile="0.75", is_leader="true", type="all"} 1.0
starrocks_fe_txn_publish_ack_latency_ms{quantile="0.95", is_leader="true", type="all"} 20.0
starrocks_fe_txn_publish_ack_latency_ms{quantile="0.98", is_leader="true", type="all"} 28.0
starrocks_fe_txn_publish_ack_latency_ms{quantile="0.99", is_leader="true", type="all"} 33.0
starrocks_fe_txn_publish_ack_latency_ms{quantile="0.999", is_leader="true", type="all"} 54.0
starrocks_fe_txn_publish_ack_latency_ms_sum{is_leader="true", type="all"} 589451.9656462707
starrocks_fe_txn_publish_ack_latency_ms_count{is_leader="true", type="all"} 182182
r
These metrics look very healthy and actually provide some good "peace of mind" regarding your current cluster state. Here is the breakdown of why these numbers are encouraging for your re-piping plan: 1. The "Plumbing" is Clear (
ack_latency
) * Metric: 
starrocks_fe_txn_publish_ack_latency_ms
(p99.9 = 54ms) * What it means: This is the time the FE spends waiting for the CN nodes to acknowledge that they’ve received the new version info. * Rocky’s Take: This is fantastic. It means your CN nodes are highly responsive. If there was a "jam" or if the CNs were overwhelmed by the 800GB of stale cache/metadata, you would see this number climb into the thousands of milliseconds. The "metadata highway" between your FE and CNs is wide open. 2. Excellent Baseline Performance * Metric: 
starrocks_fe_txn_publish_latency_ms
(p75 = 260ms) * What it means: In a Shared-Data (Lake) architecture, every "Publish" involves the FE writing metadata to the storage backend (StarOS/Metastore). * Rocky’s Take: Seeing 75% of your transactions finish the publish phase in under 300ms is a very strong baseline for a Lake cluster. It suggests your FE isn't struggling with its local BDB JE metadata store or its connection to the remote metastore. 3. Understanding the p99.9 Spikes (~3.4s - 5.2s) * Metric: 
starrocks_fe_txn_publish_execute_latency_ms
(p99.9 = 5284ms) * What it means: You are seeing occasional spikes where a publish takes about 5 seconds. * Why this happens: In StarRocks, the "Execute" phase of a publish is where the FE holds a metadata lock to update the versioning for the tablets. * Rocky’s Take: These spikes are common during periods of high concurrency (many small loads hitting the same table) or when the FE performs a Java Garbage Collection (GC) pause. As long as your p95 (370ms) and p98 (478ms) stay sub-second, these p99.9 spikes are just "background noise" and won't prevent your re-pipe from succeeding. 4. The "Ready to Finish" Check * Metric: 
starrocks_fe_txn_publish_can_finish_latency_ms
(0.0 across all quantiles) * What it means: This measures if transactions are getting stuck in a "waiting to be ready" state. * Rocky’s Take: A consistent 0.0 is perfect. It means as soon as the work is done, the transaction is immediately eligible to be marked visible. Verdict for your Re-pipe: Proceed with confidence. Your FE is currently managing transactions efficiently, and your CNs are talking back to the FE almost instantly. Dropping the partitions and re-piping should be a smooth operation from a transaction/to a metadata perspective. The metrics suggest there is no "metadata debt" or backlog that would cause the new pipe to fail. References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • faq/loading_faq.md • administration/management/BE_configuration.md References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • sql-reference/information_schema/be_txns.md
g
@Rocky all the pipes are running and we are generally good for a while here then at some point we start seeing that error
r
The "generally good for a while, then suddenly errors" pattern is a classic symptom of Garbage Collection (GC) racing against your metadata in a Shared-Data (Lake) architecture. Since you are using 
persistent_index_type = "CLOUD_NATIVE"
, your Primary Key indices are stored as SST files in Azure Blob Storage. The "Not Found" error occurs when a CN node tries to load a specific version of an index file that the FE thinks should exist, but the file has already been deleted or "vacuumed" from Azure. Here are the most likely reasons why this starts happening after a few hours: 1. The "5-Hour Hardcoded GC" (Known Issue) In several versions of StarRocks (notably around v3.1.x and v3.2.x), there was a background task in shared-data mode that specifically cleaned up Primary Key index directories every 5 hours (#40745). * The Problem: This task sometimes ignored the 
lake_autovacuum_grace_period_minutes
setting. * The Result: Even if you set your grace period to 7 hours, this "hardcoded" cleaner might have been nuking index files after 5 hours, leading exactly to the "Not Found" errors you observed after a period of stability. 2. Compaction "Version Jump" When your pipes are running, StarRocks is constantly doing Cumulative Compaction. 1. Load 1 creates Version 10 of the index. 2. Load 2 creates Version 11. 3. Compaction merges them into Version 12. 4. If a CN node is still finishing an "Apply" task or a query that started when Version 10 was current, but the 
lake_autovacuum
or the internal PK GC deletes the Version 10 SST files from Azure, the CN will fail with "Not Found." 3. "Cloud Native" Metadata Drift In a 
CLOUD_NATIVE
setup, the index metadata is technically decoupled. If the FE loses track of which index version is tied to which rowset (due to a rare metadata race), it might point a CN to a non-existent file path in Azure. How to verify this is happening Next time the error occurs, check the BE/CN logs (
<http://be.INFO|be.INFO>
or 
be.WARNING
) for a stack trace like this:
Copy code
text
W0508 ... lake_primary_index.cpp:215] Fail to load primary index: Not found: .../index.l1.6440.0.sst: No such file or directory
If you see a specific 
.sst
file path that is missing, you can check your Azure Blob Storage. If the file is truly gone, it confirms that the Vacuum/GC mechanism is too aggressive. Recommended Action Plan 1. Version Check: Ensure you are on a stable release (v3.2.10+ or v3.3.x). Many of these "PK Index GC" bugs were fixed in recent maintenance releases. 2. The "Repair" Command: If you see "Not Found" errors but don't want to drop the whole partition, you can try the cloud-native repair tool (available in newer versions):
Copy code
sql
    ADMIN REPAIR TABLE your_table_name;
This coordinates the CNs to probe historical metadata and try to reconstruct the index mapping. 3. Monitor the "Vacuum" Metrics: Watch the 
lake_vacuum_remove_file_total
metric. If you see a massive spike in file deletions exactly 5 or 7 hours after you started your pipes, you’ve found the trigger. Rocky's Summary: Your "re-pipe" (dropping partitions) is the right move to clear any current metadata corruption, but if the errors return after ~5-7 hours, it's almost certainly the background GC task being too "helpful." Upgrading to the latest maintenance release of your major version is the best permanent fix for those internal races. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • best_practices/primarykey_table.md • administration/management/compaction.md • deployment/shared_data/feature-support-shared-data.md
g
@Rocky where is this metric 
lake_vacuum_remove_file_total
r
The metric 
lake_vacuum_remove_file_total
 is an FE (Frontend) metric. In the Shared-Data (Lake) architecture, the FE acts as the "brain" for data lifecycle management. While the CN nodes do the heavy lifting for queries and loads, the FE is responsible for the Vacuum process—periodically scanning the remote storage (Azure Blob, S3, etc.) and deleting files that are no longer referenced by any metadata version. You can find this metric in two primary places: 1. The FE Metrics HTTP Endpoint (Prometheus Format) The most common way to see it is by hitting the FE's monitoring port (default 
8030
). * URL: 
http://<FE_IP>:8030/metrics
* Search for: 
starrocks_fe_lake_vacuum_remove_file_total
2. Using SQL via 
information_schema
 If you don't have direct browser/curl access to the FE nodes, you can query it directly through your SQL client:
Copy code
sql
SELECT * 
FROM information_schema.fe_metrics 
WHERE name LIKE '%lake_vacuum_remove_file_total%';
────────── What to look for in this metric: * The Delta (Rate): Because this is a counter, the absolute number isn't as important as the rate of change. * The "Spike" Correlation: If you see this metric jump significantly (thousands of files deleted) at the exact same time your pipes start throwing "Not Found" errors, it is the smoking gun. It means the FE's vacuum thread is identifying those index files as "garbage" and deleting them from Azure while your CN nodes are still trying to use them. * Vacuum Failures: You may also want to check 
starrocks_fe_lake_vacuum_fail_total
. If the vacuum is failing, it might lead to a backlog, followed by a "burst" of deletions once it succeeds, which often causes more stability issues than a steady stream of deletions. Why is it in the FE? In the shared-data model, the FE maintains the Tablet Meta (the list of which SST/Segment files belong to which version). Only the FE knows for sure when a file is "safe" to delete across the entire cluster. When the 
lake_autovacuum_grace_period_minutes
expires, the FE sends the delete commands to the object store and increments this counter. References • administration/management/monitoring/metrics.md • sql-reference/information_schema/fe_metrics.md • administration/management/FE_configuration.md • deployment/shared_data/feature-support-shared-data.md References • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md • sql-reference/information_schema/fe_metrics.md • administration/management/BE_configuration.md
g
@Rocky i don't see those specific metrics anywhere
r
If you aren't seeing 
lake_vacuum_remove_file_total
in your 
/metrics
or 
information_schema.fe_metrics
, it is likely because the naming convention in your specific StarRocks version (likely v3.1 or v3.2+) uses slightly different internal labels or the metrics are only exposed on the Leader FE. In a Shared-Data (Lake) cluster, look for these exact Prometheus names instead: 1. The Correct Metric Names (Prometheus) Search your FE metrics for the 
lake_vacuum
 prefix. The standard names are: * 
starrocks_fe_lake_vacuum_delete_total
: This is the counter for how many "Vacuum Delete" tasks have been initiated. * 
starrocks_fe_lake_vacuum_delete_failed_total
: This counts how many of those delete tasks failed (e.g., Azure Blob permission issues or timeouts). * 
starrocks_fe_lake_vacuum_delete_latency_ms
: This shows the summary/quantile latency of the vacuum process. 2. Why you might not see them: * Follower FE vs. Leader FE: Vacuum is a background maintenance task managed only by the Leader FE. If you are querying the 
/metrics
endpoint of a Follower FE node, these specific "Lake" metrics will often show as 0 or be absent entirely. Ensure you are hitting the Leader FE. * StarOS/Starlet Layer: In some deployments, the low-level file deletion is handled by a sub-service called Starlet. You might see these under: * 
starrocks_fe_starlet_vacuum_delete_total
* 
fslib_create_files
/ 
fslib_open_files
(found in the "Starlet Dashboard") 3. Check the "Vacuum Task" status directly via SQL If the metrics are being elusive, you can bypass them and see the actual background job status using this "hidden" procedure in the FE:
Copy code
sql
SHOW PROC '/vacuum';
(Note: This command gives you a real-time view of the vacuum queue size and the last time a vacuum successfully completed. If the queue size is huge, it means deletions are falling behind.) Summary for your "Not Found" Investigation: If you are trying to correlate the "Not Found" errors with the vacuum process, watch for any metric starting with 
starrocks_fe_lake_vacuum
on your Leader FE. If you see 
starrocks_fe_lake_vacuum_delete_total
increasing rapidly at the same time your pipes fail, it confirms that the FE is cleaning up files that the CN still needs. In that case, you should increase your grace period:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "720"); -- Increase to 12 hours
References • administration/management/monitoring/metrics-shared-data.md • administration/management/FE_configuration.md • faq/shared_data_faq.md • administration/management/monitoring/metrics.md References • administration/management/monitoring/metrics-shared-data.md • administration/management/FE_configuration.md • sql-reference/information_schema/fe_metrics.md • deployment/shared_data/feature-support-shared-data.md References • administration/management/monitoring/metrics-shared-data.md • faq/shared_data_faq.md • deployment/shared_data/feature-support-shared-data.md
g
@Rocky not seeing much in the logs, how about
Copy code
# HELP starrocks_fe_vacuum_files_count total files have been vacuumed
# TYPE starrocks_fe_vacuum_files_count counter
starrocks_fe_vacuum_files_count 692696
# HELP starrocks_fe_edit_log_write counter of edit log write into bdbje
# TYPE starrocks_fe_edit_log_write counter
starrocks_fe_edit_log_write 693214
r
That metric is the smoking gun. 
starrocks_fe_vacuum_files_count
is the FE's counter for how many physical files (data segments and Primary Key index SSTs) it has successfully deleted from your Azure storage. Seeing nearly 700,000 files vacuumed is a very high number. It confirms that your cluster is extremely active in "cleaning up" after your pipes. Here is what that tells us: 1. The Correlation If you check this metric again and see it incrementing rapidly right when your "Not Found" errors start, it confirms the FE is deleting files that your CN nodes are still trying to access. * In Shared-Data mode, when a pipe runs, it creates new files. * Compaction then merges these into newer, larger files. * The FE eventually "vacuums" the old, unneeded files. * The Error: If the vacuum happens too fast, a CN node looking for an older "version" of an index SST file will find it gone from Azure. 2. Why the 
edit_log_write
is there The 
starrocks_fe_edit_log_write
metric tracks internal metadata changes (BDBJE). Every time a vacuum happens or a load finishes, the FE writes an edit log. Seeing these numbers move together is normal; it just shows the FE is busy updating the system state. It isn't the cause of your errors, just a symptom of high activity. 3. Immediate Fix: Increase the "Grace Period" You need to tell the FE to be less aggressive. By default, it might be cleaning up files too quickly. You should increase the 
lake_autovacuum_grace_period_minutes
. This setting tells the FE: "Even if a file is no longer needed, wait X minutes before actually deleting it from Azure." Run this on your Leader FE:
Copy code
sql
-- Increase grace period to 12 hours (720 minutes) 
-- to give your CNs plenty of time to finish tasks
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "720");
4. Why did it work "for a while"? The reason it was "generally good for a while" is that the vacuum process usually has a cold-start delay or waits for the 
grace_period
to expire for the first time. Once that initial timer hits, it starts a cycle of deletions. If your ingestion rate is high, it creates a "rolling window" of deletions that eventually catches up to your active queries/loads. Recommendation: 1. Increase the 
lake_autovacuum_grace_period_minutes
. 2. Monitor 
starrocks_fe_vacuum_files_count
over the next few hours. You want to see it moving in smaller, steadier increments rather than huge bursts that coincide with your errors. 3. If you still see the error, check the BE/CN logs specifically for which file it couldn't find (e.g., 
.../index/12345/index.l0.6.sst
). If that file path matches the storage being vacuumed, the grace period is your primary lever. References • administration/management/monitoring/metrics.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky I still can't find any evidence of anything taking longer than the 30 minute graceperiod I had before so I'm not optimistic 12 hours is actually gonna help
r
I understand the skepticism. If your ingestion pipes and queries all complete in under 5 minutes, a 30-minute grace period should theoretically be plenty. However, in a Shared-Data architecture, the "Not Found" error often isn't caused by a single long-running query. It is usually caused by Version Churn and the Metadata-to-Data lag. Here is why 30 minutes frequently fails in high-concurrency environments (like those using Pipes), even if tasks are "fast": 1. The "Compaction Chain" Problem When using Pipes, you are likely creating hundreds or thousands of small versions per hour. * Version 10 is created. * Version 11 (Compaction) merges Version 10 into a larger segment. * Version 12 (Compaction) merges Version 11 into an even larger one. The FE sees Version 10 as "dead" as soon as Version 11 is published. The 30-minute timer for Version 10's physical files starts then. If a Pipe or a background task (like a consistency check or a secondary compaction) is still resolving a dependency that originated in Version 10—even if the current query is on Version 50—it will trigger a "Not Found" error if the FE reaches that 30-minute mark first. 2. CN Metadata Cache Lag CN nodes (Compute Nodes) cache the tablet metadata to avoid hitting the FE for every single read. 1. FE decides to vacuum a file because it’s 31 minutes old. 2. FE sends the delete command to Azure. 3. A CN node, still holding a slightly stale metadata entry in its cache or mid-execution of a task that started moments ago, tries to open that file. 4. Result: Azure returns 
404 Not Found
, and the CN throws an error. Increasing the grace period to 12 hours effectively "widens the safety margin" so that even the stalest cache or the most complex compaction chain has cleared before the physical file is deleted. 3. The "Burst" Vacuum Effect The metric you found (
starrocks_fe_vacuum_files_count
at 692,696) shows a massive volume of deletions. If the Vacuum thread gets backed up (check 
SHOW PROC '/vacuum'
), it doesn't delete files one-by-one as they expire. It might delete 50,000 files in a single 10-second burst. During that burst, the pressure on the object storage API and the internal metadata locks can cause a "race condition" where a file is deleted just milliseconds before a CN node tries to read it. How to prove it's the Vacuum (without waiting 12 hours): If you want hard evidence before committing to the 12-hour change, look at these two things simultaneously: 1. Monitor the Counter Delta:
Copy code
sql
    -- Run this every 10 seconds during your peak "Not Found" period
    SELECT * FROM information_schema.fe_metrics WHERE name = 'starrocks_fe_vacuum_files_count';
2. Match the Timestamp: If your Pipe fails at 
14:05:22
and your 
fe_vacuum_files_count
jumped by several thousand at 
14:05:21
, the vacuum is 100% the cause. Recommendation The reason the default was recently increased from 5 to 30 minutes in newer StarRocks versions is exactly because 30 minutes was found to be too aggressive for production "Pipe" workloads. Try setting it to 120 (2 hours) as a middle ground. If the errors stop, you've found your culprit. There is no performance penalty to a longer grace period; it only uses a bit more storage space on Azure for a few extra hours. References • faq/shared_data_faq.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md References • faq/shared_data_faq.md
g
@Rocky is this a good metric for this case? 
starrocks_fe_txn_total_latency_ms{quantile="0.999", is_leader="true", type="all"} 87438.0
r
This metric is highly relevant and, quite frankly, alarming. A value of 87,438 ms (~87.4 seconds) for the 99.9th percentile of 
txn_total_latency_ms
means that while most transactions might be fast, a subset of your ingestion/metadata operations are stalling for nearly a minute and a half. In a high-frequency ingestion environment (like one using StarRocks Pipes), this is a "red alert" for the following reasons: 1. What this metric tells us This metric measures the time from when a transaction is "Prepared" to when it is "Visible" (Finished). In a healthy cluster, this should be under 1–2 seconds. * 87 seconds indicates a massive bottleneck in the FE Transaction Publish Daemon. * This usually happens because the FE is struggling to update the metadata in its internal BDBJE store or because it is waiting on the StarOS/Lake layer to acknowledge file changes. 2. How this causes "Not Found" errors The "Not Found" error is a race condition. When your transaction latency is that high, the "state" of the cluster becomes inconsistent: * Version Drift: The Leader FE might have logically "committed" a new version, but because the Publish phase is stalling (taking 87 seconds), the CN nodes are still operating on an older metadata version. * The Vacuum Race: The Vacuum process (which deletes files) runs independently. If the FE's internal metadata state thinks a file is "obsolete" because a new version was committed, but the publish (which makes it safe for CNs to move to the new version) is stuck in that 87-second lag, the Vacuum might delete the file while a CN is still trying to finish a query or a sub-task on the "old" version. 3. Why "30 minutes" isn't enough in this state You mentioned you haven't seen anything take longer than 30 minutes. However, when the FE experiences 87-second stalls in transaction processing, it's often a symptom of FE Lock Contention. When the FE is "locked up" processing a heavy transaction or a massive batch of metadata, other threads (like the one that manages the vacuum grace period) might behave unpredictably. If the "clock" the vacuum thread uses and the "clock" the transaction thread uses get out of sync due to lock wait times, the 30-minute grace period can "shrink" in real-time. 4. What to check next To confirm this, look for these companion metrics: * 
starrocks_fe_txn_publish_latency_ms
: If this is where the 87s is coming from, your FE is overwhelmed by the number of "Publish" tasks. * 
starrocks_fe_slow_lock_wait_time_ms
: If this is high, it confirms that the FE is locking up, which explains why metadata and physical files are getting out of sync. * 
starrocks_fe_tablet_report_queue_size
: If this is large, the FE is being flooded with metadata updates from the CNs. Recommendation Even if you don't think you need 12 hours, increasing the grace period is your "safety net" against these 87-second stalls. It buys the system enough time to resolve these metadata lags without deleting the underlying data. If you want to solve the 87s latency itself, you should: 1. Reduce Ingestion Frequency: If you are using Pipes, try to ensure they aren't committing every few seconds. 2. Check FE Health: Ensure the Leader FE has enough CPU and fast disk (SSD) for its 
meta/
directory, as BDBJE writes are often the bottleneck for 
txn_latency
. References • administration/management/monitoring/metrics.md • faq/shared_data_faq.md • administration/management/monitoring/metrics-shared-data.md • administration/management/FE_configuration.md References • faq/shared_data_faq.md
g
@Rocky it seems like maybe 12 hours wasn't even enough? I'm getting the same error on a partition that was created sometime yesterday. Can I see when the last operation was to a partition?
r
If 12 hours isn't stopping the "Not Found" errors, we need to determine if the files being deleted are actually "stale" or if there is a more aggressive cleanup happening. To answer your specific question: Yes, you can see exactly when a partition was last updated. 1. Check the Last Update Time You can query the 
information_schema.partitions_meta
table. This is the most reliable way to see the metadata state for a partition.
Copy code
sql
SELECT 
    DB_NAME, 
    TABLE_NAME, 
    PARTITION_NAME, 
    VISIBLE_VERSION, 
    VISIBLE_VERSION_TIME, -- This is the timestamp of the last successful write/commit
    STORAGE_PATH
FROM information_schema.partitions_meta
WHERE TABLE_NAME = 'your_table_name'
  AND PARTITION_NAME = 'your_partition_name';
* 
VISIBLE_VERSION_TIME
: If this time is more than 12 hours ago, the partition is considered "stale" by the vacuum daemon. * Check Inactivity: StarRocks has a setting 
lake_autovacuum_stale_partition_threshold
(default 12 hours). If a partition hasn't been updated in 12 hours, the vacuum thread stops scanning it to save resources. If your files are being deleted after this window, it's very strange. ────────── 2. Why 12 hours might still fail If 
VISIBLE_VERSION_TIME
shows the partition was updated "yesterday" (e.g., 20 hours ago), and your grace period is 12 hours, then any files created before that 20-hour mark that were superseded by compaction are already eligible for deletion. However, if your queries are "fast" (not taking 12 hours), the only reason they would fail is if they are trying to access a version of the data that no longer exists. This usually happens for two reasons: A. Stale CN Metadata Cache If a Compute Node (CN) has cached the metadata for 
Version 100
, but the FE has already moved to 
Version 200
and vacuumed the files for 
Version 100
(because they were 12+ hours old), the CN might still try to use its cached 
Version 100
plan. * Fix: You can try to clear the cache on the CNs or check if the CNs are having trouble communicating with the FE to get the latest version. B. Primary Key Index Issues (The SST files) If this is a Primary Key table, the "Not Found" error often refers to the 
.sst
files in the 
index/
directory, not the 
.dat
files in the 
data/
directory. * In older versions of StarRocks (pre-3.1.9), there was a known issue where the Persistent Index directory was cleaned up too aggressively (every 5 hours) regardless of the grace period. * Check: Are you on a version older than 3.1.10 or 3.2.4? If so, you are likely hitting a bug where the index files are deleted early. C. Multiple Clusters / "Ghost" Vacuum Are there any other StarRocks clusters (even a dev/test cluster) configured to point to the same Azure storage path? * If two independent FE groups point to the same Azure container/path, Cluster A's vacuum will see Cluster B's data as "unreferenced" and delete it immediately. This is a common cause of "Not Found" errors that ignore grace periods. Next Steps to Debug: 1. Run the SQL query above and compare 
VISIBLE_VERSION_TIME
to the time of your error. 2. Check the exact error message in the CN/BE logs. Does it fail on a 
.dat
file or an 
.sst
file? * 
.../data/123/456.dat
-> Data file (standard vacuum). * 
.../index/123/index.l0.sst
-> PK Index file (special handling). 3. Check the Vacuum Status:
Copy code
sql
    SHOW PROC '/vacuum';
This will show you which partitions are currently being vacuumed and if there are errors in the vacuum process itself. Are you using a Primary Key table? And which version of StarRocks are you running? This will help narrow down if you're hitting one of the specific bugs related to Azure index deletion.** References • administration/management/FE_configuration.md
g
@Rocky 1. 
SHOW PROC '/vacuum';
isn't a valid path 2. i'm on version 4.0.9 3. yes this is a primary key table
r
This is a classic "perfect storm" in the Shared-Data (Lake) architecture, especially for Primary Key tables on Azure. Since you are on a very recent version (4.0.9), there are specific mechanics at play that explain why a 12-hour grace period might still feel "too short." Here is the technical breakdown of what is likely happening and how to prove it: 1. The "Stale Partition" Conflict There are two distinct timers in the FE that control data cleanup, and they are likely colliding: * 
lake_autovacuum_grace_period_minutes
: (Your 12 hours) How long to keep a file after it becomes "obsolete" (superseded by a new version/compaction). * 
lake_autovacuum_stale_partition_threshold
: (Default 12 hours) If a partition has no updates for this long, the FE stops scanning it for vacuuming to save resources. The Race: If your partition was updated "yesterday" (e.g., 13 hours ago), it just crossed the stale threshold. If there was a pending deletion that got stuck or delayed due to your 87-second transaction latency, the system might have finally executed a "cleanup" right as the partition was being marked stale, or the CN node is holding a version that the FE just finalized for deletion. Action: Increase the stale threshold to stay ahead of your grace period:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_stale_partition_threshold" = "48"); -- 48 hours
2. It’s likely the Index, not the Data On a Primary Key table in Lake mode, the "Not Found" error usually refers to Persistent Index SST files, not the actual data segments. * StarRocks 4.0.9 uses 
CLOUD_NATIVE
persistent indexes by default. These are small metadata files stored on Azure. * If the FE is under heavy pressure (that 87s latency), it can lose track of which Index SSTs are still needed by "slow" CN nodes. How to verify: Since you are on 4.0.9, you have access to a specific new metric. Run this:
Copy code
sql
SELECT * FROM information_schema.fe_metrics WHERE name LIKE '%pk_index_sst_read_error%';
If this counter is incrementing, the "Not Found" error is definitively the Persistent Index being vacuumed too early. 3. Explaining the 87s Transaction Latency That latency is the root cause of the "drift." If it takes 87 seconds to publish a version: 1. FE thinks Version 100 is done and marks Version 99's files for deletion. 2. CN is still struggling to "see" Version 100 because the publish is slow. 3. Vacuum (running on the FE) sees that 12 hours have passed since Version 99 was superseded and deletes the files. 4. CN finally tries to query, but it’s still looking for Version 99 files that were just deleted. Why is it 87 seconds? * Azure Throttling: Check your Azure Storage "Ingress/Egress" and "Transactions" metrics. If you hit the IOPS limit, StarRocks will stall. * FE Disk: The FE's 
meta/
directory (BDBJE) must be on high-performance SSDs. If the disk is slow, the "Commit" phase of transactions will balloon. 4. Why 
SHOW PROC '/vacuum';
failed In the 3.x and 4.x Lake architecture, the vacuum process was moved from a manual process to an automated background service. _ To see what’s happening now, use: 
SHOW PROC '/compactions';
(This shows the _merging* that creates the "obsolete" files). _ To see the _cleanup* status, you must check the FE logs (
fe.log
) for lines containing 
Vacuum
or 
Garbage Collection
. Final Recommendations: 1. Widening the Window: Set 
lake_autovacuum_grace_period_minutes
to 1440 (24 hours) and 
lake_autovacuum_stale_partition_threshold
to 48 (hours). 2. Check for "Ghost" Clusters: Double-check that no other StarRocks cluster (dev/test/staging) is pointed at the same Azure container/path. If a second FE is running, it will "helpfully" vacuum the first cluster's data. 3. Metadata Cache: To stop the CN nodes from being "stubborn" about old files, you can try reducing the metadata cache TTL on the CNs (BEs), but this will increase FE load.
Summary SQL to run now:
Copy code
sql
-- Check last update time (as requested)
SELECT TABLE_NAME, PARTITION_NAME, VISIBLE_VERSION_TIME 
FROM information_schema.partitions_meta 
WHERE DB_NAME = 'your_db';

-- Check for Index failures
SELECT * FROM information_schema.fe_metrics 
WHERE name IN ('starrocks_fe_lake_pk_index_write_fail_total', 'starrocks_fe_lake_pk_index_read_fail_total');
References • release_notes/release-4.0.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md • faq/shared_data_faq.md • best_practices/primarykey_table.md References • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md • faq/shared_data_faq.md • administration/management/compaction.md
g
@Rocky
Copy code
SELECT * FROM information_schema.fe_metrics 
WHERE name IN ('starrocks_fe_lake_pk_index_write_fail_total', 'starrocks_fe_lake_pk_index_read_fail_total');
doesn't return anything, but just looking for like '%failed%' i see a value of 391 for txn_fail
r
The fact that you have 391 failed transactions (
txn_fail
) combined with 87-second latency and "Not Found" errors confirms that your FE is currently in a "distressed" state. Here is what those 391 failures and the "Not Found" errors on 24-hour-old data likely mean in the context of StarRocks 4.0.9 on Azure: 1. The Root Cause: "Zombie" Transactions With an 87-second publish latency, many of your ingestion jobs are likely hitting the 
thrift_rpc_timeout_ms
 or the load timeout. * When a transaction "fails" (aborts) after the FE has already started writing files to Azure, the FE must clean up those "orphan" files. * However, if the FE is stalling for 87 seconds per transaction, the Cleanup Thread gets backed up. _ In Primary Key tables, if an index update was partially written but the transaction aborted, a subsequent query might try to reference a version of the index that the FE _thinks* exists but was actually rolled back or deleted. 2. Why "12 Hours" isn't stopping the "Not Found" Since you are on 4.0.9, you should check for a specific metric that was added to help debug this. Search for:
Copy code
sql
SELECT * FROM information_schema.fe_metrics WHERE name LIKE '%segment_not_found%';
* If this is increasing, it means the Data Files are missing. * If it's not increasing, but you still get the error, it is almost certainly the Persistent Index (SST files). The "Stale Partition" Trap: You have a partition created "yesterday" (likely > 12 hours ago). * Default Behavior: StarRocks has a parameter 
lake_autovacuum_stale_partition_threshold
which defaults to 12 hours. * If a partition hasn't been updated for 12 hours, the FE stops scanning it for vacuuming. * The Problem: If there was a "pending" deletion for that partition that got stuck in your 87-second latency queue, and then the partition hit the 12-hour "stale" mark, the state can become "frozen" in an inconsistent way. 3. Diagnosing the 391 Failures You need to see why they failed. Run this to see the last few failed transactions:
Copy code
sql
SHOW PROC '/transactions';
-- Then find your DB ID and run:
SHOW PROC '/transactions/<db_id>/finished';
Look at the Reason column. You will likely see one of these: * 
TIMEOUT
: Your 87s latency is exceeding the internal 30s/60s RPC timeouts. * 
TOO_MANY_VERSIONS
: Compaction is failing to keep up because the FE is too slow to coordinate it. * 
PUBLISH_VERSION_FAILED
: The FE couldn't confirm the data was visible on Azure. 4. Immediate Actions To stabilize the cluster while you investigate the 87s latency: 1. Increase the Stale Threshold: This prevents the FE from "giving up" on managing the metadata of older partitions.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_stale_partition_threshold" = "72"); -- 72 hours
2. Check FE Metadata Disk: An 87s transaction latency almost always points to the FE's internal metadata store (BDBJE). * Is your FE 
meta/
directory on a standard HDD or a slow Azure Managed Disk? It must be on a high-performance SSD (Premium SSD or Ultra Disk). * Check 
starrocks_fe_bdb_write_latency
(if available) or look at your OS disk I/O wait times. 3. Azure Throttling: Check your Azure Storage account metrics for "Throttling Errors". If StarRocks is hitting Azure's IOPS limits, the FE will stall, transactions will time out (the 391 failures), and files might not be fully committed before a query tries to find them. Summary: Your "Not Found" errors are a symptom of the high transaction latency causing the FE to lose sync with the physical files on Azure. Solve the 87s latency (likely Disk I/O on the FE or Azure throttling), and the "Not Found" errors will disappear. References • administration/management/FE_configuration.md
g
@Rocky I'm seeing this error message on some transactions but they still seem to have finished 
wait for publishing partition 6771949 version 14774. self version: 14773. table 6771947
r
The message you are seeing is a smoking gun for why your transactions are taking 87 seconds and why you are seeing "Not Found" errors. In StarRocks Shared-Data (Lake) mode, a transaction isn't truly "done" until its data is Visible. The error you seeing happens during the Publish Version phase. 1. What that error message means The log 
wait for publishing partition 6771949 version 14774. self version: 14773
means that your current transaction (Version 14773) is stuck in a queue. * Sequential Consistency: StarRocks requires versions to be published in strict numerical order per partition. If Version 14772 is taking a long time to write its metadata to Azure, Version 14773 must wait. * The Version Jump: In your specific log, it says it's waiting for 14774 while the current is 14773. This usually indicates a concurrent transaction backlog. If a higher version (14774) has already "claimed" its place in the commit graph, the FE is trying to coordinate a Batch Publish where both 14773 and 14774 are finalized together. * Why they "Finished": Your ingestion client (like Stream Load or Flink) returns "Success" once the transaction is 
COMMITTED
. However, the data is not actually queryable until it moves from 
COMMITTED
to 
VISIBLE
. The 87 seconds you see is the time the transaction spent sitting in this "Published" queue. 2. How this causes "Not Found" errors When you have an 87-second gap between a transaction being "Committed" and becoming "Visible," it creates a Metadata Drift: 1. FE State: The FE thinks the table is at Version 14772 because 14773 and 14774 are still "waiting." 2. Vacuum Daemon: The Vacuum thread sees files belonging to Version 14770. Since your partition was updated "yesterday" (long ago), the vacuum daemon decides Version 14770 is obsolete and deletes the files. 3. CN/BE Query: A query starts. It tries to read Version 14772 (the last visible one), but the files it needs were just deleted because the FE's cleanup logic is moving faster than the stalled Publish queue. 3. Why the latency is so high (87s) On Version 4.0.9 and Azure, this is almost always caused by one of two things: * Azure Storage Throttling: Azure Blob Storage has strict limits on IOPS and Transactions per second per storage account. If you are doing frequent small writes (common with Primary Key tables), Azure will "throttle" StarRocks. Each write that should take 10ms starts taking 1000ms. * FE Metadata Pressure: In 4.0.9, the FE manages the "Persistent Index" for Primary Key tables. If your FE's local 
meta/
directory is on a slow disk, it cannot keep up with the high-frequency version updates, leading to the 391 
txn_fail
counts you saw earlier. 4. Recommended Fixes A. Enable/Tune Batch Publishing (The most likely fix) Since you are on 4.0.9, you should ensure the FE is grouping these stalled transactions into single Azure writes:
Copy code
sql
-- Already true by default, but check if it was disabled
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");

-- Increase the batch size to clear the 87s backlog faster
ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "20");
B. Increase Publish Parallelism If your FE has the CPU headroom, allow more concurrent threads to talk to Azure:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_publish_version_max_threads" = "1024");
C. Check Azure Metrics Go to your Azure Storage Account -> Monitoring -> Metrics. * Filter by "Api name" and look for 
PutBlob
or 
GetBlob
. * Look at "Success" vs "Throttling Error". If you see any throttling, you need to increase your Azure Storage throughput tier or use multiple storage accounts. D. Check FE Disk Latency Verify the FE node isn't the bottleneck. Check the 
fe.log
for "BDBJE write latency" or "slow execution" warnings. If the FE disk is slow, the entire publish queue will back up.
Summary: Your "Not Found" error is not a vacuum problem—it's a traffic jam in your transaction publishing. The 87-second delay is long enough that the cleanup process is accidentally deleting data that the stalled transactions still need. Solve the publish latency, 87s latency by tuning the batch settings and checking Azure throttling, and the errors will stop. References • administration/management/FE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md
g
@Rocky I want to investigate that error, how can i find out what table a tableid corresponds to?
r
To map those IDs back to actual names, you can use the 
information_schema
views. This is much faster than digging through logs or PROC paths. 1. Find the Table Name by 
tableid
 Run this query to identify table `6771947`:
Copy code
sql
SELECT TABLE_SCHEMA, TABLE_NAME 
FROM information_schema.tables_config 
WHERE TABLE_ID = 6771947;
2. Find the Partition Name by 
partitionid
 Since your error also mentions partition 
6771949
, you can find exactly which "slice" of data is lagging:
Copy code
sql
SELECT DB_NAME, TABLE_NAME, PARTITION_NAME 
FROM information_schema.partitions_meta 
WHERE PARTITION_ID = 6771949;
3. Alternative: Using 
SHOW PROC
 If you prefer the administrative console approach: 1. Find the Database:
Copy code
sql
    SHOW PROC '/dbs'; -- Look for the DbId that matches your environment
2. List Tables in that DB:
Copy code
sql
    SHOW PROC '/dbs/<db_id>'; -- Replace with the ID from the step above
This will output a list where the first column is 
TableId
and the second is 
TableName
. What this tells you about the error The error 
wait for publishing partition 6771949 version 14774. self version: 14773
confirms that: * Sequential Bottleneck: Your ingestion is trying to commit Version 14773, but it’s literally stuck waiting for the previous version (14774 is likely a typo in the log or refers to a batch slot) to finish writing its metadata to Azure. * The "Not Found" Connection: While this specific table/partition is "stuck" in the publish queue for 87 seconds, the Vacuum cleaner (which is on a separate timer) might be looking at the metadata on Azure and seeing files that it thinks are no longer needed because the "new" version hasn't officially landed yet. Once you identify the table, check its write frequency. If you are doing hundreds of small "Stream Loads" or small "Flink" commits per minute to this specific table, you are likely saturating the FE's ability to coordinate with Azure Storage. Increasing the Batch Publish settings I mentioned in the previous response is the direct fix for this. References • sql-reference/information_schema/tables_config.md • sql-reference/information_schema/partitions_meta.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • deployment/shared_data/azure.md References • sql-reference/information_schema/partitions.md • sql-reference/information_schema/partitions_meta.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md
g
@Rocky is there anything about this DDL snippet that could be causing these issues?
Copy code
INDEX raw_import_timestamp_idx (raw_import_timestamp) USING BITMAP COMMENT "Index by import_timestamp."
    )
ENGINE=OLAP
PRIMARY KEY(location, meta_row_id, event_date)
COMMENT "Car Tracker RAW DATA"
PARTITION BY (event_date)
DISTRIBUTED BY HASH (meta_row_id) -- BUCKETS 30 -- Leave at default since default is for up to 10GB per day
PROPERTIES (
    "compression" = "LZ4",              -- Default compression level for the main table
    "enable_persistent_index" = "true", -- Allow the BE/CN to store entire indexes local in RAM
    "persistent_index_type" = "CLOUD_NATIVE",   -- Specify the index persistence type based on storage location
    "datacache.enable" = "false",   -- RAW=false for stagnant data, C=true for active tables=true. NOT valid for local. Allow the BE/CN to cache data
    "enable_persistent_index" = "true",
    "fast_schema_evolution" = "true"        -- default = true, allows FE to handle schema updates faster
    );
r
The DDL you shared contains a "smoking gun" that perfectly explains both your 87-second transaction latency and your "Not Found" errors. The critical issue is the combination of these two properties: 1. 
"persistent_index_type" = "CLOUD_NATIVE"
 2. 
"datacache.enable" = "false"
 1. The "Death Combo": Cloud Native Index + No Cache In a Primary Key table, StarRocks must perform a "Read-before-Write" for every single batch of data you import. It needs to check the Primary Key Index to see if the record already exists so it can perform an update instead of an insert. * 
CLOUD_NATIVE
: This tells StarRocks to store your primary key index (which can be many GBs) on Azure Blob Storage. * 
"datacache.enable" = "false"
: This tells the Compute Nodes (CN) not to cache anything it pulls from Azure on local disks. The Result: Every time you load data, the CN has to reach across the network to Azure Blob Storage to download the index segments it needs to perform the lookup. Because you disabled the cache, it can't "remember" the index for the next batch. * This turns a sub-millisecond local RAM/SSD lookup into a multi-second network operation. * When thousands of these lookups happen during a transaction, your latency balloons to 87 seconds. 2. Why this causes the "Not Found" Error Because the transaction takes 87 seconds to "Publish," the metadata and the physical data files on Azure get out of sync: 1. The Vacuum/Cleanup process sees a partition that hasn't been "successfully" updated in a while (because the update is stuck in that 87s lag). 2. It identifies old data files as "garbage" and deletes them from Azure. 3. Meanwhile, your "Publish" process finally finishes and tries to point the table to those files—but they are already gone. Result: "Not Found" error. 3. Other DDL Observations * Bitmap Index on 
raw_import_timestamp
: * Bitmap indexes are designed for low-cardinality columns (e.g., 
gender
, 
country
, 
status
). * A timestamp is extremely high-cardinality. Creating a Bitmap index on a timestamp can actually slow down writes and increase the size of the data significantly without providing much query benefit. You are likely better off using a Bloom Filter or simply relying on the fact that StarRocks sorts data by the 
ORDER BY
(or PK) columns. * Redundant Property: You have 
"enable_persistent_index" = "true"
listed twice. It doesn't break anything, but it’s worth cleaning up. Recommended Fixes Immediate Fix (The most important): Enable the data cache. This allows the CN to keep the "Cloud Native" index on local SSDs after the first time it fetches them from Azure.
Copy code
sql
ALTER TABLE your_table_name SET ("datacache.enable" = "true");
_Note: Ensure your CN nodes have local SSD space allocated for the cache (check 
storage_root_path
in 
be.conf
)._ Optimization Fix: Drop the Bitmap index if 
raw_import_timestamp
has many unique values. It is likely adding extra overhead to your already slow 87s writes.
Copy code
sql
ALTER TABLE your_table_name DROP INDEX raw_import_timestamp_idx;
Verify Key Size: Your Primary Key is 
(location, meta_row_id, event_date)
. If 
location
or 
meta_row_id
are long strings, the index will be very large. If you can use an integer ID for 
location
, the index will be much smaller and faster. Summary: Change 
"datacache.enable"
to 
"true"
. This will allow the 
CLOUD_NATIVE
index to live on the local disk of your CNs, dropping your transaction latency from 87 seconds to sub-second, which will stop the "Not Found" errors. References • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md • faq/loading/Loading_faq.md
g
@Rocky the reason we had the DDL like that was because this is a raw table that we won't use a ton after creation but should we still enable datacache?
r
Yes, you should definitely enable datacache, even for a "raw" table that isn't queried often. While it seems counter-intuitive to cache data you won't query, the issue you are facing isn't about query performance—it's about ingestion performance. Here is why your current DDL is causing the 87-second lag and "Not Found" errors: 1. The Primary Key "Read-before-Write" Tax Because this is a Primary Key table, every time you load a batch of data, StarRocks must perform a "lookup" to see if those keys already exist (to decide whether to insert a new row or update an existing one). * With 
CLOUD_NATIVE
+ 
datacache.enable = false
: The index that stores your primary keys lives on Azure. Because the cache is disabled, your Compute Nodes (CN) must download that index from Azure for every single load. * The Result: Your "Publish" phase (where the index is updated) takes 87 seconds because it's waiting on Azure network I/O. This is the "traffic jam" you see in the logs. 2. Why the "Not Found" error happens In Shared-Data mode, StarRocks has a "Vacuum" process that cleans up old data. * If a transaction takes a long time to publish (like 87 seconds), the system's metadata state becomes "stretched." * If you have many concurrent loads, the FE might try to clean up files that it thinks are "old" but are actually still needed by your 87-second-long lagging transaction. * By the time your transaction finally finishes, the physical data or index file on Azure has been deleted by the cleanup thread. Result: "Not Found" error. 3. Recommendations for your "Raw" Table Option A: Keep Primary Key (If you need Upserts) If you actually need the Primary Key features (e.g., you might get duplicate `meta_row_id`s that need to be overwritten), you must enable the cache so the index lookups are fast:
Copy code
sql
ALTER TABLE your_table SET ("datacache.enable" = "true");
Note: This only caches the "hot" data and the index. It won't fill up your disks with the entire historical raw table if you aren't querying it. Option B: Switch to Duplicate Key (If you DON'T need Upserts) If this is truly a "raw" table where you just want to dump data and you don't care about de-duplicating rows based on the Primary Key, Duplicate Key tables are much better for raw ingestion: * They have zero index overhead during ingestion. * They don't require 
datacache
for writes. * They will never have this "87-second publish lag" because there is no index lookup. Option C: Remove the Bitmap Index You have a Bitmap index on 
raw_import_timestamp
. * The Problem: Timestamps are high-cardinality (thousands/millions of unique values). * The Impact: Bitmap indexes on high-cardinality columns are very "heavy" to build during ingestion and provide almost zero query benefit. This is likely adding several seconds to your 87-second delay. * The Fix: Drop it. StarRocks is already very fast at filtering by time without a bitmap index. Summary The 
datacache.enable = false
setting is essentially starving your ingestion process of the index it needs to complete the transaction. Enabling the cache (or switching to a Duplicate Key table) will resolve the 87s latency, which in turn will stop the "Not Found" errors. References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/FE_configuration.md
g
@Rocky what disk types do you recommend for PVCs in a shared data deployment? I currently have pvs for both data and logs on the fe nodes and the cn nodes
r
For a shared-data deployment, your disk performance directly impacts the "Publish" latency (FE) and query performance (CN). Since you are on Azure (based on your previous mention of Azure Blob), here are the specific recommendations for PVC types: 1. Frontend (FE) Nodes FE nodes are the brain of the cluster. They use BDBJE (Berkeley DB) to manage metadata. This process is extremely sensitive to disk I/O latency. * For Metadata (
fe-meta
PVC): * Recommended Disk Type: Azure Premium SSD (LRS) or Premium SSD v2. * Why: BDBJE performs frequent small writes to the edit log. If these writes are slow (Standard SSD or HDD), you will see "Latch timeouts" and the FE will become unresponsive, leading to the "87-second wait for publishing" errors you saw earlier. * Size: 100GB is usually sufficient, but performance (IOPS) matters more than size. * For Logs (
fe-log
PVC): * Recommended Disk Type: Azure Standard SSD. * Why: Logs are sequential writes. You don't need high IOPS here, just a reliable place to store the history. 2. Compute Nodes (CN) In shared-data mode, CNs don't store "permanent" data, but they use local disks for the Data Cache (caching your Azure Blob data). * For Data Cache (
storage_root_path
/ 
cn-data
PVC): * Recommended Disk Type: Azure Premium SSD or Local NVMe (Ephemeral). * Why: This disk acts as a local mirror for your "Cloud Native" primary key indexes and raw data. If this disk is slow, your "Read-before-Write" lookups during ingestion will be slow, causing the transaction lag. * The "Best" Choice: If your Azure VM size supports it (e.g., Lsv2-series or Lsv3-series), using the Local NVMe (ephemeral storage) is significantly faster than any PVC. * If using PVCs: Use Premium SSD and ensure the volume is large enough to get a high IOPS tier (Azure tiers IOPS by disk size). * For Logs (
cn-log
PVC): * Recommended Disk Type: Azure Standard SSD. Summary Table (Azure Examples) Node Component | PVC Purpose | Recommended Azure Disk Type | Priority FE | Metadata | Premium SSD (P10 or higher) | High (Stability) FE | Logs | Standard SSD | Low CN | Data Cache | Premium SSD (or Local NVMe) | High (Performance) CN | Logs | Standard SSD | Low Key Configuration Tip for Kubernetes When using the StarRocks Operator or Helm chart, ensure your 
StorageClass
is set to one that uses SSDs. If you are seeing "waiting for publishing" errors, check the IOPS usage on your FE metadata disk. If you hit the ceiling of a small Premium SSD (like a 4GB P1), Azure will throttle you, and StarRocks will experience the massive latency spikes you've been seeing. Moving to a larger disk (e.g., 128GB P10) often provides the IOPS "headroom" needed for smooth out of the box. One final note: If you stay with 
"datacache.enable" = "false"
, the CN disk type won't matter for query performance, but your cluster will remain unstable due to the network-bound index lookups. no matter what disk you buy. Enable the FE uses.** References • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/hdfs.md • deployment/shared_data/s3.md • data_source/data_cache_troubleshooting.md
g
@Rocky if the cache is ephemeral, won't it need to be recreated any time a cn restarts or a new one comes up?
r
Yes, that is exactly what happens. If you use ephemeral storage (like local NVMe disks or an 
emptyDir
in Kubernetes), the cache is wiped whenever the pod/node restarts. However, because you are using 
CLOUD_NATIVE
 indexing and Shared-Data mode, this is a "performance" penalty, not a "data loss" penalty. Here is the breakdown of what happens during that "recreation" phase: 1. It is "Re-cached," not "Re-built" With your current setting of 
persistent_index_type = "CLOUD_NATIVE"
, the master copy of your Primary Key index lives on Azure Blob Storage. * The Good News: You don't have to wait for the CN to scan all the data and "rebuild" the index from scratch. * The Process: When a new CN starts up with an empty disk, the first time you try to load data, the CN says, "I don't have the index for this table locally." It then downloads just the specific blocks of the index it needs from Azure to its local disk. 2. The "Cold Start" Penalty The downside is what we call a Cold Start. * First Load after Restart: The very first ingestion job after a CN restart will be slow (potentially that 80+ second lag you saw) because it has to fetch the index blocks over the network from Azure. * Subsequent Loads: Once those index blocks are cached on the local disk, the next loads will be back to sub-second speeds. 3. PVCs vs. Local NVMe (The Performance Trade-off) Since you mentioned you already have PVCs for the CN nodes, you have a choice to make: * Option A: Persistent PVCs (Middle Performance, High Stability) If your PVCs are backed by Azure Premium SSD, the cache will survive a pod restart. When the CN pod comes back up and mounts the same PVC, the cache is still there. You avoid the "Cold Start" penalty entirely. * Option B: Local NVMe / Ephemeral (Highest Performance, Lower Stability) Local NVMe is much faster than a PVC (lower latency for those index lookups). However, you will hit that "Cold Start" delay every time the pod moves to a new physical host. Rocky's Recommendation For your specific scenario (avoiding those 87-second timeouts and "Not Found" errors): 1. Use Persistent PVCs for CN Cache: Keep using the PVCs you have, but ensure they are Premium SSD. This ensures that once the index is downloaded once, it stays there even if the pod restarts. 2. Size the PVC Appropriately: Make sure the PVC is large enough to hold your index. If the disk fills up, StarRocks will start evicting the oldest index blocks, and you'll go back to having network-driven latencies. 3. Check your ReclaimPolicy: Ensure your Kubernetes 
PersistentVolume
has a 
reclaimPolicy: Retain
or that you are using a 
StatefulSet
so the pod always gets its data back. Summary: If your cache is ephemeral, it must be re-downloaded from Azure on restart. If you use persistent PVCs for your 
storage_root_path
, the cache survives restarts and you avoid the ingestion "Cold Start" performance hit." References • deployment/shared_data/azure.md • best_practices/primarykey_table.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md References • best_practices/primarykey_table.md • data_source/data_cache.md • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md
g
@Rocky my cache disk is always full on the CNs but isn't that desired? it seems you would want it to hold as much as it can.
r
You are absolutely right—in a perfect world, you want your cache to be full. A full cache means you are maximizing your local high-speed storage and minimizing expensive trips to Azure Blob Storage. However, there is a technical distinction between a "Full Cache" (which is good) and a "Full Disk" (which can be dangerous). Here is how StarRocks manages this and why your disk looks the way it does: 1. The "Overwrite" vs. "Delete" Behavior StarRocks' Data Cache uses a Block Cache mechanism. * Why it looks full: Once StarRocks writes a piece of data to the cache disk, it never "deletes" that file to free up space. Instead, when the cache reaches its limit (default is 80% of the disk), it uses an LRU (Least Recently Used) policy to decide which data to kick out. * The "Trick": It doesn't delete the old block; it simply overwrites it with new data. * The Result: From the perspective of your OS or Kubernetes (e.g., 
df -h
), the disk will look permanently full at the maximum allowed level. This is by design and does not mean you are out of space for new cache; it just means the cache is "warmed up." 2. Cache vs. Persistent Index (The "Roommate" Problem) On a Compute Node (CN), the 
storage_root_path
(your PVC) holds two very different things: 1. Data Cache (
/datacache
): The hot data from your tables. This is managed by the 80% threshold. 2. Persistent Index (
/persist
): This is the local cache of your Primary Key index. The Risk: If your Data Cache is allowed to take 100% of the disk, there will be no room for the Persistent Index to grow. If the index cannot write to disk, your ingestion will fail with those "Not Found" or "Publish" errors you've been seeing. 3. How to tell if it's "Healthy Full" or "Dangerous Full" Run this command on your CN pod:
Copy code
bash
du -h --max-depth=1 /path/to/your/storage_root/
* Healthy: The 
datacache
folder is large (around 80% of your PVC) and the 
persist
folder has a few GBs of space to breathe. * Dangerous: The total disk usage is 95% - 100%. This usually happens if logs, core dumps, or a very large Primary Key index have eaten the "safety buffer" StarRocks tries to leave. 4. Recommended Configuration To ensure your cache doesn't choke your ingestion (Primary Key index), check these settings: * 
starlet_star_cache_disk_size_percent
: This defaults to 80. It tells the Data Cache: "Only use 80% of the disk, leave 20% for the OS, logs, and the Persistent Index." * If your disk is at 99%: You should consider lowering this to 70 to give your Primary Key index more room to breathe. Summary * Is it desired for the cache to be full? Yes. It means your "raw" table queries and ingestion lookups will be fast. * Should the disk be 100% full? No. You need a "safety margin" (usually 10-20%) for the Persistent Index to function. Without that margin, your ingestion will stall, leading back to the 87-second lags you've been troubleshooting. Rocky's Tip: If your PVC is 100GB, StarRocks will try to use 80GB for data cache. If your Primary Key index grows to 25GB, you now have 105GB of data trying to fit in 100GB. In this case, you either need a larger PVC or you need to lower the 
starlet_star_cache_disk_size_percent
. References • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • best_practices/primarykey_table.md References • best_practices/primarykey_table.md • administration/management/BE_configuration.md • introduction/Architecture.md
g
@Rocky the pvc size for starrocks-storage is 1tb, this is the du -h command
Copy code
800G    /opt/starrocks-storage/datacache
16K     /opt/starrocks-storage/lost+found
58M     /opt/starrocks-storage/meta
4.0K    /opt/starrocks-storage/persistent
4.6M    /opt/starrocks-storage/error_log
120K    /opt/starrocks-storage/tmp
4.0K    /opt/starrocks-storage/data
800G    /opt/starrocks-storage/
r
This 
du -h
output is actually a perfectly healthy sign for a StarRocks shared-data cluster. It confirms that your configuration is working exactly as intended. Here is the breakdown of why your disk looks this way and why it’s actually a good thing: 1. The "80% Cap" is working Your 
datacache
folder is exactly 800G, which is precisely 80% of your 1TB PVC. * This is controlled by the parameter 
starlet_star_cache_disk_size_percent
, which defaults to 80. * StarRocks pre-allocates or fills this space and then manages it using an LRU (Least Recently Used) policy. Once it hits 800GB, it won't grow further; it will just overwrite the oldest data with the newest. * Is it "Full"? From the perspective of the StarRocks Cache, yes. From the perspective of the Azure Disk, no. You still have 200GB of physical headroom for logs, OS operations, and metadata. This is the "safety buffer" I mentioned earlier. 2. Why is the 
/persistent
folder empty (4.0K)? Since you are using 
persistent_index_type = "CLOUD_NATIVE"
, you won't see large files in the 
/persistent
folder. * In this mode, the Primary Key index is stored as blocks in your Azure Blob storage. * When the CN needs those index blocks, it downloads them into the 
/datacache
 folder, just like regular data. * This is why your 
datacache
is 800GB and your 
persistent
folder is empty. The "Cloud Native" index is literally "hiding" inside the data cache blocks. 3. Is 800GB enough? The fact that your cache is "full" (at its 80% limit) means your cluster has successfully cached 800GB of your most frequently accessed data and index blocks. * How to tell if you need more: Check your Cache Hit Ratio in the metrics (or via 
ADMIN SHOW FRONTEND CONFIG
/ Prometheus). * If your Hit Ratio is high (e.g., >90%), 1TB is plenty. * If your Hit Ratio is low (e.g., <50%), your "working set" of data is larger than 800GB. This causes "Cache Thrashing," where StarRocks is constantly deleting and re-downloading data from Azure, which might be causing those latency spikes. Rocky's Verdict Your current disk usage is optimal. You have maximized your cache usage while maintaining a 200GB safety margin. One recommendation: If you continue to see the "87-second wait" or "Publish" latencies: 1. Check IOPS: Ensure your 1TB Azure PVC is a Premium SSD (P30). Standard SSDs at 1TB may not have the throughput to handle the 800GB of cache churn. 2. Increase Cache Size: If your dataset is much larger than 1TB, consider increasing the PVC to 2TB. This will give you 1.6TB of cache and, more importantly, double the IOPS on Azure (P40 tier), which will make index lookups much faster. Summary of your folders: * 
datacache
(800G): Your hot data + your Primary Key index blocks. * 
meta
(58M): CN-specific metadata (healthy). * 
persistent
(4K): Empty because you are using 
CLOUD_NATIVE
indexing. * 
error_log
(4.6M): Very small, which is good. References • administration/management/BE_configuration.md • sql-reference/information_schema/be_tablets.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky caching is tough in backfill situations, if we backfill a bunch of data it will force the cache to have things we won't use again. Is that just how it is?
r
That is a common concern, often referred to as "Cache Poisoning." If you backfill 10TB of historical data into a 1TB cache, you would normally expect all your "hot" data to be evicted to make room for data you might never query again. However, StarRocks has specific mechanisms to prevent this from happening. You don't just have to "deal with it." 1. The "SLRU" Savior (Segmented LRU) Starting in more recent versions (defaulting to 
slru
in v3.4+), StarRocks uses Segmented LRU instead of a simple LRU. * How it works: The cache is split into two segments: Eviction and Protection. * The Backfill Filter: When new data (like your backfill) is read, it first enters the Eviction segment. It is only promoted to the Protection segment if it is accessed again. * The Result: One-time backfill data will churn through the small Eviction segment and get discarded, while your truly "hot" data (which is frequently queried) stays safe in the Protection segment. * Check your config: Look for 
datacache_eviction_policy = slru
in your CN/BE configs. 2. Manual Control: 
populate_datacache_mode
 If you are doing a massive backfill via a SQL script or a specific session, you can explicitly tell StarRocks: "Do not put this data in the cache." Before running your backfill 
INSERT
or query, run:
Copy code
sql
SET populate_datacache_mode = 'never';
This ensures that the data read/written during that session bypasses the local disk cache entirely, leaving your 800GB of "hot" data untouched. 3. Smart Defaults for Ingestion In StarRocks v3.3.2+, the system is already "backfill-aware": * 
INSERT INTO SELECT
: By default, this statement does not populate the cache. StarRocks assumes that if you are moving data internally, you don't necessarily want it to take over the high-speed cache. * Full Table Scans: If a query scans a massive number of partitions (typical of a backfill), StarRocks will often automatically set 
populate: false
to protect the cache. 4. Table-Level TTL (
partition_duration
) If your backfill is for a specific table or historical partitions, you can use the 
datacache.partition_duration
property.
Copy code
sql
ALTER TABLE my_table SET ("datacache.partition_duration" = "30 DAY");
With this setting, StarRocks will only allow data from the last 30 days to live in the cache. Even if you backfill data from 2022, it will be written to Azure but ignored by the cache logic, preventing it from ever kicking out your current hot data. Summary: What you should do 1. Trust the SLRU: Ensure your CNs are using 
slru
as the eviction policy. 2. Use the Session Variable: For your big backfill jobs, explicitly 
SET populate_datacache_mode = 'never'
. 3. Monitor the Hit Rate: Keep an eye on your 
DataCacheHitRatio
during the backfill. If it stays high for your regular queries while the backfill is running, the protection is working. Rocky's Take: You're right that "dumb" caching is tough for backfills, but StarRocks' "smart" caching (SLRU + Population Modes) is designed specifically so your 10TB backfill doesn't break your 1TB performance tier-one performance. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • sql-reference/System_variable.md • administration/management/BE_configuration.md References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • sql-reference/System_variable.md
g
@Rocky so in this scenario, chaning 
"datacache.enable" = "true"
might not make much difference since it's a backfill?
r
The short answer is: Yes, it makes a huge difference. If you leave 
"datacache.enable" = "true"
(which is the default) during a massive backfill, you risk "poisoning" your cache with terabytes of historical data that you might never query again. Here is why 
datacache.enable
is critical in a backfill scenario and how to handle it: 1. The "Write-Through" Behavior In a StarRocks shared-data cluster, 
datacache.enable
isn't just about reading; it's about the initial load. * When `true`: As you backfill data (via 
INSERT
, 
Broker Load
, etc.), StarRocks writes that data to Azure Blob Storage and simultaneously writes it to your local CN cache disk. * The Catch: During this initial load, StarRocks ignores your 
partition_duration
(TTL). Even if you are backfilling data from 2010, it will be written into your local 800GB cache and will stay there until it is eventually kicked out by the LRU (Least Recently Used) policy. * The Consequence: Your backfill could evict all your current "hot" data (e.g., today's sales data) to make room for data from 2010. 2. Is it "just how it is"? (No, you have options) You don't have to let the backfill ruin your cache hit ratio. You have three main strategies to protect your cache during backfills: Option A: The Session-Level "Mute" (Recommended) Before running your backfill 
INSERT
or load script, set this session variable:
Copy code
sql
SET populate_datacache_mode = 'never';
This tells StarRocks: "For this specific session, do not write anything I'm touching into the local cache." This is the cleanest way to backfill 10TB of data without touching your 800GB of "hot" cache. Option B: Table-Level Disabling If the table you are backfilling is only for deep history and will rarely be queried, you can set the property to 
false
permanently:
Copy code
sql
ALTER TABLE your_table SET ("datacache.enable" = "false");
This ensures that data for this table never touches the local disk, always reading directly from Azure Blob Storage. Option C: Trust the SLRU (v3.3+) If you are on a recent version of StarRocks, the cache uses Segmented LRU (SLRU). * Data being written for the first time enters a "Probation" segment. * It only moves to the "Protected" segment if it is queried at least twice. * This means a one-time backfill will pass through the cache quickly and get evicted without ever touching your truly "hot" data that lives in the Protected segment. Rocky’s Recommendation for your 800GB Cache: Since your 1TB PVC is already 80% full, you have no room for "mistakes." If you backfill a large amount of data: 1. Check your StarRocks version. If you are on v3.3.2 or later, 
INSERT INTO SELECT
already defaults to not populating the cache. 2. Use the session variable. To be safe, always wrap your backfill scripts in 
SET populate_datacache_mode = 'never';
. 3. Monitor the Hit Rate. If you see your 
DataCacheHitRatio
drop from 95% to 20% during a backfill, it means your backfill is indeed poisoning the cache. Summary: Changing 
datacache.enable
to 
false
(or using the session variable) is the difference between your cluster staying "hot" and your cluster having to re-warm its entire cache from Azure once the backfill is over. References • using_starrocks/caching/block_cache.md • sql-reference/System_variable.md • deployment/shared_data/s3.md • data_source/data_cache.md References • sql-reference/System_variable.md • using_starrocks/caching/query_cache.md • data_source/data_cache.md
g
@Rocky but back here you were thinking that cache being disabled was one of the main sources of my problems https://starrocks.slack.com/archives/C02FACZSNJV/p1778504005409089?thread_ts=1778164536.668869&amp;cid=C02FACZSNJV
r
You are right to call that out! It might seem like a contradiction, but it’s actually two sides of the same performance coin: Steady-state performance vs. Operational safety. Here is how those two pieces of advice reconcile: 1. Why I said "Disabling it is a problem" (The Baseline) In a shared-data cluster, if 
datacache.enable
is 
false
globally or for your main tables: * Latency Spikes: Every single query—and more importantly, the Primary Key index lookups during ingestion—must go to Azure Blob storage. * The "87-second wait": During the "Publish" phase of a load, StarRocks needs to check for duplicate keys. If the cache is disabled, it has to pull index blocks from Azure. If Azure throttles you or has high latency, your "Publish" version times out or hangs, leading to those massive wait times you saw. * Verdict: You MUST have the cache enabled for your hot data to keep the cluster responsive. 2. Why I'm saying "Be careful now" (The Backfill Exception) A backfill is a special event. If you are loading 5TB of historical data into a cluster that only has 800GB of local cache: * The Problem: The backfill will "win" the LRU (Least Recently Used) race. It will kick out all your current "hot" data (the data that makes your dashboards fast) to make room for 2021 data that no one will ever look at again. * The Result: Your backfill finishes, but your cluster becomes "cold" and slow for all your users. The Resolution: The "Session Variable" is the Key The advice isn't to flip the table property 
datacache.enable
to 
false
. Instead, the best practice is to leave the table-level property as 
true
(so your hot data stays fast), but bypass the cache specifically for your backfill jobs. Instead of changing the table, run this in your backfill session:
Copy code
sql
-- This tells StarRocks: "For this session, don't write to the cache"
SET populate_datacache_mode = 'never';

-- Then run your big backfill
INSERT INTO my_table SELECT * FROM historical_source;
Summary of the Strategy: Scenario | 
datacache.enable
 | 
populate_datacache_mode
 | Why? Standard Daily Load | 
true
| 
default
| You want today's data to be "hot" in the cache immediately. User Queries | 
true
| 
default
| You want fast sub-second dashboard performance. Massive Backfill | 
true
| 
'never'
 | You want the data in Azure, but you don't want to kick out your "hot" data. Rocky's Bottom Line: I still stand by my earlier point—disabling the cache entirely is likely what caused your latency spikes. But for a backfill specifically, you want to be "surgical" so you don't accidentally ruin the cache you just fixed! References • using_starrocks/caching/block_cache.md • sql-reference/System_variable.md • data_source/data_cache.md • deployment/shared_data/feature-support-shared-data.md References • using_starrocks/caching/block_cache.md • sql-reference/System_variable.md • data_source/data_cache.md • deployment/shared_data/feature-support-shared-data.md • best_practices/primarykey_table.md • deployment/shared_data/s3.md • faq/shared_data_faq.md References
g
@Rocky I want to be clear about what's happening 1. ingest a bunch of data via pipe load 2. much later (next day) a partition with a VISIBLE_VERSION_TIME of the previous day is running into the starlet err object 3. this has happened several times in a row and increasing 
lake_autovacuum_grace_period_minutes
only delayed the problem
r
This sounds like a classic "Stale Metadata vs. Aggressive Vacuum" conflict, which is a known challenge in shared-data (Lake) architectures. Based on your observation that increasing 
lake_autovacuum_grace_period_minutes
only delayed the issue, we can confirm that the StarRocks Vacuum process is indeed deleting the objects. The "starlet err object" (usually 
[SL02] Object not found
) confirms those files are gone from Azure Blob. The million-dollar question is: Why is your system still trying to read files from yesterday if they have been superseded by a newer version? Here is the breakdown of what is likely happening: 1. The Metadata "Lag" (The Likely Root Cause) In StarRocks shared-data mode, the Leader FE handles all writes and compactions. When a compaction or load finishes: 1. The Leader FE commits a new Version (e.g., v10 -> v11). 2. The files for the old Version (v10) are marked as "stale." 3. The Vacuum thread waits for the 
grace_period
(default 30m) and then deletes the v10 files. 4. The Catch: If you are querying a Follower FE that has lagged or is stuck replaying the metadata journal, that Follower still thinks the "Current Version" is v10. 5. The Follower plans the query using v10 metadata $\rightarrow$ CN tries to fetch v10 files $\rightarrow$ Object Not Found Error. Why the "Next Day" timing? If your 
VISIBLE_VERSION_TIME
shows yesterday, but the Leader FE has already moved the partition to a new version (due to a background compaction or a successful Pipe commit), then the metadata you are seeing is stale. If that lag exceeds your grace period, the files are deleted before the follower even knows they are old. 2. Compaction "Invisible" Versions Compaction in StarRocks Lake mode increments the 
VISIBLE_VERSION
, but in some older versions, it might not update the 
VISIBLE_VERSION_TIME
if the underlying data hasn't changed (just the file structure). * The Scenario: A background compaction runs today at 10 AM. It merges files from yesterday's load. * It creates a new version. The old files are vacuumed at 10:30 AM. * If your query or your FE is still trying to reference the "un-compacted" file list from yesterday, you hit the error. 3. Primary Key Persistent Index (If applicable) If this is a Primary Key table, StarRocks stores a persistent index in the cloud. * Sometimes, the index points to old segment files. * If a compaction happens and the index isn't updated correctly or the CN is using a cached, stale version of the index, it will look for files that the vacuum process has already recycled. ────────── Recommended Troubleshooting Steps 1. Identify the Metadata Gap Run this on all your FEs (Leader and Followers):
Copy code
sql
SHOW PARTITIONS FROM your_table WHERE PartitionName = 'your_failing_partition';
* Do they all show the same 
VisibleVersion
? * If the Follower shows a lower version than the Leader, you have a metadata replication lag. 2. Check FE Health/Lag Run 
SHOW FRONTENDS;
. Check the 
Join
and 
Alive
columns. More importantly, check the 
fe.log
on your follower FEs for any errors related to 
BDBJE
or "journal replay." If the followers are stuck, they will never see the new versions and will keep trying to query deleted files. 3. Check for "Ghost" Compaction Check if compactions are running and failing or finishing without the followers knowing:
Copy code
sql
SELECT * FROM information_schema.be_cloud_native_compactions 
WHERE TABLE_NAME = 'your_table' 
ORDER BY START_TIME DESC;
4. Temporary Workaround (The "Safe" Buffer) If you can't find a metadata lag, set the grace period much higher (e.g., 
1440
for 24 hours). This isn't a fix, but it provides a safety net for any "long-tail" metadata propagation issues.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "1440");
5. Pipe-Specific Check Since you are using Pipe, verify the status of the pipe:
Copy code
sql
SHOW PIPE STATUS FOR pipe_name;
If the Pipe is in a 
STALLED
state or is repeatedly retrying a file that was vacuumed, you may need to 
RESTART PIPE
. Rocky’s Summary: The fact that the 
VISIBLE_VERSION_TIME
is from yesterday while objects are being deleted today strongly suggests your Frontend metadata is out of sync with the actual storage state. Either the Follower FE is lagging, or there’s a bug in how that specific version’s metadata was published. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md
g
@Rocky the thing is, with this table we don't care much about performance. we load it once (in backfills) or a few times (as data comes in) then use it to create cleaned/enriched tables and this raw is never touched again. should we really use cache?
r
This is a very insightful question. If you only care about "one-and-done" ETL, it is tempting to disable the cache to save disk space and avoid "poisoning" your CN nodes with raw data. However, for a Primary Key table (which your error messages suggest this is), disabling the cache entirely is often a "trap" that leads to more stability issues, specifically the 
starlet err object
(Object Not Found) you are currently fighting. Here is the breakdown of why you should probably keep the cache enabled for your raw table, even if you don't care about query speed: 1. The "Safety Net" for Object Not Found The 
starlet err object
occurs when your FE (Frontend) tells a CN (Compute Node) to read a file that no longer exists in Azure Blob. * With Cache Enabled: If that file was recently loaded or compacted, a copy likely exists in the local CN cache. The CN will serve the file from the cache, and your ETL succeeds without ever knowing the Azure file was deleted prematurely. * With Cache Disabled: The CN is forced to fetch from Azure. If the file is gone due to an aggressive vacuum or a metadata lag, the process fails instantly. * Conclusion: Disabling the cache removes your only "buffer" against the very error you are seeing. 2. Compaction relies on the Cache In a StarRocks shared-data cluster, compaction is the background process that merges small "rowsets" (from your Pipe loads) into larger, healthier files. * For Primary Key tables, compaction is intensive. It has to read old segments to verify which rows are still "current." * If the cache is disabled, every background compaction task must download data from Azure. This makes compaction significantly slower. * The Result: Compaction falls behind, your "compaction score" skyrockets, and your Pipe loads eventually stall or fail with "too many versions" or "compaction score too high." 3. Downstream ETL Performance When you run your `INSERT INTO cleaned_table SELECT ... FROM raw_table`: * If the raw table is not cached, StarRocks has to pull the entire dataset from Azure. * Depending on your Azure Blob throughput limits, this can make your "enrichment" step take 10x longer, potentially causing timeouts in your ETL pipeline. ────────── What is actually causing your "Object Not Found" error? The fact that increasing 
lake_autovacuum_grace_period_minutes
only delayed the problem is the smoking gun. It confirms that the StarRocks Vacuum process is deleting files that your system still thinks are "Visible." This is almost always caused by Metadata Lag on Follower FEs: 1. The Leader FE performs a compaction, creates a new Version, and marks the old files for deletion. 2. The Vacuum process waits for the grace period and deletes the files. 3. A Follower FE (which might be where your query/Pipe is running) is lagging behind on the metadata journal. It still thinks the "Current Version" is the old one. 4. The Follower FE tries to plan a job using those old files $\rightarrow$ Object Not Found. Rocky’s Recommended Path Forward: 1. Don't disable 
datacache.enable
. Keep it 
true
to ensure compaction stays healthy and to mask transient object-not-found errors. 2. Use the "Never" Mode for Backfills: To prevent the backfill from ruining the cache for your other tables, run your backfill/load scripts with:
Copy code
sql
    SET populate_datacache_mode = 'never';
This allows the system to read from the cache (if data is there) but prevents the massive backfill from writing new historical data into the cache. 3. Investigate Follower FE Lag: Run 
SHOW FRONTENDS;
and check if any FEs have a 
LastHeartbeat
or metadata version that is lagging behind the Leader. 4. Set a "Safe" Grace Period: If your metadata lag is unpredictable, set the grace period to at least 2-4 hours (120-240 mins) to give all nodes plenty of time to sync.
Summary: Disabling the cache for your raw table won't fix your errors; it will likely make them persistent and unrecoverable. Keep the cache "on" for stability, but use session variables to keep it "clean." References • best_practices/primarykey_table.md • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md • deployment/shared_data/s3.md
g
@Rocky this fe.warn log looks important to this issue
Copy code
2026-05-08 00:00:51.925Z ERROR (publish-task-166144|303528) [PublishVersionDaemon.publishPartitionBatch():601] publish partition batch partition.getVisibleVersion() + 1 != version.get(0) 28722707 122687 122687
r
That log entry is the "smoking gun." It explains exactly why your data is being vacuumed and why you are seeing 
Object Not Found
errors. 1. What the Error Means The log 
publish partition batch partition.getVisibleVersion() + 1 != version.get(0) 28722707 122687 122687
translates to: * Partition ID: 28722707 * Current Visible Version (in FE memory): 122687 * Version being Published: 122687 StarRocks requires versions to be strictly sequential ($V_{next} = V_{current} + 1$). The FE is trying to publish Version 122687, but the partition is already at 122687. Because $122687 + 1 \neq 122687$, the Publish task fails. 2. The Chain Reaction (Why you get "Object Not Found") 1. The Failed Publish: Your Pipe load or a background compaction finishes writing files to Azure and tries to commit the metadata. Because of the version mismatch, the FE rejects the commit. 2. Metadata Stalling: Since the commit fails, the partition’s 
VISIBLE_VERSION
stays at yesterday’s version (which explains why your 
VISIBLE_VERSION_TIME
is stale). 3. Vacuum Kicks In: The StarRocks Vacuum process looks at the files in Azure. It sees files that were written by the failed transaction. Since those files aren't linked to a "Visible" version, and they are older than your 
lake_autovacuum_grace_period_minutes
, Vacuum deletes them. 4. The Error: Later, a query or a retry of that load tries to reference those files. But they are gone. Result: 
starlet err object [SL02] Object not found
. 3. Why is this happening? This is typically caused by a bug or a race condition in the Batch Publish feature in shared-data mode (introduced in v3.1/v3.2). The FE gets confused and tries to publish a version that was already committed by a previous thread or a different FE. 4. How to Fix It Step 1: Disable Batch Publishing (The Workaround) Many users have found that disabling the "batching" logic in the Publish daemon stops these version collisions. Run this on your FE:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
This will force the FE to publish transactions one by one, which is slightly slower but much safer. Step 2: Check for Frontend Metadata Inconsistency If you have multiple FEs (a Leader and Followers), it is possible the Leader thinks the version is $X$ while the Follower thinks it's $Y$. * Run 
SHOW FRONTENDS;
and ensure all are 
Alive
and the 
LastHeartbeat
is recent. * If one FE is consistently throwing this error, it may need to be restarted to resync its metadata journal. Step 3: Manually "Kick" the Metadata (If Stuck) If a specific partition is permanently stuck and won't accept new loads, you can try to manually set the version to the next expected number, but be very careful with this. Only do this if the partition is totally unresponsive:
Copy code
sql
-- Replace with your table/partition names and the target version
ADMIN SET TABLE <table_name> PARTITION (<partition_name>) VERSION TO 122688;
Step 4: Check your StarRocks Version This specific 
PublishVersionDaemon
logic saw significant fixes in: * v3.1.x: Fixed in 3.1.10+ * v3.2.x: Fixed in 3.2.5+ * v3.3.x: Fixed in 3.3.0+ If you are on an older sub-version (e.g., 3.1.2 or 3.2.1), you are likely hitting a known bug that was addressed in later patches. Rocky's Recommendation: Disable 
lake_enable_batch_publish_version
immediately. That usually stops the version collision errors, which will stop the Vacuum process from deleting your "in-flight" data files. References • faq/shared_data_faq.md • data_source/datalake_faq.md • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md
g
@Rocky similar error 
2026-05-08 00:02:06.583Z ERROR (publish-task-166158|303543) [PublishVersionDaemon.publishPartition():983] Fail to publish partition 28722707 of txn 30269111: Fail to publish version for tablets:[], error msg: starlet err grpc.GetShard(shardId=28722709) error: Deadline Exceeded, host: kube-starrocks-cn-3.kube-starrocks-cn-search.kube-starrocks.svc.cluster.local
r
The two error logs you provided are the "cause and effect" of your problem. They confirm that your cluster is suffering from a Publish Version Bottleneck that is causing metadata corruption and triggering the Vacuum process prematurely. 1. The Trigger: gRPC "Deadline Exceeded" The log 
starlet err grpc.GetShard... Deadline Exceeded
means the Frontend (FE) tried to communicate with a Compute Node (
kube-starrocks-cn-3
) to finalize the data version, but the CN didn't respond within the default timeout (usually 5 seconds). Why this happens: * CN Overload: Your CN nodes might be struggling with high CPU or IO during the backfill, causing the Starlet service (internal storage manager) to lag. * Metadata Pressure: If you have thousands of tablets/shards, the 
GetShard
call can take longer than 5 seconds. * Network Latency: In Kubernetes, transient network spikes can easily exceed a 5s window. 2. The Result: Version Sequence Break When that gRPC call times out, the Publish task fails. However, because of a known issue in the "Batch Publish" logic (in versions 3.1.x and 3.2.x), the FE can get confused about what the "next" version should be. This leads to your first error: 
partition.getVisibleVersion() + 1 != version.get(0)
. * The FE expects to publish version 122688. * But due to the previous timeout/failure, it's trying to publish 122687 again. * The system rejects this, the transaction stays "stalled," and the partition metadata never updates. 3. Why files are missing ("Object Not Found") Because the transaction failed to "Publish," the data files written to Azure Blob are considered orphans by the FE. 1. The Pipe load writes the files. 2. The Publish step fails (due to the timeout). 3. The FE never records these files as part of a "Visible Version." 4. The Vacuum process sees these files in Azure, sees they aren't linked to any successful version, waits for your 
grace_period
, and deletes them. 5. If a retry or a downstream process later tries to look for those files, it hits the 
starlet err object [SL02]
. ────────── Recommended Fixes (Apply in order) Phase 1: Stability Configurations (Run these immediately) The most effective way to stop this loop is to increase the communication timeouts and simplify the publish logic. Run these on your FE:
Copy code
sql
-- 1. Increase the gRPC timeout for starlet (Default is 5s, increase to 30s)
ADMIN SET FRONTEND CONFIG ("starmgr_grpc_timeout_seconds" = "30");

-- 2. Disable Batch Publishing
-- This prevents the "version mismatch" logic error that occurs after a timeout.
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- 3. (Optional) Increase publish threads if you have many partitions
ADMIN SET FRONTEND CONFIG ("lake_publish_version_max_threads" = "1024");
Phase 2: Clean up the "Stuck" Partitions If a partition is still showing a 
VISIBLE_VERSION_TIME
from yesterday and won't update even after the config changes, the metadata might be "stuck." 1. Check the status: 
SHOW BACKENDS;
and 
SHOW FRONTENDS;
to ensure all nodes are healthy. 2. Restart the Leader FE. This often clears out the internal "Publish" queue and forces a resync of the metadata from the BDBJE journal. Phase 3: Optimize for Backfills Since you mentioned you don't care about performance for this table: * Lower the frequency of Pipe loads: If you are loading tiny files every few seconds, you are creating massive version pressure. Try to batch your backfill into larger files (100MB+). * Increase Grace Period during Backfills: Set 
lake_autovacuum_grace_period_minutes
to 
2880
(48 hours) specifically while the backfill is running to ensure no files are deleted until you are sure the metadata has caught up.
Summary: Your system is timing out during the final "handshake" of the load. This causes the FE to lose track of the version sequence, which causes the Vacuum process to delete the "unclaimed" files. Increasing the 
starmgr_grpc_timeout_seconds
is the most direct fix for the 
Deadline Exceeded
error. References • faq/shared_data_faq.md • administration/management/FE_configuration.md • data_source/datalake_faq.md • faq/loading/Loading_faq.md • sql-reference/System_variable.md
g
@Rocky interesting though, the partition and table mentioned above don't have any errors like the table i'm running into errors for
r
That is a very sharp observation, and it actually points to the root cause of why your cluster is struggling. The reason you see these errors for a "healthy" table while your "problematic" table is the one failing with 
Object Not Found
is because the Publish Version process is a global bottleneck. Here is what is happening under the hood: 1. The "Neighbor from Hell" Effect In StarRocks shared-data mode, all tables share the same Publish Version Daemon and the same gRPC communication channels between the FE and CN nodes. * What you see in the logs: Table A (the healthy one) is likely under heavy load or has many partitions. It is triggering a lot of 
GetShard
requests. * The Bottleneck: Because the default 
starmgr_grpc_timeout_seconds
is very low (5 seconds), and the FE might be struggling to keep up with the metadata for all tables, Table A starts timing out (
Deadline Exceeded
). * The Collateral Damage: When the Publish Daemon gets stuck or times out on Table A, it delays the processing of Table B (your raw table). If Table B’s transaction doesn't get "Published" in time, its files are left in a "pending" state. 2. Why Table B gets the "Object Not Found" but Table A doesn't This comes down to Vacuum vs. Visibility: * Table A (The "Healthy" one): Even if a few versions fail to publish, Table A might have enough "old" healthy versions that queries still work, or its data isn't being vacuumed as aggressively. * Table B (The "Raw" table): If this table is being hit with a high-frequency backfill (e.g., via Pipe or Stream Load), it is creating hundreds of tiny transactions. If the "Publish" step for these transactions is delayed because the FE is busy fighting with Table A's timeouts, those files sit in Azure Blob without a "Visible" owner. * The Janitor: The StarRocks Vacuum process doesn't care which table is which. It sees files in Azure that are older than your 
grace_period
and aren't linked to a successful 
VISIBLE_VERSION
. It deletes them. Result: Table A looks "broken" in the logs (timeouts), but Table B is the one that actually "loses" data (Object Not Found) because its files were deleted before the FE could finish the handshake. 3. The "Version Mismatch" is a Logic Bug The error 
partition.getVisibleVersion() + 1 != version.get(0)
is a known race condition in the Batch Publish feature (v3.1 and v3.2). When a publish task times out (the 
Deadline Exceeded
you saw), the FE sometimes loses track of the "sequence" of versions. It tries to commit Version 100 when the table is already at Version 100. This "stalls" the partition's metadata forever until the FE is restarted or the config is changed. ────────── Rocky’s "Global Fix" Plan To stop this cross-table interference, you need to give the FE more "breathing room" and disable the buggy batching logic. Run these on your FE: 1. Increase the communication "patience" This stops the 
Deadline Exceeded
errors that are currently clogging your logs.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("starmgr_grpc_timeout_seconds" = "30");
2. Disable the buggy Batch Publish logic This ensures that if one transaction fails, it doesn't "break the chain" for the next ones with that weird 
+1 != version
error.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
3. Increase the Publish Workers If you have a lot of tables/partitions, the default worker count might be too low, causing "head-of-line blocking" where Table A's tasks prevent Table B's tasks from starting.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_publish_version_max_threads" = "1024");
Why this answers your "Cache" question from earlier:
If you disable the cache on your raw table while the cluster is having these "Publish" timeouts, you are removing your only safety net. If the cache is ON, the CN nodes can often find the data locally even if the Azure file was prematurely deleted by the Vacuum process. If the cache is OFF, every single metadata hiccup becomes a hard query failure. Summary: The errors you see for other tables are the symptom (system-wide latency), and the 
Object Not Found
on your raw table is the consequence. Fix the global timeouts and batching, and both tables will stabilize. References • faq/shared_data_faq.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md
g
@Rocky found this in the fe warn logs, this is for a partition that is referencing an old .dat 
2026-05-08 21:23:22.728Z WARN (autovacuum-pool1-t5|295) [AutovacuumDaemon.vacuumPartitionImpl():280] Vacuumed mydb.mytable.30871159 with error: starlet err grpc.GetShard(shardId=30871162) error: Deadline Exceeded
r
This 
Autovacuum
log is the last piece of the puzzle. It confirms that the system-wide metadata bottleneck is now affecting your "garbage collector" (Autovacuum), which is exactly why your data files are disappearing. 1. What this log specifically reveals The log 
Vacuumed mydb.mytable... with error: starlet err grpc.GetShard... Deadline Exceeded
tells us two critical things: 1. The Vacuum process is indeed targeting your problematic table. Even though it "errored out" in this specific log entry, it implies that the Autovacuum daemon is actively trying to identify and delete files for that partition. 2. The Metadata Bottleneck is Global. The same 
Deadline Exceeded
error that crashed your 
PublishVersion
(preventing new data from becoming visible) is also hitting your 
Autovacuum
. 2. The "Object Not Found" Smoking Gun Here is the sequence of events that leads to your 
.dat
files being missing: 1. Stalled Publish: Because of the 
Deadline Exceeded
and the 
visible version + 1
error you saw earlier, the Frontend (FE) failed to commit the new data versions. 2. Unprotected Files: Because the commit failed, the new 
.dat
files in Azure are considered "unclaimed" or "orphaned" by the metadata. 3. Aggressive Vacuum: The Autovacuum process runs. It checks Azure for files. If it sees files that are not linked to a successful "Visible Version" and they are older than your 
lake_autovacuum_grace_period_minutes
(default is often 1 hour or less depending on version), it deletes them. 4. The "Old .dat" Reference: Your FE metadata is stuck on an old version that still wants to look at those files (or a compaction task is trying to merge them). But the Vacuum process has already deleted them from Azure. 5. Result: You get the 
Object Not Found
error. 3. Why the "Deadline Exceeded" is happening This is a known issue in StarRocks shared-data mode when the FE is under heavy metadata pressure (common during backfills). The 
ShardManager
inside the FE uses a global lock. When you have many partitions or many tiny loads: * The FE gets "lock contention." * Every request (Publishing data, Vacuuming data, or even Shard heartbeats) has to wait for this lock. * The default timeout is 5 seconds. If the lock isn't granted in 5s, you get 
Deadline Exceeded
. 4. How to stop the "Data Loss" immediately You need to tell the FE to be more "patient" and less "aggressive" with deletions. Step 1: Stop the aggressive deletion (Buy yourself time) Increase the grace period so the Vacuum process won't delete "unclaimed" files for 24 hours. This gives the metadata time to recover without losing data.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "1440");
Step 2: Fix the timeout bottleneck Increase the gRPC timeout from 5 seconds to 30 or 60 seconds to stop the 
Deadline Exceeded
cycle.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("starmgr_grpc_timeout_seconds" = "60");
Step 3: Disable the buggy Batch Publish logic As mentioned before, this prevents the FE from getting into that weird "Version + 1" logic loop after a timeout occurs.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
Why you see it for a partition referencing an "old" .dat: If a partition is stuck on an old version, and you are running a backfill or compaction, the system is constantly generating new files. If the "Publish" fails, the system tries to clean up. But if the metadata is inconsistent, it might accidentally clean up a file that a "stalled" transaction or an old version still thinks it needs. Recommendation: Once you apply these three 
ADMIN SET
commands, you should ideally restart the Leader FE. This will flush the metadata lock queues and allow the Publish Version daemon to start fresh with the new, longer timeouts. References • administration/management/FE_configuration.md
g
@Rocky if i change those, should i also change 
lake_autovacuum_stale_partition_threshold
which is at the default of12 hours
r
The short answer is no, you typically do not need to change 
lake_autovacuum_stale_partition_threshold
 to fix your current "Object Not Found" and "Deadline Exceeded" issues. In fact, leaving it at 12 hours actually provides an extra layer of safety while you are troubleshooting. Here is the breakdown of how these two parameters interact and why the default is likely fine for you right now: 1. 
lake_autovacuum_grace_period_minutes
(The "Safety Buffer") * What it does: This controls how long "orphaned" files (files not linked to a successful version) sit in your Azure Blob storage before the Vacuum process deletes them. * Your current issue: Your Publish tasks are failing/timing out, so files are being created but not "claimed" by the metadata. Because your grace period was likely low (default is 30 mins in newer versions, 5 mins in older), Vacuum was deleting them before the FE could finish the handshake. * The Fix: Setting this to 1440 (24 hours) ensures that even if your backfill stalls or the FE is under heavy load, the files stay safe in Azure for a full day. 2. 
lake_autovacuum_stale_partition_threshold
(The "Ignore Timer") * What it does: If a partition has had zero successful updates (no loads, deletes, or compactions) for this long (default 12 hours), StarRocks stops running Vacuum on it entirely to save resources. * The Interaction: * If your partition is "stuck" due to those 
version + 1
errors and hasn't had a successful commit in over 12 hours, it becomes "stale." * The Benefit: Once it's "stale," the Vacuum daemon skips it. This means it stops trying to delete files from that partition, which is actually what you want right now! It prevents further data loss while the partition is in a broken state. * The Recovery: As soon as you fix the bottleneck (by increasing gRPC timeouts and disabling batch publish) and a new load succeeds, the partition is no longer "stale," and the 12-hour timer resets. When would you change it? You would only increase 
lake_autovacuum_stale_partition_threshold
if: * You have tables that are updated very infrequently (e.g., once a day) and you want to make sure Vacuum still cleans up old compaction residues even during the long "quiet" periods. * In your current high-frequency backfill scenario, this isn't your bottleneck. Summary of what to run: Keep the stale threshold at 12h, and focus on these three to stop the immediate "Deadline Exceeded" and data loss cycle:
Copy code
sql
-- 1. Give the FE more time to talk to CNs (Fixes Deadline Exceeded)
ADMIN SET FRONTEND CONFIG ("starmgr_grpc_timeout_seconds" = "60");

-- 2. Stop the buggy version logic (Fixes the +1 != version error)
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- 3. Protect files from being deleted too fast (Fixes Object Not Found)
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "1440");
Rocky's Pro-Tip: After applying these, check 
SHOW BACKLOGS;
or look at your FE 
sys_log
. If the 
PublishVersion
tasks are still failing with the same version mismatch error, a restart of the Leader FE is the fastest way to clear the "stuck" state in the metadata memory-resident ShardManager. References • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md • administration/management/BE_configuration.md
g
@Rocky one last time, these are pure raw tables and queried infrequently, should i definitely enable cache?
r
Even for pure raw tables that are queried infrequently, you should definitely keep the cache enabled (
"datacache.enable" = "true"
). In a StarRocks shared-data (lake) architecture, the cache isn't just for speeding up your SQL queries—it is a critical component for the health of the background system. Here is why you should keep it on for your raw tables: 1. It stabilizes the "Publish" and "Compaction" process When you load data (the backfill you are doing), StarRocks writes the data to the local CN cache at the same time it writes to Azure Blob. * Compaction: Shortly after loading, the system will try to "compact" those raw files. If the cache is on, the CN node can perform this merge using local disk. * If the cache is off: Every compaction task must download that raw data back from Azure Blob over the network. This adds massive overhead to your network and contributes to the 
Deadline Exceeded
gRPC timeouts you are currently fighting. 2. It mitigates your "Object Not Found" risk As you've seen, your cluster is currently struggling with metadata sync. * If the cache is on, a query might still find the data locally on the CN even if there is a transient issue or a delay in communicating with Azure. * If the cache is off, you are 100% dependent on the Azure Blob gRPC call succeeding every single time. Given your current 
Deadline Exceeded
logs, turning off the cache will make your "Object Not Found" errors significantly more frequent. 3. The "Infrequent Query" Solution: 
partition_duration
 If your concern is that these raw tables are "wasting" expensive NVMe disk space on your CN nodes that should be used for "hot" tables, don't turn the cache off. Instead, use the TTL (Time To Live) property:
Copy code
sql
ALTER TABLE my_raw_table SET ("datacache.partition_duration" = "12 HOUR");
* What this does: It keeps the data in the cache for 12 hours (enough time for ingestion and compaction to finish smoothly). * After 12 hours: The data is eligible to be evicted from the disk to make room for other tables. If you query it a week later, it will just fetch from Azure (cold read). Summary Recommendation 1. Keep 
datacache.enable
= 
true
: This ensures your background loads and compactions don't clog your network. 2. Use 
datacache.partition_duration
: Set this to a low value (like 
12 HOUR
or 
1 DAY
) for your raw tables to prevent them from hogging the cache indefinitely. 3. Focus on the FE Configs: The 
starmgr_grpc_timeout_seconds
and 
lake_enable_batch_publish_version
changes I mentioned earlier are the real fix for the "Object Not Found" errors. Turning off the cache would likely make those errors harder to debug. Rocky's Final Word: In StarRocks Lake mode, the CN nodes are designed to be "Cache-First." Disabling it turns them into simple proxies, which often exposes the FE to the exact metadata bottlenecks you are seeing now. References • deployment/shared_data/azure.md • deployment/shared_data/feature-support-shared-data.md • table_design/data_cache.md • administration/management/BE_configuration.md • faq/shared_data_faq.md References • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/s3.md • deployment/shared_data/gcs.md • deployment/shared_data/azure.md
g
@Rocky neat i shouldn't do anything with these tables after 15 days, what about putting the partition_duration at that?
r
Setting 
datacache.partition_duration
to 15 days is an excellent idea for your scenario. It aligns perfectly with how StarRocks manages cache life cycles in a shared-data (lake) environment. Here is why this is the "sweet spot" for your raw tables: 1. How it works (The 15-day Logic) StarRocks uses this property to determine what qualifies as "Hot Data." * Day 0 to 15: When you load data or the system runs background compaction, the data is written to the local NVMe/SSD cache. Since it's within the 15-day window, it is considered "hot." If the cache fills up and this data is evicted, a query will automatically pull it back from Azure and re-cache it. * After Day 15: The data is now "cold." If the cache gets full, StarRocks will prioritize evicting this data to make room for newer partitions. If you happen to query a 20-day-old partition, StarRocks will stream it directly from Azure to the user without saving it to the local disk. This protects your disk space for the data you actually care about. 2. It supports your Backfill/Compaction As we discussed with your 
Deadline Exceeded
issues, the most "violent" part of the data lifecycle is the first 24–48 hours (Loading -> Compaction -> Initial Queries). By setting the duration to 15 days: * You ensure that Compaction always finds the data in the local cache, avoiding massive "read-back" traffic from Azure. * You keep the data "warm" for any immediate validation queries you run after the backfill. 3. Immediate Implementation You can apply this to your existing tables without a restart:
Copy code
sql
ALTER TABLE my_raw_table SET ("datacache.partition_duration" = "15 DAY");
One Final "Rocky" Recommendation: Since you mentioned these are raw tables and you are doing a backfill, also check your Storage Volume settings. If you are using Azure Blob, ensure you haven't set a super aggressive lifecycle policy on the Azure side that might delete files before StarRocks' own 
lake_autovacuum_grace_period_minutes
(which we set to 24 hours) can manage them. Summary: * 
datacache.enable
= 
true
: Keep it on to prevent network bottlenecks. * 
datacache.partition_duration
= 
15 DAY
: Perfect for raw data that "goes cold" after two weeks. It saves your local disk space while keeping the system performant during the active ingestion phase. References • deployment/shared_data/azure.md • table_design/data_cache.md • deployment/shared_data/s3.md • sql-reference/sql-statements/data-definition/ALTER_TABLE.md References • deployment/shared_data/s3.md • table_design/data_distribution/expression_partitioning.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky how can i tell what tables/partitions/etc take up what space in the cn cache?
r
In StarRocks (Lake/Shared-data mode), the Data Cache (also known as Starlet Cache or Block Cache) is managed at a global block level on the CN/BE nodes. Currently, there is no single SQL command that gives a precise "bytes-in-cache" breakdown for every individual table or partition. However, you can get the information you need by combining node-level metrics with table-level configuration metadata. Here is how you can tell what is going on: 1. Check Node-Level Usage (Total Space) To see how much total space is being used on your CN nodes for the cache, use:
Copy code
sql
-- Method 1: High level view
SHOW BACKENDS\G

-- Look for the 'DataCacheMetrics' field. 
-- It shows something like: Status: Normal, DiskUsage: 120.45GB/500.00GB, MemUsage: 2.10GB/16.00GB
Copy code
sql
-- Method 2: System Table view
SELECT * FROM information_schema.be_datacache_metrics;

-- This gives you:
-- DISK_USED_BYTES: The actual cache space used on disk.
-- DISK_QUOTA_BYTES: The maximum allowed cache size for that node.
2. Identify "Eligible" Tables and Partitions Since you can't see the exact bytes cached per table, you instead look for which tables are eligible to be in the cache. In StarRocks, data is cached if 
ENABLE_DATACACHE
is true and it falls within the 
partition_duration
.
Copy code
sql
SELECT 
    DB_NAME, 
    TABLE_NAME, 
    PARTITION_NAME, 
    DATA_SIZE, 
    ENABLE_DATACACHE
FROM information_schema.partitions_meta
WHERE ENABLE_DATACACHE = 1
ORDER BY DATA_SIZE DESC;
* DATA_SIZE: This is the size of the partition in Azure Blob. * If 
ENABLE_DATACACHE
is 
1
and you have queried this partition recently (or it was just loaded/compacted), it is likely residing in the cache up to its full 
DATA_SIZE
. 3. Advanced Monitoring (Grafana) For real-time visibility into which tables are actually being hit in the cache, you should use the StarRocks Grafana dashboard. Look for the "Data Cache" or "Block Cache" section: * Cache Hit/Miss Rate: If you see a high hit rate during a query on a specific table, that table is effectively cached. * Metric Name: 
starrocks_be_block_cache_hit_bytes
and 
starrocks_be_block_cache_miss_bytes
. 4. Summary of the "15-Day" Logic Since you set your 
partition_duration
to 15 days, you can effectively "tell" what is taking up space by running:
Copy code
sql
SELECT 
    TABLE_NAME, 
    SUM(DATA_SIZE) as potential_cache_usage
FROM information_schema.partitions_meta
WHERE ENABLE_DATACACHE = 1
  -- Note: VISIBLE_VERSION_TIME helps you identify if the data is within your 15-day window
  AND VISIBLE_VERSION_TIME > NOW() - INTERVAL 15 DAY
GROUP BY TABLE_NAME;
This query gives you the most realistic estimate of which tables are occupying your CN local disks. Rocky's Tip: If your CN disk usage is unexpectedly high, check for compaction activity. Compaction temporary files are also written to the local disk before being uploaded to Azure, which can sometimes look like "cache" usage. References • data_source/data_cache_observe.md • sql-reference/information_schema/partitions_meta.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md References • sql-reference/information_schema/partitions_meta.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md • sql-reference/information_schema/partitions.md
g
@Rocky this seems problematic
Copy code
W20260511 21:27:01.922827 140161455216192 vacuum.cpp:165] Fail to delete: Internal error: starlet err grpc.GetShard(shardId=27537636) error: Deadline Exceeded
r
This log entry is a "smoking gun" that confirms your cluster is struggling with metadata pressure between the CN nodes and the FE during your backfill. Specifically, the 
vacuum
process (which cleans up old data files from Azure Blob after compaction) is trying to ask the FE for shard details (
GetShard
), but the FE is taking longer than 5 seconds to respond, causing the CN to give up. Why this is happening: 1. Backfill Pressure: You are loading a lot of data. This creates many new "versions" and triggers frequent Compactions. 2. Vacuum Backlog: Every compaction creates "junk" files (the old un-compacted versions). The Vacuum process is trying to delete these to save you money and keep the storage clean. 3. Metadata Bottleneck: The FE is likely busy recording new data and managing transaction logs. When the CN asks for shard info to perform the vacuum, the request gets queued, hits the 5-second limit, and fails. How to fix it: You need to give the FE more "breathing room" for these metadata requests. Run these commands on your FE: 1. Increase the gRPC Timeout The default is 5 seconds. Under heavy backfill, this is often too short. Bump it to 30 or 60 seconds:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("starmgr_grpc_timeout_seconds" = "60");
2. Batch the Publish process (If not done already) This reduces the number of small metadata updates the FE has to handle, which speeds up the FE's response time for other requests like 
GetShard
.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
3. Throttle the Vacuum (Optional) If you see the FE CPU is pinned at 100%, you can tell the CN nodes to be less aggressive about cleaning up old files. On the CN nodes (in 
cn.conf
), you can check/adjust: * `lake_vacuum_worker_count`: (Default is usually 4). If it's higher, lower it to 2 to reduce the number of concurrent requests to the FE. * Note: If the errors persist after changing the timeout, just let it be for a while. The vacuum will retry automatically. Is this "Problematic"? * Short term: No. It won't crash your cluster or corrupt your data. It just means some "junk" files in Azure Blob aren't being deleted immediately. * Long term: Yes. If the vacuum never succeeds, your Azure Blob storage costs will grow because old data versions aren't being removed. Rocky's Advice: Apply the 
starmgr_grpc_timeout_seconds
fix first. It is the most direct solution to the "Deadline Exceeded" error you're seeing in the logs. Once that is increased, those 
vacuum.cpp
errors should disappear. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • administration/management/BE_configuration.md • deployment/shared_data/azure.md References • sql-reference/System_variable.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md
Open in Slack
PreviousNext
Powered by