@Rocky How to check backup jobs in progress.

To check the status of backup jobs in StarRocks, you can use the

SHOW BACKUP

command. Check Status via SQL Run the following command to view the last backup task for a specific database:

sql
SHOW BACKUP FROM <db_name>;

Key fields in the output: * State: Shows the current phase of the job. Common "in-progress" states include: * `PENDING`: Job submitted but not started. * `SNAPSHOTING`: Creating data snapshots. * `UPLOADING`: Uploading snapshots to the repository. * `SAVE_META`: Creating local metadata. * `UPLOAD_INFO`: Uploading metadata to the repository. * Progress: Displays the percentage of completion for the snapshot uploading task. * UnfinishedTasks: Shows IDs of subtasks still running during the snapshotting or uploading phases. Important Considerations * Last Job Only: StarRocks only retains information for the most recent backup task in each database. * Concurrency: Only one BACKUP or RESTORE job can run at a time within a single database. * Shared-data Clusters: Note that data backup and restore are currently not supported for StarRocks shared-data clusters. Monitoring Metrics For cluster-wide monitoring, you can check the following FE metric: * `starrocks_fe_unfinished_backup_job`: Indicates the number of currently running backup tasks. References • sql-reference/sql-statements/backup_restore/SHOW_BACKUP.md • administration/management/monitoring/metrics.md • sql-reference/sql-statements/backup_restore/SHOW_RESTORE.md • sql-reference/sql-statements/backup_restore/BACKUP.md • loading/minio.md • loading/azure.md

@Rocky My backup jobs have been failing because of this error rsion 1921) failed tablet:12771608 #version:682 [8561.1 9083.1@681 9083.1] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771614: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771614 #version:687 [8559 9083.1@686 9083.1] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771612: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771612 #version:474 [8718.1 9082.1@472 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771618: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771618 #version:576 [8639 9083@575 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771616: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771616 #version:681 [8564 9083@680 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771622: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771622 #version:706 [8546 9083@705 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771620: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771620 #version:745 [8516.1 9083@744 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771626: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771626 #version:514 [8691 9082.1@513 9082.1] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771624: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771624 #version:670 [8565 9082@668 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771630: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771630 #version:471 [8719 9082@469 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771628: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771628 #version:461 [8727 9082@460 9082] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771634: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771634 #version:703 [8545 9082@702 9082] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771632: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771632 #version:684 [8561.1 9083@683 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771638: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771638 #version:684 [8559 9083@683 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771636: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771636 #version:473 [8719 9082.1@471 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771642: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771642 #version:578 [8639.1 9083@577 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771640: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771640 #version:682 [8564 9083@681 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771646: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771646 #version:709 [8546 9083@708 9083] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12771644: make_snapshot failed. status: Not found: get_applied_rowsets(version 1921) failed tablet:12771644 #version:748 [8516 9083.1@747 9083.1] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)] | [OK] | 86400 | In the logs, it just shows2026-07-16T021020.536Z ERROR Failed to run backup {"error": "1: another StarRocks backup is already running for database \"requests\": snapshot=\"requests_backup_p2026071600\" state=\"SNAPSHOTING\" job_id=\"12789140\""}

The errors you are seeing indicate two separate but related issues: a version consistency failure during the snapshot process and a concurrency conflict preventing new jobs from starting. 1. Understanding the

get_applied_rowsets

Error The error

Not found: get_applied_rowsets(version 1921) failed tablet:... make_snapshot failed

means that by the time the Backend (BE) tried to create a data snapshot for the backup, the specific data version requested by the Frontend (FE) was no longer available. Common Causes: * Aggressive Compaction: StarRocks automatically merges data files (compaction). If compaction finishes and deletes the old "stale" rowsets before the backup job can snapshot them, this error occurs. * High Ingestion Frequency: If you are loading data very frequently (e.g., every few seconds), versions are created and recycled quickly, leaving a very small window for backups to "catch" a consistent version. * Backup Delay: If the cluster is under heavy load, the time between the FE starting the job and the BEs receiving the snapshot request may exceed the rowset retention period. Recommended Fixes: * Increase Stale Rowset Retention: You can give the backup more time to finish snapshotting by increasing how long the BE keeps old rowsets after compaction. Modify the following configuration in your

be.conf

and restart the BE (or use the HTTP API to update it dynamically):

properties
    # Default is 1800 (30 minutes). Try increasing to 3600 or more.
    tablet_rowset_stale_sweep_time_sec = 3600

* Batch Your Loads: If you are doing many small inserts/loads, try to batch them into larger, less frequent transactions to reduce version churn. ────────── 2. Resolving "another StarRocks backup is already running" StarRocks only allows one backup or restore operation per database at a time. Because your previous job failed or got stuck in the

SNAPSHOTING

state, it is blocking new attempts. How to fix: 1. Cancel the stuck job:

sql
    CANCEL BACKUP FROM requests;

2. Verify it is cleared: Run

SHOW BACKUP FROM requests;

and ensure the

State

is no longer

SNAPSHOTING

or

PENDING

. 3. Retry the backup: Once the state is clear, you can start a new backup job. If it continues to fail with the same version error, apply the BE configuration change mentioned above. Summary of Commands Action | Command Check Job Status |

SHOW BACKUP FROM requests;

Stop Stuck Job |

CANCEL BACKUP FROM requests;

Monitor Metrics | Check

starrocks_fe_unfinished_backup_job

on FE. References • sql-reference/sql-statements/backup_restore/BACKUP.md • administration/management/Backup_and_restore.md

@Rocky | 12825502 | requests_backup_p2026071614 | requests | SNAPSHOTING | [requests.requestsPARTITIONS (p2026071614) AS requests] | 2026-07-16 154437 | NULL | NULL | NULL | 12821443=11001, 12821441=11249433, 12821447=203701, 12821445=171908, 12821451=95876, 12821449=11249444, 12821455=203697, 12821453=203699, 12821459=203700, 12821457=7563068, 12821463=7563071, 12821461=110783, 12821467=11001, 12821465=11249433, 12821471=203701, 12821469=171908, 12821475=95876, 12821473=11249444, 12821479=203697, 12821477=203699, 12821483=203700, 12821481=7563068, 12821487=7563071, 12821485=110783, 12821491=11001, 12821489=11249433, 12821495=203701, 12821493=171908, 12821499=95876, 12821497=11249444, 12821503=203697, 12821501=203699, 12821379=95876, 12821377=11249444, 12821383=203697, 12821381=203699, 12821387=203700, 12821385=7563068, 12821391=7563071, 12821389=110783, 12821395=11001, 12821393=11249433, 12821399=203701, 12821397=171908, 12821403=95876, 12821401=11249444, 12821407=203697, 12821405=203699, 12821411=203700, 12821409=7563068, 12821415=7563071, 12821413=110783, 12821419=11001, 12821417=11249433, 12821423=203701, 12821421=171908, 12821427=95876, 12821425=11249444, 12821431=203697, 12821429=203699, 12821435=203700, 12821433=7563068, 12821439=7563071, 12821437=110783, 12821315=203700, 12821313=7563068, 12821319=7563071, 12821317=110783, 12821323=11001, 12821321=11249433, 12821327=203701, 12821325=171908, 12821331=95876, 12821329=11249444, 12821335=203697, 12821333=203699, 12821339=203700, 12821337=7563068, 12821343=7563071, 12821341=110783, 12821347=11001, 12821345=11249433, 12821351=203701, 12821349=171908, 12821355=95876, 12821353=11249444, 12821359=203697, 12821357=203699, 12821363=203700, 12821361=7563068, 12821367=7563071, 12821365=110783, 12821371=11001, 12821369=11249433, 12821375=203701, 12821373=171908, 12821507=203700, 12821505=7563068, 12821511=7563071, 12821509=110783, 12821515=11001, 12821513=11249433, 12821519=203701, 12821517=171908, 12821523=95876, 12821521=11249444, 12821527=203697, 12821525=203699, 12821531=203700, 12821529=7563068, 12821279=203701, 12821533=110783, 12821283=95876, 12821281=11249444, 12821287=203697, 12821285=203699, 12821291=203700, 12821289=7563068, 12821295=7563071, 12821293=110783, 12821299=11001, 12821297=11249433, 12821303=203701, 12821301=171908, 12821307=95876, 12821305=11249444, 12821311=203697, 12821309=203699 | | [12821443: make_snapshot failed. status: Not found: /mnt/starrocks/data1/be/storage/data/314/12821443/113881555/0200000000a0a536694dbe32e0c4544f89a8493c5d387383_0_2.ivt: No such file or directory be/src/fs/fs_util.cpp:30 fs->iterate_dir(path, [&](std::string_view name) { auto full_path = fmt::format("{}/{}", path, name); auto is_dir = fs->is_directory(full_path); if (!is_dir.ok()) { st = is_dir.status(); return false; } if (*is_dir && dirs != nullptr) { dirs->emplace(name); } else if (!*is_dir && files != nullptr) { files->emplace(name); } return true; }) be/src/storage/rowset/rowset.cpp:467 fs::list_dirs_files(src_inverted_file_path, nullptr, &files)], [12821441: make_snapshot failed. status: Not found: /mnt/starrocks/data2/be/storage/data/300/12821441/113881555/02000000009ddb58f24a5f076bbe3d6ffa4f12802feb4f9d_0_2.ivt: No such file or directory be/src/fs/fs_util.cpp:30 fs->iterate_dir(path, [&](std::string_view name) { auto full_path = fmt::format("{}/{}", path, name); auto is_dir = fs->is_directory(full_path); if (!is_dir.ok()) { st = is_dir.status(); return false; } if (*is_dir && dirs != nullptr) { dirs->emplace(name); } else if (!*is_dir && files != nullptr) { files->emplace(name); } return true; }) be/src/storage/rowset/rowset.cpp:467 fs::list_dirs_files(src_inverted_file_path, nullptr, &files)], [12821447: make_snapshot failed. status: Not found: /mnt/starrocks/data2/be/storage/data/291/12821447/113881555/0200000000a1080af149cab38c18b6fac748d7b36cb9bdbf_0_2.ivt: No such file or directory be/src/fs/fs_util.cpp:30 fs->iterate_dir(path, [&](std::string_view name) { auto full_path = fmt::format("{}/{}", path, name); auto is_dir = fs->is_directory(full_path); if (!is_dir.ok()) { st = is_dir.status(); return false; } if (*is_dir && dirs != nullptr) { dirs->emplace(name); } else if (!*is_dir && files != nullptr) { files->emplace(name); } return true; }) be/src/storage/rowset/rowset.cpp:467 fs::list_dirs_files(src_inverted_file_path, nullptr, &files)], [12821445: make_snapshot failed. status: Not found: /mnt/starrocks/data4/be/storage/data/286/12821445/113881555/0200000000a165ad72415fff5b6000f8b21bfcf66f51f8a6_0_2.ivt: No such file or director Snapshotting is still failing

The error

Not found: ... .ivt: No such file or directory

during the

SNAPSHOTING

phase is a more specific version of the

get_applied_rowsets

failure you saw earlier. The

.ivt

file is the Inverted Index file for a specific data rowset. This error means that while the StarRocks Backend (BE) metadata expects an inverted index to exist for a particular tablet version, the physical file is missing from the disk when the backup tries to copy it. Why this is happening This usually occurs due to a "race condition" between your backup job and the Compaction/Garbage Collection (GC) process: 1. Backup Initiation: The Frontend (FE) starts a backup and identifies a specific "consistent version" of the data to snapshot. 2. Compaction: In the background, StarRocks merges small data files (rowsets) into larger ones. Once merged, the old rowsets (and their

.ivt

index files) are marked as "stale." 3. Garbage Collection: After a certain period (defined by

tablet_rowset_stale_sweep_time_sec

), the BE physically deletes these stale files. 4. Snapshot Failure: If the BE takes too long to reach the snapshot step (due to cluster load or high task concurrency), the GC might delete the

.ivt

files before the backup can snapshot them. Recommended Solutions 1. Increase Stale Rowset Retention (Most Likely Fix) You need to give the backup job a larger window to finish snapshotting before files are deleted. Increase the retention time for stale rowsets in your

be.conf

on all BE nodes:

properties
# Increase from default 1800 (30 mins) to 3600 (1 hour) or 7200 (2 hours)
tablet_rowset_stale_sweep_time_sec = 3600

Note: This requires a BE restart to take effect, or you can update it dynamically via the BE HTTP API. 2. Check and Clear the Stuck Job Since your logs show

another StarRocks backup is already running

, you must clear the failed/stuck task before retrying:

sql
CANCEL BACKUP FROM requests;

3. Optimize Ingestion Frequency If you are performing very frequent, small loads (e.g., every few seconds), the cluster generates versions rapidly, making it harder for backups to stay "ahead" of the GC. Try to: * Batch your loads into larger chunks. * Increase the

min_cumulative_compaction_num_singleton_deltas

in

be.conf

to reduce the frequency of compaction triggers. 4. Verify Tablet Health If the error persists even after increasing the sweep time, check if the specific tablets mentioned in your error (e.g.,

12821443

) are actually healthy:

sql
ADMIN SHOW TABLET 12821443;

Look for the

IsBad

column or any version inconsistencies. If the table uses inverted indexes, you can also try to rebuild them or check if the table was recently altered to add indexes while a backup was attempted. Summary The missing

.ivt

file is almost certainly being deleted by StarRocks' internal cleanup before the backup can grab it. Increasing

tablet_rowset_stale_sweep_time_sec

is the standard fix for this behavior during periods of high data churn. References • administration/management/BE_configuration.md • faq/Others.md • administration/management/Backup_and_restore.md • administration/cluster_snapshot.md • sql-reference/sql-statements/backup_restore/BACKUP.md

@Rocky THe issue is not being resolved. WHen I checked the directories, I cant find any IVT files, only one like index.l0.215.0. I don't see any files ending with ivt

Based on the file names and errors you're seeing, there is a mismatch between what StarRocks metadata thinks should exist and what is actually on your disk. 1. The difference between

.ivt

and

index.l0

*

.ivt

files/directories: These are Inverted Index files (used for full-text search). In StarRocks, if you have a

GIN

or

Inverted Index

defined on a table, the system creates these files. *

index.l0

files: These are Persistent Index files for Primary Key tables. They are used to track primary key locations in memory/disk to speed up updates. The Problem: Your backup is failing because it is explicitly looking for Inverted Index (

.ivt

) data to back up, but those files are missing from the tablet directory. 2. Why is this happening? There are three likely scenarios: Scenario A: The Table has a "Ghost" Inverted Index You may have a table that was created with an inverted index (or one was added via

ALTER TABLE

), but for some reason, the index files were never successfully built or were deleted by a system error. * Action: Check your table schema:

sql
    SHOW CREATE TABLE requests;

Look for any

INDEX ... USING GIN

or

INDEX ... USING INVERTED

definitions. If you find one but don't actually need it, dropping the index may resolve the backup issue:

sql
    ALTER TABLE requests DROP INDEX <index_name>;

Scenario B: Inverted Index on Primary Key Table (v3.3 - v4.0) Full-text inverted indexes on Primary Key tables are a very new feature (fully supported in v4.0+). If you are on an earlier version and tried to use them, or if there was a partial failure during a schema change, the metadata might be corrupted. * Action: If this is a Primary Key table and you recently added an index, the index might be in a "Bad" state. Check the tablet health:

sql
    ADMIN SHOW TABLET <tablet_id>; -- Use one from your error, e.g., 12821443

Scenario C: Compaction deleted the files too fast As mentioned previously, if the cluster is under heavy load, the backup job might be trying to snapshot a version of the inverted index that has already been merged/compacted away. However, since you said you never see

.ivt

files, Scenario A or B is more likely. 3. Recommended Steps to Resolve 1. Check for the index: Run

SHOW CREATE TABLE requests

. If you see an inverted index, that explains why the backup is looking for

.ivt

files. 2. Verify file presence: Go to one of the BE paths mentioned in your error:

/mnt/starrocks/data1/be/storage/data/314/12821443/113881555/

Check if there are any directories or files that look like the rowset ID but without the

.ivt

extension. 3. Repair the Index: If the index is supposed to be there but files are missing, you can try to trigger a re-build by dropping and re-adding the index, or use:

sql
    -- If supported in your version
    ALTER TABLE requests REBUILD INDEX <index_name>;

4. Workaround: If you don't need the inverted index, dropping it is the fastest way to get your backups working again. StarRocks will stop looking for the missing

.ivt

files immediately. Note: If you are using a shared-data (cloud-native) cluster, CLucene-based inverted indexes are not supported, which could also lead to initialization issues if someone tried to create one. References • table_design/indexes/inverted_index.md • sql-reference/sql-statements/data-definition/ALTER_TABLE.md • administration/management/Backup_and_restore.md • table_design/indexes/vector_index.md References

@Rocky STARROCKS-CORE-SERVICES-BE-1 34.201.124.123 // (172.30.87.186) mnt starrocks data4 be storage 127 $ pwd /mnt/starrocks/data4/be/storage STARROCKS-CORE-SERVICES-BE-1 34.201.124.123 // (172.30.87.186) mnt starrocks data4 be storage $ find data -type f -name "*.ivt" STARROCKS-CORE-SERVICES-BE-1 34.201.124.123 // (172.30.87.186) mnt starrocks data4 be storage $ THat means no .ivt files exists in the storage path at all be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12821313: make_snapshot failed. status: Not found: get_applied_rowsets(version 2074) failed tablet:12821313 #version:1101 [2169 3100@1100 3100] #pending:0 cost (0/0/1) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12821319: make_snapshot failed. status: Not found: get_applied_rowsets(version 2074) failed tablet:12821319 #version:1119 [2161.1 3100.1@1118 3100.1] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12821317: make_snapshot failed. status: Not found: get_applied_rowsets(version 2074) failed tablet:12821317 #version:709 [2514 3103@708 3103] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12821323: make_snapshot failed. status: Not found: get_applied_rowsets(version 2074) failed tablet:12821323 #version:875 [2381 3100@874 3100] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12821321: make_snapshot failed. status: Not found: get_applied_rowsets(version 2074) failed tablet:12821321 #version:1094 [2187 3100@1093 3100] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12821327: make_snapshot failed. status: Not found: get_applied_rowsets(version 2074) failed tablet:12821327 #version:1175 [2107 3100@1174 3100] #pending:0 cost (0/0/0) be/src/storage/snapshot_manager.cpp:480 tablet->capture_consistent_rowsets(Version(0, snapshot_version), &snapshot_rowsets)], [12821325: make_snapshot failed. status: Not found: /mnt/starrocks/data3/be/storage/data/336/12821325/113881555/0200000000a0ebbd72415fff5b6000f8b21bfcf66f51f8a6_0_2.ivt: No such file or directory be/src/fs/fs_util.cpp:30 fs->iterate_dir(path, [&](std::string_view name) { auto full_path = fmt::format("{}/{}", path, name); auto is_dir = fs->is_directory(full_path); if (!is_dir.ok()) { st = is_dir.status(); return false; } if (*is_dir && dirs != nullptr) { dirs->emplace(name); } else if (!*is_dir && files != nullptr) { files->emplace(name); } return true; }) be/src/storage/rowset/rowset.cpp:467 fs::list_dirs_files(src_inverted_file_path, nullptr, &files)], [12821331: make_snapshot failed. status: Not found: get_applied_rowsets(version 2074) failed tablet:12821331 #version:770 [2463 3103@769 3103] #pending:0 cost (0/0/0) Why is it trying to find any ivt files

The reason StarRocks is looking for

.ivt

files is because Inverted Indexes (full-text search) use this file format. Even if you did not intentionally create an inverted index, the backup process attempts to snapshot all components of a rowset defined in the metadata. Your logs show two distinct failures happening simultaneously across different tablets: 1. Version Mismatch (

get_applied_rowsets failed

): This tablet (e.g.,

12821313

) has already compacted the version the backup was looking for. The data moved faster than the backup snapshot could complete. 2. Missing Index File (

.ivt: No such file or directory

): This tablet (e.g.,

12821325

) has metadata that indicates an inverted index exists (or the code is incorrectly attempting to find one), but the physical file is missing from the disk. Why is it looking for

.ivt

files if you don't have them? There are a few specific reasons this happens in StarRocks (especially in versions 3.3.0 and later): 1. "Ghost" Inverted Indexes If you at any point ran a command to add an index (even if it failed or was partially interrupted), or if the table was created from a template that included one, the FE metadata might still think the index exists. * Action: Run

SHOW CREATE TABLE requests;

. Check if there is any line starting with

INDEX ... USING GIN

or

INDEX ... USING INVERTED

. If it exists, drop it:

sql
    ALTER TABLE requests DROP INDEX <index_name>;

2. Vector Index or Beta Features Are you using Vector Search (

VECTOR

index)? Vector indexes in some StarRocks versions share parts of the inverted index infrastructure. If a vector index is corrupted or partially built, it might trigger the search for

.ivt

files. 3. Bug in Version 3.x / 4.x Snapshot Logic There is a known edge case in some v3.x releases where the

make_snapshot

process incorrectly attempts to list or copy an inverted index directory even for tables that don't have one, or fails to handle the absence of the directory gracefully if the rowset is in the middle of being merged. Recommended Troubleshooting Steps Step 1: Confirm the Table Schema Run the following and check for any index definitions other than the Primary Key:

sql
SHOW CREATE TABLE requests;

Step 2: Check for Indexing Jobs Check if there are any background index building tasks that are stuck or failed:

sql
SHOW ALTER TABLE COLUMN;

Step 3: Increase Retention to bypass the "get_applied_rowsets" error Since you are seeing both errors, the first thing to stabilize is the version retention. This prevents the "Not found: get_applied_rowsets" error and gives the BE more time to handle the file snapshotting. In

be.conf

on all BE nodes:

properties
tablet_rowset_stale_sweep_time_sec = 7200

(After changing this, restart the BE or apply it via the HTTP API). Step 4: Validate Tablet Consistency Pick one of the tablets failing with the

.ivt

error (like

12821325

) and check its status:

sql
ADMIN SHOW TABLET 12821325;

If the tablet is "Bad" or has a version mismatch in the

SHOW TABLET

output, you may need to trigger a repair:

sql
ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "12821325", "be_id" = "<BE_ID>", "status" = "bad");

This will force StarRocks to re-clone a healthy replica from another BE. Summary: If

SHOW CREATE TABLE

shows no inverted index, you are likely hitting a bug where the backup logic is over-eager. Upgrading to the latest maintenance release of your major version (e.g., moving from 3.3.0 to 3.3.5) often resolves these snapshot race conditions. References • administration/management/Backup_and_restore.md • sql-reference/sql-statements/backup_restore/BACKUP.md • faq/Exporting_faq.md

@Rocky I currently have an array<INT> that is being looked up using array_contains. What kind of index can I introuduce to optimize lookup?

To optimize lookups for an

ARRAY<INT>

column (specifically

array_contains

), the most effective approach in StarRocks currently depends on your version and use case. 1. Inverted Index (GIN) - Primarily for Strings StarRocks introduced Inverted Indexes (GIN) in version 3.3. However, it currently primarily supports

CHAR

,

VARCHAR

, and

STRING

data types. It does not natively support the

ARRAY

data type for GIN indexes in the current stable releases (v3.3 - v3.5). If you were using a string column that contained space-separated or comma-separated values, you could use GIN, but for a true

ARRAY<INT>

, this index is not currently applicable. 2. Generated Columns (Recommended Workaround) If you frequently query for a specific value or a specific property of the array, you can use Generated Columns (supported since v3.1). * How it works: You create a virtual column that pre-calculates a boolean or a specific value from the array. * Benefit: You can then create a Bitmap Index on that generated column to make the filter near-instant.

sql
-- Example: Pre-calculating if an array contains the value '100'
ALTER TABLE your_table ADD COLUMN contains_100 BOOLEAN AS (array_contains(your_array_column, 100));

-- Add a bitmap index to the boolean column
CREATE INDEX idx_contains_100 ON your_table (contains_100) USING BITMAP;

3. Schema Design: The "Mapping Table" Strategy If your arrays are large and

array_contains

performance is a bottleneck for arbitrary value lookups, the most performant "StarRocks-native" way is to normalize the data: 1. Create a separate Duplicate Key table that maps

id -> element

. 2. Create a Bitmap Index on the

element

column in that table. 3. Join this table back to your main table. Bitmap indexes in StarRocks are highly optimized for high-cardinality point lookups on scalar columns (like

INT

), making this approach significantly faster than scanning array elements. 4. Query-Level Optimization:

UNNEST

+

IN

In some scenarios (especially in version 3.4+), users have reported that rewriting the query to use

UNNEST

in a subquery or CTE can sometimes leverage the optimizer better than a raw

array_contains

call, though this is more of a query hint than a physical index optimization. Summary Index Type | Supported for

ARRAY

? | Best Use Case Inverted Index (GIN) | No (Strings only) | Full-text search on VARCHAR. Bitmap Index | No | Scalar columns (INT, VARCHAR, etc.). Generated Column | Yes | Accelerating specific, frequent array checks. Bloom Filter | No | Scalar equality checks. Verdict: If you need to optimize arbitrary lookups on an

ARRAY<INT>

, consider normalizing the elements into a separate table with a Bitmap Index, or use a Generated Column if the search criteria are predictable. References • sql-reference/sql-functions/array-functions/array_contains.md • sql-reference/sql-functions/array-functions/array_contains_all.md • sql-reference/sql-functions/array-functions/array_contains_seq.md • knowledge/usage/best-practice.md • table_design/indexes/inverted_index.md • sql-reference/sql-functions/json-functions/json-query-and-processing-functions/json_contains.md

@Rocky Isn't this a bug in starrocks v4.1.3? Because the built in GIN is built into the .dat segment files, but the backup is looking for a CLucene style inverted index which is stored in an .ivt file. So the issue is the backup is assuming all GIN requires an ivt file but that is not the case for the latest native GIN?

Yes, you have identified a very specific and likely valid bug in the StarRocks v4.1.x snapshot logic for the new Built-in Inverted Index. Why this is happening In StarRocks, there are now two distinct implementations of the Inverted Index (GIN), and they handle files very differently: 1. CLucene Implementation (Default in Shared-Nothing): * Files: Creates a separate directory and files ending in

.ivt

(e.g.,

rssid_0_2.ivt

). * Mechanism: Uses the external CLucene library to manage index files outside of the standard segment format. 2. Built-in Implementation (New in v4.1, Native): * Files: Does not use

.ivt

files. Instead, it is integrated into the

.dat

segment files (leveraging StarRocks' native bitmap-style indexing). * Mechanism: A native C++ implementation designed to work with both Shared-Nothing and Shared-Data (Cloud Native) architectures. The Bug in v4.1.3 The error you are seeing—`No such file or directory ... .ivt`—indicates that the

BACKUP

(snapshot) logic is still following the old "CLucene" path. It is probing the tablet directory for

.ivt

files because it sees a GIN index in the metadata, failing to realize that a Built-in index doesn't create those files. This is a known "growing pain" of the v4.1 release where the backup subsystem hasn't been fully decoupled from the assumption that "GIN = CLucene = .ivt files." How to Confirm Check your table schema to see which implementation is being used:

sql
SHOW CREATE TABLE <table_name>;

Look at the

INDEX

definition. If you see

"imp_lib" = "builtin"

, or if you are on a Shared-Data cluster (where

builtin

is the only supported type), then the system should absolutely not be looking for

.ivt

files. Recommended Actions 1. Check for "Ghost" Indices: If you recently switched implementation types or upgraded, verify that there isn't a stale index definition in the FE metadata. 2. Workaround: Since the Built-in GIN index is currently in Beta for v4.1, the most stable workaround to get your backups running is to: * Drop the index:

ALTER TABLE <name> DROP INDEX <index_name>;

* Run your backup. * Re-create the index if needed for queries. 3. Upgrade/Patch: Check the StarRocks GitHub Releases for versions v4.1.4+ or v4.2. Fixes for the snapshot logic of native GIN are high-priority items in these maintenance cycles. 4. Report to Community: If you are on the StarRocks Slack, this is a perfect case to provide to the maintainers (mentioning that

make_snapshot

is incorrectly looking for

.ivt

for

imp_lib=builtin

). Summary: You are correct. The backup process is incorrectly assuming the existence of CLucene-style

.ivt

files for a native built-in index that stores its data inside the

.dat

segments. References • table_design/indexes/inverted_index.md • release_notes/release-3.2.md • administration/management/Backup_and_restore.md • table_design/indexes/vector_index.md References • table_design/indexes/inverted_index.md • table_design/indexes/vector_index.md

@Rocky How do I delete the index then

To delete the index (Inverted, Bitmap, or Vector), you can use the

ALTER TABLE

syntax. Step 1: Find the Index Name If you don't know the exact name of the index, run:

sql
SHOW CREATE TABLE <table_name>;

Look for a line that starts with

INDEX index_name (column_name) ...

. Step 2: Drop the Index Use the following command:

sql
ALTER TABLE <table_name> DROP INDEX <index_name>;

Alternatively, for most index types, you can also use:

sql
DROP INDEX <index_name> ON <table_name>;

Important Notes: * Asynchronous Operation: Dropping an index is usually fast because it primarily updates metadata, but the physical cleanup of the data files (or the removal of references in the

.dat

segments for native GIN) happens in the background. * Verify: You can verify it is gone by running

SHOW CREATE TABLE <table_name>;

again. * Backup Bug: Once the index is dropped, the metadata that triggers the

make_snapshot

process to look for

.ivt

files will be removed. This should allow your

BACKUP

or snapshot process to complete without the "No such file or directory" error you were seeing. If you are trying to resolve the specific bug in v4.1.3 where the system looks for non-existent

.ivt

files for a native built-in index, dropping and then recreating the index (if necessary after the backup) is the standard workaround until you can upgrade to a version with the fix. References • sql-reference/sql-statements/table_bucket_part_index/DROP_INDEX.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • table_design/indexes/vector_index.md • using_starrocks/SQL_plan_manager.md • using_starrocks/Cost_based_optimizer.md