Hey there, lots of people (including myself) are talking to Rocky. The bot is very good, but I have a question for you guys working with Starrocks: I'm working on a Shared Data instance where we store a lot of different databases (~200). We'd like to understand how to have consistent backups of both data and metadata. From what I gather from the doc, there is only the trash that protects from data loss (a drop database for instance). There's nothing permitting the restoration of the data at a previous point in time. Performing cluster snapshots gives me a consistent metadata snapshot. But if the data changes while the snapshot is done, what happens if I have to restore the snapshot ? I have thought about a way to have a 30 days window of restoration by using AWS Backup: • I perform a daily cluster snapshot and send it to the same bucket as the data, in another folder • I backup the bucket using AWS Backup (triggered when the snapshot is done) => I have a metadata snapshot, and the state of the data at that point in time • I keep 30 days of point in time snapshots of the bucket If I need to restore the data, I just restore the bucket from the backups, and then restore the cluster with the included snapshot. In this case, will I have a consistent state? (metadata + data) Do you have real user feedback on this ?

Yes. In this case, you will have a consistent state across metadata and data. This consistency is part of the cluster snapshot design: when a snapshot is created, StarRocks ensures that the data files required by the corresponding metadata snapshot are retained, and during restore it aligns the data back to that metadata snapshot. So even if new data is written while the snapshot process is running, the restored cluster should recover to a consistent point: metadata and data are matched to the snapshot state.

Thank you @Kate Shao-Community Manager. So let's say I manually trigger the cluster snapshot, wait for it to fiinish, and then I can then create a point in time "freeze" version of the data in the S3 bucket, and this would be consistent with the cluster snap? how much time does the cluster keep this data consistency after the snapshot (considering I'm adding/overwriting data conttinuously)?

Yes — once the cluster snapshot job has finished successfully, the snapshot metadata should match the set of data files that StarRocks needs for that snapshot. So if you then create a point-in-time backup/freeze of the entire bucket, including both the cluster snapshot metadata and the data files, that backup should be restorable to a consistent StarRocks state. The important nuance is that the cluster snapshot itself is not a full independent copy of all data files. It preserves consistency by making sure the data files referenced by the snapshot metadata are not garbage-collected while that snapshot is retained. So the consistency is guaranteed for as long as the cluster snapshot is retained and not deleted/expired, not for an arbitrary fixed time after creation. If you continuously insert/overwrite data after the snapshot, StarRocks will write new files and update newer metadata, but the older files needed by the snapshot should remain protected until the snapshot is removed or expires. Once the snapshot is removed/expired and GC is allowed to clean old files, that point-in-time state is no longer guaranteed from the live bucket alone. Therefore, for your 30-day recovery design, the safest process is: 1. Trigger the manual cluster snapshot. 2. Wait until it completes successfully. 3. Immediately create the S3/AWS Backup point-in-time backup of the whole relevant bucket/prefix. 4. Make sure bucket lifecycle rules or object deletion policies do not remove objects needed by the backup. 5. For restore, restore the bucket/prefix from AWS Backup first, then restore StarRocks using the matching cluster snapshot metadata from that same backup. In that model, the 30-day retention is provided by AWS Backup, while StarRocks provides the metadata/data consistency at the snapshot point.

Thank you, it is clear!