<@U0A71G31CDV> are there known bugs or did youl ha...
# questions-and-troubleshooting
e
@Rocky are there known bugs or did youl have bugs where inserting/updating a row of a table in iceberg s3 caused data to get deleted/modify the row not how was asked (other columns, other values and so on) ?
r
Based on known issues and documentation for StarRocks and Iceberg, there are a few specific scenarios where you might encounter data inconsistencies, "silent" data loss, or unexpected row behavior. 1. Silent Data Loss with Equality Deletes (Bug #73901) There is a documented bug regarding equality deletes in partitioned Iceberg tables. If an Iceberg table uses equality deletes and the partition column is not part of the equality identifier keys, StarRocks may incorrectly apply those deletes across all partitions during a read operation. * Effect: Rows that should be live are "dropped" from the query result, appearing as if they were deleted even though they still exist in the storage. * Context: This primarily affects reads of Iceberg v2 tables that have been modified by engines (like Flink or Spark) using equality-based deletes. 2. Stale Data Due to Metadata Caching (#51157, #65827) A common issue involves the StarRocks Frontend (FE) metadata cache. If an Iceberg table is updated frequently by an external engine (e.g., Spark), StarRocks might continue to use a stale snapshot for a period of time (sometimes 10–20 minutes, or until an FE restart in some bug scenarios). * Effect: You might not see newly inserted rows, or you might see rows that were recently deleted/overwritten. * Resolution: Frequent use of
REFRESH EXTERNAL TABLE <table_name>
is usually required, although some bugs prevented this from working correctly in versions prior to 4.0.6/4.1. 3. Column Sequencing and "Corruption" (#53494) There is a known issue when reading Iceberg tables where different Parquet files have different column sequences (schema evolution). * Effect: StarRocks might fail to decode the Parquet page header or return errors like
Corruption: Failed to decode parquet page header
. While the data isn't physically deleted, it becomes unreadable by StarRocks until the layout is unified. 4. Support for UPDATE and MERGE INTO It is important to note that native
UPDATE
and
MERGE INTO
support for Iceberg tables is not available in current stable versions
(up to v4.1). * Users typically perform updates via
INSERT OVERWRITE
or by using external engines like Spark. * StarRocks v4.1 added support for
DELETE
on Iceberg, but
UPDATE
is currently targeted for v4.2. 5. Fixed I/O Sink Errors (#46979) In earlier versions (fixed in v3.2.15), there was a bug in how StarRocks handled I/O errors during a sink (
INSERT INTO
) to Iceberg. * Effect: If a network or storage error occurred during the write, the metadata commit might fail or become inconsistent, potentially leading to data that was written but not tracked, or vice versa. Summary of Recommendations: * Check your version: Ensure you are on at least v3.2.15 (for 3.x) or v4.0.6+ (for 4.x) to avoid fixed sink and caching bugs. * Refresh Metadata: If you see "missing" or "old" data after an external update, run
REFRESH EXTERNAL TABLE <table_name>
. * Avoid Equality Deletes on Partitioned Tables: Until bug #73901 is fully resolved in your version, be cautious with Iceberg tables that rely heavily on equality deletes across different partitions. If you are experiencing specific "modified but not how asked" behavior (e.g., wrong values in columns), this is less common and often related to schema mapping issues during an
INSERT INTO
where columns are mapped by position rather than by name. Ensure your
SELECT
statement in an
INSERT
matches the Iceberg table's column order exactly.
Yes, there have been documented bugs and specific architectural behaviors in StarRocks that could lead to data being incorrectly deleted, duplicated, or appearing as stale/incorrect when performing DML (Insert/Update/Delete) on Iceberg tables. The most relevant issues reported are: 1. Silent Data Loss: Over-deletion across partitions (#73901) A significant bug was identified where StarRocks' implementation of Equality Deletes could cause "silent data loss." * The Issue: If you have a partitioned Iceberg table and perform a delete using an equality key that does not include the partition column, StarRocks might apply that delete to all partitions instead of just the intended one. * Result: Rows with matching keys in other partitions are incorrectly filtered out (deleted) during reads. This happens because the
IcebergEqualityDeleteRewriteRule
in some versions lacks proper partition scoping. 2. Stale Data due to Data Cache If you update or overwrite an Iceberg table using an external engine (like Spark) and then query it in StarRocks, you might see "incorrect" (old) values if the Data Cache is enabled. * The Cause: StarRocks uses file names and modification times to manage its cache. Since Iceberg files are immutable and the metadata modification time can sometimes be reported as 0, StarRocks may incorrectly treat a new manifest/data file as a cached version of an old one. * Fix/Workaround: Disable the Data Cache (
SET enable_scan_datacache = false;
) or manually run
REFRESH EXTERNAL TABLE <table_name>;
. 3. Duplicate Data Bug (#44753) Users have reported seeing duplicate records in Iceberg v2 tables when using
upsert
or
merge
logic from Spark. This is often related to how StarRocks handles the "Merge-on-Read" (MoR) delete files. If the delete files (position or equality) are not interpreted correctly by the StarRocks scanner, both the original "deleted" row and the new "updated" row may be returned simultaneously. 4. DML Support Limitations It is important to note that
UPDATE
and
MERGE INTO
for Iceberg were historically not supported or were highly experimental in StarRocks. * In older versions, users were forced to use
INSERT OVERWRITE
(which replaces entire partitions) or a manual
DELETE
followed by an
INSERT
. *
DELETE
support for Iceberg was only fully introduced in more recent versions (e.g., v4.1). Using these operations in older versions often required specific configurations and had known edge cases where column mappings could fail if the schema had evolved. 5. NULL values in Equality Deletes (#67321) There were instances where rows deleted using an equality condition on a column containing NULL values were not correctly masked, causing deleted data to reappear in query results. Recommendation If you are experiencing data integrity issues: 1. Check your version: Many of these fixes (especially for over-deletion and NULL handling) are in v3.2.10+, v3.3.3+, and v4.x. 2. Verify the Cache: Try running
SET enable_scan_datacache = false;
to see if the "incorrect" data is actually just a stale cache. 3. Inspect Metadata: Use the metadata tables to see if the delete files exist:
Copy code
sql
    SELECT * FROM <catalog>.<db>.<table>$files WHERE content != 'DATA';
4. Community Support: For specific "wrong value" bugs where columns seem swapped or values are mutated, these are often linked to Schema Evolution mismatches. Ensure your StarRocks catalog is fully refreshed. References • data_source/feature-support-data-lake-analytics.md • data_source/catalog/iceberg/DML.md • data_source/catalog/iceberg/iceberg_catalog.md