@Rocky We are running a staging StarRocks cluster on version 4.0.5 and recently set up a new production cluster on version 4.1. We attempted to migrate a table from staging to production using the standard BACKUP and RESTORE commands:

CREATE TABLE test.test_table (
  id varchar(36) NOT NULL,
  metadata VARCHAR(65533) NULL
) ENGINE=OLAP
PRIMARY KEY (id)
DISTRIBUTED BY HASH(id)
PROPERTIES (
  "replication_num" = "3"
);

BACKUP DATABASE test SNAPSHOT test_table
TO migration_repo;

RESTORE SNAPSHOT test_table
FROM migration_repo
  DATABASE test
  PROPERTIES("backup_timestamp" = "2026-06-01-02-01-31-035");

SHOW RESTORE;

After the restore process completed, we started noticing data consistency issues with queries targeting a specific column (metadata). When running

SELECT id, metadata
FROM test.test_table
WHERE metadata IS NULL
ORDER BY id;
+------+----------+
| id  | metadata |
+------+----------+
| 1024 | NULL   |
+------+----------+

The query returns rows, and in those specific results, the metadata column is displayed as NULL. However, if we take an id from those exact results and query it directly, the metadata column is actually fully populated with its real data

SELECT id, metadata 
FROM test.test_table 
WHERE id = 1024;
+------+---------------------------------------+
| id  | metadata               |
+------+---------------------------------------+
| 1024 | {"user_id": 5501, "status": "active"} |
+------+---------------------------------------+

We resolved this issue by creating a fresh table and repopulating it using an INSERT INTO ... SELECT * statement from the restored table. What was the problem here? Is recreating the table via INSERT INTO ... SELECT a safe and correct way to resolve this problem?

The issue you encountered—where data appears as

NULL

when filtered but is correctly populated when looked up by ID—is a known symptom of metadata or index inconsistency (specifically regarding Zone Maps or Column Statistics) that can occur during cross-version migrations of Primary Key tables. What caused this problem? The root cause is likely a compatibility gap in how StarRocks v4.1 interprets or rebuilds the metadata of Primary Key tables backed up from v4.0.5. 1. Engine Overhaul in v4.1: StarRocks v4.1 introduced significant changes to the Primary Key storage engine, including a new tablet layout, parallel compaction, and redesigned persistent index mechanisms. 2. Stale Zone Maps: In StarRocks, each data segment stores "Zone Maps" (metadata such as min/max values and null flags) to accelerate queries. When you run

WHERE metadata IS NULL

, the engine uses these Zone Maps to decide which rows to return. If the restore process failed to correctly migrate or refresh these maps, the query executor might rely on stale metadata from the 4.0.5 snapshot that incorrectly flags rows as containing

NULL

. 3. PK Index vs. Scan: When you query by

id = 1024

, StarRocks uses the Primary Key Index to jump directly to the latest physical version of that row. Since the actual raw data was restored correctly, the point lookup succeeds. The inconsistency only appears during "scans" that rely on the secondary column-level metadata. Is the

INSERT INTO ... SELECT

fix safe and correct? Yes, this is a safe and recommended way to resolve the issue. * Why it works: A

SELECT *

on the restored table forces the engine to resolve the latest version of every row using the Primary Key Index. By piping this into a brand-new table (

INSERT INTO

), you are forcing the v4.1 cluster to generate fresh, native metadata, Zone Maps, and Primary Key Indexes. * Safety: As long as the

SELECT *

(without filters) returns the correct total row count and data values, the new table will be a perfectly consistent and healthy v4.1 native table. Recommendations for Future Migrations * Verify with

ANALYZE

: If you encounter this again, you can try running

ANALYZE TABLE <table_name>

to see if refreshing the statistics resolves the query behavior, though for structural metadata issues like Zone Maps, a table rewrite is more reliable. * Version alignment: For production migrations between major versions (like 4.0 to 4.1), it is often safer to perform a fresh

INSERT INTO ... SELECT

via a Cross-Cluster Data Migration (using the

pipe

or

External Table

approach) rather than

BACKUP/RESTORE

, as it avoids potential binary-level metadata incompatibilities. * Check Persistent Index: Ensure that

enable_persistent_index

is set consistently. If you are on a shared-data cluster, v4.1 changed the default behavior for cloud-native persistent indexes. For further verification or if you notice this recurring, we recommend checking the FE/BE logs for any "tablet meta" or "index version mismatch" warnings during the restore process, or reaching out to the StarRocks community with the specific BE logs from/to build IDs for individual tablet health status of the tablets (

SHOW TABLET ABLET FROM <tablet_id>

). References • table_design/table_types/primary_key_table.md • faq/loading/Loading_faq.md • best_practices/primarykey_table.md