@Rocky We are seeing inconsistent results across replicas after upgrading to

4.1.3-8a8e186

recently The affected table is a native OLAP Primary Key table with:

PRIMARY KEY (tenant_id, record_id)
DISTRIBUTED BY HASH(tenant_id) BUCKETS 24
replication_num = 3
replicated_storage = true
enable_persistent_index = true

A query filtering by tenant, phone, and a non-null timestamp sometimes returns the expected row and sometimes returns zero rows:

SELECT record_id
FROM <table>
WHERE tenant_id = '<tenant-id>'
  AND phone = '<phone>'
  AND created_at IS NOT NULL;

Profiling tied the result to a specific replica of tablet `175346`:

be-0 / BackendId 10004: returns row
be-1 / BackendId 10002: returns row
be-2 / BackendId 10001: returns zero rows

All replicas report

NORMAL

, the same visible version, no failed version, and no error state. The row is present on `be-2`: a Primary Key lookup and a scan without the phone predicate both return the row and correct phone value. The problem occurs only when the string predicate is evaluated through predicate-column late materialization. With the default setting:

SET enable_predicate_col_late_materialize = true;
Query ID: 01a04490-e388-7fd6-b837-f3c117d35264
Backend: be-2
RawRowsRead: 26056
PredFilterRows: 26056
Result: zero rows

With the optimization disabled:

SET enable_predicate_col_late_materialize = false;
Query ID: 01a04490-f5ec-763d-8c6a-e5f410241c08
Backend: be-2
RawRowsRead: 26056
PredFilterRows: 26055
Result: expected row

We repeated this multiple times, and disabling late materialization consistently fixes the result. Query cache was disabled. Our current workaround is:

SET GLOBAL enable_predicate_col_late_materialize = false;

This looks related to the page/dictionary filtering paths in PRs

#64600

and

#68258/#68348

, although those fixes appear to be included in the 4.1 branch. Could you confirm whether this is a known issue in 4.1.3, whether a patch is available, and whether rebuilding tablet replica

175346

on BackendId

10001

is recommended? We can provide additional profiles, BE logs, or tablet metadata privately.

It appears you are encountering a highly specific edge-case bug in the predicate-column late materialization logic, likely triggered by the interaction between the data layout in your Primary Key table and the optimizations introduced in recent versions. Current Status of the Issue Based on your observation that disabling

enable_predicate_col_late_materialize

consistently fixes the result, this confirms the issue is within the optimization's execution path (likely involving dictionary-encoded page filtering or segment-level pruning) rather than data corruption. While the PRs you mentioned (

#64600

,

#68258

, and

#68348

) were intended to optimize and fix issues in the page/dictionary filtering paths, the behavior you are seeing on 4.1.3-8a8e186 suggests a regression or a remaining edge case specifically affecting Primary Key tables with replicated storage and persistent indexes. Recommendations 1. Workaround As you have already discovered, the most reliable workaround currently is to disable the optimization at the global or session level:

sql
SET GLOBAL enable_predicate_col_late_materialize = false;

This bypasses the late materialization path, ensuring that all predicates are evaluated correctly during the initial scan. 2. Rebuilding Tablet Replica

175346

Regarding your question about rebuilding the replica on

BackendId 10001

(be-2): * Is it recommended? In this specific case, likely not as a permanent fix. * Why? Since you mentioned that a PK lookup and a scan without the

phone

predicate both return the correct data on

be-2

, the row physically exists and the persistent index is healthy. Rebuilding (marking the replica as

Bad

to trigger a clone) will likely result in the same segment files being downloaded (due to

replicated_storage = true

). If the bug is triggered by the physical layout of those specific segments (e.g., a specific dictionary state), the new replica will likely exhibit the same behavior. * When to try it? Only if you suspect local disk corruption on that specific BE, but your profiling evidence (late materialization dependency) points strongly toward a software logic bug. 3. Investigation & Reporting Because this version is very recent, this may be a newly discovered regression in the 4.1.3 release. * Check for 4.1.4: Monitor the StarRocks releases for a 4.1.4 version, as fixes for regressions found in 4.1.3 are typically prioritized there. * Provide Diagnostics: The StarRocks engineering team will likely need the Query Profile (which you already have) and potentially the segment metadata or the specific segment file for tablet

175346

from

be-2

to reproduce the issue. * Next Steps: Since you have detailed profiles and backend IDs, it is highly recommended to: 1. Open a GitHub Issue if you haven't already. 2. Share the Profile and BE logs in the #support or #general channel of the StarRocks Community Slack, tagging maintainers involved in the Storage/Execution engine. Summary * Confirmed: It is a bug in the late materialization optimization path. * Patch: No official patch beyond 4.1.3 is confirmed yet for this specific scenario; continue using the workaround. * Rebuild: Likely unnecessary, as the data itself is present and correct; the issue lies in the filtering logic during query execution. References • administration/management/BE_configuration.md • sql-reference/sql-statements/dictionary/CREATE_DICTIONARY.md • using_starrocks/query_acceleration_with_auto_increment.md

@Rocky How can I report this issue?

Given the level of detail you have already gathered, you should report this through the following official channels to ensure it reaches the StarRocks storage and execution engine maintainers. 1. File a GitHub Issue (Primary) The most effective way to track this is by opening a bug report on the StarRocks GitHub Repository. When filing the issue, please include the following details from your investigation: * Version:

4.1.3-8a8e186

* Table Schema: The DDL you provided (Primary Key table, replication, persistent index). * Reproduction Query: The specific

SELECT

with the phone predicate. * The Key Finding: Explicitly state that

SET enable_predicate_col_late_materialize = false

consistently returns the correct result, while

true

returns zero rows. * Comparative Profiles: Attach or link the query profiles for both the failing and succeeding cases. The discrepancy between

RawRowsRead

and

PredFilterRows

on

be-2

is critical evidence. * Tablet/BE Details: Mention tablet

175346

and Backend

10001

(be-2) as the specific location of the inconsistent replica. 2. Join the StarRocks Community Slack For faster visibility and real-time follow-up with the engineers, join the StarRocks Slack. * Post a summary in the #support or #general channel. * Include a link to the GitHub issue you created. * Mention that you can provide the BE logs and tablet metadata if requested. 3. Information for Maintainers Since you mentioned you can provide BE logs and tablet metadata privately, be prepared to share: * Segment Files: If requested, providing the specific segment files for tablet

175346

from

be-2

will allow developers to reproduce the filtering bug locally. *

meta

file: The tablet metadata for that specific tablet. By providing the specific comparison between the two

enable_predicate_col_late_materialize

settings, you have already pinpointed the bug to the vectorized filtering logic, which will significantly speed up the fix. References • developers/How_to_Contribute.md • developers/development-environment/dev-env-setup.md • project_help/slack.md