@Rocky Can we confirm whether our StarRocks/Flink connector version is expected to support the combination of

sink.ignore.update-before=false

and

sink.properties.merge_condition='ts_ns'

for PK tables? In our flink

ingestion

pipeline, enabling both causes merge-commit load aborts with

INTERNAL_ERROR: delete with condition column ts_ns

(label

merge_commit_*

), which suggests

UPDATE_BEFORE

rows (treated as DELETE) are incompatible with conditional merge logic. Given we need: • delete old PK row when one of the PK time col changes (`UPDATE_BEFORE`behavior), and • stale-event protection via`ts_ns`ordering (`merge_condition`behavior), what is the recommended supported pattern here?

Based on your description and the current StarRocks documentation, we can confirm that the combination of

sink.ignore.update-before=false

and

sink.properties.merge_condition

is not currently supported for Primary Key tables. Why this happens 1. Operation Conflict: When

sink.ignore.update-before

is set to

false

, the Flink connector translates

UPDATE_BEFORE

records into DELETE operations (

__op=1

). 2. Conditional Update Limitation: In StarRocks, the

merge_condition

property is designed for "Conditional Updates" (stale-event protection for Upserts). The StarRocks backend explicitly does not support conditional checks for DELETE operations by default. 3. The Error: When a load batch contains both a

merge_condition

and a

DELETE

operation (from the

UPDATE_BEFORE

row), the BE (Backend) returns the

INTERNAL_ERROR: delete with condition column ...

because it cannot apply version-based logic to a hard delete. Recommended Patterns Given you need both PK change cleanup and stale-event protection, consider the following alternatives: 1. Use "Logical Deletes" (Best for high-consistency needs) Instead of relying on the connector to issue a hard

DELETE

for the old PK: * Add a

is_deleted

(boolean) or

__op

column to your schema. * In Flink, transform

UPDATE_BEFORE

messages into an

UPSERT

where

is_deleted = true

. * Since every message is now an

UPSERT

, the

merge_condition='ts_ns'

will work correctly for both "live" updates and "deletion" updates. _ _Note: You will need to filter out rows where

is_deleted = true

in your StarRocks queries or via a View.* 2. Enable

enable_delete_with_condition

(For StarRocks v3.5+) A dynamic BE configuration parameter,

enable_delete_with_condition

, was recently introduced to allow conditional deletes. If you are on a very recent version (v3.5.x or later), you can try setting this on your BE nodes:

bash
curl -X POST http://<be_host>:<be_http_port>/api/update_config?enable_delete_with_condition=true

Note: This is a relatively new feature; verify its stability in your specific version's release notes before use in production. 3. Use Flink Materialization + Serial Loading If you cannot change your schema or use v3.5+, you may have to handle ordering in Flink rather than the DB: * Set

sink.ignore.update-before = false

. * Disable

sink.properties.merge_condition

. * Ensure in-order delivery by setting

sink.merge-commit.max-concurrent-requests = 0

(or

1

) to prevent concurrent out-of-order writes from the same subtask. * Use Flink's

upsert-materializer

(automatically enabled in many Flink SQL plans) to ensure

-U

and

+U

are correctly ordered before the sink. _ _Risk: This handles out-of-order within the Flink pipeline but does not protect against stale events arriving from external sources compared to data already in the DB.* Summary The

INTERNAL_ERROR

confirms that the backend is rejecting the

DELETE

operation because it contains a versioning condition. For immediate resolution, the Logical Delete pattern is the most robust way to support both PK changes and version-based stale-event protection simultaneously. References • sql-reference/sql-statements/table_bucket_part_index/DELETE.md • loading/Load_to_Primary_Key_tables.md

@Rocky Is there any other recommended approaches as the ones mentioned may not be possible for us?

The combination of

sink.ignore.update-before=false

and

sink.properties.merge_condition

is not supported for Primary Key (PK) tables in StarRocks. This is a limitation of the StarRocks storage engine:

DELETE

operations (including

UPDATE_BEFORE

) cannot be used with conditional updates. When

sink.ignore.update-before

is

false

, Flink sends a

DELETE

operation for the old record when a primary key changes. Since the

merge_condition

header is applied to the entire load batch, StarRocks encounters a

DELETE

row while a condition is active and aborts the load with the

INTERNAL_ERROR

you observed. Since the standard approaches (sequential loading or soft deletes) are not feasible for you, here are two alternative patterns to achieve both PK change handling and stale-event protection: 1. Flink-Side Deduplication (Recommended) Instead of relying on the StarRocks engine to handle out-of-order data via

merge_condition

, perform the versioning logic within Flink. This allows you to remove the

merge_condition

from the sink while still ensuring only the latest data reaches StarRocks. * Logic: Use Flink’s

DEDUPLICATE

(or a

ProcessFunction

with state) to keep only the record with the maximum

ts_ns

for each primary key. * Sink Config: *

sink.ignore.update-before = false

(to handle PK changes). * Remove

sink.properties.merge_condition

. * Benefit: StarRocks will process the retractions (

UPDATE_BEFORE

) successfully because there is no

merge_condition

header, and Flink ensures that stale records are filtered out before they even leave the pipeline. 2. Dual Sink Routing If you must use

merge_condition

for updates but need the

DELETE

for PK changes, you can split the stream into two different StarRocks sinks in your Flink job. * Logic: 1. Sink A (Updates): Route

INSERT

and

UPDATE_AFTER

rows here. Enable

sink.properties.merge_condition = 'ts_ns'

. 2. Sink B (Deletes): Route

UPDATE_BEFORE

and

DELETE

rows here. Do not enable

merge_condition

. * Flink SQL Example:

sql
    -- Updates Sink
    INSERT INTO sr_sink_updates
    SELECT * FROM source WHERE ROW_KIND() IN ('+I', '+U');

    -- Deletes Sink
    INSERT INTO sr_sink_deletes
    SELECT * FROM source WHERE ROW_KIND() IN ('-D', '-U');

* Trade-off: While this prevents the

INTERNAL_ERROR

, it means your

DELETE

operations are no longer protected by the

ts_ns

version check. In most PK-change scenarios, this is acceptable because the

UPDATE_BEFORE

is meant to clean up a specific old PK value. 3. Consider Unique Key Tables If native server-side stale-event protection for all operation types (including deletes) is a mandatory requirement, you may need to use a Unique Key table instead of a Primary Key table. * Unique Key tables support a

sequence_column

(the equivalent of

merge_condition

) which is natively compatible with

DELETE

operations. * Trade-off: Unique Key tables generally have lower query performance (Merge-on-Read) compared to Primary Key tables (Delete+Insert). Summary of Support Table Type |

merge_condition

+

DELETE

| Reason Primary Key | Unsupported | Storage engine's Delete+Insert logic does not support conditional deletes. Unique Key | Supported | Uses

sequence_column

to determine version across all row types. References • loading/Load_to_Primary_Key_tables.md • deployment/shared_data/feature-support-shared-data.md • table_design/table_types/primary_key_table.md • table_design/table_types/table_capabilities.md