@Rocky I cannot find clear documentation for when data becomes queryable when sinking from apache flink using the V2 sink api. Supposedly its bound to the flink checkpoint, but with my flush interval being shorter than the checkpoint interval, I'm seeing the data in my table updating with the flush interval, not the checkpoint interval. What is the actual behaviour?

The behavior you are seeing is expected and depends on the

sink.semantic

configuration of the StarRocks Flink connector. When using the Flink Sink V2 API, data visibility is determined by the loading semantic you have chosen (or the default). Here is the breakdown of how and when data becomes queryable: *1. At-Least-Once (Default)* If

sink.semantic

is set to

at-least-once

(which is the default value), the connector flushes data based on the most restrictive of several triggers: *

sink.buffer-flush.interval-ms

: The time-based threshold (default is 300,000ms or 5 minutes). *

sink.buffer-flush.max-bytes

: The size-based threshold. * Flink Checkpoint: A flush is also triggered when a Flink checkpoint starts. Why you see data every flush interval: In at-least-once mode, every flush translates to a completed and committed Stream Load transaction. As soon as the flush finishes, the data is committed in StarRocks and becomes immediately queryable. It does not wait for the Flink checkpoint to complete. *2. Exactly-Once* If you want data visibility to be strictly bound to the Flink checkpoint, you must set

sink.semantic

to

exactly-once

. * Visibility behavior: In this mode, the connector uses a Two-Phase Commit (2PC) protocol. * During Checkpoint: Data is flushed to StarRocks (Pre-commit), but the transaction remains in a

PREPARED

state and is not yet queryable. * After Checkpoint Success: Once Flink confirms the checkpoint is successful, the connector sends a commit signal to StarRocks. Only then does the data become queryable. * Constraint: In

exactly-once

mode, the

sink.buffer-flush.interval-ms

parameter is ignored. Data is only flushed when a checkpoint is triggered. *3. Merge Commit (Important Note)* If you have enabled Merge Commit (

sink.properties.enable_merge_commit = true

), please note the following: * Semantic Restriction: Merge Commit only supports at-least-once semantics. It does not currently support exactly-once. * Visibility: If Merge Commit is on, data visibility is governed by both the connector's flush interval and the StarRocks server's merge window (

sink.properties.merge_commit_interval_ms

). Summary Table Feature | At-Least-Once (Default) | Exactly-Once Data Visibility | Immediately after flush | Only after successful checkpoint Flush Trigger | Interval, Max Bytes, or Checkpoint | Only at Checkpoint Commit Type | One-phase (immediate) | Two-phase (prepared -> committed)

interval-ms

| Used to trigger frequent commits | Ignored Recommendation If your goal is to ensure data only appears in the table when a Flink checkpoint succeeds: 1. Set

sink.semantic

to

exactly-once

. 2. Ensure you have a unique

sink.label-prefix

set (this is critical for 2PC to clean up abandoned transactions). 3. Ensure Merge Commit is disabled, as it is incompatible with exactly-once. References • loading/Flink-connector-starrocks.md

@Rocky So I can keep my data fresh while using the V2 api?

Yes, you can absolutely keep your data fresh while using the Flink Sink V2 API. The behavior you are observing—where data is queryable based on the flush interval rather than the checkpoint interval—is the intended and standard behavior for the default configuration. Here is how you can control and optimize data freshness: 1. Maintain High Freshness with

at-least-once

If you do not strictly require

exactly-once

semantics, you should stick with the default

at-least-once

(or explicitly set

sink.semantic

to

at-least-once

). * Freshness Control: Your data freshness is controlled by

sink.buffer-flush.interval-ms

. You can set this to a low value (e.g.,

1000

for 1 second or

5000

for 5 seconds) to ensure data is visible almost immediately. * Why it works: In this mode, the connector does not wait for a Flink checkpoint to commit. It finishes a Stream Load transaction every time the buffer interval or size limit is reached. 2. High Freshness + High Throughput (Merge Commit) If you want very high freshness (low flush intervals) but are worried about creating too many small transactions in StarRocks, you should enable Merge Commit (available in Connector v1.2.14+ and StarRocks 3.2.4+). * How it works: Multiple Flink subtasks can flush frequently, but StarRocks will "merge" these small writes into a single transaction on the server side. * Configuration:

sql
    'sink.properties.enable_merge_commit' = 'true',
    'sink.properties.merge_commit_interval_ms' = '5000', -- Server-side merge window
    'sink.buffer-flush.interval-ms' = '1000'           -- Connector-side flush

* Result: You get the freshness of a 1-second flush with the efficiency of a 5-second commit. 3. The "Checkpoint Bound" Misconception The documentation often mentions that the sink is "bound to the Flink checkpoint" because the Sink V2 API is designed to support Two-Phase Commit (2PC). However: * In

at-least-once

: A checkpoint also triggers a flush, but it's just one of several triggers. The interval timer is the primary driver of freshness. * In

exactly-once

: Freshness is strictly bound to the checkpoint. Data is pre-committed during the checkpoint but only becomes visible after the checkpoint successfully completes. If you need high freshness here, you are forced to reduce your Flink checkpoint interval, which can increase overhead on your Flink cluster. Summary Checklist for Freshness * Use

sink.semantic

=

at-least-once

(Default). * Lower

sink.buffer-flush.interval-ms

to your desired latency (e.g.,

5000

). * Lower

sink.buffer-flush.max-bytes

if you have very high throughput and want to flush based on data volume. * Avoid

exactly-once

unless your use case cannot tolerate occasional duplicate records during a job failover (e.g., financial calculations). Verdict: Your current observation is correct and is the best way to achieve sub-minute freshness in StarRocks. References • loading/Flink-connector-starrocks.md