@Rocky We have a use case where we need to sink data from Flink into StarRocks. We’re currently planning to use the Flink–StarRocks connector, but we’ve seen users raise a few issues with it. Have those issues been fixed/addressed? Another option we’re considering is sinking the data into an intermediate Kafka topic and then using load routines to ingest the data into StarRocks. Which of these two approaches would be preferred, and why? Any guidance on the trade-offs between them would be appreciated.

Both the Flink–StarRocks Connector and the Kafka + Routine Load approach are production-ready, but they serve different architectural needs. Which one is preferred? * Choose the Flink–StarRocks Connector if you are already doing complex ETL, multi-table joins, or windowing in Flink. It is more direct and avoids the cost/latency of an intermediate Kafka topic. * Choose Kafka + Routine Load if you need a "buffer" between your data source and StarRocks, or if you want a decoupled architecture where multiple downstream systems (not just StarRocks) consume the same data. ────────── Comparison & Trade-offs Feature | Flink–StarRocks Connector | Kafka + Routine Load Simplicity | Higher. Direct path from Flink to StarRocks. | Lower. Requires managing Kafka topics and Routine Load jobs. Exactly-Once | Supported via 2PC (requires Flink 1.2.4+ and SR 2.5+). | Supported natively by Routine Load. Backpressure | Managed by Flink's checkpointing mechanism. | Kafka acts as a natural buffer, shielding StarRocks from spikes. Transformation | Full Flink SQL/DataStream API power. | Limited to basic transformations (JSON path, simple expressions). Overhead | Dependent on Flink checkpoint intervals. | Very low; Routine Load is highly optimized for StarRocks. Addressing Known Issues The "issues" users frequently encountered with the Flink connector were largely related to its early implementation of exactly-once semantics. These have been significantly addressed in recent versions: 1. Redesigned Transaction Interface: Since Flink Connector 1.2.4 and StarRocks 2.5, the connector uses the Stream Load transaction interface. This fixed the high memory usage and checkpoint overhead seen in older versions. 2. Lingering Transactions: A common issue was "zombie" transactions (status

PREPARED

) staying open after Flink job failures. This was addressed in Connector 1.2.8 by introducing

sink.label-prefix

. By setting a unique prefix, the connector can now automatically find and abort lingering transactions upon job restart. 3. Timeout Management: Newer versions (1.2.12+) allow you to configure

sink.properties.prepared_timeout

, giving you better control over how long transactions stay in a prepared state before timing out. Guidance * If you use the Flink Connector: Ensure you use version 1.2.8 or later and always set a

sink.label-prefix

. This is critical for stability and preventing "too many transactions" errors in StarRocks. * If you use Kafka: This is often the "safer" route for high-volume streaming because Kafka handles the persistence. If Flink or StarRocks goes down, the data remains safely in Kafka without putting pressure on the source. Recommendation: For most modern StarRocks deployments, the Flink–StarRocks Connector (v1.2.8+) is the preferred method because it is more efficient and easier to maintain within a Flink-centric pipeline. Only move to Kafka if you specifically need the buffering or multi-consumer capabilities it provides. References • faq/loading/Flink_connector_faq.md • unloading/Flink_connector.md • loading/Flink-connector-starrocks.md

@Rocky what about the Internal IP and routing Problem

The Internal IP and Routing problem is a common networking hurdle when using the Flink–StarRocks connector from a separate network (like a different VPC, or outside a Kubernetes cluster). What is the problem? When you use the Flink Connector, the process works via Stream Load: 1. Flink sends a load request to the StarRocks Frontend (FE) using the

load-url

. 2. The FE selects a Backend (BE) to handle the data and sends an HTTP 307 Redirect back to Flink. 3. The Issue: By default, the FE redirects Flink to the Internal IP that the BE used to register with the cluster. If your Flink cluster cannot "see" that internal IP (because it's on a private network), the connection times out, and you see errors like

None of hosts in load_url could be connected

or

Connect timed out

. ────────── How to Solve it (Routing & IP) 1. Use

priority_networks

In your

be.conf

and

fe.conf

, you can set the

priority_networks

parameter. This forces StarRocks nodes to bind to a specific IP range. If your environment has a network interface that is accessible to Flink, ensure the CIDR matches that network.

conf
# Example: force binding to the 10.10.x.x network
priority_networks = 10.10.0.0/16

2. FQDN Mode (Recommended for K8s/Docker) In recent versions, you can start StarRocks nodes with the flag

--host_type FQDN

. This causes the FE to redirect Flink using the hostname of the BE instead of its IP. * Action: You must then ensure the Flink workers can resolve these hostnames (e.g., via DNS or by adding the BE hostnames and IPs to the

/etc/hosts

file of every Flink node). 3. Proxy/Load Balancer If you are on Kubernetes, you can use a Proxy (like the

starrocksFeProxySpec

in the StarRocks Operator) to handle the redirection. The proxy can be exposed via a LoadBalancer, allowing Flink to communicate with a single stable entry point that handles the internal routing to the BEs. ────────── Why Routine Load "Fixes" This If routing is a major pain point for your infrastructure team, Routine Load is often the better choice because: * Internal Communication: The Routine Load task runs inside the StarRocks BEs. It uses the internal cluster network to communicate between the FE and BE. It never has to deal with the "307 Redirect to an unreachable IP" issue because it's already "home." * Simple External Edge: The only external networking you need to worry about is ensuring the StarRocks BEs can reach your Kafka brokers. Usually, it is easier to give a database cluster access to Kafka than to expose every single BE node's HTTP port to an external Flink cluster. Summary of Guidance * Stick with Flink if you can resolve the network routing (using FQDNs or a flat network/VPC peering). It provides more power. * Switch to Kafka + Routine Load if your Flink and StarRocks clusters are in strictly isolated networks and you don't want to manage complex NAT/Gateway rules for every BE node-to-node connectivity. References • faq/loading/Flink_connector_faq.md • quick_start/shared-data.md • loading/StreamLoad.md • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md