@Rocky what is the basic concept, note limit of flink connector starrocks. Also give me all the issue of flink connector Starrocks, get information from both document of starrocks and github

The StarRocks Flink Connector is a high-performance bridge that allows Apache Flink to read data from and write data to StarRocks. Below is a summary of its basic concepts, key limits, and notable issues gathered from official documentation and GitHub. 1. Basic Concepts * Flink Sink (Writing to StarRocks): * Mechanism: It buffers data in memory and flushes it in batches to StarRocks via Stream Load. * Semantics: Supports at-least-once (triggered by buffer size, row count, or time intervals) and exactly-once (triggered by Flink checkpoints using the Stream Load transaction interface). * V2 Sink: Newer versions use a "Merge Commit" mechanism to aggregate multiple small writes into a single transaction to reduce server-side overhead. * Flink Source (Reading from StarRocks): * Mechanism: Unlike standard JDBC, it reads in parallel from multiple Backends (BEs). It obtains a query plan from the Frontend (FE), which is then distributed across parallel Flink tasks to fetch data directly from BEs. 2. Notable Limits and Restrictions Based on the documentation and usage notes, several restrictions apply: Reading (Source) Limits: * SQL Syntax: Supports only basic

SELECT ... FROM ... WHERE ...

queries. * Aggregates: Only the

count

aggregate function is supported; others are not. * LIMIT Clause: The

LIMIT

statement is not supported in Flink SQL for this connector. * Checkpoints: StarRocks does not support the checkpointing mechanism for read tasks; if a read task fails, it must be restarted from the beginning as consistency isn't guaranteed. * Schema Mapping: The order of fields in your Flink DDL must match the order in the StarRocks table exactly. Writing (Sink) Limits: * Buffer Limits:

sink.buffer-flush.max-bytes

must be between 64 MB and 10 GB. * Parallelism: If the Flink sink parallelism is set to 1, the "Merge Commit" feature provides no performance benefit and may even increase latency. * Memory Management: Large batch sizes (controlled by

sink.buffer-flush.max-bytes

) can improve performance but increase memory usage and latency. 3. Common Issues (Documentation & GitHub) The following are common issues reported in the StarRocks FAQ and GitHub repository: * Network & Connectivity: * Internal IP Redirection: A known issue where the FE returns internal BE IP addresses that are unreachable from the Flink cluster (especially in Docker/K8s environments). Solution involves ensuring BE IPs are reachable or using NAT/Proxy. * Unreachable Load URL: Errors like

None of hosts in load_url could be connected

often require increasing

sink.connect.timeout-ms

. * Data Integrity & Formatting: * Timezone Offset: Users often report an 8-hour time difference when sinking timestamps. This is fixed by setting

server-time-zone

in the DDL. * JSON Batch Size: Large JSON loads may fail with

The size of this batch exceed the max size [104857600]

. This requires setting

sink.properties.ignore_json_size = true

. * Partial Updates: Flink CDC to StarRocks often fails with "NULL value in non-nullable column" if

sink.properties.partial_update

is not explicitly enabled for Primary Key tables. * Performance & Stability: * Scan Context Expiration: The error

Failed to get next from be

occurs when a scan task takes too long. Solution: Increase the BE config

scan_context_gc_interval_min

. * Shared-Data Mode: Some issues have been reported (e.g., GitHub #33368) where the connector fails to sink data correctly when StarRocks is in shared-data mode. For a full list of active bugs and feature requests, you can visit the official StarRocks Flink Connector GitHub Issues page. References • unloading/Flink_connector.md