Hi everyone, I'm currently working on a data pipeline where Apache Flink consumes messages from Apache Kafka and writes them to StarRocks using the StarRocks Sink (Stream Load).
One concern I have is that each Stream Load operation creates a new tablet version in StarRocks. Since tablet versions are generated sequentially, a high-frequency streaming workload could result in a very large number of tablet versions over time. From what I understand, this may put significant pressure on the BE nodes and, in extreme cases, could even cause BE instability or crashes.
Has anyone encountered this issue in a production environment? What are the recommended best practices to avoid excessive tablet version growth while still maintaining near real-time ingestion? For example, should I tune the Stream Load batch size, merge commit settings, flush interval, or consider a different ingestion approach?
Any suggestions or experience would be greatly appreciated. Thank you!