http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> Is it possible to decode Protobuf ...
# questions-and-troubleshooting
a
@Rocky Is it possible to decode Protobuf in Routine Load?
r
Currently, Routine Load in StarRocks does not natively support decoding Protobuf formatted data. According to the latest documentation, Routine Load supports the following formats: * CSV * JSON * Avro (Supported since v3.0.1, requires Schema Registry) Protobuf support for Routine Load is currently on the roadmap ("To be supported"). Alternatives for Protobuf Data If you need to load Protobuf data from Kafka into StarRocks, you have two primary options: 1. StarRocks Kafka Connector: Unlike Routine Load, the StarRocks Kafka Connector (which runs as a plugin in a Kafka Connect cluster) does support Protobuf starting from version 3.0. This is the recommended approach if you need to consume Protobuf messages directly from Kafka. 2. Preprocessing: You can use a stream processing framework (like Flink or Spark Streaming) to consume the Protobuf data from Kafka, decode it, and then sink it into StarRocks using the StarRocks connector for that framework (e.g., StarRocks Flink Connector). Summary Table (Format Support for Kafka) Feature | CSV | JSON | Avro | ***Protobuf*** Routine Load | Yes | Yes | Yes (v3.0.1+) | To be supported Kafka Connector | Yes | Yes | Yes | Yes (v3.0+) References • loading/RoutineLoad.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • loading/loading_introduction/feature-support-loading-and-unloading.md • loading/load_from_pulsar.md • developers/code-style-guides/protobuf-guides.md
a
@Rocky Does it support Kafka Connector? I think we need a Kafka Connector guide.
r
Yes, StarRocks supports a Kafka Connector (the StarRocks Sink Connector), which allows you to continuously consume data from Kafka and load it into StarRocks. Here is a quick guide to getting started with the StarRocks Kafka Connector. 1. Overview The StarRocks Kafka Connector is a sink connector designed to work with the Apache Kafka Connect framework. It can handle: * Standard JSON/CSV data. * CDC (Change Data Capture) data (e.g., from Debezium) for synchronization into StarRocks Primary Key tables. 2. Prerequisites * A running StarRocks cluster. * An Apache Kafka cluster with Kafka Connect installed. * The StarRocks Kafka Connector JAR file (you can find this on Maven Central or the StarRocks downloads page). 3. Basic Configuration (Standalone Mode) To run the connector in standalone mode, you need to configure two main files: A. Configure the Connector (
connect-StarRocks-sink.properties
) Create a configuration file for the StarRocks sink:
Copy code
properties
name=starrocks-kafka-connector
connector.class=com.starrocks.connector.kafka.StarRocksSinkConnector
topics=your_kafka_topic
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=false

# StarRocks FE HTTP URL (default port 8030)
starrocks.http.url=192.168.x.x:8030
starrocks.database.name=your_db
starrocks.username=root
starrocks.password=your_password

# Mapping Kafka topics to StarRocks tables
starrocks.topic2table.map=your_kafka_topic:your_starrocks_table
sink.properties.strip_outer_array=true
B. Configure Kafka Connect (
connect-standalone.properties
) Ensure the 
plugin.path
points to the directory containing the StarRocks connector JAR:
Copy code
properties
bootstrap.servers=kafka-broker:9092
offset.storage.file.filename=/tmp/connect.offsets
plugin.path=/path/to/starrocks-kafka-connector
4. Running the Connector You can start the connector using the following command:
Copy code
bash
CLASSPATH=/path/to/connector/* bin/connect-standalone.sh \
  config/connect-standalone.properties \
  config/connect-StarRocks-sink.properties
5. Handling CDC Data (Debezium) If you are loading CDC data into a Primary Key table, you need to add specific transformations to your connector config to handle operations like 
upsert
and `delete`:
Copy code
properties
transforms=addfield,unwrap
transforms.addfield.type=com.starrocks.connector.kafka.transforms.AddOpFieldForDebeziumRecord
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
transforms.unwrap.drop.tombstones=true
transforms.unwrap.delete.handling.mode=rewrite
Key Parameters * `starrocks.http.url`: The HTTP URL of the Frontends (FE). * `starrocks.topic2table.map`: Maps Kafka topics to StarRocks tables if they have different names. _ `sink.properties._`: You can pass additional Stream Load properties (like 
column_separator
) by prefixing them with 
sink.properties.
. For more advanced setups (like Distributed Mode via REST API), please refer to the official documentation on loading data with the Kafka connector. References • loading/Kafka-connector-starrocks.md
a
@Rocky is it require 
confluent schema registry
to decode protobuf?
r
a
@Rocky If I use kafka-connector-starrocks, is it require 
schema registry
to decode protobuf?
r
Yes, if you use 
kafka-connector-starrocks
 (the Kafka Connect sink connector) to load Protobuf data, a Schema Registry is typically required. Here is the breakdown of why and how it differs from other methods: 1. Kafka Connector Mechanism The 
kafka-connector-starrocks
runs within the Kafka Connect framework. It relies on Kafka Connect Converters to deserialize data from Kafka topics into a format the connector can understand. * To decode Protobuf messages, you would normally use the 
io.confluent.connect.protobuf.ProtobufConverter
. * This converter is designed to work with the Confluent Schema Registry to fetch the schema needed to deserialize the binary Protobuf bytes into structured data. * Configuration typically looks like this:
Copy code
properties
  value.converter=io.confluent.connect.protobuf.ProtobufConverter
  value.converter.schema.registry.url=<http://localhost:8081>
2. Difference from Routine Load It is important to distinguish the Kafka Connector from StarRocks' built-in Routine Load feature: * Routine Load: Supports Protobuf and allows you to provide a local 
.proto
file directly in the 
CREATE ROUTINE LOAD
statement. In this case, a Schema Registry is not required. * Kafka Connector: Does not natively parse 
.proto
files. It expects the Kafka Connect framework to handle the conversion. Since the standard Protobuf converter for Kafka Connect requires a Schema Registry, the connector effectively requires one too. Summary Table Feature | Protobuf Support | Schema Registry Required? | How it works Kafka Connector | Yes | Yes | Uses Kafka Connect converters (e.g., Confluent ProtobufConverter). Routine Load | Yes | No | You upload/reference a 
.proto
file in the load job. Recommendation: * If you already have a Schema Registry in your ecosystem, the Kafka Connector is a great choice as it integrates seamlessly. * If you do not want to use a Schema Registry and have the 
.proto
files available, using Routine Load is the simpler and more direct approach. References • loading/Kafka-connector-starrocks.md
3 Views
Open in Slack
PreviousNext
Powered by