http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> Does the routine load support Kafka...
# questions-and-troubleshooting
s
@Rocky Does the routine load support Kafka TLS and Kafka authentification to the broker?
r
Yes, StarRocks Routine Load supports both Kafka TLS (SSL) and multiple Kafka authentication mechanisms (SASL). These configurations are passed using the 
property.
prefix in the 
FROM KAFKA
clause, which maps to 
librdkafka
parameters. 1. Kafka TLS/SSL Support To use SSL, set the security protocol and provide the certificate paths. If the certificates are stored in StarRocks (via the 
CREATE FILE
command), use the 
FILE:
prefix. Example SSL Configuration:
Copy code
sql
FROM KAFKA
(
    "kafka_broker_list" = "host:port",
    "kafka_topic" = "my_topic",
    "property.security.protocol" = "ssl",
    "property.ssl.ca.location" = "FILE:ca-cert.pem",
    -- If client authentication is required:
    "property.ssl.certificate.location" = "FILE:client.pem",
    "property.ssl.key.location" = "FILE:client.key",
    "property.ssl.key.password" = "your_password"
)
2. Kafka Authentication (SASL) StarRocks supports several SASL mechanisms over both 
sasl_plaintext
and 
sasl_ssl
protocols. * SASL/PLAIN (Username/Password):
Copy code
sql
    "property.security.protocol" = "sasl_plaintext",
    "property.sasl.mechanism" = "PLAIN",
    "property.sasl.username" = "admin",
    "property.sasl.password" = "password"
* SASL/SCRAM: Supports 
SCRAM-SHA-256
and 
SCRAM-SHA-512
. * SASL/GSSAPI (Kerberos): Supported since StarRocks v3.1.4.
Copy code
sql
    "property.security.protocol" = "sasl_plaintext",
    "property.sasl.mechanism" = "GSSAPI",
    "property.sasl.kerberos.service.name" = "kafka",
    "property.sasl.kerberos.keytab" = "/path/to/starrocks.keytab",
    "property.sasl.kerberos.principal" = "starrocks@DOMAIN.COM"
Note: Kerberos requires SASL modules (like 
cyrus-sasl-gssapi
) to be installed on the BE nodes. Important Tips: * Custom Properties: You can pass any 
librdkafka
property by prefixing it with 
property.
. * Certificate Errors: If you encounter 
certificate verify failed
, you can bypass endpoint identification by adding: 
"property.ssl.endpoint.identification.algorithm" = "none"
* File Management: Use the 
CREATE FILE
command to upload certificates to the StarRocks cluster so they can be referenced by the Routine Load job. References • faq/loading/Routine_load_faq.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • loading/automq-routine-load.md • quick_start/routine-load.md
s
@Rocky Supports Kafka 4 too?
r
Yes, StarRocks supports Kafka 4.0 for Routine Load. Based on recent updates (as of early 2026), StarRocks has verified compatibility with Kafka 4.0. Because StarRocks uses 
librdkafka
 as its underlying C++ client for Routine Load, it benefits from the library's broad compatibility across Kafka broker versions. Key Considerations for Kafka 4.0: * KRaft Mode: Kafka 4.0 is the first major version to operate entirely without Apache ZooKeeper (KRaft-only). Since Routine Load communicates with the Kafka cluster via the 
kafka_broker_list
(bootstrap servers) and not ZooKeeper, the transition to KRaft does not change how you configure Routine Load jobs. * Protocol Compatibility: While Kafka 4.0 removed support for some very old client protocol versions (e.g., versions prior to 0.11), StarRocks uses modern versions of 
librdkafka
that are fully compatible with the newer protocols required by Kafka 4.0. * Authentication & TLS: All authentication mechanisms (SASL/PLAIN, SCRAM, Kerberos) and TLS configurations mentioned previously continue to work with Kafka 4.0 brokers. Configuration remains the same: You do not need special parameters for Kafka 4.0. A standard configuration will look like this:
Copy code
sql
CREATE ROUTINE LOAD my_db.my_job ON my_table
PROPERTIES ("format" = "json")
FROM KAFKA
(
    "kafka_broker_list" = "broker1:9092,broker2:9092",
    "kafka_topic" = "my_topic",
    "property.security.protocol" = "sasl_ssl",
    -- Kafka 4.0 still supports standard SASL/SSL properties
    "property.sasl.mechanism" = "PLAIN",
    "property.sasl.username" = "user",
    "property.sasl.password" = "pass"
);
If you are using StarRocks v3.x or v4.0, you can confidently connect to Kafka 4.0 clusters. References • loading/RoutineLoad.md • faq/loading/Routine_load_faq.md • sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md • quick_start/routine-load.md
2 Views
Open in Slack
PreviousNext
Powered by