Hi All, what is the way to configure two brokers i...
# general
s
Hi All, what is the way to configure two brokers in druid?
"bootstrap.servers": "xxxx,xxxx",
Will adding them separated by comma work if rest of the consumerProperties are same?
Also, how to know data belong to which broker?
t
The
consumerProperties
object must contain a
bootstrap.servers
property with a list of Kafka brokers in the form:
<BROKER_1>:<PORT_1>,<BROKER_2>:<PORT_2>,...
Please refer more details https://druid.apache.org/docs/latest/ingestion/kafka-ingestion/
s
I went through the docs but still not able to read data from two brokers. @John Kowtko could you please guide what needs to be done?
Getting following error while trying to read from second broker:
Copy code
WARN [task-runner-0-priority-0] org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner - OffsetOutOfRangeException with message
We also added "resetOffsetAutomatically": true but still getting errors.
j
Hi Suprabhat, Here's a Stack Overflow blog about it ... https://stackoverflow.com/questions/64729122/how-does-apache-kafka-work-with-multiple-brokers-and-single-broker It looks like you need to have replicas set up across the brokers ... I am not sure if this is automatic when creating multipe brokers in your cluster, or if this is something you have to explicitly configure. Do you have all of this setup and still getting offset errors? Thanks. John
s
thank you John.