Hi community, we had a high throughput kafka topic, is there any guidance about how to correctly scale the kafka source connectors. Or if we have multiple connections reading same topic with same consumer group id? thanks

Hi. The question is to vague - try to explain what you wish to achieve

Basically we publish about 10k messages per second over Kafka and we want to write all of them to S3 using Airbyte. We created a connection using source Kafka connector and destination S3 connector and we run the connection at 1 hr intervals. … It seems our connection is not able to consume all the messages that were published to Kafka in last one hour . Status shows that it is able to sync only about 500k records and then it stops. On checking the logs it shows that Kafka connector was not able to poll more messages but the Kafka dashboard shows that offset is still 22 million messages behind.

Got it- it seems that kafka is written in such a way that it cant handle big data logic. you have 2 options: 1. open an issue and wait 2. create your own kafka source and use dask or pyspark when writing it to handle these amounts of data- it`s not much btw so it should be quite fine

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2679 to ensure timely resolution!

@Uttam Garg do you mind sharing the logs so we can understand why the Kafka connector is stopping reading the messages even when there are more messages in the poll?