Facing issue with gms. Kafka consumer group is always rebalancing and is kind of stuck. I saw a previous thread https://datahubspace.slack.com/archives/C029A3M079U/p1650921142481779?thread_ts=1650871366.562249&cid=C029A3M079U but no resolution. Here is the analysis done so far: ========================================================================= To process the 10 partitions, we are seeing 3 active kafka clients. For every group that is getting created, 2 clients are being assigned. Then the third client tries to join the group and the whole group is dissolved to re-balance. In the next group created, again we see only 2 clients and the third client becomes spoil sport, whole process repeats. Here is the simplified log showing the issue. Team, Can you help ?

Hi @bland-appointment-45659, thanks for your patience here- if you’re still experiencing this issue I think @brainy-tent-14503 may be able to help out

Try reducing the number of records being polled, the log indicates the processing is taking too long. The default size for the client is 500, try reducing to like 10 to see if that helps. Then increase to optimize throughput. This is set using the spring kafka environment variable

SPRING_KAFKA_CONSUMER_PROPERTIES_MAX_POLL_RECORDS

@brainy-tent-14503 We did reduce the KAFKA listener concurrency to 1 and reduced max poll records to 10. Even with 1 consumer, the rebalancing is happening. Noticed was that every time the neo4j graphservice is removing and adding all connections. For entities having large number of connections, this could be taking long time.

Yes, neo4j is not something we use internally, however I’ve been told that graph updates/writes take a very long time. This was one of the reasons we moved to Elasticsearch graph implementation.

is there an option to switch between elasticSearch vs neo4j graph in ingestion ?

The option is set when you setup your datahub environment. Specifically the helm chart value is set here.