Hi Team, We had recently deployed datahub on eks and wanted to move kafka from kube to confluent cloud managed service. While setting it up i noticed that the kafka setup job by default uses a single partition for all topics. Does this mean that we need to make some sort of changes from producer/consumer side for a performance upgrade (im guessing there is a single mae/mce consumer). New to kafka, so any help is appreciated

Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!

Found a thread here on slack which said you can scale the consumers. Is there some doc on this?

Scaling the number of consumers is done through changing the replica count on your deployment. For topic configurations, you can customize your partitioning directly on the topic. Confluent has UI based configuration that allows you to change the partition numbers after deploying to scale as needed. It's recommended to have number of partitions relative to the number of consumers you want to deploy.

Hey @orange-night-91387 Thanks for getting back on this Does this mean ive got to increase the replica count on gms as the consumers are part of it or do i have to deploy the consumers separately and then increase the replica count on them

Generally, if you're hitting the point where you need to start scaling up components we recommend that you switch to the standalone consumer deployment model so that you can scale in a more targeted way. Scaling up multiple GMS instances can be more costly if you don't need the increased bandwidth for the serving layer due to its higher resource cost.

ok thanks