hello! posting this here just in case it's a better place to grab a hand if poss! 🙂 https://datahubspace.slack.com/archives/CV2UVAPPG/p1675351168523089

hey Nick! gonna look to see if I can get someone with more experience in this area to help out soon 🙂

thank you!

hey Nick! yes I asked around to find someone who's a little more familiar with this, and I believe Ryan responded to your original post

thanks! i almost wish i hadn't asked. what a stupid mistake 😄 thank you!

Happens to the best of us 🙂 no sweat, sometimes you just need a second pair of eyes and our community is happy to be that.

Hey @orange-night-91387! Having seemingly connected to MSK and got the topics created, i seem to be getting this error in the kafka-setup-job:

Topic Creation Complete.
Error while executing config command with args '--command-config /tmp/connection.properties --bootstrap-server <http://b-1.xxx:9098,b-2.xxx:9098,b-3.xxx9098|b-1.xxx:9098,b-2.xxx:9098,b-3.xxx9098> --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
	at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:552)
	at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:322)
	at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:302)
	at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
	at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException:

Like i said, i can see the topics are created in kafka. The setup job did seem to work initially; the above error seems to have occurred after I attempted to rename the topics using

extraEnvs

. I've since reverted to default topic names but the setup job is still failing. Have you seen this before, or have any idea what may be causing it?

The _schemas topic usually has to do with Schema Registry, what version are you deploying with? We recently took out schema registry as a requirement

app version

0.9.6.1

and chart

0.2.138

Have you run the prerequisites? Is Schema Registry up?

yes it is, although i think you may have helped me find the culprit here. possibly the values.yml here:

kafka:
    bootstrap:
      server: "<<bootstrap-server endpoint>>"
    zookeeper:
      server:  "<<zookeeper endpoint>>"
    schemaregistry:
      url: "<http://prerequisites-cp-schema-registry:8081>"
    partitions: 3
    replicationFactor: 3

Yeah if you have your own schema registry you'll need to change that

we don't at the moment, but given that it isn't a requirement, do we need it at all in the config?

There's a PR targeted for a future release that removes schema registry, not any released versions without it so the requirement is still there.