A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi guys! I am facing some issues when trying to upgrade from v0.8.41 to v0.8.43, particularly with the kafka-setup job

the `datahub-kafka-setup-job` is failling with:
```Error while executing config command with args '--command-config /tmp/connection.properties --bootstrap-server ... --entity-type topics --entity-name _schemas --alter --add-config cleanup.policy=compact'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
	at kafka.admin.ConfigCommand$.getResourceConfig(ConfigCommand.scala:552)
	at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:322)
	at kafka.admin.ConfigCommand$.processCommand(ConfigCommand.scala:302)
	at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:97)
	at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
Caused by: org.apache.kafka.common.errors.UnknownTopicOrPartitionException:```

it seems like it is trying to modify a topic named ‘_schemas’

```DataHubUsageEvent_v1
FailedMetadataChangeEvent_v4
FailedMetadataChangeProposal_v1
MetadataAuditEvent_v4
MetadataChangeEvent_v4
MetadataChangeLog_Timeseries_v1
MetadataChangeLog_Versioned_v1
MetadataChangeProposal_v1
PlatformEvent_v1
__amazon_msk_canary
__consumer_offsets```

afaik, _schemas topic is added by Confluent, should I create it manually?

oh, I guess this is related to the Schema Registry right? I am using the AWS Glue SR aswell

Yes, this topic should be created by the Schema Registry, a similar command should have run in 0.8.41 also referencing that topic so if it's failing in 0.8.43 it should have also failed then

it failed with the 42, but not with the 41

<@UV5UEC3LN> do we support AWS Glue as Schema Reg?

fwiw I’m using Glue and had to create the `_schemas` Kafka topic manually too

Hi everyone here, can you please provide further details for this _schema topic? :pray::skin-tone-3: having the same issue, running on version (v0.10.2) - deployed on k8s, using self-hosted bitnami Kafka

Hey, also facing same issue, it would be nice to share the command specifics for topic `_schemas`  manually.

just follow the kafka documentation, this is not datahub specific