Hi, I’m having a problem with creating datahub for the first time using the new helm chart, using

INTERNAL

schema registry. The system update job and the gms pod fail. System update job log:

2023-06-09 07:16:51,099 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Executing Step 4/5: DataHubStartupStep...
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: java.io.IOException: No schema registered under subject!

GMS pod cannot connect to itself:

2023-06-09 07:21:57,361 [R2 Nio Event Loop-1-1] WARN  c.l.r.t.h.c.c.ChannelPoolLifecycle:139 - Failed to create channel, remote=localhost/127.0.0.1:8080
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.base/java.lang.Thread.run(Thread.java:829)

@orange-night-91387 might be able to speak to this! 🙂

Just my hunch, but it’s probably due to circular dependency: • For every installation, a

datahub-system-upgrade

job is created to bootstrap Datahub system. This needs to talk to Schema Registry. •

datahub-gms

is also created in parallel.

datahub-gms

needs the above job to finish to be in Healthy state. This service host the “internal” schema registry that the above job requires. • That means

datahub-gms

Schema Registry is never up, which blocks

datahub-system-upgrade

job, which blocks

datahub-gms

, and so on.

What version are you on? https://github.com/datahub-project/datahub/pull/7999

I was on

0.10.3

@orange-night-91387 - Facing this issue in prod, and our application is down. anysuggestions on how to fix this. we are on 10.1

Your issue has to be different @handsome-football-66174, 10.1 does not support removing schema registry.

@orange-night-91387 - this is the error - we are getting :

org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching <schema-registry URL> found.
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[na:na]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:353) ~[na:na]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:296) ~[na:na]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:291) ~[na:na]

Can you make a new thread for this please? 🙂

Sure

Hey there @orange-night-91387, I'm working through an EKS deployment and coming across a similar issue:

2023-08-01 17:19:35,465 [main] ERROR c.l.d.u.s.e.steps.DataHubStartupStep:40 - DataHubStartupStep failed.
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: java.io.IOException: No schema registered under subject!
        at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestVersion(MockSchemaRegistryClient.java:261)
        at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestSchemaMetadata(MockSchemaRegistryClient.java:310)
...

Are there any additional prerequisites that we need to do prior to the system update step when we're using internal schema registry?

Is this a new deployment? Or did you update to internal schema registry from a previous deployment?:

Progress toward Removing Confluent Schema Registry requirement -- Helm & Quickstart simplifications to follow
NOTE: this will only work for new deployments of DataHub; If you have already deployed DataHub with Confluent Schema Registry, you will not be able to disable it

https://github.com/datahub-project/datahub/releases/tag/v0.10.3

Yes, this is a brand new deployment. We are on v0.10.4.

We are also running into this error with v0.10.5.

Are you customizing your Kafka topic names?

Experiencing this on upgrade from from 0.10.4 -> 0.10.5

Are you customizing your Kafka topic names?

Nope (related thread: https://datahubspace.slack.com/archives/CV2UVAPPG/p1691079347732439)

I am also having this problem,

No schema registered under subject!

just following the steps here: https://datahubproject.io/docs/deploy/kubernetes/ It appears the datahub chart is datahub-0.2.181 and the prereqs are datahub-prerequisites-0.0.18. I haven’t done any customization other than specify a namespace in my eks cluster.

@orange-night-91387 yes, we are. We are running this on a shared kafka cluster, so are prepending

dataHub_

to most topic names.

@agreeable-musician-41252, did you get this going? Running into the same issue here - same versions.

@able-library-93578 I got it mostly deployed after fiddling with the vaules.yaml but still with some weirdness in the pod logs but I gave up on that and switched to the previous version, which seems to work fine. just used

--version 0.2.180

on the helm install