Hi, We have been running v10.3 since last week without issues. Today we decided to do a re-install of 10.3 just as a weekly maintenance activity. The deployment failed due to chart issues in the datahub cleanup job. I believe that fix is made and PR open - https://github.com/acryldata/datahub-helm/pull/328 I see last working chart version was 0.2.165 so we used the same. The deployment for GMS pod failed with the message -

2023-06-13 12:29:26,953 [ThreadPoolTaskExecutor-1] ERROR o.s.k.l.KafkaMessageListenerContainer:149 - Consumer exception

java.lang.IllegalStateException: This error handler cannot process 'SerializationException's directly; please consider configuring an 'ErrorHandlingDeserializer' in the value and/or key deserializer

at org.springframework.kafka.listener.DefaultErrorHandler.handleOtherException(DefaultErrorHandler.java:151)

at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.handleConsumerException(KafkaMessageListenerContainer.java:1815)

at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1303)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition MetadataChangeLog_Versioned_v1-0 at offset 11251397. If needed, please seek past the record to continue consumption.

Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro unknown schema for id 5

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)

at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)

at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)

at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)

We installed prerequisites with latest chart and kafka 3.4.0 and tried installing 10.3 again, but still failed with same message. Please advise on what is the correct working combination currently. Our application is down currently.

Are you using the schema registry or are you trying to use no-schema registry option?

Hello, Could you share your helm values.yaml file? Please remove sensitive data that you may have there

we got it working somehow. it seems schema registry enable status was changed from True to False recently so it was not getting deployed. We specifically enabled it and also added the KAFKA type and registry URL in the values.yaml file global options. The errors were still coming but the GMS pod did come up

@incalculable-ocean-74010 - Facing this issue in prod, unable to bring up the Application.

seeing the same error

Hi @future-yak-13169 , is it possible to share your Working chart version 0.2.165? I've been trying the older ones but couldn't get them to work.

@stocky-monkey-23735 0.2.165 changed this flag from true to false so you need to explicitly set it to true