yeah it s a cached schema registry which I

Question

< Daniel Lavoie> yeah it s a cached schema registry which I m not sure it s the source of our error

Daniel Lavoie · Answer

when you say `second table is not being able to reach the second kafka cluster s schema registry` What s the error you are observing

Alexander Vivas · Answer

```ERROR LLRealtimeSegmentDataManager mls streams stream video data updated v1 0 2 20210311T1512Z mls streams stream video data updated v1 0 2 20210311T1512Z Exception while in work org apache kafka common errors SerializationException Error deserializing Avro message for id 514 Caused by javax net ssl SSLHandshakeException PKIX path building failed sun security provider certpath SunCertPathBuilderException unable to find valid certification path to requested target at sun security ssl Alert createSSLException Alert java 131 ~ 1 8 0 282 at sun security ssl TransportContext fatal TransportContext java 324 ~ 1 8 0 282 at sun security ssl TransportContext fatal TransportContext java 267 ~ 1 8 0 282 at sun security ssl TransportContext fatal TransportContext java 262 ~ 1 8 0 282 at sun security ssl CertificateMessage$T12CertificateConsumer checkServerCerts CertificateMessage java 654 ~ 1 8 0 282 at sun security ssl CertificateMessage$T12CertificateConsumer onCertificate CertificateMessage java 473 ~ 1 8 0 282 at sun security ssl CertificateMessage$T12CertificateConsumer consume CertificateMessage java 369 ~ 1 8 0 282 at sun security ssl SSLHandshake consume SSLHandshake java 377 ~ 1 8 0 282 at sun security ssl HandshakeContext dispatch HandshakeContext java 444 ~ 1 8 0 282 at sun security ssl HandshakeContext dispatch HandshakeContext java 422 ~ 1 8 0 282 at sun security ssl TransportContext dispatch TransportContext java 182 ~ 1 8 0 282 at sun security ssl SSLTransport decode SSLTransport java 149 ~ 1 8 0 282 at sun security ssl SSLSocketImpl decode SSLSocketImpl java 1143 ~ 1 8 0 282 at sun security ssl SSLSocketImpl readHandshakeRecord SSLSocketImpl java 1054 ~ 1 8 0 282 at sun security ssl SSLSocketImpl startHandshake SSLSocketImpl java 394 ~ 1 8 0 282 at sun net www protocol https HttpsClient afterConnect HttpsClient java 559 ~ 1 8 0 282 at sun net www protocol https AbstractDelegateHttpsURLConnection connect AbstractDelegateHttpsURLConnection java 185 ~ 1 8 0 282 at sun net www protocol http HttpURLConnection getInputStream0 HttpURLConnection java 1570 ~ 1 8 0 282 at sun net www protocol http HttpURLConnection getInputStream HttpURLConnection java 1498 ~ 1 8 0 282 at java net HttpURLConnection getResponseCode HttpURLConnection java 480 ~ 1 8 0 282 at sun net www protocol https HttpsURLConnectionImpl getResponseCode HttpsURLConnectionImpl java 352 ~ 1 8 0 282 at io confluent kafka schemaregistry client rest RestService sendHttpRequest RestService java 212 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client rest RestService httpRequest RestService java 256 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client rest RestService getId RestService java 486 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client rest RestService getId RestService java 479 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client CachedSchemaRegistryClient getSchemaByIdFromRegistry CachedSchemaRegistryClient java 177 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client CachedSchemaRegistryClient getBySubjectAndId CachedSchemaRegistryClient java 256 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client CachedSchemaRegistryClient getById CachedSchemaRegistryClient java 235 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka serializers AbstractKafkaAvroDeserializer deserialize AbstractKafkaAvroDeserializer java 107 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka serializers AbstractKafkaAvroDeserializer deserialize AbstractKafkaAvroDeserializer java 79 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka serializers KafkaAvroDeserializer deserialize KafkaAvroDeserializer java 55 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot plugin inputformat avro confluent KafkaConfluentSchemaRegistryAvroMessageDecoder decode KafkaConfluentSchemaRegistryAvroMessageDecoder java 114 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot plugin inputformat avro confluent KafkaConfluentSchemaRegistryAvroMessageDecoder decode KafkaConfluentSchemaRegistryAvroMessageDecoder java 120 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot plugin inputformat avro confluent KafkaConfluentSchemaRegistryAvroMessageDecoder decode KafkaConfluentSchemaRegistryAvroMessageDecoder java 53 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot core data manager realtime LLRealtimeSegmentDataManager processStreamEvents LLRealtimeSegmentDataManager java 471 ~ pinot all 0 7 0 SNAPSHOT jar with dependencies jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot core data manager realtime LLRealtimeSegmentDataManager consumeLoop LLRealtimeSegmentDataManager java 402 ~ pinot all 0 7 0 SNAPSHOT jar with dependencies jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot core data manager realtime LLRealtimeSegmentDataManager$PartitionConsumer run LLRealtimeSegmentDataManager java 538 pinot all 0 7 0 SNAPSHOT jar with dependencies jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at java lang Thread run Thread java 748 1 8 0 282 Caused by sun security validator ValidatorException PKIX path building failed sun security provider certpath SunCertPathBuilderException unable to find valid certification path to requested target at sun security validator PKIXValidator doBuild PKIXValidator java 456 ~ 1 8 0 282 at sun security validator PKIXValidator engineValidate PKIXValidator java 323 ~ 1 8 0 282 at sun security validator Validator validate Validator java 271 ~ 1 8 0 282 at sun security ssl X509TrustManagerImpl validate X509TrustManagerImpl java 315 ~ 1 8 0 282 at sun security ssl X509TrustManagerImpl checkTrusted X509TrustManagerImpl java 223 ~ 1 8 0 282 at sun security ssl X509TrustManagerImpl checkServerTrusted X509TrustManagerImpl java 129 ~ 1 8 0 282 at sun security ssl CertificateMessage$T12CertificateConsumer checkServerCerts CertificateMessage java 638 ~ 1 8 0 282 at sun security ssl CertificateMessage$T12CertificateConsumer onCertificate CertificateMessage java 473 ~ 1 8 0 282 at sun security ssl CertificateMessage$T12CertificateConsumer consume CertificateMessage java 369 ~ 1 8 0 282 at sun security ssl SSLHandshake consume SSLHandshake java 377 ~ 1 8 0 282 at sun security ssl HandshakeContext dispatch HandshakeContext java 444 ~ 1 8 0 282 at sun security ssl HandshakeContext dispatch HandshakeContext java 422 ~ 1 8 0 282 at sun security ssl TransportContext dispatch TransportContext java 182 ~ 1 8 0 282 at sun security ssl SSLTransport decode SSLTransport java 149 ~ 1 8 0 282 at sun security ssl SSLSocketImpl decode SSLSocketImpl java 1143 ~ 1 8 0 282 at sun security ssl SSLSocketImpl readHandshakeRecord SSLSocketImpl java 1054 ~ 1 8 0 282 at sun security ssl SSLSocketImpl startHandshake SSLSocketImpl java 394 ~ 1 8 0 282 at sun net www protocol https HttpsClient afterConnect HttpsClient java 559 ~ 1 8 0 282 at sun net www protocol https AbstractDelegateHttpsURLConnection connect AbstractDelegateHttpsURLConnection java 185 ~ 1 8 0 282 at sun net www protocol http HttpURLConnection getInputStream0 HttpURLConnection java 1570 ~ 1 8 0 282 at sun net www protocol http HttpURLConnection getInputStream HttpURLConnection java 1498 ~ 1 8 0 282 at java net HttpURLConnection getResponseCode HttpURLConnection java 480 ~ 1 8 0 282 at sun net www protocol https HttpsURLConnectionImpl getResponseCode HttpsURLConnectionImpl java 352 ~ 1 8 0 282 at io confluent kafka schemaregistry client rest RestService sendHttpRequest RestService java 212 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client rest RestService httpRequest RestService java 256 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client rest RestService getId RestService java 486 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client rest RestService getId RestService java 479 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client CachedSchemaRegistryClient getSchemaByIdFromRegistry CachedSchemaRegistryClient java 177 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client CachedSchemaRegistryClient getBySubjectAndId CachedSchemaRegistryClient java 256 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka schemaregistry client CachedSchemaRegistryClient getById CachedSchemaRegistryClient java 235 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka serializers AbstractKafkaAvroDeserializer deserialize AbstractKafkaAvroDeserializer java 107 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka serializers AbstractKafkaAvroDeserializer deserialize AbstractKafkaAvroDeserializer java 79 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at io confluent kafka serializers KafkaAvroDeserializer deserialize KafkaAvroDeserializer java 55 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot plugin inputformat avro confluent KafkaConfluentSchemaRegistryAvroMessageDecoder decode KafkaConfluentSchemaRegistryAvroMessageDecoder java 114 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot plugin inputformat avro confluent KafkaConfluentSchemaRegistryAvroMessageDecoder decode KafkaConfluentSchemaRegistryAvroMessageDecoder java 120 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot plugin inputformat avro confluent KafkaConfluentSchemaRegistryAvroMessageDecoder decode KafkaConfluentSchemaRegistryAvroMessageDecoder java 53 ~ pinot confluent avro 0 7 0 SNAPSHOT shaded jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e at org apache pinot core data manager realtime LLRealtimeSegmentDataManager processStreamEvents LLRealtimeSegmentDataManager java 471 ~ pinot all 0 7 0 SNAPSHOT jar with dependencies jar 0 7 0 SNAPSHOT a44d0b1bb64d00d851ea6f2d8bc46ff0ab080d3e ```

Daniel Lavoie · Answer

```sun security validator ValidatorException PKIX path building failed sun security provider certpath SunCertPathBuilderException unable to find valid certification path to requested target```

Alexander Vivas · Answer

Yep but all the certs are in place

Daniel Lavoie · Answer

You need to add the schema registry root CA in the JKS of your JVM

Alexander Vivas · Answer

And the table s schema is pointing to the right truststore

Daniel Lavoie · Answer

It s going to be verbose but you can activate the SSL debug logs so you can confirm what is effectively loaded from the truststore

Daniel Lavoie · Answer

If they are getting mixed you ll have a trace

Alexander Vivas · Answer

Nice all right