We had an issue where confluent schema registry had downtime

Question

We had an issue where confluent schema registry had downtime and realtime ingestion failed with Read from kafka failed but did not recover until we restarted the servers Is this a known issue Or is there something else we could have done The realtime table was on the default tenant and I issued a rebalance that did not help old data was there but no consuming segments

Xiang Fu · Answer

i feel we should have a cache for schema

Elon · Answer

here s the stack trace we saw

Xiang Fu · Answer

for confluent schema client

Elon · Answer

```2021 02 09 20 45 06 889 ERROR ServerSegmentCompletionProtocolHandler oas integration operation event 3 331 20210209T2044Z Could not send request <http pinot us central1 contro> ller 1 pinot us central1 controller headless pinot svc cluster local 9000 segmentStoppedConsuming name=oas integration operation event 3 331 20210209T2044Z amp offset=9259054 amp instance= Server pinot us central1 server 0 pinot us central1 server headless pinot svc cluster local 8098 amp reason=org apache kafka common errors SerializationException amp streamPartitionMsgOffset= 9259054 java net SocketTimeoutException Read timed out at java net SocketInputStream socketRead0 Native Method ~ at java net SocketInputStream socketRead SocketInputStream java 115 ~ at java net SocketInputStream read SocketInputStream java 168 ~ at java net SocketInputStream read SocketInputStream java 140 ~ at shaded org apache http impl io SessionInputBufferImpl streamRead SessionInputBufferImpl java 137 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 7115670e5c2b152812a09bd0 1```

Xiang Fu · Answer

this seems to be pinot controller issue not schema registry

Elon · Answer

Oh sorry wrong log let me see if I can still get it

Elon · Answer

I lost the log message but have you ever seen an error reading from kafka ex kafka is down and then realtime ingestion stops until the servers are restarted

Xiang Fu · Answer

ic

Xiang Fu · Answer

Xiang Fu · Answer

do we configured this <https docs confluent io 5 4 0 clients confluent kafka dotnet api Confluent SchemaRegistry ISchemaRegistryClient html Confluent SchemaRegistry ISchemaRegistryClient MaxCachedSchemas|MaxCachedSchemas>

Elon · Answer

It s default to 1000

Xiang Fu · Answer

ok

Elon · Answer

I think this was more that the kafka consumer failed due to schema registry being down for ~1 hour and then ingestion stopped did not see any errors in externalview or idealstate also did not see consuming segments until I restarted the servers

Xiang Fu · Answer

< Neha Pawar> ^^

Xiang Fu · Answer

I saw that this client is inside the avrodeserializer

Elon · Answer

Yep

Xiang Fu · Answer

so if there is no msg consumed from kafka then this logic shouldn t be triggered

Neha Pawar · Answer

i think the behavior is by design The consumer was likely seeing exceptions when trying to consume from kafka with the schema registry down ```while shouldStop amp amp endCriteriaReached Consume for the next readTime ms or we get to final offset whichever happens earlier Update currentOffset upon return from this method MessageBatch messageBatch try messageBatch = partitionLevelConsumer fetchMessages currentOffset null partitionLevelStreamConfig getFetchTimeoutMillis consecutiveErrorCount = 0 catch TimeoutException e handleTransientStreamErrors e continue catch TransientConsumerException e handleTransientStreamErrors e continue catch PermanentConsumerException e segmentLogger warn Permanent exception from stream when fetching messages stopping consumption e throw e catch Exception e Unknown exception from stream Treat as a transient exception One such exception seen so far is java net SocketTimeoutException handleTransientStreamErrors e continue ``` if permanent exception or more than 5 transient exceptions we stop consuming by marking the consuming segment offline

Neha Pawar · Answer

and an operator would need to either reset the partition or restart the server

Elon · Answer

Thanks This helps

Elon · Answer

Is there a metric we can use to detect when there s a delay between kafka topic and pinot realtime table

Neha Pawar · Answer

that metric is much requested but much missing see no evil i recall it has something to do with Kafka not exposing the latest offset in an easy way

Neha Pawar · Answer

but i remember in LinkedIn we used to use ```ServerGauge LLC PARTITION CONSUMING```

Neha Pawar · Answer

this flag turns to 0 whenever the partition is not consuming

Neha Pawar · Answer

we used to have alerts that fired if this metric stayed 0 for gt 15 minutes on any partition

Elon · Answer

Thanks I ll check that one out

Elon · Answer

< Neha Pawar> that gauge works for us The cluster usually has a steady rate of ingestion so we alert on sudden decrease and so far it s working as expected

Neha Pawar · Answer

awesome