Hi everyone, Our flink job is running for sometim...
# troubleshooting
s
Hi everyone, Our flink job is running for sometime and going to failing state with below error. Can anyone please help us in finding what exactly can be the issue. The source and sink both are kafka topics.
Copy code
org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Internal server error occurred. Stack trace: 
org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:92) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:61) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.ChainingOutput.collectAndCheckIfChained(ChainingOutput.java:95) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.BroadcastingOutputCollector.collect(BroadcastingOutputCollector.java:93) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperator$ReadOnlyContextImpl.output(CoBroadcastWithNonKeyedOperator.java:230) ~[flink-dist-1.18.1.jar:1.18.1]
	at com.rakutenrewards.messaging.functions.MessageProcessor.processElement(MessageProcessor.java:165) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at com.rakutenrewards.messaging.functions.MessageProcessor.processElement(MessageProcessor.java:59) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperator.processElement1(CoBroadcastWithNonKeyedOperator.java:111) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.io.StreamTwoInputProcessorFactory$StreamTaskNetworkOutput.emitRecord(StreamTwoInputProcessorFactory.java:254) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:146) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.io.AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:110) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.io.StreamMultipleInputProcessor.processInput(StreamMultipleInputProcessor.java:85) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953) [flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932) [flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746) [flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist-1.18.1.jar:1.18.1]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:92) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) ~[flink-dist-1.18.1.jar:1.18.1]
	... 21 more
Caused by: org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1073) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:947) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.send(FlinkKafkaInternalProducer.java:80) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.flink.connector.kafka.sink.KafkaWriter.write(KafkaWriter.java:198) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:161) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) ~[flink-dist-1.18.1.jar:1.18.1]
	... 21 more
Caused by: java.lang.InterruptedException
	at java.lang.Object.wait(Native Method) ~[?:?]
	at org.apache.kafka.common.utils.SystemTime.waitObject(SystemTime.java:55) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.kafka.clients.producer.internals.ProducerMetadata.awaitUpdate(ProducerMetadata.java:119) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1132) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:981) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:947) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.send(FlinkKafkaInternalProducer.java:80) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.flink.connector.kafka.sink.KafkaWriter.write(KafkaWriter.java:198) ~[blob_p-351bb854a158b335101bc3258f451f3cbb42fb1a-94e705edd09d21767d33e282343b4df0:?]
	at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:161) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:38) ~[flink-dist-1.18.1.jar:1.18.1]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) ~[flink-dist-1.18.1.jar:1.18.1]
	... 21 more
a
A in
InterruptedException
could indicate that another error happened earlier. Are other errors mentioned in the logs?