Hello -- I'm hoping someone might be able to point...
# troubleshooting
j
Hello -- I'm hoping someone might be able to point out what I'm missing here. I've got a
JdbcStatementBuilder
that I'm working with and running into issues. If I have
Copy code
val insertOperation: JdbcStatementBuilder[DataTracking] = (ps: PreparedStatement, dt: DataTracking) => {
    ps.setString(1, dt.file_name)
    ps.setBoolean(2, dt.processed_file)
    ps.setString(3, dt.data_origin)
    ps.setString(4, dt.data_type)
    ps.setTimestamp(5, Timestamp.from(dt.received_time))
  }
and use this statement everything is working correctly and processes fine. However if I update the table to add a new column and update the insert sql to also insert into this new column such that I now have this statement:
Copy code
val insertOperation: JdbcStatementBuilder[DataTracking] = (ps: PreparedStatement, dt: DataTracking) => {
    ps.setString(1, dt.file_name)
    ps.setBoolean(2, dt.processed_file)
    ps.setString(3, dt.data_origin)
    ps.setString(4, dt.data_type)
    ps.setObject(5, dt.data_location.orNull, Types.VARCHAR)
    ps.setTimestamp(6, Timestamp.from(dt.received_time))
  }
this results in an error. I've also used
Copy code
dt.data_location match {
      case Some(value) => ps.setString(5, value)
      case None => ps.setNull(5, Types.VARCHAR)
    }
rather than the
setObject
but see the same error either way. Any ideas on what to do here? I can provide the error message -- I'll place that in the thread.
Here is the error message seen:
Copy code
[info] [11:21:12] WARN  JdbcOutputFormat - Writing records to JDBC failed.
[info] java.util.NoSuchElementException: None.get
[info] 	at scala.None$.get(Option.scala:627)
[info] 	at scala.None$.get(Option.scala:626)
[info] 	at scala.Option.orNull(Option.scala:201)
[info] 	at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:53)
[info] 	at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:48)
[info] 	at org.apache.flink.connector.jdbc.datasource.statements.SimpleJdbcQueryStatement.statement(SimpleJdbcQueryStatement.java:44)
[info] 	at org.apache.flink.connector.jdbc.internal.executor.SimpleBatchStatementExecutor.executeBatch(SimpleBatchStatementExecutor.java:66)
[info] 	at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:198)
[info] 	at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:172)
[info] 	at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.close(JdbcOutputFormat.java:214)
[info] 	at org.apache.flink.connector.jdbc.core.datastream.sink.writer.JdbcWriter.close(JdbcWriter.java:209)
[info] 	at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:257)
[info] 	at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:236)
[info] 	at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:225)
[info] 	at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.close(SinkWriterOperator.java:287)
[info] 	at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:163)
[info] 	at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:125)
[info] 	at org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1180)
[info] 	at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:257)
[info] 	at org.apache.flink.core.fs.AutoCloseableRegistry.doClose(AutoCloseableRegistry.java:72)
[info] 	at org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
[info] 	at org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUp(StreamTask.java:1084)
[info] 	at org.apache.flink.runtime.taskmanager.Task.lambda$restoreAndInvoke$0(Task.java:945)
[info] 	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
[info] 	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:945)
[info] 	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
[info] 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
[info] 	at java.base/java.lang.Thread.run(Thread.java:840)
[info] [11:21:12] WARN  Task - Source: KafkaSourceStream -> Map -> Map -> Sink: Writer -> Sink: Committer (1/1)#0 (3e8e3be1d8fa03244e959b74df13cc26_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to FAILED with failure cause:
[info] java.io.IOException: Failed to deserialize consumer record due to
[info] 	at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:56)
[info] 	at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:33)
[info] 	at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:164)
[info] 	at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:466)
[info] 	at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
[info] 	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
[info] 	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:637)
[info] 	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
[info] 	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:972)
[info] 	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:916)
[info] 	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
[info] 	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:942)
[info] 	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
[info] 	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
[info] 	at java.base/java.lang.Thread.run(Thread.java:840)
[info] 	Suppressed: java.lang.RuntimeException: Writing records to JDBC failed.
[info] 		at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.close(JdbcOutputFormat.java:217)
[info] 		at org.apache.flink.connector.jdbc.core.datastream.sink.writer.JdbcWriter.close(JdbcWriter.java:209)
[info] 		at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:257)
[info] 		at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:236)
[info] 		at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:225)
[info] 		at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.close(SinkWriterOperator.java:287)
[info] 		at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:163)
[info] 		at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:125)
[info] 		at org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1180)
[info] 		at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:257)
[info] 		at org.apache.flink.core.fs.AutoCloseableRegistry.doClose(AutoCloseableRegistry.java:72)
[info] 		at org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
[info] 		at org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUp(StreamTask.java:1084)
[info] 		at org.apache.flink.runtime.taskmanager.Task.lambda$restoreAndInvoke$0(Task.java:945)
[info] 		at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
[info] 		at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:945)
[info] 		... 3 more
[info] 	Caused by: java.util.NoSuchElementException: None.get
[info] 		at scala.None$.get(Option.scala:627)
[info] 		at scala.None$.get(Option.scala:626)
[info] 		at scala.Option.orNull(Option.scala:201)
[info] 		at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:53)
[info] 		at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:48)
[info] 		at org.apache.flink.connector.jdbc.datasource.statements.SimpleJdbcQueryStatement.statement(SimpleJdbcQueryStatement.java:44)
[info] 		at org.apache.flink.connector.jdbc.internal.executor.SimpleBatchStatementExecutor.executeBatch(SimpleBatchStatementExecutor.java:66)
[info] 		at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:198)
[info] 		at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:172)
[info] 		at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.close(JdbcOutputFormat.java:214)
[info] 		... 18 more
[info] Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:92)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
[info] 	at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:315)
[info] 	at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
[info] 	at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter$SourceOutputWrapper.collect(KafkaRecordEmitter.java:67)
[info] 	at org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:84)
[info] 	at gov.pnnl.cic.prototypeFlink.dataTypes.CirceDeserializationSchema.deserialize(CirceDeserializationSchema.scala:8)
[info] 	at org.apache.flink.connector.kafka.source.reader.deserializer.KafkaValueOnlyDeserializationSchemaWrapper.deserialize(KafkaValueOnlyDeserializationSchemaWrapper.java:51)
[info] 	at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:53)
[info] 	... 14 more
[info] Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:92)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
[info] 	at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:37)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
[info] 	... 23 more
[info] Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:92)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
[info] 	at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:37)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
[info] 	... 27 more
[info] Caused by: java.io.IOException: Writing records to JDBC failed.
[info] 	at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.writeRecord(JdbcOutputFormat.java:154)
[info] 	at org.apache.flink.connector.jdbc.core.datastream.sink.writer.JdbcWriter.write(JdbcWriter.java:137)
[info] 	at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:223)
[info] 	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
[info] 	... 31 more
[info] Caused by: java.util.NoSuchElementException: None.get
[info] 	at scala.None$.get(Option.scala:627)
[info] 	at scala.None$.get(Option.scala:626)
[info] 	at scala.Option.orNull(Option.scala:201)
[info] 	at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:53)
[info] 	at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:48)
[info] 	at org.apache.flink.connector.jdbc.datasource.statements.SimpleJdbcQueryStatement.statement(SimpleJdbcQueryStatement.java:44)
[info] 	at org.apache.flink.connector.jdbc.internal.executor.SimpleBatchStatementExecutor.executeBatch(SimpleBatchStatementExecutor.java:66)
[info] 	at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:198)
[info] 	at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:172)
[info] 	at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.writeRecord(JdbcOutputFormat.java:151)
[info] 	... 34 more
Don't understand why this would be the case but apparently using
Copy code
dt.data_location match {
      case Some(value) => ps.setString(5, value)
      case _ => ps.setNull(5, Types.VARCHAR)
    }
works. Why the
case _
works and the
case None
fails eludes me presently (especially since the
dt.data_location
is an
Option[String]
).