Jeremie Doehla
07/08/2025, 6:29 PMJdbcStatementBuilder that I'm working with and running into issues. If I have
val insertOperation: JdbcStatementBuilder[DataTracking] = (ps: PreparedStatement, dt: DataTracking) => {
ps.setString(1, dt.file_name)
ps.setBoolean(2, dt.processed_file)
ps.setString(3, dt.data_origin)
ps.setString(4, dt.data_type)
ps.setTimestamp(5, Timestamp.from(dt.received_time))
}
and use this statement everything is working correctly and processes fine. However if I update the table to add a new column and update the insert sql to also insert into this new column such that I now have this statement:
val insertOperation: JdbcStatementBuilder[DataTracking] = (ps: PreparedStatement, dt: DataTracking) => {
ps.setString(1, dt.file_name)
ps.setBoolean(2, dt.processed_file)
ps.setString(3, dt.data_origin)
ps.setString(4, dt.data_type)
ps.setObject(5, dt.data_location.orNull, Types.VARCHAR)
ps.setTimestamp(6, Timestamp.from(dt.received_time))
}
this results in an error. I've also used
dt.data_location match {
case Some(value) => ps.setString(5, value)
case None => ps.setNull(5, Types.VARCHAR)
}
rather than the setObject but see the same error either way. Any ideas on what to do here? I can provide the error message -- I'll place that in the thread.Jeremie Doehla
07/08/2025, 6:30 PM[info] [11:21:12] WARN JdbcOutputFormat - Writing records to JDBC failed.
[info] java.util.NoSuchElementException: None.get
[info] at scala.None$.get(Option.scala:627)
[info] at scala.None$.get(Option.scala:626)
[info] at scala.Option.orNull(Option.scala:201)
[info] at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:53)
[info] at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:48)
[info] at org.apache.flink.connector.jdbc.datasource.statements.SimpleJdbcQueryStatement.statement(SimpleJdbcQueryStatement.java:44)
[info] at org.apache.flink.connector.jdbc.internal.executor.SimpleBatchStatementExecutor.executeBatch(SimpleBatchStatementExecutor.java:66)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:198)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:172)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.close(JdbcOutputFormat.java:214)
[info] at org.apache.flink.connector.jdbc.core.datastream.sink.writer.JdbcWriter.close(JdbcWriter.java:209)
[info] at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:257)
[info] at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:236)
[info] at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:225)
[info] at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.close(SinkWriterOperator.java:287)
[info] at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:163)
[info] at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:125)
[info] at org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1180)
[info] at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:257)
[info] at org.apache.flink.core.fs.AutoCloseableRegistry.doClose(AutoCloseableRegistry.java:72)
[info] at org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
[info] at org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUp(StreamTask.java:1084)
[info] at org.apache.flink.runtime.taskmanager.Task.lambda$restoreAndInvoke$0(Task.java:945)
[info] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
[info] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:945)
[info] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
[info] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] [11:21:12] WARN Task - Source: KafkaSourceStream -> Map -> Map -> Sink: Writer -> Sink: Committer (1/1)#0 (3e8e3be1d8fa03244e959b74df13cc26_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to FAILED with failure cause:
[info] java.io.IOException: Failed to deserialize consumer record due to
[info] at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:56)
[info] at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:33)
[info] at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:164)
[info] at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:466)
[info] at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
[info] at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
[info] at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:637)
[info] at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
[info] at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:972)
[info] at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:916)
[info] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
[info] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:942)
[info] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:756)
[info] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:568)
[info] at java.base/java.lang.Thread.run(Thread.java:840)
[info] Suppressed: java.lang.RuntimeException: Writing records to JDBC failed.
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.close(JdbcOutputFormat.java:217)
[info] at org.apache.flink.connector.jdbc.core.datastream.sink.writer.JdbcWriter.close(JdbcWriter.java:209)
[info] at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:257)
[info] at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:236)
[info] at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:225)
[info] at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.close(SinkWriterOperator.java:287)
[info] at org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.close(StreamOperatorWrapper.java:163)
[info] at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.closeAllOperators(RegularOperatorChain.java:125)
[info] at org.apache.flink.streaming.runtime.tasks.StreamTask.closeAllOperators(StreamTask.java:1180)
[info] at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:257)
[info] at org.apache.flink.core.fs.AutoCloseableRegistry.doClose(AutoCloseableRegistry.java:72)
[info] at org.apache.flink.util.AbstractAutoCloseableRegistry.close(AbstractAutoCloseableRegistry.java:127)
[info] at org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUp(StreamTask.java:1084)
[info] at org.apache.flink.runtime.taskmanager.Task.lambda$restoreAndInvoke$0(Task.java:945)
[info] at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:963)
[info] at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:945)
[info] ... 3 more
[info] Caused by: java.util.NoSuchElementException: None.get
[info] at scala.None$.get(Option.scala:627)
[info] at scala.None$.get(Option.scala:626)
[info] at scala.Option.orNull(Option.scala:201)
[info] at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:53)
[info] at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:48)
[info] at org.apache.flink.connector.jdbc.datasource.statements.SimpleJdbcQueryStatement.statement(SimpleJdbcQueryStatement.java:44)
[info] at org.apache.flink.connector.jdbc.internal.executor.SimpleBatchStatementExecutor.executeBatch(SimpleBatchStatementExecutor.java:66)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:198)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:172)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.close(JdbcOutputFormat.java:214)
[info] ... 18 more
[info] Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:92)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
[info] at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:315)
[info] at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
[info] at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter$SourceOutputWrapper.collect(KafkaRecordEmitter.java:67)
[info] at org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:84)
[info] at gov.pnnl.cic.prototypeFlink.dataTypes.CirceDeserializationSchema.deserialize(CirceDeserializationSchema.scala:8)
[info] at org.apache.flink.connector.kafka.source.reader.deserializer.KafkaValueOnlyDeserializationSchemaWrapper.deserialize(KafkaValueOnlyDeserializationSchemaWrapper.java:51)
[info] at org.apache.flink.connector.kafka.source.reader.KafkaRecordEmitter.emitRecord(KafkaRecordEmitter.java:53)
[info] ... 14 more
[info] Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:92)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
[info] at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:37)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
[info] ... 23 more
[info] Caused by: org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException: Could not forward element to next operator
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:92)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
[info] at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:37)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
[info] ... 27 more
[info] Caused by: java.io.IOException: Writing records to JDBC failed.
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.writeRecord(JdbcOutputFormat.java:154)
[info] at org.apache.flink.connector.jdbc.core.datastream.sink.writer.JdbcWriter.write(JdbcWriter.java:137)
[info] at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:223)
[info] at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
[info] ... 31 more
[info] Caused by: java.util.NoSuchElementException: None.get
[info] at scala.None$.get(Option.scala:627)
[info] at scala.None$.get(Option.scala:626)
[info] at scala.Option.orNull(Option.scala:201)
[info] at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:53)
[info] at gov.pnnl.cic.prototypeFlink.dataTypes.DataTracking$$anon$1.accept(DataTracking.scala:48)
[info] at org.apache.flink.connector.jdbc.datasource.statements.SimpleJdbcQueryStatement.statement(SimpleJdbcQueryStatement.java:44)
[info] at org.apache.flink.connector.jdbc.internal.executor.SimpleBatchStatementExecutor.executeBatch(SimpleBatchStatementExecutor.java:66)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.attemptFlush(JdbcOutputFormat.java:198)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.flush(JdbcOutputFormat.java:172)
[info] at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.writeRecord(JdbcOutputFormat.java:151)
[info] ... 34 moreJeremie Doehla
07/10/2025, 6:21 PMdt.data_location match {
case Some(value) => ps.setString(5, value)
case _ => ps.setNull(5, Types.VARCHAR)
}
works. Why the case _ works and the case None fails eludes me presently (especially since the dt.data_location is an Option[String]).