Oscar Perez
05/31/2024, 11:08 AMOscar Perez
05/31/2024, 11:12 AMshikai ng
06/01/2024, 1:50 AM@deprecated Use {@code org.apache.flink.connector.datagen.source.DataGeneratorSource} instead.
shikai ng
06/01/2024, 1:52 AMNicholas Erasmus
06/04/2024, 3:35 PM1.18 while working on a new project:
val recordTag: OutputTag<KafkaRecord> = object : OutputTag<KafkaRecord>("campaign") {}
val syntheticData = generateSyntheticData(environment)
val processedData =
syntheticData.process(
object : ProcessFunction<KafkaRecord, KafkaRecord>() {
override fun processElement(
value: KafkaRecord,
ctx: Context,
out: Collector<KafkaRecord>,
) {
ctx.output(recordTag, value)
}
},
)
and then
fun generateSyntheticData(env: StreamExecutionEnvironment) =
env.fromSequence(1, 12)
.map { count ->
val recordPayload = KafkaRecord.Payload(generateSourceRecord(count.toInt()))
val recordHeaders = RecordHeaders().add("header-key", "header-value".toByteArray())
KafkaRecord(recordPayload, recordHeaders)
}
... etc.
maybe that helps