hei, I noticed that sourcefunction is deprecated. ...
# troubleshooting
o
hei, I noticed that sourcefunction is deprecated. We are running into a use case where we would like to create mock data in an unbounded manner for load testing. What is the simplest workaround to use instead SourceFunction? I have seen that there is MockSource but I think this is for another purposes. Thanks for the help
also DataGeneratorSource is deprecated 😞
s
i think you still can use datagen source, just need to change its maven package to this : https://mvnrepository.com/artifact/org.apache.flink/flink-connector-datagen
@deprecated Use {@code org.apache.flink.connector.datagen.source.DataGeneratorSource} instead.
or, following this person’s example is also a good start for creating your own custom source: https://github.com/SelimAbidin/CustomFlinkSource/blob/main/src/main/scala/org/example/custom/source/IntSource.scala
n
I did this recently in
1.18
while working on a new project:
Copy code
val recordTag: OutputTag<KafkaRecord> = object : OutputTag<KafkaRecord>("campaign") {}    

val syntheticData = generateSyntheticData(environment)

    val processedData =
        syntheticData.process(
            object : ProcessFunction<KafkaRecord, KafkaRecord>() {
                override fun processElement(
                    value: KafkaRecord,
                    ctx: Context,
                    out: Collector<KafkaRecord>,
                ) {
                    ctx.output(recordTag, value)
                }
            },
        )
and then
Copy code
fun generateSyntheticData(env: StreamExecutionEnvironment) =
    env.fromSequence(1, 12)
        .map { count ->
            val recordPayload = KafkaRecord.Payload(generateSourceRecord(count.toInt()))
            val recordHeaders = RecordHeaders().add("header-key", "header-value".toByteArray())
            KafkaRecord(recordPayload, recordHeaders)
        }
... etc.
maybe that helps