Hi everyone, i am migrating some legacy Java code....
# troubleshooting
a
Hi everyone, i am migrating some legacy Java code. We are moving from DataSet to DataStream and I can't find a match for SingleInputUdfOperator.withBroadcastSet(DataSet, String). Whats the way to go here? We have something like this:
Copy code
class SomeRichMapFunction : RichMapFunction<Data, OutputData>() {
    private lateinit var oneFairlyMediumSizedElement: OtherData

    override fun open(parameters: Configuration) {
        oneFairlyMediumSizedElement = runtimeContext.getBroadcastVariable<OtherData>("oneFairlyMediumSizedElementBroadcastSet").single()
    }

    override fun map(value: Data): OutputData {
        TODO("Do some funky calc with value and oneFairlyMediumSizedElement")
    }
}


fun calculate(huge: DataSet<Data>, oneFairlyMediumSizedElement: DataSet<OtherData>) =
    huge.map(SomeRichMapFunction())
        .withBroadcastSet(oneFairlyMediumSizedElement, "oneFairlyMediumSizedElementBroadcastSet")
d
When migrating from DataSet to DataStream API in Apache Flink, the concept of broadcast variables is still available but the method to set them up changes slightly. The withBroadcastSet method you’re familiar with from the Dataset API doesn’t directly translate to the DataStream API. Instead, you should use connect to create a BroadcastStream, and then use map or flatmap with a BroadcastProcessFunction.
so something like
Copy code
class SomeBroadcastProcessFunction : BroadcastProcessFunction<Data, OtherData, OutputData>() {

    override fun processElement(data: Data, ctx: Context, out: Collector<OutputData>) {
        val oneFairlyMediumSizedElement = ctx.getBroadcastState(...).value() // Access the broadcast variable
        // TODO: Do some funky calc with data and oneFairlyMediumSizedElement, emit result with out.collect(...)
    }

    override fun processBroadcastElement(otherData: OtherData, ctx: Context, state: BroadcastState<...>) {
        // Update the broadcast state when new elements arrive
        state.put("key", otherData) // The key could be anything that identifies this element, adjust as needed
    }
}
Thats basically the approach
❤️ 1