Flavio V.
07/08/2024, 4:27 PMDataStream<SampleInstancePOJO> inputStream = env
.fromSource(myConsumer, WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofMinutes(1)), "KafkaSourceCDC")
.setParallelism(15)
.assignTimestampsAndWatermarks(watermarkStrategy)
.filter(jsonValidator)
.setParallelism(15)
.rebalance()
.keyBy(cpe -> cpe.elementId)
.window(TumblingProcessingTimeWindows.of(Time.minutes(60)))
.aggregate(new AggregateDevice(), new MyProcessWindowFunction())
.setParallelism(46)
.name("Sampler")
.uid("UID-CONSUMER");
inputStream
.rebalance()
.sinkTo(topic)
.setParallelism(20)
.name("Sampleador")
.uid("UID-PRODUCER");
env.execute("POC * v1.0");
public static class MyProcessWindowFunction
extends ProcessWindowFunction<SampleInstancePOJO, SampleInstancePOJO, String, TimeWindow> {
private static final long serialVersionUID = -4288330487045760101L;
public MyProcessWindowFunction() {
}
@Override
public void process(String sampleInstancePOJO,
ProcessWindowFunction<SampleInstancePOJO, SampleInstancePOJO, String, TimeWindow>
.Context context, Iterable<SampleInstancePOJO> input, Collector<SampleInstancePOJO> output)
throws Exception {
for (SampleInstancePOJO in: input) {
output.collect(in);
}
}
}
If anyone can give me any clue about why I have this kind of behavior between my consumer and producer task I will thanksfully!!
Regards
FlavioGeorge Leonard
07/08/2024, 6:43 PMFlavio V.
07/08/2024, 7:49 PM<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kafka</artifactId>
<version>3.0.2-1.18</version>
</dependency>Flavio V.
07/08/2024, 10:38 PMFlavio V.
07/09/2024, 3:46 PMFlavio V.
07/09/2024, 3:47 PMFlavio V.
07/09/2024, 8:59 PM