Hi all, I'm here because I'm developing a proof of...
# troubleshooting
f
Hi all, I'm here because I'm developing a proof of concept with Flink for real time processing using Kafka, so, my application has a consumer and a producer with a time window of 60 minutes. I am consuming all the time and every 60 minutes my application produces a result after parsing the JSON objects received in a topic. This is the behavior of my application in a Grafana view: See attach 1 please. This information corresponds to the last 12 hours. This is about the last 2 hours: See attach 2 please. When my application is producing information my consumer goes down to zero, so, the delay grows, then, when my producer finishes the task the consumer starts again and the same behavior is repeated every 60 minutes. I don’t know why I have this behavior. I think it is because the producer is somehow blocking the rest of the tasks but I don't understand why this is happening. I work with Flink 1.19.1 in java This is my stream:
DataStream<SampleInstancePOJO> inputStream = env
.fromSource(myConsumer, WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofMinutes(1)), "KafkaSourceCDC")
.setParallelism(15)
.assignTimestampsAndWatermarks(watermarkStrategy)
.filter(jsonValidator)
.setParallelism(15)
.rebalance()
.keyBy(cpe -> cpe.elementId)
.window(TumblingProcessingTimeWindows.of(Time.minutes(60)))
.aggregate(new AggregateDevice(), new MyProcessWindowFunction())
.setParallelism(46)
.name("Sampler")
.uid("UID-CONSUMER");
inputStream
.rebalance()
.sinkTo(topic)
.setParallelism(20)
.name("Sampleador")
.uid("UID-PRODUCER");
env.execute("POC * v1.0");
public static class MyProcessWindowFunction
extends ProcessWindowFunction<SampleInstancePOJO, SampleInstancePOJO, String, TimeWindow> {
private static final long serialVersionUID = -4288330487045760101L;
public MyProcessWindowFunction() {
}
@Override
public void process(String sampleInstancePOJO,
ProcessWindowFunction<SampleInstancePOJO, SampleInstancePOJO, String, TimeWindow>
.Context context, Iterable<SampleInstancePOJO> input, Collector<SampleInstancePOJO> output)
throws Exception {
for (SampleInstancePOJO in: input) {
output.collect(in);
}
}
}
If anyone can give me any clue about why I have this kind of behavior between my consumer and producer task I will thanksfully!! Regards Flavio
g
you mention/show Grafana... did you prometheus instrument your code, or are you using say kafka connect framework to sink metrics into a TSDB ?
f
Thanks George for your interest! 🙂 I'm using kafka connect, this is my dependency:
Copy code
<dependency>
	    <groupId>org.apache.flink</groupId>
	    <artifactId>flink-connector-kafka</artifactId>
	    <version>3.0.2-1.18</version>
	</dependency>
sorry, I missed understood your question, the info in grafana is generated by like you said, some connector in our kafka server, but is not under my administration, so, I don't know how the information is captured. I have my users for only my topics, in this case I have an user for read from a topic and a user for write in another topic and I can use tools like grafana for see the behavior and kouncil for see the messages in topics.
👍 1
I have made some progress on my own, it seems that the problem is with the JSON size, maybe it is too big for our kafka because I sent a tiny JSON and everything works fine. My JSON is between 7 and 8 Kb. I will keep working with this to come to a better conclusion.
What do you think? 8Kb JSON is big?
I finally found the root of my problem, I had been working with a hashmap string, I mean a: Hashmap<String,Hashmap<String,Object>> but when I try to serialize my POJO object it required a lot of Flink effort, because I was using two "for", one to iterate over the first hasmap and one to iterate the second, so, I refactored my code and now it works better, now I'm doing some tests.