I'm looking for some pointers on how to do perform...
# troubleshooting
d
I'm looking for some pointers on how to do performance analysis on my operator. My operator which is 100% busy has about 10 steps in it which include: • convert from stream to table • Perform simple UDF calculation on input rows • Join with external data, using cached lookup join • Convert back to stream The performance I consider terrible, around 16 - 50 messages in/out per second per subtask. I would expect at least 10x this given what is being asked of Flink(all per partition computation with cached data). This really feels like a IO problem, but i'm completely in the dark with how to narrow it down. Splitting this into more partitions doesn't seem to help either as my workload requires partitioning data by ID, and processing 100k-700k records for a given ID. With the current rate it takes approximately 40 minutes to reprocess this amount of data.
r
You may try exploring this config. https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/config/#pipeline-object-reuse In flink's terminology, I think you are referring to the task being 100% busy. Operators are chained to form a task. The data exchange between consecutive operators (even within the same task) goes through serde layer. I was also facing a similar issue where kryo serde was heavily slowing down my task. After enabling object re-use, the throughput of my task just went into different orbit 🚀 !!
d
That’s very interesting