I'm looking for some pointers on how to do performance analysis on my operator. My operator which is 100% busy has about 10 steps in it which include:
• convert from stream to table
• Perform simple UDF calculation on input rows
• Join with external data, using cached lookup join
• Convert back to stream
The performance I consider terrible, around 16 - 50 messages in/out per second per subtask. I would expect at least 10x this given what is being asked of Flink(all per partition computation with cached data). This really feels like a IO problem, but i'm completely in the dark with how to narrow it down. Splitting this into more partitions doesn't seem to help either as my workload requires partitioning data by ID, and processing 100k-700k records for a given ID. With the current rate it takes approximately 40 minutes to reprocess this amount of data.