Just recently started experiencing this after a fr...
# troubleshooting
r
Just recently started experiencing this after a fresh deployment of a job after enabling DEBUG-level logging and am trying to diagnose/get the job healthy:
Copy code
last log message from the stuck taskmanager `2024-07-12 14:50:42,394 DEBUG org.apache.flink.runtime.throughput.BufferDebloater          [] - my-operator (1/4)#0 (ac9b4d5dbbf364eb45963a4057452142_8423c94737d0e0db49aa6dbc6e8ac97c_0_0): Buffer size recalculation: gateIndex=0, currentSize=256, newSize=256, instantThroughput=0, desiredBufferSize=0, buffersInUse=25, estimatedTimeToConsumeBuffers=PT1H46M40S, announceNewSize=false`
This could be a red-herring altogether as I’ve never seen it previously. I’ve restarted the job multiple times, including sans any previous state (stateless) and I continually have a single task for an operator that never comes up. The above is the only thing that I see in the logs that sounds remotely related. All other operators/tasks are successfully running however this one task appears to be stuck: Any advice?
d
Looks like you have already been debugging this for a while so I might suggest you use a Heap Dump Analysis on the task manager process and using VisualVM or YourKit to look for abnormal behavior.
Are you using trace level logging?
You should also check your flink version and dependencies for compatibility issues.
Check resource constraints for the taskmanager running this as well.
Check for recent changes to the operator as well.
The message you posted does not seem to indicate a problem its a standard message to optimize network buffer usage. You might check for other messages in task manager and job manager logs around the timeframe.
Make sure CPU, Diskspace and Network bandwidth resources are all sufficient for the taskmanager responsible for the stuck task. Insuffient resources can cause this behavior as well.