Hey folks, one of our jobs (1.15.2) got stuck in r...
# troubleshootingm
Matyas Orhidi
01/19/2023, 6:01 PMHey folks, one of our jobs (1.15.2) got stuck in resource allocation loop after a lost HB, has anyone seen this?
Copy code
Heartbeat of TaskManager with id <app-name>-2-1 timed out.
Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeout
Slot request bulk is not fulfillable! Could not allocate the required slot within slot request timeoutMatyas Orhidi
01/19/2023, 6:03 PMThe job keeps restarting in every 5 mins and gets stuck in
CREATEDMatyas Orhidi
01/19/2023, 6:04 PMWe're using native k8s, and the JM/TM pods are intact.
Matyas Orhidi
01/19/2023, 6:05 PMThe problematic TM keeps receiving the slot allocation requests:
Copy code
2023-01-19 00:34:37,465 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive slot request 3014cf663b14a298593d64574322cbd5 for job 00000000000000000000000000000002 from resource manager with leader id accbf5b4f4d6afd68065b5da23ef47b5.
2023-01-19 00:34:37,465 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Allocated slot for 3014cf663b14a298593d64574322cbd5.
2023-01-19 00:34:37,465 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive slot request d32c5af2263c6f538bfd00f8ca088416 for job 00000000000000000000000000000002 from resource manager with leader id accbf5b4f4d6afd68065b5da23ef47b5.
2023-01-19 00:34:37,465 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Allocated slot for d32c5af2263c6f538bfd00f8ca088416.
2023-01-19 00:34:37,465 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Receive slot request 8c3baf254f51a9ff0f198ab605b6ea49 for job 00000000000000000000000000000002 from resource manager with leader id accbf5b4f4d6afd68065b5da23ef47b5.