https://flink.apache.org/ logo
Join Slack
Powered by
I'm getting started running Batch SQL Jobs in Flin...
# troubleshooting
j
I'm getting started running Batch SQL Jobs in Flink, and I've run into this strange behavior a couple of times. One Task completes, a subsequent Task fails and gets cancelled, and rather than the Job failing or getting cancelled it stays in a Running state forever (never completes). Any idea what the cause is or how to keep this from happening?
If it matters, there was an out of memory error on a Task Manager that seems to have caused that stage to fail.
What I'd like to have happen is for the failed Task to cause the whole Job to fail, that way our restart policy can take over and restart the job, or it can wind up in a Failed state
I added checkpointing to this job to see if that helps stability/gives us proper restarting behavior. So far so good, but this job ran very well for two days before this happened.
m
You mean batch execution mode when you say “Batch SQL” jobs?
j
Yes, I do
m
That won’t have an effect because checkpointing does not work for batch execution https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/execution_mode/#important-considerations
💯 1
j
Any idea what's happening here @Martijn Visser?
m
I’m not too big of a batch expert. I would think of a resource issue given that batch only emits a final result, but that would be my starting point
j
This has recurred again. This time I wound up with the Tasks stuck in CREATED. We run these jobs hourly, and it happened once since Friday afternoon, so it's not frequent. It's also our staging cluster, which is not as robust as prod. Has anyone else ever seen anything similar with batch mode?
w
Could you post the log of 
jobmanager
? I need to find the reason why the failover task has not changed to the status of 
schduled
.
j
@weijie guo Yep! Fortunately the cluster wasn't doing too much else during the period in question, it's actually pretty informative
JobManager.log.txt
JobManager.log.txt
Copy code
2023-01-25 16:16:01,964 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - Discarding the results produced by task execution f72b637e4b71463d0985c55a9ddd69c5.
That line sticks out to me, but I'm not sure why that would be the case
I already took some steps to address the instability that led to us losing task managers
w
It is very strange that the failed task changes its state to cancel, but there is no reason for reporting in the log and jobMaster does not seem to trigger failover. Can you still get the log of this TM (
10-50-89-193. flik-taskmanager-query-state-optic.svc.cluster.local (dataPort=32835)
)? I want to see why TM decided to cancel the corresponding task through the log.
j
I can't, sorry. This happened on Saturday and we don't retain the logs very long
2 Views
Open in Slack
PreviousNext
Powered by