We're trying to upgrade a flink job (~using a save...
# troubleshooting
t
We're trying to upgrade a flink job (~using a savepoint~by trying to create a savepoint) that is currently stuck in a crash restart loop due to a bug triggered by the most recent input. What's the best way to deal with this? The naive solution (just stopping the job with a savepoint) fails due to the tasks already having terminated ("Failure reason: Not all required tasks are currently running.").
p
Do you use the operator?
t
The one that is "not running"? Yes, the whole job is basically just a single
KafkaSource->IcebergSink
p
But I mean, what did you use to deploy your pipeline? Isn’t it restarting because the previous resource is stuck? Do you use k8s?
t
It's running with AWS EMR (which from my understanding is really just a stand-alone cluster on a bunch of EC2 instances). It's just the cluster restart policy that is restarting the task due to an unhandled exception.
p
Can’t you force remove the running one and redeploy it with no upgrade?
t
That's out current plan, but it means manually specifying the starting offset of the Kafka source. That's OK in this case, but in a more complex setup it might not be viable, so I wanted to understand if there was a better way.
p
it means manually specifying the starting offset of the Kafka source
If you’re using a savepoint it’s not required
t
But it is precisely the savepoint it is failing to produce
p
Oh. I thought you had onse savepoint already and you were trying to restore from it
t
Sorry, sloppy wording on my part :)