Hi. Having a weird issue in one of our AWS cluster...
# troubleshooting
m
Hi. Having a weird issue in one of our AWS clusters: some flink deployments (flink:1.20.0-java17, operator 1.10) get stuck in restart loop. On startup we copy the jar to S3 (45MB), and then operator takes control. It downloads the jar from S3 (I verified the size and it's 45MB), but then job managers can't start, complaining that the jar is 0-length. Any ideas? JM pods have 2GB free disc space. I had a hypothesis that some timeout expires and operator begins to shut down the JM to retry or reconfigure and that 0-length JAR is just a red-hearing because the download of the jar gets aborted. But the default timeout for JM to start is 5 minutes and this begins to happen in shorter time.
We fixed it by setting
presto.s3.pin-client-to-current-region
to
true
Very strange behavior...
Anyone can explain why this works?