Hi everyone, My team is running into an issue with...
# troubleshooting
b
Hi everyone, My team is running into an issue with Flink operator in session mode. We deploy a
FlinkSessionJob
that consumes data from a Kafka topic and writes it to a BigQuery table. When the JobManager starts, it seems that the initialization step — possibly checking the BigQuery table — takes a long time. During this delay, the web upload temporary file appears to be deleted, causing the job to fail. We are seeing the following exception:
Copy code
Caused by: java.io.FileNotFoundException: 
/tmp/flink-web-11008d54-f3d5-4458-8b2c-531d6c88ab1a/flink-web-upload/edc628df-da1b-4b0f-b508-88b3d9891edb_flink-datastream-1.0.0.jar 
(No such file or directory)
Has anyone encountered this issue before? Is there a recommended way to prevent the uploaded jar file from being deleted during job initialization? Thanks in advance for any help! 🙏
a
Hi @Boice Lu, Did you check the
/tmp/flink-web-11008d54-f3d5-4458-8b2c-531d6c88ab1a/flink-web-upload/edc628df-da1b-4b0f-b508-88b3d9891edb_flink-datastream-1.0.0.jar
inside the jobmanager container?
b
Yes ,it is empty when exception print out.
a
Hi, As far as I understood, you use Flink API to submit your jar file to JobManager. E.g.
Copy code
curl -X POST -H "Expect:" -F "jarfile=@/Users/flink-datastream-1.0.0.jar" <http://localhost:8081/jars/upload>
the response is probably:
edc628df-da1b-4b0f-b508-88b3d9891edb_flink-datastream-1.0.0.jar
and I assumed you also saw the jar file in jobManager
/tmp/flink-web-11008d54-f3d5-4458-8b2c-531d6c88ab1a/flink-web-upload/edc628df-da1b-4b0f-b508-88b3d9891edb_flink-datastream-1.0.0.jar
Next, you send another Flink API to starts the job. For example,
Copy code
curl -X POST <http://localhost:8081/jars/edc628df-da1b-4b0f-b508-88b3d9891edb_flink-datastream-1.0.0.jar/run>
After the job got started (RUNNING state), it run the BigQuery as an initialization step, but it took long time to finish. However, during this step, the
/tmp/flink-web-11008d54-f3d5-4458-8b2c-531d6c88ab1a/flink-web-upload/edc628df-da1b-4b0f-b508-88b3d9891edb_flink-datastream-1.0.0.jar
got deleted inside your JobManager, resulting in FAIL state in the job. Please correct me if I am wrong in any step
b
Hi, Yes, your understanding is mostly correct. However, in my case, I didn’t submit the JAR directly via the Flink REST API. I’m using Flink Operator with a FlinkSessionJob to submit the job instead. As far as I know, the behavior of
FlinkSessionJob
should be equivalent to what you described — it also uploads the JAR to the JobManager and triggers job execution internally. Tanks for your confirmation
a
Hi @Boice Lu, If you use FlinkSessionJob to submit your job to Flink operator, I believe the jar file should reside in the Flink operator pod, instead of JM's. Could you help me double check?
b
Ok Thank for reply. I will check it tomorrow.