Hi Team, I am using Flink operator in Application ...
# troubleshooting
r
Hi Team, I am using Flink operator in Application mode and I am trying to figure out the operator behavior when job is in terminally failed state. I see log message below which clearly shows job status changing from
RECONCILING
to
FAILED
. However, this status change in not reflected in
FlinkDeployment
resource. Please see screenshot.
Copy code
o.a.f.k.o.l.AuditUtils        [m [32m[INFO ][samplejob/samplejob-jm-ue1a] >>> Event  | Info    | JOBSTATUSCHANGED | Job status changed from RECONCILING to FAILED
From the source code as well, it seems that the jobs status change event should propagate to update kubernetes resource here - https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-ope[…]pache/flink/kubernetes/operator/observer/JobStatusObserver.java But this does not seem to be happening
g
There seems to be some nullpointer error somwhere, can you check the opeartor logs?
👀 1
r
I could not find any reference to
NullPointerException
in operator logs. Only other error logged by the operator is
Copy code
[m[33m2023-08-14 06:02:40,028[m [36mo.a.f.k.o.o.JobStatusObserver [m [1;31m[ERROR][hubblederivedeventsjob/hubblederivedeventsjob-jm-ue1a] Job 5c8133740a6e7c5c08492d05d56963b4 failed with error: org.apache.flink.client.program.ProgramInvocationException: An error occurred while invoking the program's main method: 'com.fasterxml.jackson.core.io.ContentReference com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createContentReference(java.lang.Object)
Which is the error from the actual job.
This possibly could be thrown from here which translates to setting
NullPointer
message with
ReconciliationException
In any case, should'nt the FlinkDeployment resource status should be marked as
Failed
? The UI also shows as the job is in "Failed" state (as part of completed job list).
The expectation is that if
FlinkDeployment
resource reflects correct state and not stuck in
RECONCILING
state, we can instracut our CI/CD to delete the resource and recreate with the new spec. As of now, this does not seem to be feasible.