Hi, I'm facing a problem with a batch job in k8s where writing the json results to a filesystem sink (GCS) is very slow. At some point the task managers fail for unclear reasons and the job restarts. The logs of the pods don't give any details about why pods were terminated. It's typically just saying something like

The TaskExecutor is shutting down

. After the failure I see results partially written to the filesystem under some

.staging

prefix, the writing didn't finish though. Is there a possibility to enable more detailed logging for the flink deployment? Something like a debug mode? Or does someone have a hint about how to debug why pods were terminated after they've already been deleted? I'm using flink version 1.17.1.

Hi! This is how we enable debug mode on flink deployment(helm)

spec:
  image: {{ .image }}
  flinkVersion: v1_17

  {{- if .logLevelDebug }}
  logConfiguration:
    log4j-console.properties: |+
      logger.flink.name = org.apache.flink
      logger.flink.level = DEBUG
  {{- end }}

to be able to analyse the logs after pod is terminated, the logs should be collected centralised in K8s 🤷

Thanks for sharing. I'll try it out. I have access to the logs even after the pods are terminated, they're just not providing information why the pods were terminated.

Can you check the exit code with which the Taskmanager pods "die"?

Unfortunately that's not the case. Out of memory show indeed a clear OOMkilled event in the logs, I had one job with such an issue. However, in the majority of cases I cannot find a clear error message like that.

Is the TaskManager receiving a Signal? Flink is logging whether it receives SIGKILL or SIGTERM or something

The task manager is receiving a SIGTERM signal.

I assume you are using the FLink K8s operator with native kubernetes mode?

The debug logs are very helpful. It turns out that the job manager receives all of a sudden a MODIFIED event for a task manager that is already done uploading and just idling. As a result the job manager decides to terminate that pod because it gets confused and thinks that task manager already exists. At least that's how I understand it now. Some time later the job manager is confused about the missing task manager and restarts a lot of tasks. So the question is where that MODIFIED event is coming from.

Not sure if it will be helpful, but K8s has audit logs that track who's doing what in the api server