hello everyone. Anyone who knows how to set `clean...
# troubleshooting
m
hello everyone. Anyone who knows how to set
cleanupTerminalJmAfterTtl
in Flink operator?
l
I found this in the operator repo (I picked release 1.8 of the Flink Operator, but I saw the same code in the latest operator release): The cleanupTerminalJmAfterTtl function ends up doing this call:
Copy code
var ttl = observeConfig.get(KubernetesOperatorConfigOptions.OPERATOR_JM_SHUTDOWN_TTL);
Here is a link to where OPERATOR_JM_SHUTDOWN_TTL is defined, and it shows what I assume would be the flink-conf.yaml property name.
Copy code
@Documentation.Section(SECTION_DYNAMIC)
    public static final ConfigOption<Duration> OPERATOR_JM_SHUTDOWN_TTL =
            operatorConfig("jm-deployment.shutdown-ttl")
                    .durationType()
                    .defaultValue(Duration.ofDays(1))
                    .withDescription(
                            "Time after which jobmanager pods of terminal application deployments are shut down.");
m
oh thank you πŸ™‚ I'll try it
Actually I expect that the option works well with the situation that when the flink batch job is finished and taskamangers terminates, I want the jobmanager pod terminates automatically. Would the option(jm-deployment.shutdown-ttl) be suitable for this situation ?
I set kubernetes.operator.jm-deployment.shutdown-ttl like
60000 ms
but I don't think it seems work
l
Are you saying you tried
kubernetes.operator.jm-deployment.shutdown-ttl: 60000
or
kubernetes.operator.jm-deployment.shutdown-ttl: "60000 ms"
? I see an example from decodable and they don't include the milliseconds: https://www.decodable.co/blog/get-running-with-apache-flink-on-kubernetes-1#~text=Shorten%20the%20lifetime%2[…]while%20when%20suspending%20a%20job
Copy code
kubernetes.operator.jm-deployment.shutdown-ttl: 10000
m
actually I've tried the both (without ms, with ms)
l
Have you seen the jobmanager being shutdown after 1 day? That should happen by default according to the code/documentation.
Also, you should be able to log out the config value when you start your app. Perhaps you can get some insight into whatever the issue is by trying to log the value. It could be that the value is being formatted in an unexpected way when the job is submitted. You might get an error when trying to log the value, or at least you can verify that you can set the property.
m
no,, absolutely not ,, so that's why I tried with 60000 (1minute). I think I should take a look at the source codes how to work
l
Have you tried logging the loaded configuration in your app to see if the property is being set?
m
yeah, I tried logging and checked it, and I ended up making external script where it watches the status of flink application and make them stop πŸ™‚ thank you
l
> yeah, I tried logging and checked it, and I ended up making external script where it watches the status of flink application and make them stop What was the outcome from when you tried logging the value and checked it? Did the value get logged out? If the value was logged out, then did the logged configuration value match what you expected? Sometimes command line args might accidentally get mangled, so when you try using the Flink ParamTools to parse the command line args, then you might end up with a weird value for one of the options. For example, you might have the option key as a value for another option.
@Minsu Jo - I just noticed this: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.8/docs/concepts/controller-flow/#application-reconciler cleanupTerminalJmAfterTtl: Flink 1.15 and above we do not automatically terminate jobmanager processes after shutdown/failure to get a robust observer behaviour. However once the terminal state is observed we can clean up the JM process/deployment.