Joao Boto
03/11/2025, 12:28 PMConfigOption.
In my Configuration, I have the following value:
schemas.files -> - schemas/schema1.yml\n- docker/pipeline/schemas/schema2.yml
And the ConfigOption is defined like this:
ConfigOption<ListString> SCHEME_FILES = ConfigOptions
.key("schemas.files")
.stringType()
.asList()
.noDefaultValue();
Locally, the configuration is read correctly as a list of strings without the "- " prefix.
However, in a deployed environment, it is read with the "- " prefix at the beginning, which causes it to be invalid afterward.
Any suggestions?
I know that I can add a map to the config to clean this, but Im trying to understand the reason to work different from local to a deployment and what should be the "expected result"Aleksandr Pilipenko
03/11/2025, 12:54 PMconf/flink-conf.yaml file present on remote environment and/or confirm in the logs that Flink is indeed falls back to old configuration style.Joao Boto
03/11/2025, 12:57 PMJoao Boto
03/11/2025, 12:59 PMAleksandr Pilipenko
03/11/2025, 1:02 PMJoao Boto
03/11/2025, 1:03 PMJoao Boto
03/11/2025, 10:39 PMflink-conf.yaml file, will Flink fall back to the legacy YAML parser?
I believe I've already migrated all configurations to the new format, but the legacy YAML parser is still being used. Additionally, I don't see any deprecated configurations in the logs.
Is there a way to validate which parser Flink is using?Aleksandr Pilipenko
03/12/2025, 10:05 AMflink-conf.yaml is detected - Flink will fallback to old configuration style, from the docs:
Once the legacy configuration fileis detected, Flink will prioritize using it as the configuration file.flink-conf.yaml
Aleksandr Pilipenko
03/12/2025, 10:12 AMJoao Boto
03/12/2025, 10:17 AMAleksandr Pilipenko
03/12/2025, 11:21 AMConfiguration class where you can override parser behaviour: https://github.com/apache/flink/blob/release-1.20/flink-core/src/main/java/org/apache/flink/configuration/Configuration.java#L92-L97
thought this is for testing/debugging purposes onlyJoao Boto
03/13/2025, 10:36 AMGlobalConfiguration.setStandardYaml(false);
As Im only force that tests have the same behaviour that will have on clusters, this solve my problem..
Thanks Aleksandr for the help..