I'm encountering a somewhat rare issue when readin...
# troubleshooting
j
I'm encountering a somewhat rare issue when reading configurations using
ConfigOption
. In my
Configuration
, I have the following value:
Copy code
schemas.files -> - schemas/schema1.yml\n- docker/pipeline/schemas/schema2.yml
And the
ConfigOption
is defined like this:
Copy code
ConfigOption<ListString> SCHEME_FILES = ConfigOptions
            .key("schemas.files")
            .stringType()
            .asList()
            .noDefaultValue();
Locally, the configuration is read correctly as a list of strings without the "- " prefix. However, in a deployed environment, it is read with the "- " prefix at the beginning, which causes it to be invalid afterward. Any suggestions? I know that I can add a map to the config to clean this, but Im trying to understand the reason to work different from local to a deployment and what should be the "expected result"
a
How your cluster in deployed environment is configured? One likely reason for this is that local cluster uses new yaml configuration (available from 1.19) and remote environment uses old yaml-like config, which has different notion of lists representation as string. You can check if you have
conf/flink-conf.yaml
file present on remote environment and/or confirm in the logs that Flink is indeed falls back to old configuration style.
j
the cluster is deployed using flink kubernetes operator in version 1.10 and flink in version 1.20.1
ok, is using old style.. INFO [] - Using legacy YAML parser to load flink configuration file from /opt/flink/conf/flink-conf.yaml. changing the configs to the new style should solve this??
a
new style config will respect yaml formatting, yes
j
Thanks
If the Flink Kubernetes Operator only generates the
flink-conf.yaml
file, will Flink fall back to the legacy YAML parser? I believe I've already migrated all configurations to the new format, but the legacy YAML parser is still being used. Additionally, I don't see any deprecated configurations in the logs. Is there a way to validate which parser Flink is using?
a
If
flink-conf.yaml
is detected - Flink will fallback to old configuration style, from the docs:
Once the legacy configuration file
flink-conf.yaml
is detected, Flink will prioritize using it as the configuration file.
j
Can I force in any way in local to load the legacy YAML parser. Right now I have a beaveour in local and another pretty different in the cluster..
a
Haven't tried this myself, but I see that for 1.19 and 1.20 there is a testing constructor override for
Configuration
class where you can override parser behaviour: https://github.com/apache/flink/blob/release-1.20/flink-core/src/main/java/org/apache/flink/configuration/Configuration.java#L92-L97 thought this is for testing/debugging purposes only
j
I saw that I can also use
Copy code
GlobalConfiguration.setStandardYaml(false);
As Im only force that tests have the same behaviour that will have on clusters, this solve my problem.. Thanks Aleksandr for the help..
πŸ‘ 1