Jark Wu
05/19/2022, 1:39 PMJark Wu
05/19/2022, 1:57 PMJark Wu
05/19/2022, 1:57 PMJark Wu
05/19/2022, 1:57 PMJark Wu
05/19/2022, 1:57 PMXintong Song
05/20/2022, 11:12 AMJeremy Ber
06/02/2022, 5:35 PMBenjamin Carlton
06/02/2022, 10:38 PMJeff Levesque
06/03/2022, 8:34 PMprint
connector. My intention is to develop flink applications locally, to test the sliding window functionality with the print
connector. If all goes well, I'd promote the PyFlink application to AWS. However, during my testing, the application just hangs, as if the blackhole
connect was used instead of the print
connector. I've created a StackOverflow post to hopefully better link the corresponding script/resources:
• https://stackoverflow.com/questions/72311800/implement-print-connector-with-sliding-window-sinkHKB
06/04/2022, 2:31 AMMarco Villalobos
06/04/2022, 3:32 AMMarco Villalobos
06/04/2022, 3:33 AMMarco Villalobos
06/04/2022, 5:28 AMSucheth Shivakumar
06/05/2022, 4:47 PMSucheth Shivakumar
06/05/2022, 4:47 PMczchen
06/06/2022, 12:45 AMspec.serviceAccount
is needed. This is not mentioned in document.
• The initialSavepointPath
in https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.0/docs/operations/upgrade/#upgrading-with-existing-flinkdeployments is point to /.../_metadata
. However, we always point to _metadata
parent directory. Are they both supported?sri hari kali charan Tummala
06/06/2022, 2:17 AMsri hari kali charan Tummala
06/06/2022, 2:19 AMJeesmon Jacob
06/06/2022, 12:53 PMValidatingWebhookConfiguration
is cluster scoped, it is not allowed. So wondering what we will miss if we don't run the validation webhook and instead add some checks in our CI/CD pipelines. I see the DefaultValidator
is called from both webhook and controller. So even if CR is admitted without validation through webhook, controller will still validate it as part of Reconcile logic? Thanks.Jeesmon Jacob
06/06/2022, 1:56 PMJeesmon Jacob
06/07/2022, 12:20 AMFlinkDeployment
CR, is there an option to resolve a flinkConfiguration
from a ConfigMap or Env loaded from a ConfigMap. For example, in our side s3 bucket is created by another pipeline and endpoint and bucket name are exposed in a ConfigMap. We would like to load that ConfigMap to JM/TM pods using podTemplate and use the bucket name for state.savepoints.dir
, state.checkpoints.dir
and high-availability.storageDir
instead of hardcoding it in FlinkDeployment
. Thanks.czchen
06/07/2022, 6:28 AM/opt/flink/bin/flink run-application --target kubernetes-application -Denv.java.opts.taskmanager='-XX:+UseG1GC -verbose:class'
However, when the application starts, the log indicates that it does not load this option correctly:
2022-06-07 06:24:13,214 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: env.java.opts.taskmanager, '-XX:+UseG1GC
Any idea how to solve this problem?czchen
06/07/2022, 10:13 AMflink-s3-fs-hadoop-1.15.0.jar
and flink-gs-fs-hadoop-1.15.0.jar
are incompatible in Flink 1.15. Anyone know how to solve this issue? (original discuss in mailing list: https://www.mail-archive.com/user@flink.apache.org/msg47950.html)
The following are detail:
We got the following error when migrating to Flink 1.15
Caused by: java.lang.ClassCastException: class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem cannot be cast to class org.apache.hadoop.fs.FileSystem (com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem and org.apache.hadoop.fs.FileSystem are in unnamed module of loader 'app')
Add -verbose:class
to Java options show the following:
[2.074s][info][class,load] org.apache.hadoop.fs.FileSystem source: file:/opt/flink/opt/flink-s3-fs-hadoop-1.15.0.jar
...
[8.094s][info][class,load] com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem source: file:/opt/flink/opt/flink-gs-fs-hadoop-1.15.0.jar
Looks like Flink are trying to cast com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
in flink-gs-fs-hadoop-1.15.0.jar
to org.apache.hadoop.fs.FileSystem
in flink-s3-fs-hadoop-1.15.0.jar
without success.aromal
06/07/2022, 10:37 AMVeeramani Moorthy
06/07/2022, 11:17 AMVeeramani Moorthy
06/07/2022, 11:17 AMVeeramani Moorthy
06/07/2022, 11:17 AMVeeramani Moorthy
06/07/2022, 11:18 AMVeeramani Moorthy
06/07/2022, 11:19 AMliwei li
06/07/2022, 12:29 PM