Adrian Chang
06/22/2023, 3:57 PMINFO org.apache.flink.client.python.PythonEnvUtils [] - Starting Python process with environment variables:
I am using Flink Kubernetes Operator v1.5.0 and setting the secrets using secrets Kubernetes section in the yml file.
Is there any way to prevent Flink logging sensitive information ?
ThanksCmx Uni
06/22/2023, 5:07 PMWatermarkStrategy*.*forMonotonousTimestamps*()*
for watermark generation. Right after the sourceStream, the elements are sinked into Kafka. My expectation was that the sink would not emit the elements with the latest timestamp because the current watermark is latestTimestamp - 1
, but the latest elements are indeed sinked to kafka before the watermark is emitted.
Are sinks not aware of watermarks? Do I need to write a dummy ProcessFunction that holds these elements back?
Thanks! Sorry, I could not find anything explaining this in the docs..Rajat Ahuja
06/22/2023, 6:47 PMCameron Miller
06/22/2023, 11:21 PMSumit Nekar
06/23/2023, 7:11 AMAmenreet Singh Sodhi
06/23/2023, 9:55 AMArun Karthick
06/23/2023, 11:45 AMPankaj Singh
06/23/2023, 1:50 PMRichard Diekema
06/23/2023, 10:04 PMLeon Xu
06/24/2023, 4:16 AMOscar Perez
06/24/2023, 8:48 PMSeason
06/25/2023, 5:59 AMSumit Nekar
06/26/2023, 4:16 AMAbhishek Joshi
06/26/2023, 6:35 AMHJK nomad
06/26/2023, 6:56 AMYaroslav Bezruchenko
06/26/2023, 10:09 AMTong Yue
06/26/2023, 10:34 AMCheguri Vinay Goud
06/26/2023, 11:01 AMOscar Perez
06/26/2023, 11:04 AMEric Shieh
06/26/2023, 7:05 PMFileSource
(S3) -> KafkaSink
with a JobListner.onJobExecuted
to "mv" the source file to a new folder/bucket (cleanup logic). When the job runs, one or more of the fetchers throw FileNotFoundException
indicating the source file is missing; if I remove the JobListener
code, the job completes successfully. It appears that the source file was moved to the new folder/bucket while the SplitFetcher
is still polling for the records. Is there a way to perform clean-up logic after all streaming tasks have been completed? ps. this seems to occur when running in session mode (1 JobManager, 1 TaskManager); when I ran it in my IDE, the job completes successfully with the cleanup logic.Amenreet Singh Sodhi
06/26/2023, 7:21 PMFaisal A. Siddiqui
06/26/2023, 7:48 PMakka.client.timeout: 10min
and akka.ask.timeout:60 s
but still seeing this error. Am i missing something ??
java.lang.Exception: Cannot deploy task GroupAggregate[109] -> Calc[110] (5/8) (47e7dd1ba4f795e0dc9357f834c09f55_0122d1f6290e823b79ad614da62c0d8f_4_33) - TaskManager (xxx.xx.x.xx:6122-ca433e @ xxx-xx-x-xx.flink-taskmanager.xxx-flink-xxxx.svc.cluster.local (dataPort=33833)) not responding after a rpcTimeout of 10000 ms
...
...
....
Caused by: java.util.concurrent.TimeoutException: Invocation of [RemoteRpcInvocation(TaskExecutorGateway.submitTask(TaskDeploymentDescriptor, JobMasterId, Time))] at recipient [<akka.tcp://flink@xxx.xx.x.xx:6122/user/rpc/taskmanager_0>] timed out. This is usually caused by: 1) Akka failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase akka.ask.timeout.
at org.apache.flink.runtime.jobmaster.RpcTaskManagerGateway.submitTask(RpcTaskManagerGateway.java:59)
Saj Yogeswaran
06/27/2023, 12:16 AMFaisal A. Siddiqui
06/27/2023, 4:47 AM'properties.group.id' = 'testGroup'
and i noticed this consumer group is quite behind in Kafka.
Q#1: Is this information reliable ??
Q#2: What is the purpose of this property when checkpointing is available?Adam Richardson
06/27/2023, 6:05 AMBoris Litvak
06/27/2023, 7:53 AMYang osmondy
06/27/2023, 8:46 AMCarlos Santos
06/27/2023, 10:50 AMGintaras Matulas
06/27/2023, 11:44 AMjava.lang.UnsupportedOperationException: Currently it is not supported to update the CommittableSummary for a checkpoint coming from the same subtask. Please check the status of FLINK-25920
Prathit Malik
06/27/2023, 12:38 PMemr-6.10.0
Our EMR currently contains only master & core nodes. We are trying to use Task Node but we are facing an issue as listed here.
Error : Caused by: java.nio.file.AccessDeniedException: /var/lib/flink
Since the stackoverflow question has no resolution for the same, can someone check if they faced the same issue & were able to resolve it ?