Yuval Itzchakov
12/14/2023, 6:52 AMMani Kanthi (C)
12/14/2023, 10:23 PM2023-12-07 17:55:21,721 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Trying to recover job with job id 12c749a0e9c311d1106cc4bf42d733a6.
2023-12-07 17:55:21,733 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Dispatcher <akka.tcp://flink@10.244.132.89:6123/user/rpc/resourcemanager_0> completed job recovery and leadership confirmation with fencing token d5db40b0-3d56-4276-b73a-3c933bf8641d
2023-12-07 17:55:21,919 INFO org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Stopping SessionDispatcherLeaderProcess.
2023-12-07 17:55:21,921 INFO org.apache.flink.runtime.jobmanager.DefaultJobGraphStore [] - Stopping DefaultJobGraphStore.
2023-12-07 17:55:21,922 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal error occurred in the cluster entrypoint.
java.util.concurrent.CompletionException: org.apache.flink.util.FlinkRuntimeException: Could not recover job with job id 12c749a0e9c311d1106cc4bf42d733a6.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_322]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) [?:1.8.0_322]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Caused by: org.apache.flink.util.FlinkRuntimeException: Could not recover job with job id 12c749a0e9c311d1106cc4bf42d733a6.
at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.recoverJob(SessionDispatcherLeaderProcess.java:144) ~[flink-dist_2.11-1.13.6-splunk-2.jar:1.13.6-splunk-2]
at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.recoverJobs(SessionDispatcherLeaderProcess.java:122) ~[flink-dist_2.11-1.13.6-splunk-2.jar:1.13.6-splunk-2]
at org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198) ~[flink-dist_2.11-1.13.6-splunk-2.jar:1.13.6-splunk-2]
at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.recoverJobsIfRunning(SessionDispatcherLeaderProcess.java:113) ~[flink-dist_2.11-1.13.6-splunk-2.jar:1.13.6-splunk-2]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) ~[?:1.8.0_322]
... 3 more
Caused by: com.facebook.presto.hive.s3.PrestoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 1701971721882933269; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null (Path: <s3://flink-state/flink/default/submittedJobGraph8402b4d82ab6>)
Rabih Hajj
12/15/2023, 11:26 AMPrabhu Joseph
12/17/2023, 10:32 AM朱雪琰
12/18/2023, 7:29 AM刘伯栋
12/19/2023, 1:11 PMJacob Barrieault
12/19/2023, 8:52 PMstandalone-job
which requires a .jar (doesn’t support sql client, to my understanding).
I’d like to make something for my team that makes it easy to write and run a PoC Flink SQL app in a development environment.
I can script may way to a solution, but wanted to know if there’s an out-of-the-box option I missed.Hunter
01/02/2024, 1:25 PMWei Ding
01/05/2024, 2:29 AMkafka-lag-exporter
to explore Kafka consumer lag, however it seems that Flink manages the offsets in it's own way that the consumer lag metrics could not be picked up.. (Correct me if I'm wrong.)
Now I'm trying to use Flink Prometheus metrics with Grafana, I'd like to confirm if there a Grafana dashboard for Kafka connector already? (If no I gonna build the wheel...)Ola Theander
01/11/2024, 12:21 PMKeyedProcessFunction
thus evaluating an incoming message and if I want to keep state for that message increment this counter and when it as some point goes out of scope decrement the counter. The point being able to monitor how many "active" state items there is across all keys.
One thing in particular is unclear; will metrics be part of a checkpoint/save-point? I.e. when restoring from a checkpoint will the counter retain it's value or will it be zeroed? It matters because if it's zeroed the counter will no longer reflect the true number of "active" state items.
Another approach I'm investigating is to use a RichSinkFunction
implementing CheckpointedFunction
whose purpose is to collect increment/decrement messages and keep count in a state and expose it as a Gauge. That way I believe the accurate count will survive a checkpoint restore since it keeps the count in a state that will be included in the checkpoint. However, this is more complex than a simple counter so if the counter can be confirmed to survive a restore then this approach is probably overkill.Haymant Mangla
01/14/2024, 7:52 AMSlackbot
01/15/2024, 8:46 PMVISHAL KUMAR
01/22/2024, 9:34 AMSlackbot
01/22/2024, 1:18 PMMaz Jindeel
01/23/2024, 7:14 PMJim Dickson
01/24/2024, 2:46 AMSingh
01/31/2024, 10:48 PMKarim Rakia
02/05/2024, 11:28 AMFidea Lidea
02/06/2024, 11:54 AMWei Ding
02/08/2024, 4:30 AMorg.apache.flink.sql.parser.impl.ParseException
when I have --
inside the define condition.
So, this case is good
...
define
A as A.name = 'aaa'
And this hits exception
...
define
A as A.name = 'aaa--bbb'
Is this a known issue? Any workarounds?
Thanks in advance.Barak Ben-Nathan
02/13/2024, 7:17 AMSamrat Deb
03/04/2024, 4:10 AM邢云
03/05/2024, 1:16 PMVISHAL KUMAR
03/06/2024, 4:13 PMregisterCatalog
function within the Flink Table API. However, we noticed it's marked as deprecated. While it currently functions, we'd like to leverage the latest Flink features for this new connector.
Could you please point us toward the recommended approach for registering catalogs in current Flink versions? Additionally, any insights on the deprecation of registerCatalog
would be appreciated.
Thanks in advance for your assistance!rmoff
03/08/2024, 4:30 PMSai Sharath Dandi
03/09/2024, 9:05 AMAri Huttunen
03/09/2024, 2:11 PMPerfect Stranger
03/10/2024, 6:32 PMWasim
03/23/2024, 10:56 AMYonatan Becker
03/28/2024, 3:26 PM