Grzegorz Liter
06/26/2024, 10:38 AMSpecificAvroBulkFormat is a copy of AvroGenericRecordBulkFormat that we made to read bytes directly into specific avro class.
SpecificAvroBulkFormatEventRecord, FileSourceSplit inputFormat = new SpecificAvroBulkFormat<>(EventRecord.class);
FileSourceEventRecord fileSource = FileSource.forBulkFileFormat(inputFormat, sourcePaths)
.build();
return env.fromSource(fileSource, WatermarkStrategy.noWatermarks(), EVENT_RECORDS_SOURCE_V3)
.returns(EventRecord.class)
.uid(EVENT_RECORDS_SOURCE_V3)
.name(EVENT_RECORDS_SOURCE_V3)
.setParallelism(parallelism);
The problem is when it loads a lot of file the actual split enumeration happens in JobManager causing it to be unresponsive and crash due to Pekko ask time out. We need to set really high timeouts to work around it.
2024-06-26 12:05:02.584
2024-06-26 10:05:02,584 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting split enumerator for source Source: event-records-source-v3.
2024-06-26 12:05:02.661
2024-06-26 10:05:02,660 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:05:02.868
2024-06-26 10:05:02,866 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:05:03.287
2024-06-26 10:05:03,286 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:05:04.111
2024-06-26 10:05:04,111 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:05:05.726
2024-06-26 10:05:05,726 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:05:08.946
2024-06-26 10:05:08,946 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:05:15.367
2024-06-26 10:05:15,367 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:05:28.187
2024-06-26 10:05:28,187 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:05:53.807
2024-06-26 10:05:53,807 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:06:45.027
2024-06-26 10:06:45,026 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:08:27.446
2024-06-26 10:08:27,446 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:10:50.877
2024-06-26 10:10:50,876 INFO org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting split enumerator for source Source: untracked-order-records-source.
2024-06-26 12:11:52.267
2024-06-26 10:11:52,266 INFO org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
2024-06-26 12:15:10.685
2024-06-26 10:15:10,669 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler [] - Unhandled exception.
2024-06-26 12:15:10.685
java.util.concurrent.TimeoutException: Invocation of [LocalRpcInvocation(RestfulGateway.requestMultipleJobDetails(Time))] at recipient [pekko.tcp://flink@10.117.12.83:6123/user/rpc/dispatcher_1] timed out. This is usually caused by: 1) Pekko failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase pekko.ask.timeout.Grzegorz Liter
06/26/2024, 10:39 AMpekko.ask.timeout there seems to be multiple timeouts involvedGrzegorz Liter
06/26/2024, 10:40 AMrest.connection-timeout: "900000"
heartbeat.timeout: "900000"
But to my surprise when I raised all of them from 300 to 900 the exception is saying that timeout is set to 600s
Ask timed out on [Actor[pekko://flink/user/rpc/dispatcher_1#-720744125]] after [600000 ms].
which is nowhere in my configurationGrzegorz Liter
06/26/2024, 10:45 AMGrzegorz Liter
06/26/2024, 11:12 AMweb.timeout: "900000"
which by default is 600sGrzegorz Liter
06/26/2024, 11:58 AMGrzegorz Liter
06/26/2024, 2:04 PM