Hi, wanted to check with you if I am using this co...
# troubleshooting
g
Hi, wanted to check with you if I am using this correct. I have a batch job with FileSource. The file format
SpecificAvroBulkFormat
is a copy of
AvroGenericRecordBulkFormat
that we made to read bytes directly into specific avro class.
Copy code
SpecificAvroBulkFormatEventRecord, FileSourceSplit inputFormat = new SpecificAvroBulkFormat<>(EventRecord.class);
FileSourceEventRecord fileSource = FileSource.forBulkFileFormat(inputFormat, sourcePaths)
                    .build();
return env.fromSource(fileSource, WatermarkStrategy.noWatermarks(), EVENT_RECORDS_SOURCE_V3)
                    .returns(EventRecord.class)
                    .uid(EVENT_RECORDS_SOURCE_V3)
                    .name(EVENT_RECORDS_SOURCE_V3)
                    .setParallelism(parallelism);
The problem is when it loads a lot of file the actual split enumeration happens in JobManager causing it to be unresponsive and crash due to Pekko ask time out. We need to set really high timeouts to work around it.
Copy code
2024-06-26 12:05:02.584	
2024-06-26 10:05:02,584 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting split enumerator for source Source: event-records-source-v3.
	
	
2024-06-26 12:05:02.661	
2024-06-26 10:05:02,660 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:05:02.868	
2024-06-26 10:05:02,866 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:05:03.287	
2024-06-26 10:05:03,286 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:05:04.111	
2024-06-26 10:05:04,111 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:05:05.726	
2024-06-26 10:05:05,726 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:05:08.946	
2024-06-26 10:05:08,946 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:05:15.367	
2024-06-26 10:05:15,367 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:05:28.187	
2024-06-26 10:05:28,187 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:05:53.807	
2024-06-26 10:05:53,807 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:06:45.027	
2024-06-26 10:06:45,026 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:08:27.446	
2024-06-26 10:08:27,446 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:10:50.877	
2024-06-26 10:10:50,876 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Starting split enumerator for source Source: untracked-order-records-source.
	
	
2024-06-26 12:11:52.267	
2024-06-26 10:11:52,266 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Registering job manager 94b2266f1195132457b9b3dd064b4934@pekko.tcp://flink@10.117.12.83:6123/user/rpc/jobmanager_2 for job b087c94d3ef6319671c6432fd12ad0c9.
	
	
2024-06-26 12:15:10.685	
2024-06-26 10:15:10,669 ERROR org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler [] - Unhandled exception.
	
	
2024-06-26 12:15:10.685	
java.util.concurrent.TimeoutException: Invocation of [LocalRpcInvocation(RestfulGateway.requestMultipleJobDetails(Time))] at recipient [pekko.tcp://flink@10.117.12.83:6123/user/rpc/dispatcher_1] timed out. This is usually caused by: 1) Pekko failed sending the message silently, due to problems like oversized payload or serialization failures. In that case, you should find detailed error information in the logs. 2) The recipient needs more time for responding, due to problems like slow machines or network jitters. In that case, you can try to increase pekko.ask.timeout.
Also I would like to raise that it is not enough to just raise
pekko.ask.timeout
there seems to be multiple timeouts involved
I had to also raise:
Copy code
rest.connection-timeout: "900000"
    heartbeat.timeout: "900000"
But to my surprise when I raised all of them from 300 to 900 the exception is saying that timeout is set to 600s
Copy code
Ask timed out on [Actor[pekko://flink/user/rpc/dispatcher_1#-720744125]] after [600000 ms].
which is nowhere in my configuration
So the question: is it expected that JM will be going through all the files to do split enumeration itself? Why it cannot delegate to TMs?
Found why it was 600s. Seems that it was capped by this settings:
Copy code
web.timeout: "900000"
which by default is 600s
managed to write implementation with ForkJoinPool to do that in parallel. Its is much much faster now.