Hi all, I'm new to Flink. We are planning to bring...
# troubleshooting
s
Hi all, I'm new to Flink. We are planning to bring this to our Org. Trying to do a POC. - Installed Flink 1.19.0. Web portal is running - Followed some tutorial, built a jar file to read a file, count the words & write to an output. - When I run the jar file I’m running into a weird issue and there is not much help on this error ( Could not connect to BlobServer at address localhost). Can you please suggest how to fix the config for this? I’m running the jar file on the same server (RHEL, Java 17) as Flink. [root@ae1 flink-1.19.0]# ./bin/flink run --class flink.First /opt/apps/flink-0.0.1-SNAPSHOT.jar --input file:///opt/apps/test.txt --output file:///opt/apps/testoutput.txt ------------------------------------------------------------ The program finished with the following exception: org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:108) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270) at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367) at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph. at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) at org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1069) at org.apache.flink.client.program.ContextEnvironment.executeAsync(ContextEnvironment.java:144) at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:73) at flink.First.main(First.java:43) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ... 9 more Caused by: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1064) ... 17 more Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph. at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$12(RestClusterClient.java:486) at java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990) at java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:272) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614) at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1163) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.apache.flink.runtime.rest.util.RestClientException: [org.apache.flink.runtime.rest.handler.RestHandlerException: Could not upload job files. at org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.lambda$uploadJobGraphFiles$4(JobSubmitHandler.java:201) at java.base/java.util.concurrent.CompletableFuture.biApply(CompletableFuture.java:1311) at java.base/java.util.concurrent.CompletableFuture$BiApply.tryFire(CompletableFuture.java:1280) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.apache.flink.util.FlinkException: Could not upload job files. at org.apache.flink.runtime.client.ClientUtils.uploadJobGraphFiles(ClientUtils.java:86) at org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.lambda$uploadJobGraphFiles$4(JobSubmitHandler.java:195) ... 10 more Caused by: java.io.IOException: Could not connect to BlobServer at address localhost/127.0.0.1:37225 at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:103) at org.apache.flink.runtime.rest.handler.job.JobSubmitHandler.lambda$null$3(JobSubmitHandler.java:199) at org.apache.flink.runtime.client.ClientUtils.uploadJobGraphFiles(ClientUtils.java:82) ... 11 more Caused by: java.net.ConnectException: Connection refused at java.base/sun.nio.ch.Net.connect0(Native Method) at java.base/sun.nio.ch.Net.connect(Net.java:579) at java.base/sun.nio.ch.Net.connect(Net.java:568) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:593) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at org.apache.flink.runtime.blob.BlobClient.<init>(BlobClient.java:97) ... 13 more ] at org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:644) at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$6(RestClient.java:628) at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ... 4 more
k
That looks like it might be a comms issue between your JobManager and TaskManager(s).
r
when you say "Installed Flink 1.19.0. Web portal is running", how exactly is Flink installed? Is it running standalone with
./bin/start-cluster.sh
? Is the configuration default or has it been modified?
s
It is a standalone installation (no docker). I updated only one value in config.yaml by changing bind-host from localhost to an IPAddress to access the web portal from outside the Linux VM. Rest all is the default configuration. What am I missing? TIA.
r
is that the jobmanager bind-host? In the code I saw that the BlobServer also binds to that host. I think you might be missing some configuration because there is this other option:
Copy code
jobmanager:
  ... snip ...
  rpc:
    # The external address of the host on which the JobManager runs and can be
    # reached by the TaskManagers and any clients which want to connect. This setting
    # is only used in Standalone mode and may be overwritten on the JobManager side
    # by specifying the --host hostname parameter of the bin/jobmanager.sh executable.
    # In high availability mode, if you use the bin/start-cluster.sh script and setup
    # the conf/masters file, this will be taken care of automatically. Yarn
    # automatically configure the host name based on the hostname of the node where the
    # JobManager runs.
    address: localhost
which configures the external address of the jobmanager. poking through the code it looks like it uses the
rpc.address
when it's attempting to connect to the blobserver. So options could be to set job-manager bind-host to
0.0.0.0
so you are bound to all interfaces, including local. So it's connectable on localhost and your other interface. Or try setting
rpc.address
to the same ip you are binding to.
hmm I tried it locally, binding to a specific interface and I got the same exception as you when submitting jars. Setting the various address configurations didn't help. It looks like internally it uses the hostname of the PekkoRpcService which defaulted to
localhost
here. maybe there's some built in assumption that the JobManager can connect to itself on localhost, at least in this standalone configuration (even if you configure it to bind to a specific ip).
s
Thanks for checking. Does it mean it is not possible to run in a stand alone configuration?
r
to clarify, are you setting the jobmanager bind-host or the rest bind-address? I found changing the jobmanager bind-host and address caused problems, but that configuring:
Copy code
rest:
  address: 192.168.178.91
  bind-address: 192.168.178.91
to bind the rest/web to a specific interface meant I can access the web UI on that interface and
./bin/flink run examples/batch/WordCount.jar
succeeds. Maybe easier if you could show your config.yaml