How would I go about adding Hadoop to the classpat...
# troubleshooting
k
How would I go about adding Hadoop to the classpath in dockerized Flink job? Context: I'm trying to create a
FileSource
using an s3 path. I've enabled `flink-s3-fs-hadoop`(getting past the "no implementation for 's3'" issue), but during startup I see:
Copy code
jobmanager-1  | 2024-05-31 17:25:51,270 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] -  Starting StandaloneApplicationClusterEntryP
oint (Version: 1.19.0, Scala: 2.12, Rev:eaffd22, Date:2024-03-06T15:18:44+01:00)
jobmanager-1  | 2024-05-31 17:25:51,270 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] -  OS current user: flink
jobmanager-1  | 2024-05-31 17:25:51,270 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] -  Current Hadoop/Kerberos user: <no hadoop dependency found>
And the jobmanager just hangs after initializing the job:
Copy code
jobmanager-1  | 2024-05-31 17:25:53,098 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Starting execution of job 'TEST READ' (38bae99cbfab3d01bc2c80d00289a3f1) under job master id 00000000000000000000000000000000.
jobmanager-1  | 2024-05-31 17:25:53,101 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Connecting to ResourceManager pekko.tcp://flink@jobmanager:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
jobmanager-1  | 2024-05-31 17:25:53,103 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Resolved ResourceManager address, beginning registration
jobmanager-1  | 2024-05-31 17:25:53,104 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager 00000000000000000000000000000000@pekko.tcp://flink@jobmanager:6123/user/rpc/jobmanager_2 for job 38bae99cbfab3d01bc2c80d00289a3f1.
jobmanager-1  | Job has been submitted with JobID 38bae99cbfab3d01bc2c80d00289a3f1
jobmanager-1  | 2024-05-31 17:25:53,108 INFO  org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager 00000000000000000000000000000000@pekko.tcp://flink@jobmanager:6123/user/rpc/jobmanager_2 for job 38bae99cbfab3d01bc2c80d00289a3f1.
jobmanager-1  | 2024-05-31 17:25:53,109 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000.
jobmanager-1  | 2024-05-31 17:25:53,130 INFO  org.apache.flink.fs.s3.common.token.AbstractS3DelegationTokenReceiver [] - Updating Hadoop configuration
jobmanager-1  | 2024-05-31 17:25:53,131 INFO  org.apache.flink.fs.s3.common.token.AbstractS3DelegationTokenReceiver [] - Updated Hadoop configuration successfully
jobmanager-1  | 2024-05-31 17:25:53,191 WARN  org.apache.hadoop.metrics2.impl.MetricsConfig                [] - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
jobmanager-1  | 2024-05-31 17:25:53,198 INFO  org.apache.hadoop.metrics2.impl.MetricsSystemImpl            [] - Scheduled Metric snapshot period at 10 second(s).
jobmanager-1  | 2024-05-31 17:25:53,198 INFO  org.apache.hadoop.metrics2.impl.MetricsSystemImpl            [] - s3a-file-system metrics system started
jobmanager-1  | 2024-05-31 17:25:53,210 WARN  org.apache.hadoop.util.NativeCodeLoader                      [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
There's no further output.