I am trying to download a file from gcs bucket in ...
# troubleshooting
s
I am trying to download a file from gcs bucket in one of the map functions in my flink pipeline. when executing this StorageOptions storageOptions = StorageOptions.getDefaultInstance() .toBuilder() .setProjectId(gcsBlobConfig.getProjectId()) .build(); storageOptions.getService(); Added dependencies <dependency> <groupId>com.google.cloud</groupId> <artifactId>google-cloud-storage</artifactId> <version>2.48.1</version> </dependency> <dependency> <groupId>com.google.cloud</groupId> <artifactId>google-cloud-storage-control</artifactId> <version>2.48.1</version> </dependency> Getting the following error . Does flink need anything in the runtime ? 064355.130 [flink-pekko.actor.default-dispatcher-18] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - BID_GCP_JAPAN_EAST_BLOB_FILE_READER (4/4) (64ab7f5f8b67c55fb250e6f1e8bcc0b6_3e396ec1b502651ca7ed141a5f6b4dee_3_5) switched from INITIALIZING to RUNNING. 064355.148 [flink-pekko.actor.default-dispatcher-18] INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - BID_GCP_JAPAN_EAST_BLOB_FILE_READER (3/4) (64ab7f5f8b67c55fb250e6f1e8bcc0b6_3e396ec1b502651ca7ed141a5f6b4dee_2_5) switched from RUNNING to FAILED on pipeline-realtime-reporting-taskmanager-1-1 @ 10.39.136.139 (dataPort=34483). java.lang.NoClassDefFoundError: Could not initialize class com.google.api.services.storage.Storage at com.google.api.services.storage.Storage$Builder.build(Storage.java:15606) ~[?:?] at com.google.cloud.storage.spi.v1.HttpStorageRpc.<init>(HttpStorageRpc.java:151) ~[?:?] at com.google.cloud.storage.spi.v1.HttpStorageRpc.<init>(HttpStorageRpc.java:126) ~[?:?] at com.google.cloud.storage.HttpStorageOptions$HttpStorageRpcFactory.create(HttpStorageOptions.java:431) ~[?:?] at com.google.cloud.storage.HttpStorageOptions$HttpStorageRpcFactory.create(HttpStorageOptions.java:405) ~[?:?] at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:602) ~[?:?] at com.google.cloud.storage.HttpStorageOptions.getStorageRpcV1(HttpStorageOptions.java:88) ~[?:?
p
Are you adding those dependencies to your fat jar?
s
Hi Pedro, Thanks for replying. I was able to debug this. The error thrown here was not completely correct. Yes there was an issue in the fat jar. Client properties file was not getting bundled. Fixed it. Works now