Hi Team While checkpointing to Azure Blob Storage using Flin Apache Flink #troubleshooting

Hi Team, While checkpointing to Azure Blob Storage...

Dheeraj Panangat

01/18/2023, 1:33 PM

Hi Team, While checkpointing to Azure Blob Storage using Flink, we get the following error :

Copy code

Caused by: Configuration property <accoutnname>.<http://dfs.core.windows.net|dfs.core.windows.net> not found.
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)

We have given the configurations in core-site.xml too for following

Copy code

fs.hdfs.impl
fs.abfs.impl -> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
fs.file.impl
fs.azure.account.auth.type
fs.azure.account.oauth.provider.type
fs.azure.account.oauth2.client.id
fs.azure.account.oauth2.client.secret
fs.azure.account.oauth2.client.endpoint
fs.azure.createRemoteFileSystemDuringInitialization -> true

On debugging found that flink reads from core-default-shaded.xml, but even if the properties are specified there, the default configs are not loaded and we get a different exception as :

Copy code

Caused by: Unable to load key provider class.
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:540)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1136)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)

Is anyone familiar with this issue ? Appreciate any help on this. Thanks

Martijn Visser

01/18/2023, 1:39 PM

Could you file a Jira for this?

Dheeraj Panangat

01/18/2023, 2:53 PM

sure @Martijn Visser Ticket : FLINK-30745

sairam yeturi

01/18/2023, 4:40 PM

Ack checking

sairam yeturi

01/20/2023, 1:53 AM

what key provider are you using… this looks a classpath issue… configured in "fs.azure.account.keyprovider.<accountname>" note there in class path.

Dheeraj Panangat

01/20/2023, 11:56 AM

Hi @sairam yeturi, Might have found the resolution to this. Commented on the ticket : FLINK-30745 Can you please take a look and let us know your thoughts ? Thanks

Dheeraj Panangat

01/20/2023, 2:20 PM

Hi @sairam yeturi @Martijn Visser, We were able to get it working on our local machine, where we are just executing the main class. When we deployed the same solution on Kubernetes using Flink Native Kubernetes command, it gives the same error there. The core-default-shaded.xml is included in the docker image jar which is deployed, but still seeing the same issues there. Can you help check what could be the possible reasons ?

Open in Slack

Previous Next