Hi Team, DOES PINOT SUPPORTS ADLS Gen 2 (wasbs) ...
# troubleshooting
m
Hi Team, DOES PINOT SUPPORTS ADLS Gen 2 (wasbs) or it only supports abfss I am writing parquet files to the azure container, using wasbs method. Then i use this ingestionconfig to ingest it to pinot, using minions: "ingestionConfig": { "batchIngestionConfig": { "segmentIngestionType": "APPEND", "segmentIngestionFrequency": "DAILY", "consistentDataPush": False, "batchConfigMaps": [ { "input.fs.className": "org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS", "input.fs.prop.authenticationType": "ACCESS_KEY", "input.fs.prop.accountName": "wzanalyticsdatastoreprod", "input.fs.prop.accessKey": "xxxxxxxxxxx", "input.fs.prop.fileSystemName": tenant_id, "inputDirURI": f"wasbs://{tenant_id}@wzanalyticsdatastoreprod.blob.core.windows.net/pinot", "includeFileNamePattern": "glob:*/.parquet", "excludeFileNamePattern": "glob:*/.tmp", "inputFormat": "parquet" } ] } But I think pinot is not able to look into specified BLOB. I cant use abfss in this container cause does not support BlobStorageEvents or SoftDelete In my DEV container, i was writing the parquet in abfss methos, it is still working.. Is something wrong in my ingestionConfig, using wasbs. can someone pls help!!
does Pinot only supports ADLS Gen 2 (
abfss
)
m
Yes Pinot supports ADLS Gen2, what impl are you looking for?
m
ADLS Gen 2 wasbs ADLS Gen 2 (
abfss
) is working..
m
wasbs isn’t supported
m
okay. thanks