Hi Team,
DOES PINOT SUPPORTS ADLS Gen 2 (wasbs) or it only supports abfss
I am writing parquet files to the azure container, using wasbs method.
Then i use this ingestionconfig to ingest it to pinot, using minions:
"ingestionConfig": {
"batchIngestionConfig": {
"segmentIngestionType": "APPEND",
"segmentIngestionFrequency": "DAILY",
"consistentDataPush": False,
"batchConfigMaps": [
{
"input.fs.className": "org.apache.pinot.plugin.filesystem.ADLSGen2PinotFS",
"input.fs.prop.authenticationType": "ACCESS_KEY",
"input.fs.prop.accountName": "wzanalyticsdatastoreprod",
"input.fs.prop.accessKey": "xxxxxxxxxxx",
"input.fs.prop.fileSystemName": tenant_id,
"inputDirURI": f"wasbs://{tenant_id}@wzanalyticsdatastoreprod.blob.core.windows.net/pinot",
"includeFileNamePattern": "glob:*/.parquet",
"excludeFileNamePattern": "glob:*/.tmp",
"inputFormat": "parquet"
}
]
}
But I think pinot is not able to look into specified BLOB. I cant use abfss in this container cause does not support BlobStorageEvents or SoftDelete
In my DEV container, i was writing the parquet in abfss methos, it is still working..
Is something wrong in my ingestionConfig, using wasbs. can someone pls help!!