@User Is there a support for multiple directories for FS? If Yes, we can extend that to multiple buckets.

@User How do you want to split data across buckets?

@User No, I was thinking if users can provide a list of subFolders/s3buckets, we can pick one randomly or hash it based on segment name

Randomly at the time of creating the segments?

Wouldn't that disrupt the query execution?

no, we just store the uri along with segment metadata in ZK

it can point to anything

actually, this is a problem only with real-time where we create the URI

with batch ingestion, user can provide any URI

We don’t have any specific requirement around how to slit data across buckets.

Ok. Then I believe the change needs to be done in the handling of ingestion config and then picking a random directory while creating segments S3 filesystem implementation won't need any change unless the buckets are located in different regions

all the buckets are co located.

Yash, is this realtime or offline

Right now it is only offline.

then, you dont need any thing for now

I am guessing you will use the ingestion-job to generate the segments

Yeah I realised that too. I am very new to this so sorry for any troubles 🙂

no worries, this is a good feature to have. if you dont mind, can you create an issue

@User has left the channel

@User has left the channel

Hi, this sounds like the right channel for my question, but please redirect me if not - I believe my question is a little different from what i see in history. I am trying to use org.apache.pinot.plugin.filesystem.S3PinotFS for ingesting data, but I need to ingest using read only credentials from one account and write credentials to a bucket in a second account. Something like this:

inputDirURI: '<s3://bucket_with_read_only_credentials/data/>'
outputDirURI: '<s3://bucket_with_separate_read_write_credentials/pinot/segments>'

Is it possible to configure separate read and write credentials?

@aj Did you ever get an answer to this question? I am having the same problem.