Hi, I wonder if any of you tried ingesting data from S3 from different AWS regions? Most of the buckets that I use are in us-east zone, but I have a new one in

us-west

and when I've pasted the s3 prefix in a spec I got:

prefix[normalized/event/event/]; S3 error: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed

Looks like you weren’t specific and it is expecting

us-west-2

- also note, this needs to go in the JVM properties https://druid.apache.org/docs/latest/development/extensions-core/s3#aws-region

but if majority of my s3 buckets are in

us-east-1

region, can I provide mutliple regions in JVM settings?

@Kyle Hoondert I touched the same approach on #C0309C9L90D where I share a bit more details

You might be able to set up a separate tier of MiddleManagers which have the different JVM config and use Overlord Dynamic Properties to specify those machines for a different region

Hmmm, but I wonder if overlord will accept a task from s3 source that it can't connect to -

<https://druid-router.company.com/druid/indexer/v1/sampler?for=connect>

will return this 400 error. I guess we will find out.

overlord should only try to connect to this to run the sampler (wizard) Ingestion will happen on the MMs only

Yeah, okay. I will try to setup a MM with different region and I will report back on the status.

Sweet! 👍