Can Pinot be hosted in one AWS account and read a stream from another account? Perhaps a way to give the table config an account number.

Should be possible, cc: @User?

you will simply need VPC peering with the other account

Here's the initial error that makes me think there needs to be a way to explicitly give it the account number:

{"code":500,"error":"shaded.software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException: Stream stream-name under account 123 not found.

stream-name

is actually in account 456, not 123.

Ok, that’s a specific kinesis question. Sorry I was having kafka in mind.

i assume you have already given the right accessKey/secretKey?

I was able to skip that because it is on EKS. Those permissions have worked to get my first stream connected (in the same account as EKS).

pinot ingestion can read from other stream/account as long as you are fine putting the credentials

I see, so the accessKey/secretKey will tell Pinot which account to look for. Is there another way? We use SSO, and don't make users.

for s3, you can give the corresponding accessKey and accessSecret

this is for kinesis ingestion, Xiang

yes, for kinesis as well you can provide region secret key and access kry

@User any alternative to providing that? can giving the endpoint work here?

Maybe try to follow this link: https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html

@User I'd love to give it a shot. I'm going to create a work around on my end to finish my proof-of-concept and then I can swing back to this. I'll create an issue.

sgtm 👍 you should be able to modify current code to do so