https://pinot.apache.org/ logo
#general
Title
# general
r

Ryan Clark

07/22/2021, 6:55 PM
Can Pinot be hosted in one AWS account and read a stream from another account? Perhaps a way to give the table config an account number.
m

Mayank

07/22/2021, 6:56 PM
Should be possible, cc: @User?
🙌 1
d

Daniel Lavoie

07/22/2021, 7:00 PM
you will simply need VPC peering with the other account
Ensure you don’t have overlapping CIDR
Pinot doesn’t really care about account ownership. It’s only ip networking 🙂
r

Ryan Clark

07/22/2021, 7:02 PM
Here's the initial error that makes me think there needs to be a way to explicitly give it the account number:
Copy code
{"code":500,"error":"shaded.software.amazon.awssdk.services.kinesis.model.ResourceNotFoundException: Stream stream-name under account 123 not found.
stream-name
is actually in account 456, not 123.
d

Daniel Lavoie

07/22/2021, 7:03 PM
Ok, that’s a specific kinesis question. Sorry I was having kafka in mind.
n

Neha Pawar

07/22/2021, 7:04 PM
i assume you have already given the right accessKey/secretKey?
@User we recently fixed a similar scenario for minion ingestion from S3 in another account right?
r

Ryan Clark

07/22/2021, 7:05 PM
I was able to skip that because it is on EKS. Those permissions have worked to get my first stream connected (in the same account as EKS).
I also gave IAM permissions for EKS to read the stream in the second account
x

Xiang Fu

07/22/2021, 7:06 PM
pinot ingestion can read from other stream/account as long as you are fine putting the credentials
r

Ryan Clark

07/22/2021, 7:07 PM
I see, so the accessKey/secretKey will tell Pinot which account to look for. Is there another way? We use SSO, and don't make users.
x

Xiang Fu

07/22/2021, 7:07 PM
for s3, you can give the corresponding accessKey and accessSecret
yes, and it’s local to that ingestion job
n

Neha Pawar

07/22/2021, 7:08 PM
this is for kinesis ingestion, Xiang
i’m guessing it should work the same
k

Kartik Khare

07/22/2021, 7:11 PM
yes, for kinesis as well you can provide region secret key and access kry
n

Neha Pawar

07/22/2021, 7:16 PM
@User any alternative to providing that? can giving the endpoint work here?
🙌 1
x

Xiang Fu

07/22/2021, 8:03 PM
We may need to add a few more params(
credentialProvider
,
roleArn
and
roleSessionName
) to make it happen, according to the example code: https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html#examples-cross-app
@User since you are asking for this, are you willing to add the support for this 😛
r

Ryan Clark

07/22/2021, 10:09 PM
@User I'd love to give it a shot. I'm going to create a work around on my end to finish my proof-of-concept and then I can swing back to this. I'll create an issue.
x

Xiang Fu

07/22/2021, 10:09 PM
sgtm 👍 you should be able to modify current code to do so