This message was deleted.
# generals
Slackbot
03/28/2023, 5:24 PMThis message was deleted.
s
Sergio Ferragut
03/28/2023, 5:38 PMrun a simple batch ingestion and see if you get segments in the bucket
Sergio Ferragut
03/28/2023, 5:39 PMinitially, streaming ingestion will not publish segments, it will publish at handoff which occurs normally at the end of the task duration (or at intermediateHandoffPeriod).
a
Adam Peck
03/28/2023, 5:40 PMI normally test with the wikipedia sample from the web console loader.
v
Vamsi Gandrapu
03/28/2023, 5:42 PM👍 will give a try, thanks!
Vamsi Gandrapu
03/28/2023, 6:28 PMAuth failure now. And I see
WebIdentityTokenCredentialsProvider Copy code
2023-03-28T17:55:54,309 WARN [[index_parallel_wikipedia_nhgflfaj_2023-03-28T17:55:31.446Z]-appenderator-merge] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 4) in 1,008ms.
java.io.IOException: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: NETB5ZBG7E7FH9Z7; S3 Extended Request ID: iZluZGHneBhidh5UvQYDvTyJL6iznH89s3WPtoZB2BakmnLm5QblX1SuIKEIrjX1f7md18/sTDcyC7Dx3cKgEw==; Proxy: null), S3 Extended Request ID: iZluZGHneBhidh5UvQYDvTyJL6iznH89s3WPtoZB2BakmnLm5QblX1SuIKEIrjX1f7md18/sTDcyC7Dx3cKgEw==
at org.apache.druid.storage.s3.S3DataSegmentPusher.pushToPath(S3DataSegmentPusher.java:108) ~[?:?]
at org.apache.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:82) ~[?:?]
at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:956) ~[druid-server-25.0.0.jar:25.0.0]a
Adam Peck
03/28/2023, 6:31 PMIf you describe the pod this is running on do you see the AWS related environment variables set?
Did you use the Druid Helm chart to install or some other method? Did you specify the IRSA annotation on the service account for all service types?
v
Vamsi Gandrapu
03/28/2023, 6:39 PMyes, pod does have env vars. I am using operator to create this cluster, yes, druid helm charts..
serviceaccount has an annotation
Copy code
Annotations: <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::<<myaccount>>:role/kubernetes_services/<<myrole>>a
Adam Peck
03/28/2023, 6:39 PMDoes the role have access to read from the S3 bucket on both the bucket and the role?
Adam Peck
03/28/2023, 6:39 PMWell read and write most likely is required.
v
Vamsi Gandrapu
03/28/2023, 6:41 PMyes, I am able to read/write from a test pod, but not druid
Vamsi Gandrapu
03/28/2023, 6:50 PMwondering if there is a specific rbac needed for the pods
f
Frank Cao
03/28/2023, 10:15 PMhave you set up the druid service account in druid-operator?eg. serviceAccount: druid-application-sa, you need to add S3 permissions to the service account.
v
Vamsi Gandrapu
03/28/2023, 10:17 PMnever mind, just enabled trace logging on
com.amazonawsVamsi Gandrapu
03/28/2023, 10:28 PM Copy code
2023-03-28T21:57:58,829 DEBUG [[index_parallel_wikipedia_acdefclb_2023-03-28T21:57:40.714Z]-appenderator-merge] com.amazonaws.auth.AWSCredentialsProviderChain - Unable to load credentials from WebIdentityTokenCredentialsProvider: Unable to locate specified web identity token file: /var/run/secrets/eks.amazonaws.com/serviceaccount/token