Can someone help with the below S3 access issue ```Copy tmp Apache Pinot #troubleshooting

Can someone help with the below S3 access issue ``...

Rakesh Bobbala

06/22/2022, 7:59 PM

Can someone help with the below S3 access issue

Copy code

Copy /tmp/pinot-tmp-data/fileUploadTemp/rakesh_test__0__0__20220622T1821Z.a804f229-8b20-4b26-be2a-36c97eecd56b from local to <s3://test_bucket/test_key_1/data/rakesh_test/rakesh_test__0__0__20220622T1821Z.tmp.295786f8-ca70-4929-bacc-8685d7eed4d5>
Response to segmentUpload for segment:rakesh_test__0__0__20220622T1821Z is:{"offset":143796,"status":"UPLOAD_SUCCESS","isSplitCommitType":false,"segmentLocation":"<s3://test_bucket/test_key_1/data/rakesh_test/rakesh_test__0__0__20220622T1821Z.tmp.295786f8-ca70-4929-bacc-8685d7eed4d5>","streamPartitionMsgOffset":"143796","buildTimeSec":-1}
Handled request from 10.0.1.187 POST <http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000/segmentUpload?segmentSizeBytes=151514&buildTimeMillis=119&streamPartitionMsgOffset=143796&instance=Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098&offset=-1&name=rakesh_test__0__0__20220622T1821Z&rowCount=1000&memoryUsedBytes=510612>, content-type multipart/form-data; boundary=XMJs_YLCxWLk2ADRtRKgDg5q7PaR5pB4_Bkwm3 status code 200 OK
Processing segmentCommitEndWithMetadata:Offset: -1,Segment name: rakesh_test__0__0__20220622T1821Z,Instance Id: Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098,Reason: null,NumRows: 1000,BuildTimeMillis: 119,WaitTimeMillis: 0,ExtraTimeSec: -1,SegmentLocation: <s3://test_bucket/test_key_1/data/rakesh_test/rakesh_test__0__0__20220622T1821Z.tmp.295786f8-ca70-4929-bacc-8685d7eed4d5,MemoryUsedBytes>: 510612,SegmentSizeBytes: 151514,StreamPartitionMsgOffset: 143796
Processing segmentCommitEnd(Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098, 143796)
Committing segment rakesh_test__0__0__20220622T1821Z at offset 143796 winner Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098
Committing segment file for segment: rakesh_test__0__0__20220622T1821Z
mkdir <s3://test_bucket/test_key_1/data/rakesh_test>
Copying uri <s3://test_bucket/test_key_1/data/rakesh_test/rakesh_test__0__0__20220622T1821Z.tmp.295786f8-ca70-4929-bacc-8685d7eed4d5> to uri <s3://test_bucket/test_key_1/data/rakesh_test/rakesh_test__0__0__20220622T1821Z>
Deleting uri <s3://test_bucket/test_key_1/data/rakesh_test/rakesh_test__0__0__20220622T1821Z> force true
Caught exception while committing segment file for segment: rakesh_test__0__0__20220622T1821Z
software.amazon.awssdk.services.s3.model.S3Exception: Access Denied (Service: S3, Status Code: 403, Request ID: TNEVQ80HE8YYE6QV, Extended Request ID: s2urpeZBQG+wFdNvBN/AC57hEzwBOJy4kQJMP/rKOpJZHQLsfTQMV5ghT3bF2XatKwxTqmjP0UQ=)

Neha Pawar

06/22/2022, 11:24 PM

have you tried adding the access key and secret key to both controller and server configs as explained here? https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3#controller-config

Rakesh Bobbala

06/22/2022, 11:25 PM

Yes I did

Neha Pawar

06/22/2022, 11:26 PM

can you cehck if you are able to use s3 CLI to upload something to that bucket from the instance, using same credentials. my hunch is the credentials are incorrect

Rakesh Bobbala

06/22/2022, 11:26 PM

But it still gives me this error Also the node group from where the controller and pinot runs has the IAM policies to access the bucket

Rakesh Bobbala

06/22/2022, 11:26 PM

Okay I will try that

Rakesh Bobbala

06/22/2022, 11:44 PM

i'm wondering how the "tmp" files are getting created if the credentials are wrong @Neha Pawar

Neha Pawar

06/22/2022, 11:47 PM

what’s the rest of the stack trace? maybe it doesnt have permission to do the delete

Rakesh Bobbala

06/22/2022, 11:48 PM

May be, below is the EKS nodegroup IAM policy

Copy code

{
  "Action": [
    "s3:GetBucketAcl",
    "s3:GetObject",
    "s3:ListBucket",
    "s3:PutObject"
  ],
  "Effect": "Allow",
  "Resource": "arn:aws:s3:::*"
}

Rakesh Bobbala

06/22/2022, 11:48 PM

this time i changed it to s3:* to see it works

Rakesh Bobbala

06/22/2022, 11:49 PM

also I made one more change

Copy code

object_ownership = "BucketOwnerEnforced"

earlier it was

Copy code

BucketOwnerPreferred

Rakesh Bobbala

06/22/2022, 11:51 PM

rest stack trace

Copy code

at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[pinot-s3-0.11.0-SNAPSHOT-shaded.jar:0.11.0-SNAPSHOT-3a8079f4a140f48c24cfa6b195584f0cb67771fc]

Rakesh Bobbala

06/22/2022, 11:52 PM

Copy code

Removing FSM (if present):{rakesh_test__0__0__20220622T1821Z,ABORTED,1655922135379,Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098,143796,true,<http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000>}
Response to segmentCommitEndWithMetadata for segment:rakesh_test__0__0__20220622T1821Z is:{"offset":-1,"status":"FAILED","isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1}

Rakesh Bobbala

06/23/2022, 4:59 AM

it worked @Neha Pawar. it was because of the S3 bucket ACLs enabled one last question I have setup deep storage as S3 and all the segments are uploaded to S3 bucket. So, when we run the query how the data is fetched ? does pinot pull the data from S3 and deliver the results ? or the segments are stored somewhere in pinot server (local disk) ?

Neha Pawar

06/23/2022, 5:40 AM

The latter. Deep store segnents are only for backup and restore. The serving layer (Pinot servers) keeps every segment locally in uncompressed form, memory mapped

Rakesh Bobbala

06/23/2022, 4:37 PM

Got It I checked the folder /var/pinot/server/data/segment and It is empty and /var/pinot/server/data/index has some information

Rakesh Bobbala

06/23/2022, 4:37 PM

So does index folder store the actual segments ?

Neha Pawar

06/23/2022, 5:19 PM

that’s right ^

Rakesh Bobbala

06/23/2022, 6:10 PM

Got it Thanks for the response

Open in Slack

Previous Next