Hi :wink: I’m still on an old pinot version 0.12 (...
# troubleshooting
f
Hi 😉 I’m still on an old pinot version 0.12 (migration is planned but I need a bit more robustness before) First things to look at is S3 as deepstore I’ve folowed the doc here https://docs.pinot.apache.org/release-0.12.0/users/tutorials/use-s3-as-deep-store-for-pinot From Swagger controller seems able to download the segment when I hit th download API but on server side lots of Failed to download segment absencesreport__1__5__20240527T2122Z from deep store: Download segment absencesreport__1__5__20240527T2122Z from deepstore uri s3://bucketName/segments/absencesreport_REALTIME/absencesreport__1__5__20240527T2122Z failed. Caught exception in state transition from OFFLINE -> ONLINE for resource: absencesreport_REALTIME, partition: absencesreport__1__5__20240527T2122Z Any ideas ?
Copy code
getting logs like software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region is wrong; expecting 'eu-west-1'. (Service: S3, Status Code: 400, Request ID: 184BA1F70B628FA6, Extended Request ID: 82b9e6b1548ad0837abe6ff674d1d3e982a2038442a1059f595d95962627f827)
here is my server conf for the S3 part
Copy code
# Pinot Server Data Directory
pinot.server.instance.dataDir=/var/lib/pinot_data/server/index
# Pinot Server Temporary Segment Tar Directory
pinot.server.instance.segmentTarDir=/var/lib/pinot_data/server/segmentTar
#S3
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-1
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.storage.factory.s3.bucket.name=bucketName
pinot.server.storage.factory.s3.endpoint=URL_OF_MY_S3_ENDOINT
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.segment.fetcher.s3.pathStyleAccess=true
Any ideas welcome 🙂
a
Hey, looking at the error, what stands out is a region mismatch between your configuration and the actual S3 bucket region. Your server configuration specifies us-west-1, but the error message indicates the bucket is actually in eu-west-1. It seems you are also missing some configs, could you try out these: server
Copy code
# Pinot Server Data Directory
pinot.server.instance.dataDir=/var/lib/pinot_data/server/index
# Pinot Server Temporary Segment Tar Directory
pinot.server.instance.segmentTarDir=/var/lib/pinot_data/server/segmentTar
# S3 Configuration
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=eu-west-1
pinot.server.storage.factory.s3.accessKey=YOUR_ACCESS_KEY
pinot.server.storage.factory.s3.secretKey=YOUR_SECRET_KEY
pinot.server.storage.factory.s3.disableAcl=false
pinot.controller.storage.factory.s3.disableAcl=false
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.segment.store.uri=s3://bucketName
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.s3.endpoint=URL_OF_MY_S3_ENDOINT
pinot.server.segment.fetcher.s3.pathStyleAccess=true
Controller
Copy code
controller.data.dir=s3://bucketName
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=eu-west-1
pinot.controller.storage.factory.s3.accessKey=YOUR_ACCESS_KEY
pinot.controller.storage.factory.s3.secretKey=YOUR_SECRET_KEY
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
Lastly: Verify S3 connectivity: Try to access your S3 bucket directly from the Pinot server using AWS CLI to confirm credentials and connectivity Verify segment existence: Confirm that the segment absencesreport__1__5__20240527T2122Z actually exists in the expected S3 path
f
Actually pinot.server.storage.factory.s3.accessKey and pinot.server.storage.factory.s3.secretKey came from aws cli already installed and configured
Good catch for the pinot.server.storage.factory.s3.region=eu-west-1 👍 Give it a try /restart all my server and I will came back 😉
Some buggy segments (latest commit one to check) 😄 But mots of them have now been downloaded successufully. Thx for you help 👍