https://pinot.apache.org/ logo
Join Slack
Powered by
Hi All, i am trying to configure apache pinot with...
# general
y
Hi All, i am trying to configure apache pinot with realtime stream from kafka with S3 as deep store, and all the components are TLS enabled with basic auth. I was able to see the data in pinot, as well as segments. but not able to see this data in s3 bucket. the S3 permission seems to be good as pinot was able to create directories to store data. was in this stage of implementaiton from past few days, can anyone please help !!
This is the table config for your ref. 
"REALTIME": {
"tableName": "transcript_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"schemaName": "transcript",
"timeType": "MILLISECONDS",
"replicasPerPartition": "1",
"timeColumnName": "timestampInEpoch",
"minimizeDataMovement": false
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false,
"columnMajorSegmentBuilderEnabled": false,
"optimizeDictionary": false,
"optimizeDictionaryForMetrics": false,
"noDictionarySizeRatioThreshold": 0.85,
"rangeIndexVersion": 2,
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"loadMode": "MMAP"
},
"metadata": {
"customConfigs": {}
},
"ingestionConfig": {
"continueOnError": true,
"rowTimeValueCheck": true,
"segmentTimeValueCheck": false,
"streamIngestionConfig": {
"streamConfigMaps": [
{
"stream.kafka.decoder.prop.format": "JSON",
"key.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
"streamType": "kafka",
"value.serializer": "org.apache.kafka.common.serialization.ByteArraySerializer",
"stream.kafka.consumer.type": "LOWLEVEL",
"stream.kafka.topic.name": "pinot-transcripts",
"stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
"stream.kafka.broker.list": "<https://kafka-cluster-kafka-bootstrap.services-quin.svc.cluster.local:9093>",
"sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"kafka-cluster-user\" password=\"\";",
"stream.kafka.hlc.zk.connect.string": "kafka-cluster-zookeeper-client.services-quin:2181",
"stream.kafka.zk.broker.url": "kafka-cluster-zookeeper-client.services-quin:2181",
"sasl.mechanism": "SCRAM-SHA-512",
"security.protocol": "SASL_SSL",
"ssl.truststore.location": "/keystore/truststore.p12",
"ssl.truststore.password": "",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.threshold.segment.rows": "50000",
"realtime.segment.serverUploadToDeepStore": "true",
"stream.kafka.consumer.prop.auto.offset.reset": "smallest"
}
],
"columnMajorSegmentBuilderEnabled": false
},
"transformConfigs": []
},
"isDimTable": false
}
}
m
Segments are committed to S3 periodically, are you saying that the segments are committed but don’t show up in S3? Can you share a segment metadata (swagger api), what does the download url point to in the metadata?
y
Hello Mayank, thanks for the response. There are developments in my troubleshooting after i posted this query, Pinot was able to push to S3 as soon as i updated thresholds in config, this happened when i am testing without TLS setup between pinot components and also without any authentication mechanism. When i flip back to with one-way TLS and using auth(ZK), the segments are showing still in progress. hence no push to s3. I found a server log, that says connection issue between server and controller. as i believe server using http where as we need to use https. I will post the error log, controller and server config in a seperate msg Here is the segment metadata for your ref.
Copy code
{
  "segment.realtime.numReplicas": "1",
  "segment.creation.time": "1711095326640",
  "segment.flush.threshold.size": "6",
  "segment.realtime.startOffset": "0",
  "segment.realtime.status": "IN_PROGRESS"
}
<http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.NoHttpResponseException: pinot-controller-0.pinot-controller-headless.pinot-gs.svc.cluster.local:9443 failed to respond
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org.apache.http.impl.io|org.apache.pinot.shaded.org.apache.http.impl.io>.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at <http://org.apache.pinot.shaded.org|org.apache.pinot.shaded.org>.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at org.apache.pinot.common.utils.http.HttpClient.sendRequest(HttpClient.java:282) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentCompletionProtocolRequest(FileUploadDownloadClient.java:1131) ~[pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.sendRequest(ServerSegmentCompletionProtocolHandler.java:209) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.segmentConsumed(ServerSegmentCompletionProtocolHandler.java:176) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.postSegmentConsumedMsg(RealtimeSegmentDataManager.java:1198) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run(RealtimeSegmentDataManager.java:719) [pinot-all-1.2.0-SNAPSHOT-jar-with-dependencies.jar:1.2.0-SNAPSHOT-e578dd2b19365dfffae3ade93edca290833ce2d1]
at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Controller conf: 
dafd   controller.helix.cluster.name=pinot
controller.data.dir=<s3://bucketname/controller-data>
controller.zk.str=pinot-zookeeper:2181
<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
pinot.controller.segment.fetcher.protocols=file,https,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=eu-
controller.enable.split.commit=true
controller.task.scheduler.enabled=true
controller.segment.fetcher.auth.token=Basic
controller.access.protocols=https
controller.access.protocols.https.port=9443
controller.broker.protocol=https
controller.broker.port.override=8443
controller.vip.protocol=https
controller.vip.port=9443
controller.local.temp.dir=/tmp/pinot/controller/data
controller.tls.keystore.path=/keystore/keystore.p12
controller.tls.keystore.password=
controller.tls.truststore.path=/keystore/truststore.p12
controller.tls.truststore.password=
controller.admin.access.control.factory.class=org.apache.pinot.controller.api.access.ZkBasicAuthAccessControlFactory
Server.conf 
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
pinot.server.instance.realtime.alloc.offheap=true
pinot.server.segment.fetcher.auth.token=Basic
pinot.server.segment.uploader.auth.token=Basic
pinot.server.instance.auth.token=Basic
pinot.server.adminapi.access.protocols=https
pinot.server.adminapi.access.protocols.https.port=7443
pinot.server.netty.enabled=false
pinot.server.nettytls.enabled=true
pinot.server.nettytls.port=8089
pinot.server.tls.keystore.path=/keystore/keystore.p12
pinot.server.tls.keystore.password=
pinot.server.tls.truststore.path=/keystore/truststore.p12
pinot.server.tls.truststore.password=
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.segment.fetcher.protocols=file,https,s3
pinot.server.instance.segment.store.uri=s3://
pinot.server.instance.enable.split.commit=true
pinot.server.enable.commitend.metadata=true
pinot.server.storage.factory.s3.region=eu-
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.storage.factory.s3.httpclient.maxConnections=50
pinot.server.storage.factory.s3.httpclient.socketTimeout=60s
pinot.server.storage.factory.s3.httpclient.connectionTimeout=10s
pinot.server.storage.factory.s3.httpclient.connectionTimeToLive=0s
pinot.server.storage.factory.s3.httpclient.connectionAcquisitionTimeout=30s
m
I think 
pinot.server.segment.uploader.protocol=https
is missing?
y
Hi @Mayank, Its working, TYSM for taking your time on this, i really appreciate the help.
m
Glad to hear that
2 Views
Open in Slack
PreviousNext
Powered by