Then i changed config as mentioned in <https://d...
# getting-starteda
Amit Chopra
12/11/2020, 4:50 PMThen i changed config as mentioned in https://docs.pinot.apache.org/operators/operating-pinot/decoupling-controller-from-the-data-path. And now segments are not being written to S3. I do see segments being created, as they show up on query browser. But the segments show up as status BAD. Can someone help to point what is wrong with the configuration:
Configs:
controller.conf
------------------------------
controller.helix.cluster.name=pinot-quickstart
controller.port=9000
controller.enable.split.commit=true
controller.allow.hlc.tables=false
controller.data.dir=/tmp/pinot-tmp-data/
controller.local.temp.dir=/tmp/pinot-tmp-data/
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-west-2
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
controller.zk.str=pinot-zookeeper:2181
pinot.set.instance.id.to.hostname=true
server.conf
------------------
pinot.server.netty.port=8098
pinot.server.instance.enable.split.commit=true
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index
pinot.server.instance.segment.store.uri=s3://pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-data
pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.set.instance.id.to.hostname=true
pinot.server.instance.realtime.alloc.offheap=true
table conf
----------------------
{
“REALTIME”: {
“tableName”: “demo1_REALTIME”,
“tableType”: “REALTIME”,
“segmentsConfig”: {
“timeType”: “MILLISECONDS”,
“schemaName”: “demo1",
“timeColumnName”: “mergedTimeMillis”,
“retentionTimeUnit”: “DAYS”,
“retentionTimeValue”: “60",
“replication”: “1",
“replicasPerPartition”: “1",
“completionConfig”: {
“completionMode”: “DOWNLOAD”
},
“peerSegmentDownloadScheme”: “http”
},
“tenants”: {
“broker”: “DefaultTenant”,
“server”: “DefaultTenant”
},
“tableIndexConfig”: {
“streamConfigs”: {
“streamType”: “kafka”,
“stream.kafka.consumer.type”: “lowlevel”,
“stream.kafka.topic.name”: “demo1",
“stream.kafka.decoder.class.name”: “org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder”,
“stream.kafka.consumer.factory.class.name”: “org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory”,
“stream.kafka.zk.broker.url”: “z-1.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:2181,z-3.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:2181,z-2.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:2181",
“stream.kafka.broker.list”: “b-2.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:9092,b-1.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:9092",
“realtime.segment.flush.threshold.time”: “10m”,
“realtime.segment.flush.threshold.size”: “10000",
“stream.kafka.consumer.prop.auto.offset.reset”: “smallest”
},
“enableDefaultStarTree”: false,
“enableDynamicStarTreeCreation”: false,
“loadMode”: “MMAP”,
“autoGeneratedInvertedIndex”: false,
“createInvertedIndexDuringSegmentGeneration”: false,
“aggregateMetrics”: false,
“nullHandlingEnabled”: false
},
“metadata”: {
“customConfigs”: {}
}
}
}
x
Xiang Fu
12/11/2020, 5:04 PM
I think this controller.data.dir=/tmp/pinot-tmp-data/ should be on s3?
Xiang Fu
12/11/2020, 5:26 PM
oic, this is for split commit
Xiang Fu
12/11/2020, 5:26 PM
have you seen any logs on pinot server for not able to write to s3?
a
Amit Chopra
12/11/2020, 5:30 PMBTW - if i change controller.data.dir to s3 path, things start to work. Segments are getting created in S3. But how do i know then if it is controller or server creating and uploading the segments to S3?
Amit Chopra
12/11/2020, 5:38 PM@Xiang Fu - basically trying to understand if pinot.server.instance.segment.store.uri is set with S3 path for server config, does controller also need the S3 path set using controller.data.dir?
And if required to be set to controller too, why does it need that?
x
Xiang Fu
12/11/2020, 5:38 PM
no need
Xiang Fu
12/11/2020, 5:38 PM
in your case it's separation
Xiang Fu
12/11/2020, 5:38 PM
I think your config is fine
Xiang Fu
12/11/2020, 5:39 PM
can you check server log and see if there is any exception about saving segment to s3
👍 1
a
Amit Chopra
12/11/2020, 5:39 PMi see. so next steps is for me to remove the s3 path from controller conf. And then check the logs on server logs
x
Xiang Fu
12/11/2020, 5:40 PM
yes, controller is not on data path
a
Amit Chopra
12/11/2020, 8:15 PM@Ting Chen @Xiang Fu - i see the following in server logs (with controller.data.dir pointing to local temp dir)
2020/12/11 201346.613 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__1__125__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201347.851 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__4__126__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201348.052 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__3__126__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201349.012 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__0__125__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201349.331 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__2__126__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201349.695 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__1__125__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201350.931 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__4__126__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201351.115 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__3__126__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201352.082 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__0__125__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201352.389 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__2__126__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201352.752 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__1__125__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201354.014 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__4__126__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
2020/12/11 201354.206 WARN [LLRealtimeSegmentDataManager_demo1__3__126__20201211T1757Z] [demo1__3__126__20201211T1757Z] CommitEnd failed with response {“isSplitCommitType”false,“streamPartitionMsgOffset”null,“buildTimeSec” 1,“status”“FAILED”,“offset”:-1}
t
Ting Chen
12/11/2020, 8:44 PMcan you check the controller log to find why CommitEnd failed?
m
Mahesh Yeole
12/14/2020, 7:20 PM@Ting Chen @Xiang Fu I see lot of files are written to S3 under same timestamp but i do see error on controller as well as server.
I see on cluster manager console, segment keep showing consuming….
We are tying to use split commit feature but even setting split.commit to true for both controller and server , we do see "isSplitCommitType":false in server error.
error on server logs
[LLRealtimeSegmentDataManager_pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z] [pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z] CommitEnd failed with response {"isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"FAILED","offset":-1}
Error on controller logs
[SegmentCompletionFSM_pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z] [grizzly-http-server-1] Caught exception while committing segment file for segment: pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z
java.io.IOException: software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: E62169F11317304B, Extended Request ID: 3dlRY25FjPWIVJsA82PfQnhwlyp/26Nw1VM2xZCzlqEUvNSIXpFSexbvMewbLTR3ZuaDSHE6rq8=)
This is my controller.conf
controller.helix.cluster.name=pinot-cluster
controller.port=9000
controller.local.temp.dir=/var/pinot/controller/data
controller.data.dir=s3://pinot-cluster-segment-s3/pinot-data/pinot-s3-example/controller-data/
controller.zk.str=pinot-zookeeper:2181
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-west-2
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
controller.allow.hlc.tables=false
controller.enable.split.commit=true
pinot.set.instance.id.to.hostname=true
This is my server.conf
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
pinot.set.instance.id.to.hostname=true
pinot.server.instance.realtime.alloc.offheap=true
pinot.server.instance.segment.store.uri=s3://pinot-cluster-segment-s3/pinot-data/pinot-s3-example/controller-data/
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcherroot@pinot-server-0:/opt/pinot#
Mahesh Yeole
12/14/2020, 7:20 PM@Ting Chen @Xiang Fu Are we missing any configuration ?
x
Xiang Fu
12/14/2020, 9:09 PM
Copy code
java.io.IOException: software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: E62169F11317304B, Extended Request ID: 3dlRY25FjPWIVJsA82PfQnhwlyp/26Nw1VM2xZCzlqEUvNSIXpFSexbvMewbLTR3ZuaDSHE6rq8=)Xiang Fu
12/14/2020, 9:09 PM
do we set the credential correctly for aws
m
Mahesh Yeole
12/14/2020, 9:35 PM@Xiang Fu yes we did set env variables for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN in templates/controller/statefulset.yaml and templates/server/statefulset.yml
Mahesh Yeole
12/14/2020, 9:47 PMI am able to access S3 from server pod
Mahesh Yeole
12/14/2020, 9:48 PMmyeole-a01:pinot myeole$ aws s3 ls pinot-cluster-segment-s3/pinot-data/pinot-s3-example/controller-data/pullRequestMergedEventsAwsMskDemo/
2020-12-14 134707 0
2020-12-14 102619 855564 pullRequestMergedEventsAwsMskDemo__0__0__20201214T1817Z00d97c80-cad9-48d8-ab08-4bea626f30a3
2020-12-14 104849 855564 pullRequestMergedEventsAwsMskDemo__0__0__20201214T1817Z00da9537-502a-4f56-b6cb-85656dc743e8
x
Xiang Fu
12/14/2020, 9:49 PM
ic
Xiang Fu
12/14/2020, 9:50 PM
have you checked the pinotServer.log and pinotController.log for ERROR
Xiang Fu
12/14/2020, 9:51 PM
that should have more logs than kubectl console log
m
Mahesh Yeole
12/14/2020, 9:52 PMyes I see these errors in logs
Mahesh Yeole
12/14/2020, 9:52 PMerror on server logs
[LLRealtimeSegmentDataManager_pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z] [pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z] CommitEnd failed with response {"isSplitCommitType":false,"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"FAILED","offset":-1}
Error on controller logs
[SegmentCompletionFSM_pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z] [grizzly-http-server-1] Caught exception while committing segment file for segment: pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z
java.io.IOException: software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: E62169F11317304B, Extended Request ID: 3dlRY25FjPWIVJsA82PfQnhwlyp/26Nw1VM2xZCzlqEUvNSIXpFSexbvMewbLTR3ZuaDSHE6rq8=)
Mahesh Yeole
12/14/2020, 9:52 PMi dont see any other error
x
Xiang Fu
12/14/2020, 9:52 PM
for
<s3://pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-data>Xiang Fu
12/14/2020, 9:53 PM
is bucket
pinot-quickstart-s3Xiang Fu
12/14/2020, 9:53 PM
or it’s
s3://<bucket>/pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-dataXiang Fu
12/14/2020, 9:54 PM
can you try to fill the bucket name there as well
m
Mahesh Yeole
12/14/2020, 9:59 PMI am using this pinot.server.instance.segment.store.uri=s3://pinot-cluster-segment-s3/pinot-data/pinot-s3-example/controller-data/
x
Xiang Fu
12/14/2020, 10:02 PM
right, i mean do you have a bucket with this s3 url?
Xiang Fu
12/14/2020, 10:05 PM
i somehow feel that segment is not accessible inside pinot
Xiang Fu
12/14/2020, 10:06 PM
so
pinot-cluster-segment-s3m
Mahesh Yeole
12/14/2020, 10:06 PMyes
x
Xiang Fu
12/14/2020, 10:07 PM
ic
Xiang Fu
12/14/2020, 10:08 PM
can you check segment metadata in swagger ui
Xiang Fu
12/14/2020, 10:08 PM
for the failed segment
Xiang Fu
12/14/2020, 10:08 PM
you can give table name/ segment name
m
Mahesh Yeole
12/14/2020, 10:08 PM Copy code
{
"segment.realtime.endOffset": "9223372036854775807",
"segment.time.unit": null,
"segment.start.time": "-1",
"segment.flush.threshold.size": "10000",
"segment.realtime.startOffset": "10000",
"segment.end.time": "-1",
"segment.total.docs": "-1",
"segment.table.name": "pullRequestMergedEventsAwsMskDemo_REALTIME",
"segment.realtime.numReplicas": "1",
"segment.creation.time": "1607971877305",
"segment.realtime.download.url": null,
"segment.name": "pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z",
"segment.index.version": null,
"custom.map": null,
"segment.flush.threshold.time": null,
"segment.type": "REALTIME",
"segment.crc": "-1",
"segment.realtime.status": "IN_PROGRESS"
}Mahesh Yeole
12/14/2020, 10:15 PMthis is on cluster manager UI
Mahesh Yeole
12/14/2020, 10:15 PMfrom swagger API
Mahesh Yeole
12/14/2020, 10:15 PMI am getting this
Mahesh Yeole
12/14/2020, 10:18 PM Copy code
{
"code": 404,
"error": "Segment: pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z of table: pullRequestMergedEventsAwsMskDemo_REALTIME not found at: <s3://pinot-cluster-segment-s3/pinot-data/pinot-s3-example/controller-data//pullRequestMergedEventsAwsMskDemo_REALTIME/pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z>"
}Mahesh Yeole
12/14/2020, 10:19 PMobjects in S3 are in this format, the files are created looks like every few seconds ..why not sure
Mahesh Yeole
12/14/2020, 10:19 PMpullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z000af865-ade9-4d23-844e-e1e3c2a0d36d - December 14, 2020, 11:02 (UTC-08:00)
875.0 KB
Standard
pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z00266be0-75b2-4f9d-bd22-a0a41a0632a7 - December 14, 2020, 11:03 (UTC-08:00)
875.0 KB
Standard
pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z00278389-5ccb-4641-99a1-bb8a54170c7e - December 14, 2020, 12:33 (UTC-08:00)
875.0 KB
Standard
pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z003590a9-6c21-4707-8b76-22c24f2b577a - December 14, 2020, 14:16 (UTC-08:00)
875.0 KB
Standard
pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z0048e9d0-c701-4840-a017-7e482d422243 - December 14, 2020, 13:45 (UTC-08:00)
875.0 KB
Standard
pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z00515765-c3e2-447b-bff7-0b2f4320a8f3 - December 14, 2020, 13:11 (UTC-08:00)
875.0 KB
Standard
pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z0061a14d-e1fa-4caf-8a97-7d5ff12c47cf - December 14, 2020, 11:25 (UTC-08:00)
875.0 KB
Standard
pullRequestMergedEventsAwsMskDemo__0__1__20201214T1851Z0073c589-929a-4c45-ac96-7ed2b012a288
x
Xiang Fu
12/14/2020, 10:21 PM
hmmm, I feel those segments are created and uploaded from server
Xiang Fu
12/14/2020, 10:21 PM
but controller commit got failed
Xiang Fu
12/14/2020, 10:21 PM
@Ting Chen do u have any idea?
a
Amit Chopra
12/15/2020, 6:15 PM@Ting Chen @Xiang Fu Just to close on this thread. I deleted my k8s namespace and redeployed. This time passed s3 path to both server and controller. And things are working as expected. Thanks for all the help 🙂
I see the following logs on server instance:
2020/12/15 170225.113 INFO [LLRealtimeSegmentDataManager_demo2__0__0__20201215T1652Z] [demo2__0__0__20201215T1652Z] Successfully built segment in 312 ms, after lockWaitTime 0 ms
2020/12/15 170225.139 INFO [FileUploadDownloadClient] [demo2__0__0__20201215T1652Z] Sending request: http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000/segmentCommitStart?name=demo2__0__0__20201215T1652Z&offset=298&instance=Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098&buildTimeMillis=312&memoryUsedBytes=6327504&segmentSizeBytes=132903&rowCount=298&streamPartitionMsgOffset=298 to controller: pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local, version: Unknown
2020/12/15 170225.140 INFO [ServerSegmentCompletionProtocolHandler] [demo2__0__0__20201215T1652Z] Controller response {“streamPartitionMsgOffset”null,“buildTimeSec”-1,“isSplitCommitType”false,“status”“COMMIT_CONTINUE”,“offset”:-1} for http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000/segmentCommitStart?name=demo2__0__0__20201215T1652Z&offset=298&instance=Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098&buildTimeMillis=312&memoryUsedBytes=6327504&segmentSizeBytes=132903&rowCount=298&streamPartitionMsgOffset=298
2020/12/15 170225.767 INFO [S3PinotFS] [pool-9-thread-1] Copy /tmp/pinot-tmp/server/index/demo2_REALTIME/demo2__0__0__20201215T1652Z.tar.gz from local to s3://pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-data/demo2/demo2__0__0__20201215T1652Zb58f082e-07be-4928-af1e-a941d09101b2
2020/12/15 170225.883 INFO [PinotFSSegmentUploader] [demo2__0__0__20201215T1652Z] Successfully upload segment demo2__0__0__20201215T1652Z to s3://pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-data/demo2/demo2__0__0__20201215T1652Zb58f082e-07be-4928-af1e-a941d09101b2.
2020/12/15 170226.410 INFO [FileUploadDownloadClient] [demo2__0__0__20201215T1652Z] Sending request: http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000/segmentCommitEndWithMetadata?name=demo2__0__0__20201215T1652Z&offset=298&instance=Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098&buildTimeMillis=312&memoryUsedBytes=6327504&segmentSizeBytes=132903&rowCount=298&location=s3://pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-data/demo2/demo2__0__0__20201215T1652Zb58f082e-07be-4928-af1e-a941d09101b2&streamPartitionMsgOffset=298 to controller: pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local, version: Unknown
2020/12/15 170226.411 INFO [ServerSegmentCompletionProtocolHandler] [demo2__0__0__20201215T1652Z] Controller response {“streamPartitionMsgOffset”null,“buildTimeSec”-1,“isSplitCommitType”false,“status”“COMMIT_SUCCESS”,“offset”:-1} for http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000/segmentCommitEndWithMetadata?name=demo2__0__0__20201215T1652Z&offset=298&instance=Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098&buildTimeMillis=312&memoryUsedBytes=6327504&segmentSizeBytes=132903&rowCount=298&location=s3://pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-data/demo2/demo2__0__0__20201215T1652Zb58f082e-07be-4928-af1e-a941d09101b2&streamPartitionMsgOffset=298
Amit Chopra
12/15/2020, 6:16 PMmy final config, in case it helps for future:
Server
-------------------------
/var/pinot/server/config/pinot-server.conf
pinot.server.netty.port=8098
pinot.server.instance.enable.split.commit=true
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index
pinot.server.instance.segment.store.uri=s3://pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-data
pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-west-2
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.set.instance.id.to.hostname=true
pinot.server.instance.realtime.alloc.offheap=true
controller
-----------------------
/var/pinot/controller/config/pinot-controller.conf
controller.helix.cluster.name=pinot-quickstart
controller.port=9000
controller.enable.split.commit=true
controller.allow.hlc.tables=false
controller.data.dir=s3://pinot-quickstart-s3/pinot-data/pinot-s3-example/controller-data
controller.local.temp.dir=/tmp/pinot-tmp-data/
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-west-2
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
controller.zk.str=pinot-zookeeper:2181
pinot.set.instance.id.to.hostname=true
Table config
---------------------
{
“REALTIME”: {
“tableName”: “demo2_REALTIME”,
“tableType”: “REALTIME”,
“segmentsConfig”: {
“timeColumnName”: “mergedTimeMillis”,
“retentionTimeUnit”: “DAYS”,
“retentionTimeValue”: “60",
“completionConfig”: {
“completionMode”: “DOWNLOAD”
},
“peerSegmentDownloadScheme”: “http”,
“timeType”: “MILLISECONDS”,
“schemaName”: “demo2”,
“replication”: “1”,
“replicasPerPartition”: “1”
},
“tenants”: {
“broker”: “DefaultTenant”,
“server”: “DefaultTenant”
},
“tableIndexConfig”: {
“autoGeneratedInvertedIndex”: false,
“loadMode”: “MMAP”,
“createInvertedIndexDuringSegmentGeneration”: false,
“streamConfigs”: {
“streamType”: “kafka”,
“stream.kafka.consumer.type”: “lowlevel”,
“stream.kafka.topic.name”: “demo2",
“stream.kafka.decoder.class.name”: “org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder”,
“stream.kafka.consumer.factory.class.name”: “org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory”,
“stream.kafka.zk.broker.url”: “z-1.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:2181,z-3.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:2181,z-2.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:2181",
“stream.kafka.broker.list”: “b-2.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:9092,b-1.pinot-quickstart-msk-d.9sahwk.c7.kafka.us-west-2.amazonaws.com:9092",
“realtime.segment.flush.threshold.time”: “10m”,
“realtime.segment.flush.threshold.size”: “10000",
“stream.kafka.consumer.prop.auto.offset.reset”: “smallest”
},
“enableDefaultStarTree”: false,
“enableDynamicStarTreeCreation”: false,
“aggregateMetrics”: false,
“nullHandlingEnabled”: false
},
“metadata”: {
“customConfigs”: {}
}
}
}
t
Ting Chen
12/15/2020, 6:19 PMawesome. glad to see you get it working!
Ting Chen
12/15/2020, 6:20 PMthanks for sharing the final config. one minor optimization: you can remove this optional table config:
Ting Chen
12/15/2020, 6:20 PM Copy code
"completionConfig": {
"completionMode": "DOWNLOAD"
}Ting Chen
12/15/2020, 6:21 PMI will update our Apache doc on this too.
a
Amit Chopra
12/15/2020, 6:22 PMGot it. Thanks. Will change and try.
BTW, remove the completion config. But keep the “peerSegmentDownloadScheme”: “http”?
Amit Chopra
12/15/2020, 6:36 PM@Ting Chen trying to understand the effect of this config better. As per , replica server (non commit server) will download the segment instead of catching up. And thus your suggestion to remove it, so that download from S3 can be reduced. Am i understanding it correctly?
https://cwiki.apache.org/confluence/display/PINOT/By-passing+deep-store+requirement+f[…]145724933/Screen%20Shot%202020-02-05%20at%204.45.39%20PM.png▾
t
Ting Chen
12/15/2020, 6:36 PMyes.
a
Amit Chopra
12/15/2020, 6:37 PMok, cool. Thanks 🙂
j
Jatin Yadav
11/15/2023, 7:24 AMHi @Ting Chen I am also facing the same issue , so I followed the above conversation, created table, in s3 tmp files of controller data is created, and segment status is consuming, , though “realtime.segment.flush.threshold.time” :"03m" , please can you help me what is the issue , thanks!
t
Ting Chen
11/20/2023, 9:06 PMHmm.. this is a thread closed 3 years ago. What problem are you facing exactly?