https://pinot.apache.org/ logo
Join Slack
Powered by
guys do i need to add any configuration related to...
# troubleshooting
a
guys do i need to add any configuration related to s3 for realtime table configs i make it before work but lost my change i did most of config related to cotroller and server but not sure if i still need to add something let my tabe use s3 as a deep store segment realtime.segment.download.url: s3 path
d
The following documentation has all the required configurations to use S3 as a deep storage. https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3
a
I saw it but my question is there is any extra config you need to provide in the case of real-time table "I do not see it in any documentation"
Thank you so much, as I did this config carefully and now it is not working my realtime table. Also i knew that this conig is being appended to pinot-controller.conf but i have in the /var/pinot/,... bla bla i see other file in opt/pinot with same name that not have the s3 config appended correclt .. "I am not sure which config the pod use it "
d
Can you share your table config here?
a
next_cdc_realtime_intentions_table_config.json: |- { "tableName": "next_intentions_trial4", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts_ms", "timeType": "MILLISECONDS", "schemaName": "next_intentions_schema2", "replicasPerPartition": "1", "allowNullTimeValue": true, "completionConfig": { "completionMode": "DOWNLOAD" }, "segmentPushType": "APPEND" }, "tenants":{}, "tableIndexConfig": { "loadMode": "MMAP", "nullHandlingEnabled": true, "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "simple", "stream.kafka.topic.name": "my-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.hlc.zk.connect.string": "pinot-zookeeper.pinot-quickstart.svc:2181", "stream.kafka.zk.broker.url": "pinot-zookeeper.pinot-quickstart.svc:2181", "stream.kafka.broker.list": "my-cluster-kafka-bootstrap.kafka.svc:9092", "realtime.segment.flush.threshold.time":"0h1m", "realtime.segment.flush.threshold.rows":"2", "outputDirURI": "s3://house-development-new/pinot-data/pinot-quickstart/controller-data/", "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "us-east-1", "realtime.segment.download.url": "s3://house-development-new/pinot-data/pinot-quickstart/controller-data/", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "routing":{ "instanceSelectorType":"strictReplicaGroup" }, "metadata": { "customConfigs": {} }, "upsertConfig":{ "mode":"FULL", "hashFunction":"NONE" }, "ingestionConfig":{ "transformConfigs":[ { "columnName":"ts_ms", "transformFunction":"JSONPATHLONG(json_format(payload),'$.ts_ms')" } , { "columnName":"id", "transformFunction":"JSONPATHLONG(json_format(payload),'$.after.id')" }, { "columnName":"created", "transformFunction":"JSONPATHLONG(json_format(payload),'$.after.created')" }, { "columnName":"modified", "transformFunction":"JSONPATHLONG(json_format(payload),'$.after.modified')" }, { "columnName":"uuid", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.after.uuid')" }, { "columnName":"intention_ref", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.after.intention_ref')" }, { "columnName":"confirmed", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.after.confirmed')" }, { "columnName":"live", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.after.live')" }, { "columnName":"user_id", "transformFunction":"JSONPATHLONG(json_format(payload),'$.after.user_id')" }, { "columnName":"stream_operation", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.op')" } ] } }
next_cdc_realtime_intentions_table_config.json: |-
{ "tableName": "next_intentions_trial4", "tableType": "REALTIME", "segmentsConfig": { "timeColumnName": "ts_ms", "timeType": "MILLISECONDS", "schemaName": "next_intentions_schema2", "replicasPerPartition": "1", "allowNullTimeValue": true, "completionConfig": { "completionMode": "DOWNLOAD" }, "segmentPushType": "APPEND" }, "tenants":{}, "tableIndexConfig": { "loadMode": "MMAP", "nullHandlingEnabled": true, "streamConfigs": { "streamType": "kafka", "stream.kafka.consumer.type": "simple", "stream.kafka.topic.name": "my-topic", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder", "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory", "stream.kafka.hlc.zk.connect.string": "pinot-zookeeper.pinot-quickstart.svc:2181", "stream.kafka.zk.broker.url": "pinot-zookeeper.pinot-quickstart.svc:2181", "stream.kafka.broker.list": "my-cluster-kafka-bootstrap.kafka.svc:9092", "realtime.segment.flush.threshold.time":"0h1m", "realtime.segment.flush.threshold.rows":"2", "outputDirURI": "s3://house-development-new/pinot-data/pinot-quickstart/controller-data/", "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS", "input.fs.prop.region": "us-east-1", "realtime.segment.download.url": "s3://house-development-new/pinot-data/pinot-quickstart/controller-data/", "stream.kafka.consumer.prop.auto.offset.reset": "smallest" } }, "routing":{ "instanceSelectorType":"strictReplicaGroup" }, "metadata": { "customConfigs": {} }, "upsertConfig":{ "mode":"FULL", "hashFunction":"NONE" }, "ingestionConfig":{ "transformConfigs":[ { "columnName":"ts_ms", "transformFunction":"JSONPATHLONG(json_format(payload),'$.ts_ms')" } , { "columnName":"id", "transformFunction":"JSONPATHLONG(json_format(payload),'$.after.id')" }, { "columnName":"created", "transformFunction":"JSONPATHLONG(json_format(payload),'$.after.created')" }, { "columnName":"modified", "transformFunction":"JSONPATHLONG(json_format(payload),'$.after.modified')" }, { "columnName":"uuid", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.after.uuid')" }, { "columnName":"intention_ref", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.after.intention_ref')" }, { "columnName":"confirmed", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.after.confirmed')" }, { "columnName":"live", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.after.live')" }, { "columnName":"user_id", "transformFunction":"JSONPATHLONG(json_format(payload),'$.after.user_id')" }, { "columnName":"stream_operation", "transformFunction":"JSONPATHSTRING(json_format(payload),'$.op')" } ] } }
this is a test config not recommended as having too many segments or too little segment is a tradeoff ..what i ask and focus on. Is there is any table config needed to use S3 as a deepstore for segments . things are not working smoothly using the the doc ... I use k8s latest pinot but having the config on values.yaml folder extra config key for controller and server related to S3 "Not giving me the expected result"
m
so is the problem that you aren't seeing anything written to the S3 bucket/
✅ 1
a
yes also from the controller logs too
i do not find my bucker name or anything related to even security or so
also /opt/pinot/conf/pinot-controller.conf not have the configuration it only presented in /var/pinot/controller/config/pinot-controller.conf which make sense according the chart but i do not know which file is used as the conf to the controller
Thanks mark, i also want to mentioned that i made this work before but i lost my changes .. I was wondering if you have clear steps beside what is already provided
m
so I've been reading through the code to make sure that the config suggested in that article are correct and it looks like they are. From what I can tell, with this config the server will send the segment file to the controller, which will then upload it to S3. So it seems like the problem we have right now is understanding in which file these config need to be configured.
let me check where they are read from and reply back
one way to pass the values through is via 
StartController -config <file>
but let me see if there's another way that doesn't require a file
a
# Extra configs will be appended to pinot-controller.conf file extra: configs: |- pinot.set.instance.id.to.hostname=true controller.task.scheduler.enabled=true controller.data.dir=s3://path controller.local.temp.dir=/tmp/pinot-tmp-data/ pinot.controller.storage.factory.s3.disableAcl=false pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3.region=region pinot.controller.segment.fetcher.protocols=file,http,s3 pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
m
this is for the helm chart
✅ 1
?
a
yes i did so as following on k8s values.yaml
the issue in the helm chart args: [ "StartController", "-configFileName", "/var/pinot/controller/config/pinot-controller.conf" ] it do what you mention
m
yeh I don't think what I suggested will work very well for K8s
a
thats what i believe too as it's not working . Do you think i put the right table config needed for s3 ?
m
what you have looks like it should work (to me at least)
✅ 1
but obv it doesn't, so I guess it's not getting picked up for some reason
when you look at 
/var/pinot/controller/config/pinot-controller.conf"
you said it has the values you set right?
a
yes
m
and you said there's nothing under 
logs/pinot-all.log
on the controller?
a
no i meant the container logs kubectl logs -n=pinot-quickstart pinot-controller-0 -f
m
are you able to check the logs on one of the controller pods?
that should have more logging
a
Sure i willl check it now
i have this warning 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'realtime.segment.flush.threshold.rows' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'outputDirURI' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'stream.kafka.hlc.zk.connect.string' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'realtime.segment.download.url' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'streamType' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'input.fs.className' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'input.fs.prop.region' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'stream.kafka.consumer.type' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'stream.kafka.broker.list' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'realtime.segment.flush.threshold.time' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'stream.kafka.zk.broker.url' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'stream.kafka.consumer.factory.class.name' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'stream.kafka.consumer.prop.auto.offset.reset' was supplied but isn't a known config. 2021/12/08 141823.444 WARN [ConsumerConfig] [pool-8-thread-4] The configuration 'stream.kafka.topic.name' was supplied but isn't a known config.
I have this warning. is it related to depreication or version issue ... but how is consuming my data correct from my kafka topic ?
m
do you see this line anywhere? 
Initializing PinotFSFactory
a
nope mark 🙂
m
hmmm, ok
how about: 
Initializing SegmentFetcherFactory
a
no as well
m
hmmm, that's strange. In theory those messages should be there when the controller starts.
e.g. this is on a locally started controller:
Copy code
2021/12/03 14:38:17.684 INFO [BaseControllerStarter] [main] Initializing PinotFSFactory
2021/12/03 14:38:17.687 INFO [BaseControllerStarter] [main] Initializing ControllerFilePathProvider
2021/12/03 14:38:17.689 INFO [ControllerFilePathProvider] [main] Data directory: file:/tmp/data/PinotController
2021/12/03 14:38:17.693 INFO [ControllerFilePathProvider] [main] Local temporary directory: /tmp/data/PinotController/192.168.144.3_9000
although I haven't defined any factories so it doesn't register anything
do a grep for 
"\[BaseControllerStarter"
see what it says is happening when the controller starts
I'm not sure about the kafka config thing - those exact configs are used for a bunch of Pinot's QuickStart examples! 🤔
a
kafka is working fine .. i had all streams of data being injested successfully also create segment ever two row entries as test
root@pinot-controller-0:/opt/pinot/logs# cat pinot-all.log | grep "\[BaseControllerStarter" root@pinot-controller-0:/opt/pinot/logs#
return nothing. i mean based on your discussion i can look and see why i do not see such messages and update you .. how is that sound use your intuition as a starting point ?
and update uou
you *
m
I mean maybe they got truncated
they would only be there right at the beginning
when you started the controller
I was wondering whether the S3 provider is being registered
it should be based on the config you shared
let me check the log messages around copying segments. Let's see if we can find anything
a
Thank you ! sounds good too
m
do you find anything if you grep for 
LLRealtimeSegmentDataManager
?
you don't need to paste everything if you do
just wanted to see if that finds something
a
yes only one message .
m
what's it say?
a
starting with running frequency of 3000 secs
m
ok for sanity's sake, you don't see anything if you grep for 
HLRealtimeSegmentDataManager
a
no
actually i grep for my bucket name on the log file and i see messages related to s3 for firstime
[SegmentDeletionManager] [grizzly-http-server-0] Moved segment next_intentions_trial4__0__9__20211208T1017Z from file:/var/pinot/controller/data,s3://bucketname/pinot-data/pinot-quickstart/control ler-data/next_intentions_trial4/next_intentions_trial4__0__9__20211208T1017Z to file:/var/pinot/controller/data,s3://bucketname/pinot-data/pinot-quickstart/controller-data/Deleted_Segments/next_intentions_trial4/next_intentio ns_trial4__0__9__20211208T1017Z
m
that would be if you deleted a table at some pooint?
I guess that was when you had it wired up and working!
a
No when had it wired up was on diff table and i still have its data on s3 .. But yes while testing config with this new intention bla table i did deleted it multible times
m
oh I see
the file name looks wrong
it should be file:// or s3:///
it seems to have concatenated them
✅ 1
so it thinks our file type is local file somehow
it's missing the s3 scheme
one sec lemme see why
a
yes i recognized this as well but not know why ... Thank you *
m
does that mean it's trying to write stuff to 
file:/var/pinot/controller/data,<s3://bucketname/pinot-data/pinot-quickstart/control>
is that even a valid path. See if there's anything there?
Copy code
URI deletedSegmentDestURI = URIUtils.getUri(_dataDir, DELETED_SEGMENTS, rawTableName, URIUtils.encode(segmentId));
segmentId = 
next_intentions_trial4__0__9__20211208T1017Z
rawTableName = 
next_intentions_trial4
dataDir = 
file:/var/pinot/controller/data,<s3://bucketname/pinot-data/pinot-quickstart/controller-data>
a
well i see now two entry points controller.data.dir=/var/pinot/controller/data controller.data.dir=s3 one in the pinot-controller.conf
m
ah - delete the 
/var/pinot/controller/data
one
✅ 1
a
Yes now things are working properly .. Thank you so much @Mark Needham yes in values.yaml file the chart take both the exrtra config and another default value for the data dir
Thanks so much again sir
m
awesome! No worries, that was a good journey through the code for me 😄
❤️ 1
I guess the validation should throw an error if you define the same value twice. I think the reason it doesn't is that it delegates that to an apache library
✅ 1
a
It was my issue but actually the debug journey of you showed me alot of stuff that i googled and google to become better .. Thank so much for the support man !
❤️ 1
d
Thanks for the thread, guys! I made the same mistake, added the data dir to the extra configs, and they ended up duplicated in the final config file used by the Controller, just concatenating the data dirs. It's fixed now on my side too!
Open in Slack
PreviousNext
Powered by