Hello everyone I am trying to setup a deep store on GCS and Apache Pinot #troubleshooting

Hello everyone, I am trying to setup a deep store ...

Abdelhakim Bendjabeur

08/29/2022, 9:41 AM

Hello everyone, I am trying to setup a deep store on GCS and experiment on backup and recovery on kubernetes. I followed this doc and Pinot does not seem to upload segments to the bucket. the controller is running with the following config

Copy code

JAVA_OPTS:-XX:ActiveProcessorCount=2 -Xms256M -Xmx1G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xlog:gc*:file=/opt/pinot/gc-pinot-controller.log -Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-gcs -Dlog4j2.configurationFile=/opt/pinot/conf/log4j2.xml -Dplugins.dir=/opt/pinot/plugins

StartController -configFileName /var/pinot/controller/config/pinot-controller.conf

$ cat /var/pinot/controller/config/pinot-controller.conf
controller.helix.cluster.name=pinot-quickstart
controller.port=9000
controller.data.dir=/var/pinot/controller/data
controller.zk.str=pinot-zookeeper:2181
pinot.set.instance.id.to.hostname=true
controller.task.scheduler.enabled=true
controller.data.dir=<gs://pinot-quickstart-deep-storage/data>
controller.local.temp.dir=/temp
controller.enable.split.commit=true
<http://pinot.controller.storage.factory.class.gs|pinot.controller.storage.factory.class.gs>=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.controller.storage.factory.gs.projectId=some-project-id
pinot.controller.storage.factory.gs.gcpKey=/var/pinot/controller/config/gcp-key.json
pinot.controller.segment.fetcher.protocols=file,http,gs
pinot.controller.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

The plugin is there

Copy code

$ ls /opt/pinot/plugins/pinot-file-system/
pinot-adls  pinot-gcs  pinot-hdfs  pinot-s3

When I check the propertystore.segments in zookeeper, I see that the download url is not a gcs url

Copy code

"segment.download.url": "<http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000/segments/tickets_channels/tickets_channels__2__0__20220826T0933Z>",

Did anyone succeed at configuring this? am I missing something? Thanks a lot for your help 🙏

Abdelhakim Bendjabeur

08/29/2022, 9:46 AM

I appended the config on the helm chart. the config

controller.data.dir

should be overwritten, right?

Luis Fernandez

08/29/2022, 1:48 PM

do you get any exception?

Luis Fernandez

08/29/2022, 1:48 PM

it seems like you had configure it properly based on this https://dev.startree.ai/docs/pinot/recipes/google-cloud-storage-deep-store

Luis Fernandez

08/29/2022, 1:51 PM

in this doc they also change some configs in the server https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot

Abdelhakim Bendjabeur

08/29/2022, 2:10 PM

Thank you @Luis Fernandez for taking the time. • No exception in the logs • I also updated the server per the s3 example

Copy code

root@pinot-server-0:/var/pinot/server/config# cat pinot-server.conf 
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
pinot.set.instance.id.to.hostname=true
pinot.server.instance.realtime.alloc.offheap=true
pinot.server.instance.currentDataTableVersion=2
pinot.server.instance.enable.split.commit=true
<http://pinot.server.storage.factory.class.gs|pinot.server.storage.factory.class.gs>=org.apache.pinot.plugin.filesystem.GcsPinotFS
pinot.server.storage.factory.gs.projectId=gorgias-pipeliine-staging
pinot.server.storage.factory.gs.gcpKey=/var/pinot/server/config/gcp-key.json
pinot.server.segment.fetcher.protocols=file,http,gs

Luis Fernandez

08/29/2022, 2:12 PM

and your realtime files are getting committed yes ?

Abdelhakim Bendjabeur

08/29/2022, 2:19 PM

I am not I understand the question, This is my very first hands-on experience with Pinot. But here is an answer I recreated the table to reproduce the problem but it's "stuck". it didn't persist segments on the FS

Copy code

{
  "id": "realtime_table_name__5__0__20220829T0914Z",
  "simpleFields": {
    "segment.creation.time": "1661764482430",
    "segment.flush.threshold.size": "3000000",
    "segment.realtime.numReplicas": "1",
    "segment.realtime.startOffset": "2124",
    "segment.realtime.status": "IN_PROGRESS"
  },
  "mapFields": {},
  "listFields": {}
}

Luis Fernandez

08/29/2022, 2:20 PM

can u show the config of the realtime?

Luis Fernandez

08/29/2022, 2:21 PM

if your segment is still in_progress it makes sense you don’t see the data in the deep store

👌 1

Luis Fernandez

08/29/2022, 2:25 PM

https://docs.pinot.apache.org/basics/components/deep-store#how-do-segments-get-into-the-deep-store

Abdelhakim Bendjabeur

08/29/2022, 2:29 PM

Copy code

{
  "tableName": "table_name",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "ts_ms",
    "timeType": "MILLISECONDS",
    "segmentPushType": "APPEND",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
    "schemaName": "table_name",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "tags-use-over-time",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.prop.auto.offset.reset": "largest",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "xxx.confluent.cloud:9092",
      "sasl.mechanism": "PLAIN",
      "security.protocol": "SASL_SSL",
      "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"xxx\" password=\"xxx\";",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.time": "24h",
      "realtime.segment.flush.autotune.initialRows": "3000000",
      "realtime.segment.flush.threshold.segment.size": "500M"
    }
  },
  "metadata": {
    "customConfigs": {}
  },
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  },
  "upsertConfig": {
    "mode": "FULL"
  }
}

Luis Fernandez

08/29/2022, 2:30 PM

ok so your realtime segment has to be committed for it to make it to deep store

Luis Fernandez

08/29/2022, 2:30 PM

does that make sense?

Luis Fernandez

08/29/2022, 2:31 PM

like what it says here https://docs.pinot.apache.org/basics/components/deep-store#how-do-segments-get-into-the-deep-store

Luis Fernandez

08/29/2022, 2:31 PM

if you are just testing then just make this really small

"realtime.segment.flush.threshold.time": "24h",

Luis Fernandez

08/29/2022, 2:31 PM

if u just wanna see the thing working

Abdelhakim Bendjabeur

08/30/2022, 1:34 PM

Thank you Luis for the tip I managed to have it commit the segments 👌 But it still commits to the server

Copy code

"segment.download.url": "<http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000/segments/tickets_tags/tickets_tags__1__2__20220830T1331Z>",

I'll keep looking, I may have missed a config

Luis Fernandez

08/30/2022, 4:12 PM

feel free to share configs

Abdelhakim Bendjabeur

08/31/2022, 8:06 AM

I just managed to managed to make it work 😄 I had a typo in the server's config (google project id) and I deleted the duplicate

controller.data.dir

Thanks again Luis for your help 🙏

🍷 1

Luis Fernandez

08/31/2022, 1:37 PM

glad it worked!

🙏 1

Open in Slack

Previous Next