https://pinot.apache.org/ logo
Join Slack
Powered by
HI Team, We are doing pinot poc for offline ingest...
# troubleshooting
a
HI Team, We are doing pinot poc for offline ingestions currently. Currently facing an issue , while ingesting segment from s3 to pinot.
Copy code
2021/11/03 08:29:22.109 INFO [SegmentFetcherFactory] [HelixTaskExecutor-message_handle_thread] Segment fetcher is not configured for protocol: s3, using default
2021/11/03 08:29:22.109 WARN [PinotFSSegmentFetcher] [HelixTaskExecutor-message_handle_thread] Caught exception while fetching segment from: <s3://pinot-db/pinot-ingestion/mytable/mytable_OFFLINE_2021091800_2021091800_0.tar.gz> to: /tmp/data/pinotSegments/mytable_OFFLINE/tmp-mytable_OFFLINE_2021091800_2021091800_0-90b8d75e-b2e8-4e4f-b115-36e5528c37cf/mytable_OFFLINE_2021091800_2021091800_0.enc
java.lang.IllegalStateException: PinotFS for scheme: s3 has not been initialized
        at shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
        at org.apache.pinot.spi.filesystem.PinotFSFactory.create(PinotFSFactory.java:78) ~[pinot-all-0.8.0-jar-with-dependencies.jar:0.8.0-c4ceff06d21fc1c1b88469a8dbae742a4b609808]
Following are our conf: Server conf: pinot.server.instance.enable.split.commit=true pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.server.storage.factory.s3.region=us-east-1 pinot.server.segment.fetcher.protocols=s3 pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher Controller conf: controller.data.dir=s3://pinot-db/ controller.local.temp.dir=/tmp/pinot/ controller.enable.split.commit=true pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3.region=us-east-1 pinot.controller.segment.fetcher.protocols=file,http,s3 pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
j
@Anish Nair you could use this link and check whether you configured Injection job correctly: https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot#set-up-ingestion-jobs If it’s still occurring can you please share the 
ingestionJobSpec.yaml
k
Not sure if it matters but I see following missing on the controller conf. (I am running docker containers).
Copy code
pinot.role=controller
controller.helix.cluster.name=PinotCluster
controller.zk.str=pinot-zookeeper:2181
controller.host=
controller.port=9000
a
Hey @jagadesh, sharing jobspec
Copy code
executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
    segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'

jobType: SegmentCreationAndMetadataPush

inputDirURI: '<s3://pinot-db/my_table/stats_date_hour/2021091800/>'
outputDirURI: '<s3://pinot-db/pinot-ingestion/my_table/>'

overwriteOutput: true

pinotFSSpecs:
    - scheme: s3
      className: org.apache.pinot.plugin.filesystem.S3PinotFS
      configs:
        region: 'us-east-1'

recordReaderSpec:
    dataFormat: 'parquet'
    className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader'

tableSpec:
    tableName: 'my_table'
pinotClusterSpecs:
    - controllerURI: '<http://crontroller:9000>'

pushJobSpec:
  pushParallelism: 2
  pushAttempts: 1
k
You're able to hit S3 from that box? Using env to pass in access key and secret?
a
yes we are able to access s3 from the server,
k
I'm comparing what you've shared with working versions of my jobspec and conf files to read CSV files from S3. I noticed the jobspec is missing schemaURI and tableConfigURI under tableSpec. And the server conf. is missing
Copy code
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index
pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars
Not sure if these things are directly causing the errors but you can update and give it a shot.
a
Hey @Kamal Chavda,we have the above configs in server and controller. The ingestion is completing with success, but status of the ingested segment is showing as BAD. and upon checking the logs of server, we found this error. Will try to provide additional configs as you mentioned.
👍 1
Hey @Kamal Chavda, till the same error. can i know how you are setting the aws key and secret in server conf?
j
@Anish Nair You could set in controller config as
Copy code
pinot.controller.storage.factory.s3.accessKey=****************LFVX
pinot.controller.storage.factory.s3.secretKey=****************gfhz
k
I followed the tutorial and found it to be very helpful. I also passed the aws key and secret when starting the containers (controller, broker, server, ingestion job):
Copy code
docker create -ti \
    --name pinot-server \
    --network=pinot-demo \
    --env AWS_ACCESS_KEY_ID= \
    --env AWS_SECRET_ACCESS_KEY= \
    -e JAVA_OPTS="-Dplugins.dir=/opt/pinot/plugins -Xms32G -Xmx32G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log" \
    --mount type=bind,source=/opt/pinot,target=/tmp \
    apachepinot/pinot:0.8.0 StartServer \
    -zkAddress pinot-zookeeper:2181
a
Okay, will try this. Currently we did set creds inside server conf, but not in controller conf.
@Kamal Chavda, which segment push type did you used ?
Worked with SegmentCreationAndTarPush only. SegmentCreationAndMetadataPush and SegmentCreationAndUriPush is giving error.
k
Copy code
# executionFrameworkSpec: Defines ingestion jobs to be running.
executionFrameworkSpec:

  # name: execution framework name
  name: 'standalone'

  # segmentGenerationJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentGenerationJobRunner interface.
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'

  # segmentTarPushJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentTarPushJobRunner interface.
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'

  # segmentUriPushJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.SegmentUriPushJobRunner interface.
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'

  # segmentMetadataPushJobRunnerClassName: class name implements org.apache.pinot.spi.batch.ingestion.runner.IngestionJobRunner interface.
  segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner'

# jobType: Pinot ingestion job type.
# Supported job types are:
#   'SegmentCreation'
#   'SegmentTarPush'
#   'SegmentUriPush'
#   'SegmentCreationAndTarPush'
#   'SegmentCreationAndUriPush'
jobType: SegmentCreationAndMetadataPush
Any luck?
a
MetadataPush is not working. Only TarPush is working. All configs are in place as per the document.
Open in Slack
PreviousNext
Powered by