Hey team I created a table with this in it to attempt to use the

minion

component to ingest data. When doing a POST at tasks/schedule, it looks like the minions are doing something (talks about using AVRO in logs) but they’ll either just hang, or error out. Any insights? I also made these changes: controller.task.scheduler.enabled=true minion config:

pinot.set.instance.id.to.hostname=true
      <http://pinot.minion.storage.factory.class.gs|pinot.minion.storage.factory.class.gs>=org.apache.pinot.plugin.filesystem.GcsPinotFS
      pinot.minion.storage.factory.gs.projectId=REDACTED
      pinot.minion.storage.factory.gs.gcpKey=REDACTED
      pinot.minion.segment.fetcher.protocols=file,http,gs
      pinot.minion.segment.fetcher.gs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
      plugins.include=pinot-gcs

Added auth key to controller, server, and minion (auth worked before ssh’ing into server and running a job)

You can monitor CPU activity on the minion worker. Also,

pinotMinion.log

has more verbose logs. How big are the files you are ingesting?

The CPU goes up on the minions when I launch the jobs but no output after a few hours. Files range but the max is right under a gig. Each minion pod has 1 cpu and 5g, and java mem settings are -Xms1G -Xmx4G

If you are running a pod, get in the minion pod and look at the logs of

pinotMinion.log

in the home dir

To store the segments in deep storage?

The controller will do that on its own

ahh ok, was porting over what I had from the job

you config is also missing

"includeFileNamePattern": "glob:**/*.gz",

I’m grabbing all files in that dir does that matter?

Try “includeFileNamePattern”: “glob:*.gz” then

~150

Ok, I don’t see anything unreasonable with what you described.

let me add that, purge the log, and then kick it off and tail

Remote the outputdiruri too

I have and they have been writing there fine. all are using same auth

Ok, just try without outputdiruri. When minion uploads a segment to pinot, it will endup in deepstore thanks to the controller

There were some from a prior job but none had been added/modified. I just delete a couple segments to see if it’ll try

Can you go in the zookeeper explorer and look for the status of the sub tasks status here :

correct I changed that last night

No errors on controller and server?

what about

pinotController.log

?

This is the only error, and corresponds to when I’m doing things in the controller UI’s zookeeper page:

2021/02/18 18:15:09.643 ERROR [ZkBaseDataAccessor] [grizzly-http-server-1] paths is null or empty

I’m probably way off here, but I noticed that the minion segment configurations aren’t prefixed with

pinot.minion

. Minion: https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java#L352-L354 Controller: https://github.com/apache/incubator-pinot/blob/master/pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java#L323-L324 Ended up with a configuration like this that works with a `RealtimeToOfflineSegmentsTask`:

pinot.minion.port=9514
    storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
    storage.factory.s3.region=my-region
    segment.fetcher.protocols=file,http,s3
    segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

But I had a quite clear error message something about that the minion doesn’t have a class factory for

s3

scheme.

That’s a great catch, I just assumed that’s what it was. Let me change that and give it a go!