Hi Team I have included this in the OFFLINE table config and Apache Pinot #troubleshooting

Hi Team, I have included this in the OFFLINE table...

Nagendra Gautham Gondi

10/10/2022, 9:38 PM

Hi Team, I have included this in the OFFLINE table config, and wanted to test how tasks work. Though the table and schema are getting created, I’m not seeing any tasks in Minion Task Manager.

Copy code

"ingestionConfig": {
  "batchIngestionConfig": {
    "segmentIngestionType": "APPEND",
    "segmentIngestionFrequency": "HOURLY",
    "batchConfigMaps": [
      {
        "input.fs.className": "org.apache.pinot.plugin.filesystem.S3PinotFS",
        "input.fs.prop.region": "us-east-1",
        "inputDirURI": "<s3://masked-bucket/dataset-sample/year=2022/month=10/day=10/>",
        "includeFileNamePattern": "glob:**/*.parquet",
        "excludeFileNamePattern": "glob:**/*.tmp",
        "inputFormat": "parquet"
      }
    ]
  }
},
"tasks": {
  "taskTypeConfigsMap": {
    "SegmentGenerationAndPushTask": {
      "schedule": "0 * * * * ?"
    }
  }
},

This is what is passed to controller through values.yaml, where the task scheduler is enabled. Can you please help me how are the tasks enabled?

Copy code

extra:
  configs: |-
    pinot.set.instance.id.to.hostname=true
    controller.task.scheduler.enabled=true

Mayank

10/10/2022, 9:40 PM

Any logs in the minion instances?

Nagendra Gautham Gondi

10/10/2022, 9:43 PM

Copy code

Updating instance: Minion_pinot-minion-stateless-647f9b6557-lgtvz_9514 with hostname: pinot-minion-stateless-647f9b6557-lgtvz
No Messages to process
76 END:INVOKE CallbackHandler 0, /pinot-quickstart/INSTANCES/Minion_pinot-minion-stateless-647f9b6557-lgtvz_9514/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@663f1ebc type: CALLBACK Took: 15ms
Updating instance: Minion_pinot-minion-stateless-647f9b6557-lgtvz_9514 with default tags: []
fallbackRoot: /pinot-quickstart/HELIX_PROPERTYSTORE doesn't exist, skip creating fallback property store
Starting minion admin application on: <http://0.0.0.0:9514>
Sep 28, 2022 4:45:51 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:9514]
Sep 28, 2022 4:45:51 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
Reflections took 4265 ms to scan 1 urls, producing 74333 keys and 147122 values 
Initializing health check callback
Pinot minion started
Pinot Minion instance [Minion_pinot-minion-stateless-647f9b6557-lgtvz_9514] is Started...
Started Pinot [MINION] instance [Minion_pinot-minion-stateless-647f9b6557-lgtvz_9514] at 12.728s since launch

No update since sept 28th.

Mayank

10/10/2022, 9:49 PM

Hmm, @Haitao Zhang can you take a look?

thankyou 1

Haitao Zhang

10/11/2022, 12:42 AM

(1) Did you see the task type listed under the minion task manager? (2) Did you see any task generation errors? (grep -E "PinotTask|Quartz" should list something)

Nagendra Gautham Gondi

10/11/2022, 12:44 AM

1. The task was not showing up in Minion Task Manager. 2. Where should I check? In server?

Haitao Zhang

10/11/2022, 12:47 AM

for 2, in controller (I thought you are using quick start and all logs are mixed together?)

Nagendra Gautham Gondi

10/11/2022, 12:50 AM

No logs in controller as well 😞

Nagendra Gautham Gondi

10/11/2022, 12:53 AM

I’m still using quickstart, however I modified few values to install within EKS

Haitao Zhang

10/11/2022, 12:54 AM

can you go to

/home/pinot/data/logs/pinotController/

folder to find pinot controller full logs?

Haitao Zhang

10/11/2022, 12:54 AM

there are many logs, so you cannot get all of them using kubectl logs

Nagendra Gautham Gondi

10/11/2022, 1:08 AM

Also, when I logged into the pod and manually checked conf/pinot-controller.conf. I couldn’t see this property.

Copy code

controller.task.scheduler.enabled=true

I don’t see this value getting reflected in any of the controller templates. Do you think that could be the reason?

Nagendra Gautham Gondi

10/11/2022, 1:19 AM

Never mind, the property is enabled in the config map. I tried looking at the logs within pinot-controller pod, but couldn’t find anything with the corresponding task name. However there are some logs related to other tables

Haitao Zhang

10/11/2022, 1:24 AM

Did you get anything by

grep -E "PinotTask|Quartz"

? We should see something like "Trying to schedule task type: <different task type>"

Nagendra Gautham Gondi

10/11/2022, 1:29 AM

Copy code

2022/10/11 01:23:05.597 INFO [PinotTaskManager] [ZkClient-EventThread-121-pinot-zookeeper:2181] Cleaning up task in scheduler for table silvermineBIServiceLogs_OFFLINE
2022/10/11 01:23:05.606 INFO [PinotTaskManager] [ZkClient-EventThread-121-pinot-zookeeper:2181] Checking task config changes in table configs
2022/10/11 01:23:30.308 INFO [PinotTaskManager] [ZkClient-EventThread-121-pinot-zookeeper:2181] Checking task config changes in table configs
2022/10/11 01:23:30.309 INFO [PinotTaskManager] [ZkClient-EventThread-121-pinot-zookeeper:2181] Trying to update task schedule for table: silvermineBIServiceLogs_OFFLINE
2022/10/11 01:23:30.311 INFO [PinotTaskManager] [ZkClient-EventThread-121-pinot-zookeeper:2181] taskConfig is null, trying to remove all the tasks for table silvermineBIServiceLogs_OFFLINE if any

Nagendra Gautham Gondi

10/11/2022, 1:29 AM

This is what I got in the log file within controller

Haitao Zhang

10/11/2022, 4:27 PM

I guess the SegmentGenerationAndPushTask is configured for table

silvermineBIServiceLogs

Nagendra Gautham Gondi

10/11/2022, 4:27 PM

Yes

Haitao Zhang

10/11/2022, 4:29 PM

taskConfig is null, trying to remove all the tasks for table silvermineBIServiceLogs_OFFLINE if any

means no tasks are scheduled. Just to double confirm: 1/ do we have files in the folder? 2/ do we have aws key provided?

Nagendra Gautham Gondi

10/11/2022, 4:31 PM

Files are available. I didn’t provide aws key, since the EKS Node instance role directly has access to the S3 Bucket.

Nagendra Gautham Gondi

10/11/2022, 4:31 PM

Is providing access key mandatory, is there a way to leverage Roles to gain access to that S3 bucket?

Haitao Zhang

10/11/2022, 4:35 PM

if the EKS node has access, then it should be fine

Haitao Zhang

10/11/2022, 4:35 PM

oh, your configuration uses the wrong keyword 😃

task

tasks

Nagendra Gautham Gondi

10/11/2022, 4:40 PM

That’s a bummer! Thanks for pointing that out. @Haitao Zhang Is there a way we can leverage roles to access S3? Like Kinesis Role based access

Haitao Zhang

10/11/2022, 4:53 PM

I am not an expert on that. @Navina any thoughts?

Navina

10/11/2022, 5:33 PM

catching up on this thread. hang on

Navina

10/11/2022, 5:37 PM

@Nagendra Gautham Gondi I think you should be able to leverage roles to access S3. is it a cross-account IAM role that you have setup ? if the policies are correctly attached to your instance, then you can simply remove the accesskey and secretKey from your table config and see if it works.

Nagendra Gautham Gondi

10/11/2022, 5:38 PM

Hi Navina, Yes! I would like to assume cross account role to access S3.

Navina

10/11/2022, 5:48 PM

I have tried S3 bucket access across different accounts using "IAM policies and resource-based bucket policies" (See https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/) . Not the cross-account IAM role access for S3 . I think that will require some changes in Pinot oss to use the

AssumeRole

credential provider . From what I know, that hasn't been implemented yet.

Nagendra Gautham Gondi

10/11/2022, 5:50 PM

Got it! Yes, I am able to access with Bucket based policies in a different account. I was wondering if even assume role has been implemented. Thanks Navina

Navina

10/11/2022, 5:51 PM

yeah. that has been pending. We always welcome contributions @Nagendra Gautham Gondi 🙏

thankyou 1

Open in Slack

Previous Next