Hey folks,
I am trying to ingest CSVs from S3 wit...
# troubleshootinga
André Siefken
04/06/2022, 8:58 AMHey folks,
I am trying to ingest CSVs from S3 with hourly
SegmentGenerationAndPushTaskCan you help me find a means to trace and debug the issue? AFAIK it is not easy to monitor Minion tasks, but do you know of any hints towards where it may fail - at the configs level and/or parsing the files (although each individually seem to be valid)?"INFO": "java.lang.RuntimeException: Failed to execute SegmentGenerationAndPushTask"
André Siefken
04/06/2022, 9:00 AMLet me know if I can help you helping me, e.g. providing more info.
n
Neha Pawar
04/06/2022, 2:41 PM
Any exceptions in the minion logs before this message?
Most common failure is due to minions config not being setup for S3 access (similar to controller and server config)
Neha Pawar
04/06/2022, 5:14 PM
https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3#minion-config this is the minion config i’m talking about
a
André Siefken
04/06/2022, 5:33 PMThanks @User for trying to help, much appreciated!!! With quite some tedious bughunting I was able to find the issue - apparently Groovy was set to disable, so some transformations failed.
n
Neha Pawar
04/06/2022, 6:05 PM
i see, yes groovy has been default disabled starting latest release. Was no exception logged in the minion/controller logs?
a
André Siefken
04/06/2022, 7:16 PMUnfortunately not.
André Siefken
04/06/2022, 7:19 PMBut good to know, thanks! We are on a managed cluster, so access to logs and configs are a little non-trivial.
2 Views