When I try to run bathc ingestion I get this `doc...
# troubleshooting
j
When I try to run bathc ingestion I get this
docker exec pinot-controller ./bin/pinot-admin.sh \
>   LaunchDataIngestionJob \
>   -jobSpecFile /config/orders/orders_job_spec.json
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/opt/pinot/lib/pinot-all-1.1.0-jar-with-dependencies.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2025/05/22 18:28:38.417 ERROR [LaunchDataIngestionJobCommand] [main] Got exception to generate IngestionJobSpec for data ingestion job -
org.yaml.snakeyaml.constructor.ConstructorException: Cannot create property=recordReaderSpec for JavaBean=org.apache.pinot.spi.ingestion.batch.spec.SegmentGenerationJobSpec@3f9270ed
in 'string', line 1, column 1:
{
^
Cannot create property=inputFormat for JavaBean=org.apache.pinot.spi.ingestion.batch.spec.RecordReaderSpec@40e60ece
in 'string', line 16, column 25:
"recordReaderSpec": {
^
Unable to find property 'inputFormat' on class: org.apache.pinot.spi.ingestion.batch.spec.RecordReaderSpec
in 'string', line 17, column 22:
"inputFormat": "JSON",
^
in 'string', line 16, column 25:
"recordReaderSpec": {
^
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:283) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:169) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:320) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObjectNoCheck(BaseConstructor.java:264) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:247) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:201) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:185) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:493) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:473) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.getSegmentGenerationJobSpec(IngestionJobLauncher.java:100) ~[pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:112) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202) [pinot-all-1.1.0-jar-with-dependencies.jar:1.1.0-c2606742bbc4b15cff857eb0ffe7ec878ff181bb]
Caused by: org.yaml.snakeyaml.constructor.ConstructorException: Cannot create property=inputFormat for JavaBean=org.apache.pinot.spi.ingestion.batch.spec.RecordReaderSpec@40e60ece
in 'string', line 16, column 25:
"recordReaderSpec": {
^
Unable to find property 'inputFormat' on class: org.apache.pinot.spi.ingestion.batch.spec.RecordReaderSpec
in 'string', line 17, column 22:
"inputFormat": "JSON",
^
m
Seems like spec issue?
j
I managed to fix it somehow. I have another question can i create automatically somehow from same scheme 2 same tables one realtima and one offline
m
You mean you want to create a hybrid table where RT is ingesting data from stream and offline is from batch? That is always possible.
x
it’s expecting yaml, but I think you are passing json file