https://pinot.apache.org/ logo
Join Slack
Powered by
# general
  • d
    Is there an example of getting a batch json reader working? I tried and couldn't get it to work. I attached a tar of a directory that contains a batch-job-spec, schema, table and sample data.
    eventjson.tar.gz
  • d
    When I run "./pinot-admin.sh LaunchDataIngestionJob -jobSpecFile /tmp/event/batch-job-spec.yml", it stops outputting after "Initializing PinotFS"... and the *.log files don't contain anything.
  • d
    If I change it to csv, it works fine.
  • x
    I will take a look
  • x
    Copy code
    executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '/tmp/event/rawdata/'
outputDirURI: '/tmp/event/segments/'
overwriteOutput: true
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
  dataFormat: 'json'
  className: 'org.apache.pinot.plugin.inputformat.json.JSONRecordReader'
tableSpec:
  tableName: 'events'
  schemaURI: '<http://localhost:9000/tables/events/schema>'
  tableConfigURI: '<http://localhost:9000/tables/events>'
pinotClusterSpecs:
  - controllerURI: '<http://localhost:9000>'
  • x
    @User ^^
  • x
    the class name is 
    org.apache.pinot.plugin.inputformat.json.JSONRecordReader
    not 
    org.apache.pinot.plugin.inputformat.json.JsonRecordReader
  • x
    also for json, there is no classconfig, you can delete that
  • d
    Ah, thanks!
  • d
    Is it possible to make the pinot-admin.sh script output errors?
  • x
    yes
  • x
    you can update 
    conf/pinot-admin-log4j2.xml
    file with
  • x
    Copy code
    <?xml version="1.0" encoding="UTF-8"?>
<!--

    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

      <http://www.apache.org/licenses/LICENSE-2.0>

    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

-->
<Configuration>
  <Appenders>
    <Console name="console" target="SYSTEM_OUT" />
    <RandomAccessFile name="controllerLog" fileName="pinotController.log" immediateFlush="false">
      <PatternLayout>
        <Pattern>%d{yyyy/MM/dd HH:mm:ss.SSS} %p [%c{1}] [%t] %m%n</Pattern>
      </PatternLayout>
    </RandomAccessFile>
    <RandomAccessFile name="brokerLog" fileName="pinotBroker.log" immediateFlush="false">
      <PatternLayout>
        <Pattern>%d{yyyy/MM/dd HH:mm:ss.SSS} %p [%c{1}] [%t] %m%n</Pattern>
      </PatternLayout>
    </RandomAccessFile>
    <RandomAccessFile name="serverLog" fileName="pinotServer.log" immediateFlush="false">
      <PatternLayout>
        <Pattern>%d{yyyy/MM/dd HH:mm:ss.SSS} %p [%c{1}] [%t] %m%n</Pattern>
      </PatternLayout>
    </RandomAccessFile>
  </Appenders>
  <Loggers>
    <Root level="info" additivity="false">
      <AppenderRef ref="console"/>
    </Root>
    <Logger name="org.apache.pinot.controller.ControllerStarter" level="info" additivity="false">
      <AppenderRef ref="console"/>
    </Logger>
    <Logger name="org.apache.pinot.tools.admin" level="info" additivity="false">
      <AppenderRef ref="console"/>
    </Logger>
    <AsyncLogger name="org.reflections" level="error" additivity="false"/>
    <AsyncLogger name="org.apache.pinot.spi.plugin" level="error" additivity="false">
      <AppenderRef ref="console"/>
    </AsyncLogger>
  </Loggers>
</Configuration>
  • d
    Will do. I also needed to remove recordReaderSpec.configClassName.
  • d
    Sweet! That worked. Thanks!
  • d
    How do I create a Pinot Issue? I'm going to submit a small fix for the JSONRecordReader.
  • d
  • k
  • k
    updated the docs
  • d
    When I try running 
    git push origin jsonnpe
    , I get an a permission denied error.
  • k
    you need to push to your own repo and then submit a PR
  • d
    I'm trying to get my own realtime stream working with Pinot. I copied (with some small changes) the table config from the airlineStats and I'm hitting an error. The error is vague. How can I get a more detailed error?
  • d
    Copy code
    <http://java.io|java.io>.IOException: Server returned HTTP response code: 500 for URL: <http://192.168.0.29:9000/tables>
	at <http://sun.net|sun.net>.<http://www.protocol.http.HttpURLConnection.getInputStream0|www.protocol.http.HttpURLConnection.getInputStream0>(HttpURLConnection.java:1919) ~[?:?]
	at <http://sun.net|sun.net>.<http://www.protocol.http.HttpURLConnection.getInputStream|www.protocol.http.HttpURLConnection.getInputStream>(HttpURLConnection.java:1515) ~[?:?]
	at org.apache.pinot.tools.admin.command.AbstractBaseAdminCommand.sendRequest(AbstractBaseAdminCommand.java:89) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
	at org.apache.pinot.tools.admin.command.AbstractBaseAdminCommand.sendPostRequest(AbstractBaseAdminCommand.java:68) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
	at org.apache.pinot.tools.admin.command.AddTableCommand.sendTableCreationRequest(AddTableCommand.java:138) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
	at org.apache.pinot.tools.admin.command.AddTableCommand.execute(AddTableCommand.java:163) ~[pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:152) [pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:164) [pinot-all-0.3.0-SNAPSHOT-jar-with-dependencies.jar:0.3.0-SNAPSHOT-565374bbabd1b359f8d4f8d85d62ffd0f3d27458]
  • d
    I changed the following fields.
  • d
    Copy code
    "tableName": "events",
  "segmentsConfig": {
    "timeColumnName": "timestamp",
    "timeType": "MILLISECONDS",
  },
  "tableIndexConfig": {
    "streamConfigs": {
      "stream.kafka.topic.name": "events-realtime"
    }
  }
  • s
    Are you doing a cut-paste of the table cnfig on the ui, or did you issue a curl command?
  • s
    you can look at the logs on the controller side, perhaps it was not able to parse the table co dig
  • s
    config
  • d
    I used the 
    ./bin/pinot-admin.sh AddTable
    cli
  • d
    Ah, thanks!
1...117118119...160Latest