Is there a way to force pinot-server to load plugins without restarting? If I understand correctly, I think right now pinot only loads plugins at startup?

Hi, I ran a batch ingestion using "pinot-admin.sh LaunchDataIngestionJob" which seems to have completed successfully. In Pinot UI, I see that the reported size of the table is 17895109 (was empty before) so it seems the data was ingested properly. *Yet when I go to query console and do a simple "select * from mytable limit 10", I get no results and it says numDocsScanned=0, totalDocs=0.* Has anyone encountered something similar? how can I debug this?

I have a pinot setup using helm charts. I have setup a realtime table. The server ran out of disk space and it was stuck at the last segment (not consuming any new data). I updated the disk size and restarted the pinot server, and in the web UI, the last segment status was shown as BAD. So, I deleted the segment from the UI, but it still hasn't start consuming new data. Is there a way around this?

Hello is there a way to backup and recover all segments? In situations like, server is not recovering and you are forced to recreate the table but don’t want the indexing to happen on previously indexed data but resume from the crash point.

Hi, we have a hybrid table that is not serving realtime data (unless _REALTIME is explicitly used) and noticed that the time boundary is given in ms but the time column is a datetime field with input in ms from another field (base column in kafka) and output granularity in seconds. Anyone else experience this? Is there a way to set the time boundary?

Hi everyone. I'm currently trying to import some batch data to my Pinot cluster and I'm running into some issues with doing this. I have the latest version of Pinot (0.7.0) in a docker container, and I set everything up manually. I followed the docker version of this guide here: https://docs.pinot.apache.org/basics/getting-started/advanced-pinot-setup. I am able to configure the

baseballStats

offline table with some modifications to the files. When I am uploading my own batch data, I get the following error:

400 (Bad Request) with reason: "Cannot add invalid schema: rows_10m. Reason: null"

I currently have a CSV that's formatted like this

# /DIRECTORIES/rawdata/rows_10m.csv
id, hash_one, text_one
0, (large integer), a
1, (large integer), b
...

A schema.json that has this

# /DIRECTORIES/rows_10m_schema.json
{
    "schemaName": "rows_10m",
    "dimensionFieldSpecs": [
        {
            "datatype": "STRING",
            "name": "text_one"
        }
    ],
    "metricFieldSpecs": [
        {
            "datatype": "INT",
            "name": "id"
        },
        {
            "datatype": "INT",
            "name": "hash_one"
        }
    ]
}

and a table config that has this

# /DIRECTORIES/rows_10m_offline_table_config.json
{
    "tableName": "rows_10m",
    "tableTypes": "OFFLINE",
    "segmentsConfig": {
        "segmentPushType": "APPEND",
        "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
        "schemaName": "rows_10m",
        "replication": "1"
    },
    "tenants": {},
    "tableIndexConfig": {
        "loadMode": "HEAP",
        "invertedIndexColumns": [
            "id",
            "hash_one"
        ]
    },
    "metadata": {
        "customConfigs": {
        }
    }
}

This is very similar to what I used when I manually added the default

baseballStats

. Am I missing anything in my schema.json file?

Hi team, is the

FixedSegmentNameGenerator

supported ? From the doc https://docs.pinot.apache.org/configuration-reference/job-specification#segment-name-generator-spec , only simple and normalizedDate name generator are supported.

Hi everyone, I'm currently trying to batch import some data into a Pinot Offline table and currently running into some issues. My current Pinot version is 0.7.0, currently in a docker container. I have successfully added an

offline_table_config.json

and a

schema.json

file to Pinot, however creating a segment doesn't appear to be working. A

SEGMENT-NAME.tar.gz

file isn't being created. My current docker-job-spec.yml looks like this:

# docker-job-spec.yml

executionFrameworkSpec:
  name: 'standalone'
  segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
  segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
  segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndTarPush
inputDirURI: '/tmp/pinot-manual-test/rawdata/100k'
includeFileNamePattern: 'glob:**/*.csv'
outputDirURI: '/tmp/pinot-manual-test/segments/100k'
overwriteOutput: true
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS
recordReaderSpec:
  dataFormat: 'csv'
  className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader'
  configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig'
tableSpec:
  tableName: 'rows_100k'
  schemaURI: '<http://pinot-controller-test:9000/tables/rows_100k/schema>'
  tableConfigURI: '<http://pinot-controller-test:9000/tables/rows_100k>'
pinotClusterSpecs:
  - controllerURI: '<http://pinot-controller-test:9000>'

Some of the error messages I'm getting are

Failed to generate Pinot segment for file - file:/tmp/pinot-manual-test/rawdata/100k/rows_100k.csv
Caught exception while gathering stats
java.lang.NumberFormatException: For input string: "5842432235322161941"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:1.8.0_282]
        at java.lang.Integer.parseInt(Integer.java:583) ~[?:1.8.0_282]

Any leads on this would be appreciated. Thanks

I see the following message in pinot broker log: 2021/02/06 231541.016 INFO [BaseInstanceSelector] [ClusterChangeHandlingThread] Failed to find servers hosting segment: spanEventView__0__219__20210206T0115Z for table: spanEventView_REALTIME (all ONLINE/CONSUMING instances: [] are disabled, but find enabled OFFLINE instance: Server_pinot-server-0.pinot-server.hypertrace.svc.cluster.local_8098 from OFFLINE instances: [Server_pinot-server-0.pinot-server.hypertrace.svc.cluster.local_8098], not counting the segment as unavailable)

Hi, It seems there's some compatibility issues between Pinot and Superset in regards to the time column: In Pinot, I have it defined like this

"dateTimeFieldSpecs": [
    {
      "name": "ts",
      "dataType": "STRING",
      "format": "1:SECONDS:SIMPLE_DATE_FORMAT:\"yyyy-MM-dd HH:mm:ss\"",
      "granularity": "1:MINUTES"
    }
  ]

In Superset, I must define the string format in the Python way for it to parse correctly:

%Y-%m-%d %H:%M:%S

When I try creating a chart, I get this error:

Apache Pinot Error
unsupported format character 'Y' (0x59) at index 58


This may be triggered by:
Issue 1002 - The database returned an unexpected error.

Because the query gets translated to this (note that if I remove the "DATETIMECONVERT" and simply use "ts" column it works fine):

SELECT DATETIMECONVERT(ts, '1:SECONDS:SIMPLE_DATE_FORMAT:%Y-%m-%d %H:%M:%S', '1:SECONDS:SIMPLE_DATE_FORMAT:%Y-%m-%d %H:%M:%S', '1:DAYS'),
       AVG(metric) AS "AVG_1"
FROM schema.table
WHERE ts >= '2021-02-01 00:00:00'
  AND ts < '2021-02-08 00:00:00'
GROUP BY DATETIMECONVERT(ts, '1:SECONDS:SIMPLE_DATE_FORMAT:%Y-%m-%d %H:%M:%S', '1:SECONDS:SIMPLE_DATE_FORMAT:%Y-%m-%d %H:%M:%S', '1:DAYS')
LIMIT 50000;

Has anyone encountered something similar? What is the solution?

Hello, is there an API to reset a table when it’s offset is out of range for consumption? If not, does disabling and enabling the table trigger a offset reset?

Hey, quick question i don't find anything on the docs, i have a realtime table with a consuming segment and i would like to stop the consumtion and save it into the deep storage without creating a new consuming segment ? My use-case is simply to be able to stop ingesting new events to do some tasks like updating the server or any maintenance. Thanks !

Hi! I’m following the Code Setup instructions to setup my environment and when I run this maven command, I get the following error.

[ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:1.1:npm (npm install) on project pinot-controller: Failed to run task: 'npm install' failed. (error code 1) -> [Help 1]

The pinot-controller package seems to be a java project, so why is it trying to run

npm install

? How do I get around this? Thanks!

<!here> Created UPSERT table with composit primary key having 8 columns. Many times - few of the column value can be null. But Together all 8 columns ensure the uniqueness. Do you see concern with this ? Will upsert work properly ?

Hello Team, I was trying to work with pinot realtime table on Kafka and S3 as deep storage. My pinot-server has this error in the logs : java.lang.ClassNotFoundException: org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory. This setup is done kubernetes via helm. Earlier I was getting a different error around S3PinotFS, to fix that I updated the server config to use pinot-s3 plugin. After that it seems like Kafka plugin is not available to the server. How do I solve this?

Hi, I am having trouble getting the reload segments to work (I am running latest master code) Seeing below logs on pinot-server. Looks like

SegmentMessageHandlerFactory

is not getting registered for some reason. When I restart the server I don’t see Logs from this function beyond this point. (https://sourcegraph.com/github.com/apache/incubator-pinot/-/blob/pinot-server/src/ma[…]org/apache/pinot/server/starter/helix/HelixServerStarter.java)

Subscribing changes listener to path: /PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES, type: CALLBACK, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4b9419ff
Subscribing child change listener to path:/PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES
Subscribing to path:/PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES took:0
21 START:INVOKE /PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor@4b9419ff type: CALLBACK
Resubscribe change listener to path: /PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES, for listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4b9419ff, watchChild: false
Subscribing changes listener to path: /PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES, type: CALLBACK, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@4b9419ff
Subscribing child change listener to path:/PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES
Subscribing to path:/PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES took:1
The latency of message dbb39d17-cece-4f3c-bd88-29fb61a93863 is 922470 ms
Fail to find message handler factory for type: USER_DEFINE_MSG msgId: dbb39d17-cece-4f3c-bd88-29fb61a93863
The latency of message 4c8dffc3-68cf-417e-8cd2-a572ce4cdcb4 is 42 ms
Fail to find message handler factory for type: USER_DEFINE_MSG msgId: 4c8dffc3-68cf-417e-8cd2-a572ce4cdcb4
21 END:INVOKE /PinotCluster/INSTANCES/Server_10.0.101.11_8069/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor@4b9419ff type: CALLBACK Took: 7ms

We had an issue where confluent schema registry had downtime, and realtime ingestion failed with "Read from kafka failed" but did not recover until we restarted the servers. Is this a known issue? Or is there something else we could have done? The realtime table was on the default tenant and I issued a rebalance, that did not help (old data was there but no consuming segments).

Apologies for all the trouble today: we noticed that some tables are in a "bad" (cluster manager ui) state. Looks like it's due to an attempt by servers to download non-existent segments from deepstore. Could it be that the segment was empty and not copied to deepstore?

Hi Team, What is the recommended way of updating table schema. I used the following job to create my table

apiVersion: batch/v1
kind: Job
metadata:
  name: request-realtime-table-creation
  namespace: data2
spec:
  template:
    spec:
      containers:
        - name: request-realtime-table-json
          image: apachepinot/pinot:latest
          args: [ "AddTable", "-schemaFile", "/var/pinot/examples/request_schema.json", "-tableConfigFile", "/var/pinot/examples/request_realtime_table_config.json", "-controllerHost", "pinot2-controller", "-controllerPort", "9000", "-exec" ]
          env:
            - name: JAVA_OPTS
              value: "-Xms4G -Xmx4G -Dpinot.admin.system.exit=true"
          volumeMounts:
            - name: examples
              mountPath: /var/pinot/examples
      restartPolicy: OnFailure
      volumes:
        - name: examples
          configMap:
            name: pinot-table
  backoffLimit: 100

Hi! I'm trying to set up Pinot for the first time (as a cluster) and am starting to set up S3 following the steps in https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3. It works fine for the controller, but for the server I see this error on startup:

2021/02/10 12:22:25.506 ERROR [PluginManager] [main] Failed to load plugin [pinot-s3] from dir [/<redacted>/apache-pinot-incubating-0.6.0-bin/plugins/pinot-file-system/pinot-s3]                             
java.lang.IllegalArgumentException: object is not an instance of declaring class

Hey team thanks for setting this slack up! Would appreciate any help on this: I’m trying to do a multi line groovy script like this in a query in the Pinot Query Console:

""" 
def value = 'blah'
return value
"""

using this syntax:

select groovy('{"returnType":"STRING","isSingleValue":true}', <GROOVY MULTI LINE HERE>, my_variable) as new_variable from table

I have tried to cancel out the quotes, use single, cancel any newlines, and cannot figure out how to get this to work. Any ideas?

Hi Team, we are facing one issue with realtime consumption (from kafka). We have 4 pinot servers, 7 real time tables. Every few hours, one of the pinot server stops consuming from one kafka topic. In logs, server unable to connect to controller while finalising the segment. However, controller is up and running. • Should pinot server retry for such recoverable errors? • Are there any config levers (retries on error, etc) to fix this? Any thoughts?

Hi Team , I am running a job to import data form s3 from parquet ,getting this error

2021/02/11 06:27:24.999 ERROR [SegmentGenerationJobRunner] [pool-2-thread-1] Failed to generate Pinot segment for file - <s3://xxxx/xxxx/xxxx.0.parq>
java.lang.IllegalArgumentException: INT96 not yet implemented.

check out the includeFileNamePattern: https://docs.pinot.apache.org/basics/components/segment#job-spec-yaml

I am getting error while uploading s3 data

Failed to generate Pinot segment for file s3:xxx/xxx/1234.csv
Illegal character in scheme name at index 2: table_OFFLINE_2021-02-01 09:39:00.000_2021-02-01 11:59:00.000_2.tar.gz
at java.net.URI.create(URI.java:852) ~[?:1.8.0_282]
	at java.net.URI.resolve(URI.java:1036) ~[?:1.8.0_282]
	at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$run$0(SegmentGenerationJobRunner.java:212) ~[pinot-batch-ingestion-standalone-0.7.0-SNAPSHOT-shaded.jar:0.7.0-SNAPSHOT-162d0e61b6b1c3d51f915f7ad3e151a4fb24110a]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_282]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_282]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_282]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]

I have uploaded data to Pinot, I can see in S3 the tar files, but in query editor only 10000 files which I uploaded intially are showing up is this expected? that it would have some delay ? becasue there was no error on console from script.?

Hi Team, what are the steps to restore deleted segments back to a REALTIME table? We did the following steps but to no avail. • Copy back the deleted segments to respective tables • Restored the zookeeper metadata from zk backup to this path

/pinot/<datasource-name>/PROPERTYSTORE/SEGMENTS/<table-name>_REALTIME/…

• Restarted servers, controllers We don’t see these segments in IDEAL_STATE. Anyone has done this? Whats the right way to restore deleted segments for a REALTIME table?

Hi Team I have a column with INT in offline table , I am loading from s3 csv, value for that is '1' in csv but its loaded as '-2147483648' in table what could be the reason?

Guys, I merged the latest changes into our fork, built again the docker image, deployed and now when we try to create a table we see this error in the controller:

ClassNotFoundException: org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory

Any suggestions?

When I open the pinot controller web UI it becomes visible very briefly and then goes blank. I see a bunch of warnings in the controller logs, are these relevant?