Hello Is it possible to generate segment names following the input file names ? Say I generate 10 files for 10 "ids", I'd want segments to contain these ids, so that they can be replaced later by generating another segment with the same name. e.g.

ID1.parquet -> prefix_ID1.segment

Anyway to make this work using

segmentNameGeneratorSpec.type

? Maybe using a particular file structure like

data/ID/file.parquet

? Thanks !

Greetings! Is there a stable version of the helm chart to run? I install the latest (0.7.1) but all the components crash with messages like the following

Unrecognized VM option 'PrintGCDateStamps'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Hello! I am using Presto Pinot python connector to query Pinot. I have a requirement for a regex type predicate on one of the dimensions. I created text index on the dimension. Will this help in the performance? Will it be able to use TEXT_MATCH to query?

How realtime backward-Compatible(new column) works in Pinot. I have some events in Kafka and created one pinot realtime table .next time I have again pushed some data in Kafka with 1 new field and again executed AddTable command with updated schema(new column added in schema) and then on pinot ui did reload all segments.on ui in schema I am able to see that new column but when I am doing select * from tablename the newly added column is not showing there. Here I have added city as new column. @Jonathan Meyer

I'm able to see other column data only for newly added column it's not showing any data.this is my configuration file. Kindly suggest.

[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.094 s [INFO] Finished at: 2021-06-24T190206+05:30 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.pluginsmaven compiler plugin3.8.0:compile (default-compile) on project pinot-spi: Compilation failure [ERROR] javac: invalid target release: 11 [ERROR] Usage: javac <options> <source files> [ERROR] use -help for a list of possible options [ERROR] [ERROR] [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <args> -rf :pinot-spi

@Xiang Fu I am starting all these components on single node. How to check that Hadoop conf dir is mounted to pinot or not? Yes, pinot has access to hdfs.

Hi Everyone, Is there any way to Ingest Data from Kerberous Kafka in Pinot?.I have started Pinot with quick-start-batch.sh file and added jaas file location in quick-start-batch.sh. -Djava.security.auth.login.config="/home/hdfs/client_jaas.conf" \ -Dapp.name="quick-start-batch" \ and it worked as expected.Now I am Starting all Pinot Components seprately with the help of config file and added -Djava.security.auth.login.config="/home/hdfs/client_jaas.conf" in all existing Pinot component files (start-broker.sh,start-controller.sh,start-server.sh) and trying to create table but getting below error message. 2021/06/29 222903.708 INFO [AddTableCommand] [main] {"code":500,"error":"org.apache.kafka.common.KafkaException: Failed to construct kafka consumer"}. Someone Kindly suggest how should i use client_jaas.conf to start pinot in clusterd mode so it can read data from kerberous kafka also. @Xiang Fu @Jackie @Jonathan Meyer

Hello I'm seeing odd behavior with filtering (WHERE predicate) A column contains LONG values (single valued), and filtering only works for some of the values in that column Here's the data:

Hi, we are getting a 500 error (with no logs) when doing queries with Integer.MAX_VALUE as the limit on pinot 0.7.1 - I see from the code that should be the max limit. Is there something else we are running into? (there are no table quotas). Here is the full error, redacted hosts and table names to maintain super secrecy:) :

Unexpected response status: 500 for request {"sql" : "select AAA, BBB from CCC limit 2147483647" } to url <http://XXXX>:XXXX/query/sql, with headers {Accept=[application/json], Content-Type=[application/json]}, full response null

Hi team, I'm getting this error in broker, controller, minion, server. Kindly help on this.

Hello 🙂 In Pinot's Helm chart, is there any reason to have

.Values.{controller|broker}.external.enabled

set to

true

by default ? Maybe having it to

false

would be a safer alternative for first time users not knowing the chart well enough yet

Anyone has an idea of a workaround for that issue ? https://github.com/apache/incubator-pinot/issues/7090

Hello Team, I am observing a few issues - 1) I changed the replicas for a table from 1 to 3. All the segments that are created after the change have 3 replicas, but the pre-existing segments continue to have only 1 replica. 2) When a server node goes down, its hosted segments are not created on another node to maintain 3 replicas for the segments. 3) When I add a new server, no redistribution of segments happen automatically. What are the expectations from Pinot in the above cases?

Hello 😄 Any idea why the

/segments/{tableName}/metadata

controller API endpoint only returns metadata of one segment (a random one, it seems) and not of all of them ? The description states

Get the server metadata for all table segments

Hi All, I need help with kubernetes installation. I helm install pinot using the chart included in main pinot github (by cloning the entire pinot source code). Apparently broker-0 could not able to come up because of

Cluster structure is not set up for cluster: pinot-quickstart

What went wrong here ?

cd incubator-pinot/kubernetes/helm/pinot
helm install -n pinot-quickstart2 pinot .

Session establishment complete on server pinot-zookeeper/172.20.47.166:2181, sessionid = 0x1012fddef2b0003, negotiated timeout = 30000
zookeeper state changed (SyncConnected)
MBean HelixZkClient:Key=pinot-quickstart.Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099,Type=SPECTATOR has been registered.
MBean HelixZkClient:Key=pinot-quickstart.Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099,PATH=Root,Type=SPECTATOR has been registered.
KeeperState: SyncConnected, instance: Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099, type: SPECTATOR
Handle new session, instance: Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099, type: SPECTATOR
Handling new session, session id: 1012fddef2b0003, instance: Broker_pinot-broker-0.pinot-broker-headless.pinot-quickstart2.svc.cluster.local_8099, instanceTye: SPECTATOR, cluster: pinot-quickstart
fail to createClient.
org.apache.helix.HelixException: Cluster structure is not set up for cluster: pinot-quickstart
        at org.apache.helix.manager.zk.ZKHelixManager.handleNewSession(ZKHelixManager.java:1124) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-47a75e5093129cc280de4c118434ccb337cd3da1]

Hi I try to push data from GCS to Pinot, after submitting job it seem not doing any and no output at all, these are my job spec

executionFrameworkSpec:
    name: 'standalone'
    segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner'
    segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner'
    segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner'
jobType: SegmentCreationAndUriPush
inputDirURI: '<gs://mdm-datalake/ais/sentences/>'
outputDirURI: '/tmp/ais-pinot/sentences/'
includeFileNamePattern: 'glob:**/**.parquet'
overwriteOutput: true
pinotFSSpecs:
  - scheme: file
    className: org.apache.pinot.spi.filesystem.LocalPinotFS 
  - scheme: gs
    className: org.apache.pinot.plugin.filesystem.GcsPinotFS
    configs:
        projectId: 'aton-analytics'
        gcpKey: '/var/pinot/controller/config/gcs-datalake-key.json'
recordReaderSpec:
    dataFormat: 'parquet'
    className: 'org.apache.pinot.plugin.inputformat.parquet.ParquetRecordReader'
tableSpec:
    tableName: 'sentence'
pinotClusterSpecs:
    - controllerURI: '<http://localhost:9000>'

I tried these table and schema format I am not able see any datas in the pinot table but i can see the data in kafka-topic

@Radhika @Sadim Nadeem Please check the controller and server log and see if there is any error logged? Even if the partitioning is wrong, you should still be able to see data consumed, so that should not be the cause

Hi. We have a K8s Pinot deployment and some of our queries are taking > 10s. We found one conspicuous correlation during our investigation - Latency spikes happen when there is also a spike a YG GC count. In the following charts, spikes happened across the board at 15:28. Does this indicate a possible GC issue?

Hello Team, when I set alias in a query, it fails : select c.CustomerID from customer c limit 10 . The error is : [ { "errorCode": 190, "message": "TableDoesNotExistError" } ] While one without alias succeeds: select CustomerID from customer limit 10 Can someone please help understand what is wrong in the query with alias?

Hi All , Can Pinot realtime table data transfer to offline table directly, thx

Hi everyone! I am researching realtime table evolution. After updating pinot schema and reloading segments I see new columns in table and null values in old data. But after consuming new data with not empty newly added fields, new data are importing with null values too in new columns. Kafka messages in avro formats. When I debug the code I see that AvroRecordExtractor still uses old set of fields. Can I refresh fields set in AvroRecordExtractior and start consuming messages with new columns?

Hello everyone , i would like to understand how can backfill offline data which contains multiple segment.

If I’m using a secured Kafka using SASL_SSL. Is there any way of configuring that and use those credentials? Or there is another way of setting security from Pinot to Kafka for data ingestion?

Hello, I'm using the following tutorial to use S3 and Pinot in Docker (https://docs.pinot.apache.org/users/tutorials/use-s3-and-pinot-in-docker) but when I try to start up the controller using the the docker command provided in the tutorial I keep on getting below error:

Executing command: StartController -configFileName /tmp/controller.conf
Caught exception while starting controller, exiting.
java.lang.RuntimeException: Error: Unable to find controller config file /tmp/controller.conf

I've created the controller.conf file and placed it in the /tmp/pinot-s3-docker folder on my EC2 instance. Any help would be greatly appreciated.

I've ran into a problem with a realtime table using

"completionMode": "DOWNLOAD"

. The download failed three times on the replica server, and the segment is stuck in error state. If I try to reload it the log says it's reloading a consuming segment, but nothing changes. How can I recover from this state?

Hello, I am trying to load a csv (generated from Redshift UNLOAD) command and the dates are in this format (in schema file):

"name": "created_date",
            "dataType": "STRING",
            "format" : "1:MILLISECONDS:SIMPLE_DATE_FORMAT:YYYY-MM-dd HH24:MI:<http://SS.MS|SS.MS>",
            "granularity": "1:MILLISECONDS"

example from csv file: 2020-03-01 073108.792457. I keep on getting failed to generate pinot segment and java illegal argument exception error

java.lang.IllegalArgumentException: Illegal pattern component: I
	at org.joda.time.format.DateTimeFormat.parsePatternTo(DateTimeFormat.java:566) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.joda.time.format.DateTimeFormat.createFormatterForPattern(DateTimeFormat.java:687) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.joda.time.format.DateTimeFormat.forPattern(DateTimeFormat.java:177) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.spi.data.DateTimeFormatPatternSpec.<init>(DateTimeFormatPatternSpec.java:57) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.spi.data.DateTimeFormatSpec.<init>(DateTimeFormatSpec.java:59) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.core.indexsegment.generator.SegmentGeneratorConfig.setTime(SegmentGeneratorConfig.java:212) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.core.indexsegment.generator.SegmentGeneratorConfig.<init>(SegmentGeneratorConfig.java:138) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.plugin.ingestion.batch.common.SegmentGenerationTaskRunner.run(SegmentGenerationTaskRunner.java:95) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]
	at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$run$0(SegmentGenerationJobRunner.java:199) ~[pinot-batch-ingestion-standalone-0.7.1-shaded.jar:0.7.1-afa4b252ab1c424ddd6c859bb305b2aa342b66ed]

Has anyone run into this issue? Do I need to convert the dates to EPOCH when generating CSV?

If the format is actually simple date format then what you are looking for is likely YYYY-MM-dd HHmmss.SSS. See https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html

Does Pinot provide a way to see the execution plan of a query?