@User we curently cannot check the lag, but there are a few metrics available that can help you get there.

ServerGauge.HIGHEST_STREAM_OFFSET_CONSUMED

is one such. In addition, we have

ServerMeter.REALTIME_ROWS_CONSUMED

so you can check the consumption rate. Also see

ServerMeter.LLC_PARTITION_CONSUMING

which emits

1

to indicate the health of the consumer, and

0

whenever it stops to consume (e.g. during segment commit, or other errors). If this metric is 0 for a long time, then you have an issue

@User/@User: We have used pinot as backend for API and got very good performance. It took around 5 milliseconds for two PQL query fetching (around 3 billion records & 160 GB size) and the overall response took 500-1000 milliseconds due to some network latency. Appreciate your continuous support and thanks a lot to achieve this.

on how many machines? is this on the cloud? What is the I/O bandwidth?

I have configured a realtime table with low level kafka consumer and the consumer crashed due to avro deserialization issue. How can I restart the consumer again after fixing the avro issue?

Hi. I'm looking into both Druid and Pinot and am having trouble evaluating the differences. This was a good article, https://medium.com/@leventov/comparison-of-the-open-source-olap-systems-for-big-data-clickhouse-druid-and-pinot-8e042a5ed1c7, but seems to be outdated at this point, as it doesnt mention for instance that you can use GCS as your deep storage for Pinot. Has anyone found anything helpful in evaluating the two different OLAP databases? Also, I'm curious whether people have paired any other front end visualization pieces with Pinot besides Superset and writing their own?

And yes, Pinot does support GCS as deep storage

https://github.com/apache/incubator-pinot 2500 stars GitHub

HI: I'm trying to create a table on local filesystem by importing a CSV file using the steps provided in "Batch upload sample data" section from the documentation. I ran

bin/pinot-admin.sh AddTable

and

bin/pinot-admin.sh LaunchDataIngestionJob

commands after running

bin/quick-start-batch.sh

. I got successful messages for both

AddTable

and

LaunchDataIngestionJobs

commands. I also see the segment file created. But the table shows zero rows on the "Pinot Data Explorer". One deviation from the tutorial was that I had to pass

-<http://Djava.io|Djava.io>.tmpdir

argument to JAVA in

bin/pinot-admin.sh

and

bin/quick-start-batch

scripts as there was not enough space in

/tmp

directory to import the table. Please advise on how I could fix this.

@User 1. From the Pinot REST API, do you see the segments if you list segments for the table? 2. Do you see any exceptions in pinotController.log, pinotServer.log ?

1. I see the segment listed:

[
  {
    "OFFLINE": [
      "new_OFFLINE_1583846302903254_1584027805186276_0"
    ]
  }
]

2. I see exception in pinotServer.log:

2020/04/17 15:23:55.362 ERROR [SegmentFetcherAndLoader] [HelixTaskExecutor-mess

ge_handle_thread] Cannot load segment : new_OFFLINE_1583846302903254_15840278

05186276_0 for table new_OFFLINE

<http://java.io|java.io>.IOException: No space left on device

    

at <http://java.io|java.io>.FileOutputStream.writeBytes(Native Method) ~[?:1.8.0_232]

    

at <http://java.io|java.io>.FileOutputStream.write(FileOutputStream.java:326) ~[?:1.8.0_2

32]

    

at <http://org.apache.commons.io|org.apache.commons.io>.IOUtils.copyLarge(IOUtils.java:1793) ~[pinot-al

l-0.3.0-jar-with-dependencies.jar:0.3.0-9b2dc20c07dec6cf33df08c4444d996e8202c3ba

]

    

at <http://org.apache.commons.io|org.apache.commons.io>.IOUtils.copyLarge(IOUtils.java:1769) ~[pinot-al

l-0.3.0-jar-with-dependencies.jar:0.3.0-9b2dc20c07dec6cf33df08c4444d996e8202c3ba

]

    

at <http://org.apache.commons.io|org.apache.commons.io>.IOUtils.copy(IOUtils.java:1744) ~[pinot-all-0.3

It appears like Server is still trying to use

/tmp

directory. How can I change this?

looks like there's no way in quickstart to change that. hmm, we should make that configurable perhaps.

are you using the distribution, or have you cloned the repo?

I am running with distribution but I have clone repo too.

can you run using the source code instead of the distribution, and apply this change: https://github.com/apache/incubator-pinot/pull/5270

yes, thanks

Hi! I think my team might have found a bug. If we use the hybrid quick start server and have an offline table created where the schema name does not match the table name, the following request fails. If the table and schema name match though, it works. We updated the schemaName in both the events-offline-table.json and the -schema.json file. If I add a realtime table (with when the names do not match), everything works fine too. http://localhost:9000/tables/events_testing/schema

Any plans to integrate pinot with promethus? https://prometheus.io/docs/operating/integrations/

@User ^^

I think this is for using pinot for telemetry storage

I will take a look, mostly for pinot, read part is fine, write part for realtime points might still need to go through kafka right now

@User The data is immutable at record level, but you can refresh segments to overwrite data in that segment.

Could you elaborate a bit more in the mutablility requirement of your use case?

So, the requirement is something like … We have data e.g TS, ColA, ColB, ColC where TS is the timestamp column. We need aggregates like TS, ColA --> Fact1 TS, ColA, ColB --> Fact2 Older data can come in TS column, which means we may need to update the value of Fact1 and Fact2. This update needs to happen realtime. Let me know if this helps

I am unclear, do you want to overwrite and existing record, or just that data with older time is coming in and you want to be able to ingest? If latter, Pinot does not time partition the data in the sense that it will be able to ingest data with older time stamp.

root@pinotlocal:~/incubator-pinot/pinot-distribution/target/apache-pinot-incubating-0.4.0-SNAPSHOT-bin/apache-pinot-incubating-0.4.0-SNAPSHOT-bin/bin# ./pinot-admin.sh UploadSegment -controllerHost 40.121.159.109 -controllerPort 9000 -segmentDir /home/neela_vinod/out_put/ Executing command: UploadSegment -controllerHost 40.121.159.109 -controllerPort 9000 -segmentDir /home/neela_vinod/out_put/ Compressing segment transcript_0_0 Uploading segment transcript_0_0.tar.gz Sending request: http://40.121.159.109:9000/v2/segments?tableName to controller: pinotlocal.internal.cloudapp.net, version: Unknown Exception caught: org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Caught internal server exception while uploading segment" while sending request: http://40.121.159.109:9000/v2/segments?tableName to controller: pinotlocal.internal.cloudapp.net, version: Unknown at org.apache.pinot.common.utils.FileUploadDownloadClient.sendRequest(FileUploadDownloadClient.java:374) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-b20ace081fc8566490d5e8addc211537843c8a9a] at org.apache.pinot.common.utils.FileUploadDownloadClient.uploadSegment(FileUploadDownloadClient.java:491) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-b20ace081fc8566490d5e8addc211537843c8a9a] at org.apache.pinot.common.utils.FileUploadDownloadClient.uploadSegment(FileUploadDownloadClient.java:510) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-b20ace081fc8566490d5e8addc211537843c8a9a] at org.apache.pinot.tools.admin.command.UploadSegmentCommand.execute(UploadSegmentCommand.java:134) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-b20ace081fc8566490d5e8addc211537843c8a9a] at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:156) [pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-b20ace081fc8566490d5e8addc211537843c8a9a] at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:168) [pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-b20ace081fc8566490d5e8addc211537843c8a9a]

ℹ️ just an Fyi, When I provisioned the pinot controllers with

openjdk 11.0.3 2019-04-16

on

pinot 0.3

I got a warning

WARNING: Illegal reflective access by org.apache.pinot.spi.plugin.PluginClassLoader (file:/usr/local/pinot/lib/pinot-all-0.3.0-jar-with-dependencies.jar) to method <http://java.net|java.net>.URLClassLoader.addURL(<http://java.net|java.net>.URL)
WARNING: Please consider reporting this to the maintainers of org.apache.pinot.spi.plugin.PluginClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

this is expected

pinot is using some jdk8 apis which becomes not public in jdk11 hence we need to use this --illegal-access flag to access them

for jdk8+, pinot add extra flags in script

--add-exports java.base/jdk.internal.ref=ALL-UNNAMED