Hello everyone, I’m playing with ingestion of events from kafka to pinit. Initially was blocked on getting any events into pinot. Though thats working now, we’re noticing that ingestion arbitrarily stops after some time. Our cluster health look good, and the pinot UI also shows everything green. (we inferred that ingestion has stopped as the query console gets stuck at the same number of docs, and kafka also shows that consumption has stopped) Is there some limit to the number of events that we can push to Pinot, or can it be some configuration issue? I’m not sure how to go about debugging this.

does the

storage

quota in a dimension table refer to the uncompressed size in memory, or the compressed segment size?

Does pinot support Ldap/Oauth authentication.

Say I have a table

t1

with 200+ columns

A

,

B

, `C1`…`C200`. When I do

select max(A) from table group by B'

, I get the max A for each B. Is there to get the corresponding `C1`…`C200` for each max(A) per B? That is to say, how to achieve `select max(A), C1, ..., C200 from table group by B`without aggregating on

C

or grouping by

C

?

Hi! I have question about updating indexes without re-ingestion data in segment-store. I mean in documentation there is such information:

The updated index config won't be picked up unless we invoke the reload API.

I have added new index(

inverted

) to table config and call reload API(with

forceDownload

flag) but as I see index is not created(I checked in

reload status

and even in

metadata.properties

file of particular segment). But after I re-ingest data(with spark job) to segment-store then index is created. All these operations are done for

offline

table So the question is: is it necessary to comply some additional conditions to make reload API working? Maybe, I missed something

i was looking at https://docs.pinot.apache.org/basics/components/table#pre-aggregation. is there a plan to support aggregation functions other than

SUM

in future?

also what is the time bucket for which aggregation happens?

Is it possible to use LZ4 compression type for metric columns?

hey there 👋 Happy 2022! 🎉 I see that Pinot 0.9.3 has been released, but there is no 0.9.3 tag on docker hub. Is there a new tag I should use?

Aaaand here I am again XD We are deploying Pinot to our staging environment (on k8s). We are using the provided helm package, but we need the option to have an ingress resource. We couldn't find it inside project's the helm files (https://github.com/apache/pinot/blob/master/kubernetes/helm/) Are we missing something here?

Hey folks, can anyone confirm whether range queries will work with

JSON_MATCH

? Ex. Modifying example query from json-index docs[1]:

{
  "name": "adam",
  "age": 30,
  "country": "us",
  "addresses": [
    {
      "number": 112,
      "street": "main st",
      "country": "us"
    },
    {
      "number": 2,
      "street": "second st",
      "country": "us"
    },
    {
      "number": 3,
      "street": "third st",
      "country": "ca"
    }
  ]
}

SELECT ... 
FROM mytable 
WHERE JSON_MATCH(person, '"$.addresses[*].number">2')

I would expect that this

adam

doc/row would match. But I’m not sure if range queries are supported. Thanks! https://docs.pinot.apache.org/basics/indexing/json-index#chained-key-lookup

Hello, Pinot community! New to Pinot. I was looking for a example similar to this one : https://docs.pinot.apache.org/users/tutorials/ingest-parquet-files-from-s3-using-spark but does ingestion using apache flink. Is there one I can refer to?

Hello, I am using Pinot tables for an app which cannot have a down time. The tables are REFRESH tables. I am thinking of maintaining 2 table versions of current and old for each table. So that at any given time data is available and to add some logic to switch the tables being referenced at the API side once data gets refreshed. Is this is the best way to go about it? Or can I use segments with date and try to query the correct segment- something along those lines. Am I missing something here?

Did we do the scalability test for upsert table?

hi, is

Reported Size

in the table UI in bytes?

Hello, I am trying to create a local Pinot cluster with HDFS(running locally) as deep storage. Getting multiple class not found/method not found errors. Pinot Version -

0.9.3

and Hadoop Version

3.3.1

- Is there a compatibility matrix which I can refer?

Hi, is there any planning to add new Geospatial functions? And how i can return ST_Distance in meters? And what types of geometry i can use in GeoIndex - it's only POINT? can i use LINESTRING or GEOMETRYCOLLECTION?

How does pinot authentication works during ingestion and querying?

Hi Team, I am trying to submit a spark ingestion job but getting

main

method not found exception and I checked the class

LaunchDataIngestionJobCommand

code, it does not have a main method.

Exception in thread "main" java.lang.NoSuchMethodException: org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main([Ljava.lang.String;)
	at java.base/java.lang.Class.getMethod(Class.java:2108)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:42)
	at <http://org.apache.spark.deploy.SparkSubmit.org|org.apache.spark.deploy.SparkSubmit.org>$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Configuration -

#!/usr/bin/env bash

export HADOOP_CONF_DIR=/Users/$USER/servers/hadoop-2.7.1/etc/hadoop
export SPARK_DIST_CLASSPATH=$(/Users/$USER/servers/hadoop-2.7.1/bin/hadoop classpath)

export PINOT_VERSION=0.9.3
export PINOT_DISTRIBUTION_DIR="/Users/$USER/servers/pinot-${PINOT_VERSION}"

export SPARK_VERSION=3.2.0
export SPARK_HOME="/Users/$USER/servers/spark-${SPARK_VERSION}-bin-without-hadoop"

${SPARK_HOME}/bin/spark-submit \
  --class org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \
  --master "local[2]" \
  --deploy-mode client \
  --conf "spark.driver.extraJavaOptions=-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" \
  --conf "spark.driver.extraClassPath=${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar" local://${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with-dependencies.jar \
  -jobSpecFile ${PINOT_DISTRIBUTION_DIR}/examples/batch/airlineStats/sparkIngestionJobSpec.yaml

Meetup next week! 🙂 https://www.meetup.com/apache-pinot/events/282697873/ Presentation by @User

Hi 👋 and welcome to all the new Apache Pinot community members! We're so happy to meet you here! 🍷 Please tell us a little about yourself and what brought you here. @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User @User

Hello, does Pinot support loading from this input format :

org.apache.hadoop.mapred.TextInputFormat

?

Hello, does pinot support sql windowing?

can someone point me to the documentation for java connection pooling with pinot?

Regarding the IdSet filtering feature, is there a way for an external program to generate the base64 encoded IdSet object?

Hi folks! I'm trying to get a Pinot cluster setup in AWS, but I need to decide exactly the pieces of this cluster so that the sysops from my the company I work for can set it up, and I'd like your opinions on this. I'll explain more in this thread.

is there a way to force flush a segment?

Hello, I am a beginner with Pinot and trying to create my first POC. I have set up everything on docker and all Pinot Services is running. I just need a help to connect external Kafka topic with pinot. I am consuming my topic in a console like this. Please help me to configure this setting in pinot.

Hey all, our team is currently using a hybrid pinot table with the offline portion of the data being limited to data that is 3 days old or greater. We are wanting to better understand how to safely adjust the time boundary from 3 days to 5 days without loss of data in our query output. We attempted to do this by altering the lookback window of our offline data pipeline, but data was then missing from the gap between the 3 and 5 days. Is there any guidance in the docs on how to safely adjust this time boundary? I haven't been able to find anything myself

Need help on this error - https://stackoverflow.com/questions/70682861/apache-pinot-trying-to-add-schema-by-terminal