Hello all, I'm looking at this example for the realtimetoofflinesegmentstask and see a

schedule

config property but don't see an explanation on the doc page. Anyone have experience implementing it?

Hi folks! A question about `dateTimeFieldSpecs`: Is the

TIMESTAMP

type something that will stay in Pinot? Because @User gave me a nice hint about that but we don't see that in the docs, so I'm unsure whether I can safely use it and know that it won't get removed on future Pinot versions

Hi Getting below error while I was starting broker or an controller

[uname@adscjbhcsdc211 apache-pinot-0.8.0-bin]$ bin/pinot-admin.sh StartBroker -zkAddress localhost:2181
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.UnsupportedClassVersionError: JVMCFRE003 bad major version; class=org/apache/pinot/tools/admin/PinotAdministrator, offset=6
        at java.lang.ClassLoader.defineClassImpl(Native Method)
        at java.lang.ClassLoader.defineClassInternal(ClassLoader.java:398)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:359)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:154)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:730)
        at java.net.URLClassLoader.access$400(URLClassLoader.java:96)
        at java.net.URLClassLoader$ClassFinder.run(URLClassLoader.java:1187)
        at java.security.AccessController.doPrivileged(AccessController.java:770)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:605)
        at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:944)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:889)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:872)
        at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:507)

As per https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion

stream.[streamType].consumer.prop.auto.offset.reset

determines the offset from which to start the ingestion

smallest

largest

or timestamp in milliseconds

However, it doesn’t seem to take a timestamp?

Hi All, http://localhost:9000/v2/brokers/tenants/DefaultTenant?state=ONLINE : “_error”: “Permission is denied for access type ‘READ’ to the endpoint “. It’s happening while i’m trying to connect through pinot-jdbc 0.8.0 version client using user pwd.

Not sure what channel is best for this, but I was looking into Pinot-related issues (from other projects) on the Apache Jira site, and noticed that there’s a Pinot project with a handful of old issues. See https://issues.apache.org/jira/projects/PINOT/issues/PINOT-12?filter=allopenissues

I see version 0.9 is in rc, do we have a binary download link for this version? Some nice new features to try out.

Thanks @User! Hi Everyone 👋 I'm currently working an a set of automatic code transformations to help when migrating from Joda-Time to java.time, I noticed Issue #7499 discussing migrating from Joda to java.time so I'm interested in whether I can be of any help during this process. I am currently just trying to get more familiar with pinot and it's components as I haven't used it before. I will demo pinot to our engineering team once I have a better grasp of it. If anyone is interested in discussing #7499 then i'm keen to understand if I can be of any use!

Hello @User I am a Java software engineer. I hava experience with classic Java web applications (Spring, Hibernate), also with relation databases and Elasticsearch. Now I am considering some data engineering experience and I discovered Pinot on Twitter. I am in my first steps following some YouTube tutorials

Hi folks! I have a question about segment thresholds. If I have something like this:

'realtime.segment.flush.threshold.rows': '10000',
                    'realtime.segment.flush.threshold.time': '24h',
                    'realtime.segment.flush.desired.size': '100M',

does this mean that the first value that gets reached from the above ones determines that the segment will be flushed? Or is it the last value reached that determines that? For example, if a segment has been filling for 24h already, but has only 200 rows and 10M in size, does it get flushed because it reached the 24h mark?

That's awesome, folks, congratulations! Are there plans yet for a Docker image with the new version?

Is stream ingestion with Upsert compatible with the RealtimeToOfflineSegmentsTask? I plan to have a realtime table that uses the upsert functionality, and then have older segments moved to an offline table to make a hybrid table.

Hi Folks, Is there a constraint of dimension tables being assigned to tenants? Does lookup UDF join work when a table and dim table are in different tenants?

Are we considering upgrade the Kafka lib version to something newer like 2.4+?

Hi, do you know if it’s possible to query Pinot from Apache Drill or Dremio?

1. Servers are assigned partitions to consume from. If a server goes down it’s partitions are not auto moved to other server. This is why you need to run in HA mode (relocation).

2. For new table consuming for the first time, it will start from smallest Kafka offset. If consumption already started then offset in zk used

3. Depends on the change you made to config. Some require reload api (indexing changes), others wait until new segment consumption starts etc (streaming config changes).

Is it possible to have table sorted on a multi valued column?

Hi Team, Need your recommendation on this one. We are planning to bring in a dimension table for quick LOOKUP functionality where we JOIN our analytics table with this dim table. 1. Since dimension tables are loaded in each server's memory for a given tenant ID, do we need replication-factor >1 here? Since its replicated on each node by default, is RF of any significance here? 2. Our calculation says we would be needing ~1-2 GB memory for this dimension table. Is there any constraint on the size? Or its just controlled by the "quota" config?

Hi Team, I have some questions regarding the quota related configuration of the Dimension table: We are running a Pinot Cluster setup having 4 servers and we have created a dimension table with the storage configured to be "200 MB". We are populating this dimension table with segments having 100k records and the uncompressed size of this segment is around ~23 MB (232 byte for each record in the segment). So as per the calculation, we were expecting that the table will be able to hold around ~900k records. However, we see only 200k records in the table and when we are pushing 3rd segment of 100k records, we are getting 403 error. Can someone please help us here as to why we are getting 403 error and not able to push any more segment of 100k record.

json path expressions in query are not working. select payload.push_id from githubEvents limit 10 throws unknown column name exception. But using json functions works. @User and @User - this is using json batch quickstart setup.

One question on mergeRollup tasks - do they honor segment partitioning strategy? For example, I have setup segmentPrunerTypes with some dimension based partitioning. When merge rollup tasks produce the merged segment, will the merged segments contain data that honors the dimension based partitioning?

Can we use Java Spring Boot data JPA with Pinot? I am not able to find any resources around it.

Is there an SQL equivalent of TOP from PQL?

is there any documentation on the LASTWITHTIME aggregation function? I'm trying to use it right now but it seems that it only supports time columns that are LONG/INT?

Hi ! Does someone know if the pinot-pulsar module supports Pulsar authentication ?

Is there an efficient way to get the latest rows (based on timestamp and we want all the columns in the row) grouped by a column? I tried using LASTWITHTIME but it only takes in 1 column name at a time. Specifying LASTWITHTIME for each of the columns in the query increases latency significantly, which seems to imply that it does a linear scan for each column?

Hi all! ⭐ ⭐ ⭐ The Apache Pinot 2021 Recap and Roadmap discussion is being rescheduled!! ⭐ ⭐ ⭐ It is now scheduled for Monday, December 13th 📅 👉 MONDAY (December 13th) 10:00 PST | 13:00 EST | 18:00 UTC We're inviting you all to join us for the 2021 Annual re-cap of Apache Pinot. We will also be opening up the floor for discussion of the future roadmap and would love for you to join the discussion! Please RSVP and attend in the meetup link below. We will be doing it on Zoom, so you will be able to chime in with voice/video or text on your thoughts and feedback. Thank you so much for being part of this movement with us, and we look forward to an awesome 2022!!!! 🤩 https://www.meetup.com/apache-pinot/events/282168447/ ✅ ✅ ✅ Also, please VOTE on what you'd like to see in the future Apache Pinot roadmap! ✅ ✅ ✅ https://poll.ly/#/Pvy1rpp2 (poll closes Friday, December 10th) ❓ Questions, Comments, Concerns, Ideas? Feel free to DM me ❓ 😃

hey there 👋 I'm not able to make partial upserts work on 0.9.0 and the error messages are not helpful. the partial upserts did work previously, I have no idea what is happening. Can someone help me? This kinda blocks me to deliver my app to prod 😭 More info (such as config files and error messages) here.