waves to @User and @User

🤔 question about the documentation. On https://docs.pinot.apache.org/developers/advanced/data-ingestion, the docs say

The servers download the segments (as a cached local copy to serve queries) and load them into local memory. All segment data is maintained in memory as long as the server hosts that segment.

But Pinot supports an

mmap

mode. Is it safe to say that all segment data will not be stored on physical memory, meaning your cluster memory size does not need to be >= your segment data size.

good catch, fixing it

Hello, LinkedIn is hosting a virtual meetup for Pinot on Sept 2, 6pm. Please join the meetup page https://www.meetup.com/apache-pinot/events/272495311/. Registration link in the

Details

section.

Note, in order to get the zoom link, please register at https://linkedinpinotmeetup.splashthat.com

Hi all, We’re planning to add a feature to move segments from realtime table to offline table. The motivation is to eliminate the need for user to setup their own offline flows for a hybrid table, and let Pinot manage that. Here’s the design doc, called Pinot managed offline flows, if anyone wants to take a look : https://docs.google.com/document/d/1-e_9aHQB4HXS38ONtofdxNvMsGmAoYfSnc2LP88MbIc/edit?usp=sharing

Great article by @User on ingesting parquet files in S3 into Pinot via Spark. https://medium.com/apache-pinot-developer-blog/leverage-plugins-to-ingest-parquet-files-from-s3-in-pinot-decb12e4d09d

Pinot is taking long time to import data when the data size is huge. I am using "standalone" data load job. Trying with 80GB TPCH Lineitem data split into 600 files(each file is around 130MB). Creating segment file is taking around 3 hours on a 4 CPU 64GB RAM machine. Is this expected behavior?

How many controllers do you have? Are you pushing files sequentially?

Using deep-store with segment-uri push will help reduce the time, by avoiding to have to push the actual payload

Two controllers

Can you help in finding the documentation for "deep-store with segment-uri push" ?

Here's a sample: https://docs.pinot.apache.org/basics/data-import/pinot-file-system/import-from-hdfs#push-hdfs-segment-to-pinot-controller

Is option other than using HDFS?

yeah, you can also use gcs, s3

Ok,thanks

May I ask what are you trying to achieve? Is this a benchmark?

Actually, looks like even when using deep-store, the controller may still need to download the segments (metadata push may not be supported yet)

Ok, then we need to understand where the time is spent. Is it on index generation or actual push?

Indexing is taking time. Push is relatively faster.

Hmm then it should be easy to make the job multi threaded, if it isn’t already

There is a parallelism setting

Hi Apache Pinot Community, friendly reminder to signup for Apache Pinot virtual meetup on Sept 2. Hope to see you there: https://linkedinpinotmeetup.splashthat.com/

Hi, do we publish the latest jars from master to maven or only released ones? Needn’t be maven central but any other repo.

only release versions are published to maven

you can always publish other versions to your internal artifactory

Hi all! The one and only @User is presenting on Pinot and Kafka at the virtual Kafka Summit conference. If you signed up for free and can access the talk live, please tune in and support one of our own. https://twitter.com/ApachePinot/status/1298331477315391488?s=20

hello

Hi @User

Hi Everyone 🙂