Apache Pinot

image.png

but if I do the query with another value in limit, everything is ok

by example" SELECT player_nr, processTime, id FROM transaction_line_REALTIME LIMIT 21474836"

You are pulling a lot of data from Pinot 

Pinot is not meant to be used to pull all the data out 

Hi, there are only 3000 rows now in the table

it was working few days, and stop to work,

That should have worked... what’s the jvm memory 

I have updated now to this configuration: - JAVA_OPTS=-Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:gc-pinot-server.log

if I dont the query : select * from transaction_line limit 2147483647, it is not working

Hi, I was playing with Pinot and noticed something strange with group by on certain column (some results are missing).

```select visitor_id, count(*) from visitor_ids
group by visitor_id
order by count(*) desc```

gives:
```visitor_id	count(*)
78144a788f39b7075bb37758	44
979ba2f45c229ab908bdd829	26
db5e6a7b6b48b64d3df838a7	22
3402d34497b08b5d0172128f	21
4f212877c876191c7db40a11	20```
But if specify missing visitor_id, it is there
```select visitor_id, count(*) from visitor_ids
where visitor_id = '6d797e84d6d19d5825ef7060'
group by visitor_id
order by count(*) desc```
gives:
```visitor_id	count(*)
6d797e84d6d19d5825ef7060	66```


column.visitor_id.cardinality = 148594
column.visitor_id.totalDocs = 190000

I think we limit 100k unique visitors within a segment

if that visitor id is not found in first 100k unique ids we see in a segment it wont show up

<@UDQU92KBK> can you confirm the above behaviour^^

Yes, default number of group limits to 100k

<@UDRJ7G85T> Thank you, works as expected.

we plan to make this a bit smarter but still have a limit

Any tips on speeding up a `LoadDataIngestionJob`? I’ve tried upping Xmx and my pods have plenty of CPU and memory on the table. I notice in the logs it looks like it’s only using one thread but haven’t seen any settings in the documents to give more threads.

Also wondering how to distribute the ingestion load across `server` instances. I’ve been ssh’ing into the pod and running `LaunchDataIngestionJob` but this only runs it on the server instance I’m logged into.

the new minion task based ingestion with the new `SegmentGenerationAndPushTask` task will allow you to run scalable ingestion jobs accross multiple minion instances.

Sadly the documentation is not ready yet :disappointed:

That’s interesting thanks for sharing! I’m fine with using a single server instance right now just trying to figure out how to speed it up.

Using this new mechanism you can have 1 worker generate 1 segment in parrallel per file ingested