you guys are awesome, thank you! 🙇

Would be interested in learning more about your use case @User

excellent question. my domain is cybersecurity (bro/zeek logs). i’m specifically planning on loading lots of

conn

logs (https://docs.zeek.org/en/stable/scripts/base/protocols/conn/main.bro.html)

data looks like:

1258790493.773208	CcH8zVkCER7UopU1j	192.168.1.104	137	192.168.1.255	137	udp	dns	3.748891	350	0	S0	-	-	0	D	7	546	0	0	-	00:0b:db:4f:6b:10	ff:ff:ff:ff:ff:ff	-	-
1258790451.402091	CMucAv3aPcRqfSv8Dj	192.168.1.106	138	192.168.1.255	138	udp	-	-	-	-	S0	-	-	0	D	1	229	0	0	-	00:0b:db:63:5e:a7	ff:ff:ff:ff:ff:ff	-	-
1258790493.787448	CLa6hu4FVZF6WEkdFf	192.168.1.104	138	192.168.1.255	138	udp	-	2.243339	348	0	S0	-	-	0	D	2	404	0	0	-	00:0b:db:4f:6b:10	ff:ff:ff:ff:ff:ff	-	-
1258790615.268111	C4MQVQ3FXAlnFCOgX8	192.168.1.106	137	192.168.1.255	137	udp	dns	3.764626	350	0	S0	-	-	0	D	7	546	0	0	-	00:0b:db:63:5e:a7	ff:ff:ff:ff:ff:ff	-	-
1258790615.289842	CpVVNr3W2hrDoSO0C2	192.168.1.106	138	192.168.1.255	138	udp	-	2.251581	348	0	S0	-	-	0	D	2	404	0	0	-	00:0b:db:63:5e:a7	ff:ff:ff:ff:ff:ff	-	-
1258790620.442107	C9cjp71VsiuBn6w6P9	192.168.1.104	68	192.168.1.1	67	udp	dhcp	0.034866	311	300	SF	-	-	0	Dd	1	339	1	328	-	00:0b:db:4f:6b:10	00:19:e3:e7:5d:23	-	-

(full example file at https://github.com/dgunter/ParseZeekLogs/blob/master/examples/conn.log, for the curious)

ppl tend to put this data into elasticsearch, but performance is terrible and it seems to be a poor fit for the data model

I see, thanks for sharing

of course… i’ll try to write a blog post about it

any advice on hardware sizing requirements? i.e., how much RAM, how many nodes, etc.?

i’m using the offline quickstart for now, so it’s all on one box

What's your latency/throughput requirements?

all data is offline. query latency would ideally be <5 seconds. throughput is low: a few queries a minute at most

most queries are of the

COUNT.... GROUP BY

variety

And data size?

You could go SSD routes with limited RAM (say 64GB)

also, try star-tree for frequently used dimensions in filter and group by

unfortunately, our cluster only has HDDs 😞

but yes, i totally plan on using star indices. read the paper. very interesting

we have lots of RAM, though. 64GB is fiine

yeah, with HDD's star tree will help a lot more

@User: is there a guide for configuring/enabling star tree indices? i don’t see much at https://pinot.readthedocs.io/en/latest/index_techniques.html#star-tree-index

<!here>: should i use pinot from

master

or use the 0.1 release from March?

there was this commit which had instructions

but it was reverted

@User why did you revert this?

use master

@User Because we wanted to make the engineer blog, so pulled it off temporary. Will revisit that and submit another pr

Quick question: during the offline segment push - it looks like if the segment already exists and that the CRC is the same : we still update the segment metadata in Zk. Why is this done exactly ? FYI: this is causing issues in our setup where there's no shared filesystem mounted on the controllers and we haven't moved to using HDFS filesystem yet. So what happens is if there is a redundant upload request by the Spark job on a different controller, the Zk download URI is modified to this new controller - which doesn't have the segment 😞 😞 😞