https://pinot.apache.org/ logo
Join Slack
Powered by
I'm seeing what I believe are very large latency (...
# general
c
I'm seeing what I believe are very large latency (a few hundreds MS) for processing some test queries in a local/test Pinot 0.5 setup and I'm wondering what might be the cause for that. Can anyone please advise if I'm doing something very wrong in this setup: https://www.dropbox.com/s/dio3zi5w1vdmclo/pinot-questions.txt?dl=0 ?
m
Can you share the broker response
👍 1
c
Yeah. I'll get that. Just a sec
I updated the file in Dropbox. But here are some sample lines:
2020/10/15 122701.335 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-0] requestId=1,table=tpch_lineitem_OFFLINE,timeMs=615,docs=49313307/49313307,entries=0/49313307,segments(queried/processed/matched/consuming/unavailable):7/7/7/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);localhost_O=28,420,359,0,query=select sum(l_extendedprice) from tpch_lineitem LIMIT 1000 2020/10/15 122752.097 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-7] requestId=8,table=tpch_lineitem_OFFLINE,timeMs=5,docs=1000/49313307,entries=0/1000,segments(queried/processed/matched/consuming/unavailable):7/1/1/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);localhost_O=1,3,8337,0,query=select l_extendedprice from tpch_lineitem LIMIT 1000 2020/10/15 122816.374 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-13] requestId=14,table=tpch_lineitem_OFFLINE,timeMs=631,docs=49313307/49313307,entries=0/49313307,segments(queried/processed/matched/consuming/unavailable):7/7/7/0/0,consumingFreshnessTimeMs=0,servers=1/1,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);localhost_O=1,629,8348,1,query=select l_extendedprice from tpch_lineitem order by l_extendedprice LIMIT 1000
m
Note, some of your queries are scanning the entire data: 
49313307/49313307
c
Yeah. So, should I expect that kind of latency for this scenario?
m
Some of the TPCH queries just do selection (no aggr, or group-by). While those queries are certainly supported, we have mainly focused on optimizing for aggr/group-by.
Depending on whether it is actually the CPU work-done, or there are other factors such as paging etc, you might be able to further reduce by playing around with JVM/GC settings
c
Thanks @Mayank! This is exactly the work I'm doing now. I.e., check whether there is a GC (memory) or CPU bottleneck
As a follow up about the use of aggregation operation in queries. If I send this query "select sum(l_extendedprice) from tpch_lineitem" it takes about 700ms
Do you think this dataset is too large for a single server?
m
What's the total segment size, and the JVM settings?
c
Dataset: TPCH - 7 Segments of about 400MB - About 5M rows
Pinot: 0.5 OpenJDK: 1.8.0_252 JVM Args: -Xms8G -Xmx8G -XX:MaxDirectMemorySize=4G -XX:+ManagementServer -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=200 -XX:+ParallelRefProcEnabled -XX:+PrintAdaptiveSizePolicy -XX:+PrintGC -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintGCCause -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:log.gc
m
400MB per segment?
c
I'm using JDK 1.8 because it's what the client uses.
Yeah, at least the on disk file is about 400MB
m
Let's move to #troubleshooting ?
👍 1
Open in Slack
PreviousNext
Powered by