https://pinot.apache.org/ logo
#general
Title
# general
m

Matt

11/25/2020, 6:43 PM
Hello, I tried incubator-pinot helm chart and set replica count for server as 2 . I noticed that all the segments are going to one server and it is used heavily and the other statefulset is kind of doing nothing. Is this some sort of active passive behaviour? Also I scheduled 26G ram with jmx 4G for the server and sometimes the memory reaches max as well. How can i spread the segments and distribute the load? Also which metric can we use for autoscaling?
k

Kishore G

11/25/2020, 8:30 PM
do you have both the servers up and running?
x

Xiang Fu

11/25/2020, 9:00 PM
is this realtime or offline table? Can you check current online instances and the table idealstate? If it's realtime, what's the kafka topic partition you have.
m

Matt

11/26/2020, 12:04 AM
@Kishore G I have both servers up and running
@Xiang Fu it is a realtime table. I went to pinot controller UI and there it says both server alive. I added one more server to make it 3 and can see some segments going to the third one. But the server don't have much logs though.
x

Xiang Fu

11/26/2020, 12:07 AM
How many Kafka partitions in the topic? Which will translate to the number of consuming segments
m

Matt

11/26/2020, 12:07 AM
Topic only have 1 partition
x

Xiang Fu

11/26/2020, 12:08 AM
I suspect there is only one Kafka partition
m

Matt

11/26/2020, 12:08 AM
I want to maintain ordering
Thats why created single partition
@Xiang Fu so if there is single partition then single server will consume it?
What will happen if I create another topic with single partition?
Also I assume the search queries will be served by all the servers?
x

Xiang Fu

11/26/2020, 12:12 AM
I feel the issue is the real-time segment assignment always go to the first server. Let me check the code.
m

Matt

11/26/2020, 12:14 AM
Thanks @Xiang Fu. if thats the case I can create another topic and stream some of the data there. So another Pinot server will pick it up?
x

Xiang Fu

11/26/2020, 12:15 AM
Yes
Or add replicas ;)
If you want redundant
I will check the segment assignment logic
m

Matt

11/26/2020, 12:16 AM
you mean replication and replicasperpartition?
even if I set that to 2 for instance, still the topic will be consumed by single server right?
x

Xiang Fu

11/26/2020, 12:16 AM
Btw, for ordering, if you have primary key, then you can still have a partitioned Kafka topic
It will be two servers
Which means one segment should go to two Pinot servers
m

Matt

11/26/2020, 12:18 AM
ohh ok, make sense. I have primarykey will check kafka side on that.
x

Xiang Fu

11/26/2020, 12:18 AM
I think the issue is that segments are not evenly distributed for one Kafka topic partition table. We will look into this issue .
m

Matt

11/26/2020, 12:18 AM
I am trying to avoid query order by desc
x

Xiang Fu

11/26/2020, 12:18 AM
ic
But in Pinot query, you still need to do order by
m

Matt

11/26/2020, 12:19 AM
ohh..!! I thought Pinot will maintain order
x

Xiang Fu

11/26/2020, 12:20 AM
Pinot query across multiple segments are not ordering preserved
m

Matt

11/26/2020, 12:22 AM
In my test Pinot created around 100 segments. however seems like ordering is maintained however like you said it could be because the results are pulled from a particular segment
I will try with diff queries and see, also any tips to improve performance when using order by ?
k

Kishore G

11/26/2020, 12:40 AM
whats the query type
query pattern
m

Matt

11/26/2020, 1:15 AM
@Kishore G it is just order by timemillis
i just tried a few and seems it is performing ok
Thanks @Kishore G and @Xiang Fu
x

Xiang Fu

11/26/2020, 3:40 AM
I checked the code of method
updateInstanceStatesForNewConsumingSegment
in
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager
. Pinot reuses the same server for same Kafka topic partition consumption. I guess the intention was to reuse objects like Kafka consumers. In your case, since there is only one kafka topic partition and table is 1 replica, then all the segments will be assigned on the same host. One thing you can try is to let Pinot move completed segment to the other servers. See: https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime#moving-completed-segments-to-different-hosts
this requires tag pinot servers first then create the table
m

Matt

11/27/2020, 12:44 AM
@Xiang Fu cool this will help, it is good that there is a feature to spread segments.