https://pinot.apache.org/ logo
Join Slack
Powered by
Hello, I tried incubator-pinot helm chart and set ...
# general
m
Hello, I tried incubator-pinot helm chart and set replica count for server as 2 . I noticed that all the segments are going to one server and it is used heavily and the other statefulset is kind of doing nothing. Is this some sort of active passive behaviour? Also I scheduled 26G ram with jmx 4G for the server and sometimes the memory reaches max as well. How can i spread the segments and distribute the load? Also which metric can we use for autoscaling?
k
do you have both the servers up and running?
x
is this realtime or offline table? Can you check current online instances and the table idealstate? If it's realtime, what's the kafka topic partition you have.
m
@Kishore G I have both servers up and running
@Xiang Fu it is a realtime table. I went to pinot controller UI and there it says both server alive. I added one more server to make it 3 and can see some segments going to the third one. But the server don't have much logs though.
x
How many Kafka partitions in the topic? Which will translate to the number of consuming segments
m
Topic only have 1 partition
x
I suspect there is only one Kafka partition
m
I want to maintain ordering
Thats why created single partition
@Xiang Fu so if there is single partition then single server will consume it?
What will happen if I create another topic with single partition?
Also I assume the search queries will be served by all the servers?
x
I feel the issue is the real-time segment assignment always go to the first server. Let me check the code.
m
Thanks @Xiang Fu. if thats the case I can create another topic and stream some of the data there. So another Pinot server will pick it up?
x
Yes
Or add replicas ;)
If you want redundant
I will check the segment assignment logic
m
you mean replication and replicasperpartition?
even if I set that to 2 for instance, still the topic will be consumed by single server right?
x
Btw, for ordering, if you have primary key, then you can still have a partitioned Kafka topic
It will be two servers
Which means one segment should go to two Pinot servers
m
ohh ok, make sense. I have primarykey will check kafka side on that.
x
I think the issue is that segments are not evenly distributed for one Kafka topic partition table. We will look into this issue .
m
I am trying to avoid query order by desc
x
ic
But in Pinot query, you still need to do order by
m
ohh..!! I thought Pinot will maintain order
x
Pinot query across multiple segments are not ordering preserved
m
In my test Pinot created around 100 segments. however seems like ordering is maintained however like you said it could be because the results are pulled from a particular segment
I will try with diff queries and see, also any tips to improve performance when using order by ?
k
whats the query type
query pattern
m
@Kishore G it is just order by timemillis
i just tried a few and seems it is performing ok
Thanks @Kishore G and @Xiang Fu
x
I checked the code of method 
updateInstanceStatesForNewConsumingSegment
in 
org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager
. Pinot reuses the same server for same Kafka topic partition consumption. I guess the intention was to reuse objects like Kafka consumers. In your case, since there is only one kafka topic partition and table is 1 replica, then all the segments will be assigned on the same host. One thing you can try is to let Pinot move completed segment to the other servers. See: https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime#moving-completed-segments-to-different-hosts
this requires tag pinot servers first then create the table
m
@Xiang Fu cool this will help, it is good that there is a feature to spread segments.
Open in Slack
PreviousNext
Powered by