Ya so once we increase the pv size should the segment be manually uploaded to that server?

It should be ONLINE in the ideal state. So a server restart should fix it.

Oh okay.. that should do.. thanks a lot Mayank and Sidd👍

Hi, I am consuming data into pinot using two kafka nodes and I notice data from lot of partitions is missing (I have 64 partitions currently). I am setting following kafka related properties, initially I had the

stream.kafka.zk.broker.url

missing, added it later on. I have number of replicas set to

1

and no tagsOverrideConfig

"streamType": "kafka",
        "stream.kafka.consumer.type": "LowLevel",
        "stream.kafka.topic.name": "<TOPIC>",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.broker.list": "<node1>:9092,<node2>:9092",
        "stream.kafka.zk.broker.url": "<zk>:2191",

When I run a select count(distinct()) query on a field whose value is different for data received from each partition, I see that segments queries data shows

"numSegmentsQueried": 14,

where there are 64 kafka partitions and also /segments/{tableName} shows ~68 segments

Wondering if anyone knows if the issue is in kafka consumer setting?

Looking into it

Can you paste the ideal state

Untitled

The ideal state looks correct. What is the query you sent? Is the data partitioned by a column?

select count(distinct(<column_name>)) from <table>

column_name % 64 is by which producer decides the kafka partition to send the data into

Without any filter the query should hit all segments

Can you paste the external view of the table?

Untitled

Ah this is interesting, there’s some segments in ERROR state

And I only see 32 segments here

You might need to open the server log to see what's going wrong with the ERRORed segments

yup yup trying that

curious what does externalview imply?

You can think of Pinot cluster management as a state machine

Ideal state is the desired state, external view is the actual state

What command did you use to get the ideal state/external view?

swagger api

on the controller

ah there’s a

Caused by: java.lang.OutOfMemoryError: Direct buffer memory

I have two servers consuming for this kafka topic and ideal size set to 150MB and initial segments sizes are ~70MB per partition So, that leaves me at max (150 * 32) or if we need both the segments while swapping (150 * 64) ~9.6G Machine size is 16G and I didn’t change the default setting of the pinot servers

Any suggestions on how to think about amount of memory to allocate? or machine size?

@Neha Pawar can you point him to the provisioning tool

I don't think there's a doc for that, but this blog has all the details: https://engineering.linkedin.com/blog/2019/auto-tuning-pinot

thanks will go over it