https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • s

    Shounak Kulkarni

    06/29/2020, 4:33 PM
    Ya so once we increase the pv size should the segment be manually uploaded to that server?
  • m

    Mayank

    06/29/2020, 4:33 PM
    It should be ONLINE in the ideal state. So a server restart should fix it.
  • s

    Shounak Kulkarni

    06/29/2020, 4:35 PM
    Oh okay.. that should do.. thanks a lot Mayank and Sidd👍
  • p

    Pradeep

    06/30/2020, 12:48 AM
    Hi, I am consuming data into pinot using two kafka nodes and I notice data from lot of partitions is missing (I have 64 partitions currently). I am setting following kafka related properties, initially I had the
    stream.kafka.zk.broker.url
    missing, added it later on. I have number of replicas set to
    1
    and no tagsOverrideConfig
    Copy code
    "streamType": "kafka",
            "stream.kafka.consumer.type": "LowLevel",
            "stream.kafka.topic.name": "<TOPIC>",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
            "stream.kafka.broker.list": "<node1>:9092,<node2>:9092",
            "stream.kafka.zk.broker.url": "<zk>:2191",
  • p

    Pradeep

    06/30/2020, 1:07 AM
    When I run a select count(distinct()) query on a field whose value is different for data received from each partition, I see that segments queries data shows
    "numSegmentsQueried": 14,
    where there are 64 kafka partitions and also /segments/{tableName} shows ~68 segments
  • p

    Pradeep

    06/30/2020, 1:09 AM
    Wondering if anyone knows if the issue is in kafka consumer setting?
  • j

    Jackie

    06/30/2020, 1:41 AM
    Looking into it
  • k

    Kishore G

    06/30/2020, 1:45 AM
    Can you paste the ideal state
  • p

    Pradeep

    06/30/2020, 1:52 AM
    Untitled
    Untitled
  • j

    Jackie

    06/30/2020, 2:06 AM
    The ideal state looks correct. What is the query you sent? Is the data partitioned by a column?
  • p

    Pradeep

    06/30/2020, 2:09 AM
    Copy code
    select count(distinct(<column_name>)) from <table>
    column_name % 64 is by which producer decides the kafka partition to send the data into
  • j

    Jackie

    06/30/2020, 2:11 AM
    Without any filter the query should hit all segments
  • j

    Jackie

    06/30/2020, 2:12 AM
    Can you paste the external view of the table?
  • p

    Pradeep

    06/30/2020, 2:14 AM
    Untitled
    Untitled
  • p

    Pradeep

    06/30/2020, 2:14 AM
    Ah this is interesting, there’s some segments in ERROR state
  • p

    Pradeep

    06/30/2020, 2:14 AM
    And I only see 32 segments here
  • j

    Jackie

    06/30/2020, 2:15 AM
    You might need to open the server log to see what's going wrong with the ERRORed segments
  • p

    Pradeep

    06/30/2020, 2:16 AM
    yup yup trying that
  • p

    Pradeep

    06/30/2020, 2:16 AM
    curious what does externalview imply?
  • j

    Jackie

    06/30/2020, 2:18 AM
    You can think of Pinot cluster management as a state machine
  • j

    Jackie

    06/30/2020, 2:18 AM
    Ideal state is the desired state, external view is the actual state
  • j

    Jackie

    06/30/2020, 2:19 AM
    What command did you use to get the ideal state/external view?
  • p

    Pradeep

    06/30/2020, 2:19 AM
    swagger api
  • p

    Pradeep

    06/30/2020, 2:19 AM
    on the controller
  • p

    Pradeep

    06/30/2020, 2:28 AM
    ah there’s a
    Caused by: java.lang.OutOfMemoryError: Direct buffer memory
  • p

    Pradeep

    06/30/2020, 2:31 AM
    I have two servers consuming for this kafka topic and ideal size set to 150MB and initial segments sizes are ~70MB per partition So, that leaves me at max (150 * 32) or if we need both the segments while swapping (150 * 64) ~9.6G Machine size is 16G and I didn’t change the default setting of the pinot servers
  • p

    Pradeep

    06/30/2020, 2:36 AM
    Any suggestions on how to think about amount of memory to allocate? or machine size?
  • k

    Kishore G

    06/30/2020, 2:56 AM
    @Neha Pawar can you point him to the provisioning tool
  • n

    Neha Pawar

    06/30/2020, 4:58 AM
    I don't think there's a doc for that, but this blog has all the details: https://engineering.linkedin.com/blog/2019/auto-tuning-pinot
  • p

    Pradeep

    06/30/2020, 4:52 PM
    thanks will go over it
1...118119120...166Latest