https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • k

    Kishore G

    06/24/2020, 10:35 PM
    @Neha Pawar do we still need that property, looks like its not adding any value
  • k

    Kishore G

    06/24/2020, 10:36 PM
    maybe it was added to roll out the feature in a controlled manner
  • n

    Neha Pawar

    06/24/2020, 10:36 PM
    actually i dont know why we have that prop. i can check and remove it if i dont find any use
  • k

    Kishore G

    06/24/2020, 10:36 PM
    cool, lets create an issue and remove it
  • n

    Neha Pawar

    06/24/2020, 10:58 PM
    created issue: https://github.com/apache/incubator-pinot/issues/5615
  • d

    Dan Hill

    06/26/2020, 5:25 PM
    Hi. I'm getting the following error related to my realtime table ingesting events from kafka.
    insertions
    is just a long metric column. It's not a dimension. I'm not sure why it's being hitting this.
    Copy code
    Metrics aggregation cannot be turned ON in presence of dictionary encoded metrics, eg: insertions
  • s

    Sidd

    06/26/2020, 5:26 PM
    For realtime metrics aggregation, all metric columns should be noDictionary
  • m

    Mayank

    06/26/2020, 5:26 PM
    This ^^
  • m

    Mayank

    06/26/2020, 5:26 PM
    Also dimension columns cannot be no dict
    ➕ 1
  • s

    Sidd

    06/26/2020, 5:31 PM
    we should probably consider adding support for realtime metrics aggregation with dictionary encoded metric columns and remove this restriction unless it is infeasible
  • k

    Kishore G

    06/26/2020, 5:32 PM
    we can but it inefficient
  • d

    Dan Hill

    06/26/2020, 5:35 PM
    I don't think these metrics need to be dictionary encoded. I assumed metric columns by default would not be dictionary encoded.
  • s

    Sidd

    06/26/2020, 5:36 PM
    all columns by default are dictionary encoded AFAIK
  • d

    Dan Hill

    06/26/2020, 5:37 PM
    Looks like the docs say that. I'll update my config. Thanks!
    👍 1
  • d

    Dan Hill

    06/26/2020, 5:44 PM
    Separate topic, my
    pinot-server-0
    log is showing that my params in
    streamConfig
    were
    supplied but isn't a known config
    . Any ideas?
    Copy code
    2020/06/26 17:26:05.729 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.hlc.zk.connect.string' was supplied but isn't a known config.
    2020/06/26 17:26:05.729 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a known config.
    2020/06/26 17:26:05.737 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
    2020/06/26 17:26:05.737 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'streamType' was supplied but isn't a known config.
    ...
    Here is what my table config looks like (with some repetitive parts removed)
    Copy code
    {
      "tableName": "metrics",
      "tableType": "REALTIME",
      "segmentsConfig": {
        "timeColumnName": "timestamp",
        "timeType": "MILLISECONDS",
        "retentionTimeUnit": "DAYS",
        "retentionTimeValue": "5",
        "segmentPushType": "APPEND",
        "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
        "schemaName": "metrics",
        "replication": "1",
        "replicasPerPartition": "1" 
      },  
      "tableIndexConfig": {
        "loadMode"  : "MMAP",
        "starTreeIndexConfigs": [
          {   
            "dimensionsSplitOrder": [
              "platform_id",
              ... my other dimensions
            ],  
            "skipStarNodeCreationForDimensions": [
            ],  
            "functionColumnPairs": [
              "SUM__insertions",
              ... my other metrics
            ]   
          }   
        ],  
        "streamConfigs": {
          "streamType": "kafka",
          "stream.kafka.consumer.type": "simple",
          "stream.kafka.topic.name": "metrics-realtime",
          ... all of the configs are ignored
        }   
      },  
      "tenants": {}, 
      "metadata": {
        "customConfigs": {}
      }   
    }
  • n

    Neha Pawar

    06/26/2020, 5:46 PM
    those are coming from the kafka consumer. because we put the entire streamConfigs map from pinot into the consumer configs. those should be harmless.
  • k

    Kishore G

    06/26/2020, 5:47 PM
    a good beginner task to fix
  • d

    Dan Hill

    06/26/2020, 5:52 PM
    Cool. I was hitting an issue where my Pinot server temporarily stopped ingesting events from Kafka and was trying to figure out if there was an issue. It's working again and still logging those lines.
  • p

    Pradeep

    06/26/2020, 8:47 PM
    Hi, I am having trouble querying the recent data from Pinot,
    select * from <tablename> order by timestamp limit 10
    returns empty. I also see some of the old segments to be in error state from the broker logs
    Copy code
    Resource: <tablename>_REALTIME, partition: <tablename>__7__0__20200625T1913Z is in ERROR state
    Resource: <tablename>_REALTIME, partition: <tablename>__8__0__20200625T1913Z is in ERROR state
    Resource: <tablename>_REALTIME, partition: <tablename>__9__0__20200625T1913Z is in ERROR state
  • p

    Pradeep

    06/26/2020, 8:47 PM
    Wondering how do I go ahead about debugging this?
  • m

    Mayank

    06/26/2020, 8:48 PM
    The server log should show you what the segment went into error state.
  • p

    Pradeep

    06/26/2020, 8:49 PM
    Above is from server logs, for example:
    <tablename>__9__0__20200625T1913Z
    is one of the segment that went into error state (though it’s an older segment)
  • p

    Pradeep

    06/26/2020, 8:49 PM
    I don’t see any logs on recent segments
  • m

    Mayank

    06/26/2020, 8:51 PM
    Can you grep for one of the segments in error state in all the server logs (if you have separate logs per day)? It should show some error/exeception that caused this to happen.
  • m

    Mayank

    06/26/2020, 8:51 PM
    Alternatively, you can try to re-start the server and see if the segment still gets into ERROR state.
  • p

    Pradeep

    06/26/2020, 8:55 PM
    tried restarting, seems like a helix issue let me dig a bit more
  • m

    Mayank

    06/26/2020, 8:56 PM
    Are these old segments still within your retention period?
  • p

    Pradeep

    06/26/2020, 8:57 PM
    yeah these are within ttl
  • p

    Pradeep

    06/26/2020, 9:05 PM
    for exmple one of the exceptions I see in the server logs
    Copy code
    Sleep for 10000ms as service status has not turned GOOD: IdealStateAndCurrentStateMatchServiceStatusCallback:partition=<tablename>__57__0__20200625T191
    3Z, expected=ONLINE, found=OFFLINE, creationTime=1593202611703, modifiedTime=1593202650032, version=95, waitingFor=CurrentStateMatch, resource=<tablename>_REALTIME, numResourcesLeft=1, numTotalResources=1, minStartCount=1,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;
    Caught exception in state transition from OFFLINE -> ONLINE for resource: <tablename>_REALTIME, partition: <tablename>__57__0__20200625T1913Z
    java.lang.RuntimeException: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts
            at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:286) ~[pinot-all-0
    .4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
            at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:252) ~[pinot-all-0.4.0-jar-with-d
    ependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
            at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:132) ~[pinot-all-0.4.0-jar-with
    -dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
            at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
            at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[?:?]
  • k

    Kishore G

    06/26/2020, 9:13 PM
    did you setup deep store for Pinot?
1...116117118...166Latest