https://pinot.apache.org/ logo
Join Slack
Powered by
I am getting failures at the time of segment commi...
# troubleshooting
s
I am getting failures at the time of segment commit on controller and server keeps on retrying indefinitely. I have attached controller and server logs screenshot. (v0.3.0) Table config:
Copy code
{
  "tableName": "transcript",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "timeColumnName": "timestamp",
    "timeType": "MILLISECONDS",
    "schemaName": "transcript",
    "replicasPerPartition": "1"
  },
  "tenants": {},
  "tableIndexConfig": {
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.topic.name": "transcript-topic",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.broker.list": "localhost:9876",
      "realtime.segment.flush.threshold.time": "5m",
      "realtime.segment.flush.threshold.size": "5",
      "stream.kafka.consumer.prop.auto.offset.reset": "largest"
    }
  },
  "metadata": {
    "customConfigs": {}
  }
}
schema
Copy code
{
  "schemaName": "transcript",
  "dimensionFieldSpecs": [
    {
      "name": "studentID",
      "dataType": "INT"
    },
    {
      "name": "firstName",
      "dataType": "STRING"
    },
    {
      "name": "lastName",
      "dataType": "STRING"
    },
    {
      "name": "gender",
      "dataType": "STRING"
    },
    {
      "name": "subject",
      "dataType": "STRING"
    }
  ],
  "metricFieldSpecs": [
    {
      "name": "score",
      "dataType": "FLOAT"
    }
  ],
  "dateTimeFieldSpecs": [
    {
      "name": "timestamp",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ]
}
n
looking
did any segment commit successfully before ? or are you seeing this issue from the start?
Do you have zooinspector installed? can you share segment metadata from PropertyStore/Segments for this failing segment?
s
@Neha Pawar I have attached new snapshot of error along with zooinspector snapshot.
I think commit is failing for me from the start. Generally i had longer commit time or rows so i never checked. I got this issue when i was verifying commit by setting lower values in table config.
n
btw, any reason you’re using 0.3.0 instead of the latest 0.4.0?
and are you using the exact same data as given in the getting started guide?
s
@Neha Pawar No reason for using 0.3.0. Yes, data is same as per guide
Copy code
{"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"Maths","score":3.8,"timestamp":1571900400000}
{"studentID":205,"firstName":"Natalie","lastName":"Jones","gender":"Female","subject":"History","score":3.5,"timestamp":1571900400000}
{"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Maths","score":3.2,"timestamp":1571900400000}
{"studentID":207,"firstName":"Bob","lastName":"Lewis","gender":"Male","subject":"Chemistry","score":3.6,"timestamp":1572418800000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Geography","score":3.8,"timestamp":1572505200000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"English","score":3.5,"timestamp":1572505200000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Maths","score":3.2,"timestamp":1572678000000}
{"studentID":209,"firstName":"Jane","lastName":"Doe","gender":"Female","subject":"Physics","score":3.6,"timestamp":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"Maths","score":3.8,"timestamp":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"English","score":3.5,"timestamp":1572678000000}
{"studentID":211,"firstName":"John","lastName":"Doe","gender":"Male","subject":"History","score":3.2,"timestamp":1572854400000}
{"studentID":212,"firstName":"Nick","lastName":"Young","gender":"Male","subject":"History","score":3.6,"timestamp":1572854400000}
n
i was able to reproduce the issue in 0.3.0, i will investigate. could you try with 0.4.0 ?
s
ok sure
n
just tried 0.4.0 and it works fine
i’ve updated the docs to refer to 0.4.0 in the Getting Started
s
Yeah, 0.4.0 is working for me. Thanks.
n
it was probably not working in 0.3.0 because of “dateTimeFieldSpec” in the schema, which is supported only starting 0.4.0
Open in Slack
PreviousNext
Powered by