https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • y
    ill try making it more like the format you mentioned.
  • x
    hmm so file name is 
    00000_0
    ?
  • x
    here is some sample layout:
  • x
  • y
    I am getting the same issue after converting it into this format.
  • y
    Copy code
    <hdfs://bigredns/user/Z00290G/guest_sdr_gst_data_sgl/2019/05/18/guest_sdr_gst_data_sgl_data_2019-05-18.avro>
  • y
    Copy code
    Unable to extract out the relative path based on base input path: <hdfs://bigredns/user/Z00290G/guest_sdr_gst_data_sgl>
  • y
    I was able to log the inputfile and output dir with it.
  • y
    Copy code
    java.lang.IllegalStateException: Unable to extract out the relative path based on base input path: <hdfs://bigredns/user/Z00290G/guest_sdr_gst_data_sgl> inputFile: hdfs:/user/Z00290G/guest_sdr_gst_data_sgl/2019/05/21/guest_sdr_gst_data_sgl_data_2019-05-21.avro outputDir: <hdfs://bigredns/user/Z00290G/guest_sdr_gst_data_sgl_staging>
  • y
    it is removing the 
    <hdfs://bigredns>
    from the input file.
  • x
    I wrote a simple test
    Copy code
    @Test
  public void testGetRelativePath() {

    System.out.println(SegmentGenerationUtils.getRelativeOutputPath(
        URI.create("<hdfs://bigredns/user/Z00290G/guest_sdr_gst_data_sgl>"),
        URI.create("<hdfs://bigredns/user/Z00290G/guest_sdr_gst_data_sgl/2019/05/18/guest_sdr_gst_data_sgl_data_2019-05-18.avro>"),
        URI.create("<hdfs://bigredns/user/Z00290G/guest_sdr_gst_data_sgl_staging>")));
  }
  • x
    the relative output is 
    <hdfs://bigredns/user/Z00290G/guest_sdr_gst_data_sgl_staging/2019/05/18/>
  • x
    oh, I think the issue is that 
    inputFile
    is not under 
    base input path
  • x
    I’m using 
    URI relativePath = baseInputDir.relativize(inputFile);
    to get the relative path
  • x
    which will be empty in this case
  • e
    We upgraded to pinot-0.4.0 and one of the tables is not ingesting. The external view shows the bad segments and we see error messages like this:
    Copy code
    2020/07/23 22:02:12.981 ERROR [LLRealtimeSegmentDataManager_oas_integration_operation_event__2__1309__20200723T2158Z] [oas_integration_operation_event__2__1309__20200723T2158Z] Holding after response from Controller: {"buildTimeSec":-1,"isSplitCommitType":false,"status":"NOT_SENT","offset":-1}
  • e
    and
    Copy code
    ERROR [ServerSegmentCompletionProtocolHandler] [oas_integration_operation_event__1__1309__20200723T2200Z] Could not send request <http://pinot-controller-1.pinot-controller-headless.pinot.svc.cluster.local:9000/segmentConsumed?name=oas_integration_operation_event__1__1309__20200723T2200Z&offset=238845637&instance=Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098&reason=rowLimit&rowCount=1731228>
"
  • e
    Now when I send the same request I get the following message:
  • e
    Copy code
    {
  "status": "FAILED",
  "isSplitCommitType": false,
  "streamPartitionMsgOffset": null,
  "buildTimeSec": -1,
  "offset": -1
}
  • e
    Is there a way we can delete the realtime segments that show up in error state in the external view and get them to be reingested?
  • k
    there is a validation manager that will fix stuck segments
  • x
    segment deletion api should work for deleting the current consuming segment
  • x
    Controller should fix this
  • e
    Thanks! So I just delete those segments?
  • x
    do you get any meaningful log from the controller-1?
  • x
    Yes, you can just delete those
  • x
    so all the consuming segments are stopped?
  • k
    @Elon can probably help you on this.
    👍 1
  • e
    Thanks for the help, now we're getting a "cannot build segment" for the realtime table, only ingested a few rows. Is there something with the spec that needs to be updated to be compatible with pinot-0.4.0?
  • e
    Untitled
    Untitled
1...122123124...166Latest