Apache Pinot #troubleshooting

Join Slack

Damiano

05/05/2020, 2:16 PM

build passed

Damiano

05/05/2020, 2:16 PM

running the Quickstart.java

Damiano

05/05/2020, 2:18 PM

quickstart.java with no errors... Pinot is running

Mayank

05/05/2020, 2:26 PM

Neha Pawar

05/05/2020, 4:33 PM

@Ranveer Singh are there any exceptions in the create segment log?

Neha Pawar

05/05/2020, 4:34 PM

also, the DateTimeFieldSpecs is not stable and ready yet. To unblock yourself, you can model them as dimensions for now

Kishore G

05/05/2020, 4:35 PM

I think he is complaining about validateSchema call

Ranveer Singh

05/05/2020, 4:36 PM

I feel , somehow these are getting ignored by ingestion. When i didn't see these columns post ingestion , i thought scheam may has issue and i used Validate Schema . As part of response , i dont see dimensionFieldSpecs as well. I am not sure that is bug

Neha Pawar

05/05/2020, 4:37 PM

let me try with your schema

Ranveer Singh

05/05/2020, 4:38 PM

i build 0.4 code as well locally. I am trying using that but same response

Neha Pawar

05/05/2020, 4:44 PM

So there's 2 problems: 1. With this schema when you tried to create a segment, your segment doesn't have dimensionFieldSpecs and dateTimeFieldSpecs 2. When you validate the schema, you don't see dateTimeFieldSpec and dimensionsFieldSpecs

Neha Pawar

05/05/2020, 4:44 PM

correct?

Neha Pawar

05/05/2020, 4:48 PM

regarding the issue with validate, your schema has a typo. It should be "dateTimeFieldSpecs" - plural, not "dateTimeFieldSpec"

Neha Pawar

05/05/2020, 4:49 PM

I can see the dateTimeFieldSpecs in the schema now, after i corrected that @Ranveer Singh

Neha Pawar

05/05/2020, 4:49 PM

this will likely fix the ingestion issue, but i'm trying it out to confirm

Neha Pawar

05/05/2020, 4:51 PM

could you give me sample values for this pattern?

EEE MON dd HH:mm:ss UTC yyyy

- i'm not sure this is following the convention correctly

Neha Pawar

05/05/2020, 4:55 PM

Perhaps it should be

EEE MMM dd HH:mm:ss Z yyyy

: https://stackoverflow.com/questions/19861642/date-format-parse-exception-eee-mmm-dd-hhmmss-z-yyyy

Ranveer Singh

05/05/2020, 5:02 PM

This is not correct one.. I sent u old one.. EEE MON dd HHmmss UTC yyyy

Ranveer Singh

05/05/2020, 5:03 PM

correct one is SIMPLE_DATE_FORMAT:EEE MMM dd HHmmss z yyyy is noticed post this .. Thanks

Ranveer Singh

05/05/2020, 5:21 PM

Thanks Neha. Its working now. dimensionFieldSpecs is there .. Looks like problem was plural .. Thanks for correcting that

Neha Pawar

05/05/2020, 5:23 PM

here's my dateTimeFieldSpecs section:

Copy code

"dateTimeFieldSpecs": [
    {
      "name": "StatusRecModifyTS",
      "dataType": "STRING",
      "format": "5:MINUTES:SIMPLE_DATE_FORMAT:EEE MMM dd HH:mm:ss z yyyy",
      "granularity": "5:MINUTES"
    },
    {
      "name": "StatusRecCreateTS",
      "dataType": "STRING",
      "format": "5:MINUTES:SIMPLE_DATE_FORMAT:EEE MMM dd HH:mm:ss z yyyy",
      "granularity": "5:MINUTES"
    },
    {
      "name": "EmailSendCreate",
      "dataType": "STRING",
      "format": "5:MINUTES:SIMPLE_DATE_FORMAT:EEE MMM dd HH:mm:ss z yyyy",
      "granularity": "5:MINUTES"
    }
  ]

Ranveer Singh

05/05/2020, 5:25 PM

I changed it.. I thought it will internally convert to millis but it was throwing numberformat. I checked the code and accordingly chnaged to String. thanks for clarifying

Neha Pawar

05/05/2020, 5:27 PM

if your goal is to convert those time columns into millis, you can do that after the PR I shared. After that code is in, you can specify a transformFunction like this -

Copy code

{
      "name": "newTimeColumn",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "5:MINUTES",
      "transformFunction": "fromDateTime(StatusRecModifyTS, "5:MINUTES:SIMPLE_DATE_FORMAT:EEE MMM dd HH:mm:ss z yyyy")"
    }

Ranveer Singh

05/05/2020, 5:32 PM

great thank you

Dan Hill

05/05/2020, 10:07 PM

I'm hitting a weird issue with SUM when the Pinot Data Explorer. If I look at the values of a long metric individually, I see 0 or 1. When I

sum(value)

I get a buggy looking number.

Copy code

select impressions, count(*) from events_testing group by impressions

Copy code

"rows": [
            [1, 1492],
            [0, 169]
        ]

However when I get a sum, the value is very tiny.

Copy code

select sum(impressions) from events_testing

Copy code

"rows": [
            [5.32150281E-315]
        ]

I looked. My data just has 0 or 1 in it. The sum converts the aggregation to a DOUBLE, which seems broken.

Kishore G

05/05/2020, 10:14 PM

there is some problem with the data ingested?

Kishore G

05/05/2020, 10:15 PM

Copy code

select sum(impressions) from events_testing where impressions IN (0,1)

Dan Hill

05/05/2020, 10:16 PM

Hmm, weird, I updated Pinot to a more recent image and it fixed it.

Neha Pawar

05/06/2020, 5:06 PM

Moving this question from DM to here, to get help: @kish is trying to create a segment from files in parquet format. He's seeing this. Any ideas?

Copy code

Finished records indexing in IndexCreator!
Got exception to kick off standalone data ingestion job -
<http://java.io|java.io>.IOException: Input/output error
    at java.nio.MappedByteBuffer.force0(Native Method) ~[?:1.8.0_232]
    at java.nio.MappedByteBuffer.force(MappedByteBuffer.java:203) ~[?:1.8.0_
232]
    at org.apache.pinot.core.segment.memory.PinotByteBuffer.flush(PinotByteB
uffer.java:327) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPS
HOT-7fd2110de66c2bb0b49ee340bac492fafe1f0693]
    at org.apache.pinot.core.segment.memory.PinotDataBuffer.close(PinotDataB
uffer.java:271) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPS
HOT-7fd2110de66c2bb0b49ee340bac492fafe1f0693]
    at <http://org.apache.pinot.core.io|org.apache.pinot.core.io>.util.PinotDataBitSet.close(PinotDataBitSet.j
ava:247) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-7fd
2110de66c2bb0b49ee340bac492fafe1f0693]
    at <http://org.apache.pinot.core.io|org.apache.pinot.core.io>.util.FixedBitIntReaderWriter.close(FixedBitI
ntReaderWriter.java:67) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4
.0-SNAPSHOT-7fd2110de66c2bb0b49ee340bac492fafe1f0693]
    at <http://org.apache.pinot.core.io|org.apache.pinot.core.io>.writer.impl.v1.FixedBitSingleValueWriter.clo
se(FixedBitSingleValueWriter.java:45) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-depend
encies.jar:0.4.0-SNAPSHOT-7fd2110de66c2bb0b49ee340bac492fafe1f0693]
    at org.apache.pinot.core.segment.creator.impl.fwd.SingleValueUnsortedFor
wardIndexCreator.close(SingleValueUnsortedForwardIndexCreator.java:47) ~[pinot-a
ll-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-7fd2110de66c2bb0b49ee
340bac492fafe1f0693]
    at org.apache.pinot.common.utils.FileUtils.close(FileUtils.java:95) ~[pi
not-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-7fd2110de66c2bb0
b49ee340bac492fafe1f0693]
    at org.apache.pinot.core.segment.creator.impl.SegmentColumnarIndexCreato
r.close(SegmentColumnarIndexCreator.java:577) ~[pinot-all-0.4.0-SNAPSHOT-jar-wit
h-dependencies.jar:0.4.0-SNAPSHOT-7fd2110de66c2bb0b49ee340bac492fafe1f0693]
    at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriver
Impl.handlePostCreation(SegmentIndexCreationDriverImpl.java:223) ~[pinot-all-0.4
.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-7fd2110de66c2bb0b49ee340bac
492fafe1f0693]

Neha Pawar

05/06/2020, 5:09 PM

@Jackie @Jack ^^