<https apache pinot slack com archives CDRCA57FC p1602711095 Apache Pinot #troubleshooting

<https://apache-pinot.slack.com/archives/CDRCA57FC...

murat migdisoglu

10/14/2020, 9:49 PM

https://apache-pinot.slack.com/archives/CDRCA57FC/p1602711095152100 I'm following up my thread related to the real time ingestion here..

Neha Pawar

10/14/2020, 9:51 PM

can you share your table config and schema?

murat migdisoglu

10/14/2020, 9:58 PM

revenue-schema.json,revenue-table-rt.json

revenue-table-rt.json revenue-schema.json

Neha Pawar

10/14/2020, 10:01 PM

the issue might be the schema name. the table config has

"schemaName": "revenue",

whereas the schema has

"schemaName": "revenue_test_murat",

Neha Pawar

10/14/2020, 10:01 PM

Does your pinot-server log show absolutely no warning/error/exception message?

Neha Pawar

10/14/2020, 10:02 PM

and which version of Pinot are you on? The newer versions should have blocked creating the table config with misssing schema

murat migdisoglu

10/14/2020, 10:05 PM

I'll reverify your point with schema mismatch

murat migdisoglu

10/14/2020, 10:07 PM

we're running 0.5.0

murat migdisoglu

10/14/2020, 10:18 PM

ok I verified the schema @Neha Pawar the table's schema is revenue_test_murat

murat migdisoglu

10/14/2020, 10:18 PM

I'm tailing the server log

murat migdisoglu

10/14/2020, 10:19 PM

and it doesn't print anythin

Neha Pawar

10/14/2020, 10:21 PM

did you delete and recreate table config after fixing the schema name?

murat migdisoglu

10/14/2020, 10:22 PM

I did and I can retry.. But if that was the issue, why would it ingest only the first 50000 rows? "segment.flush.threshold.size": "50000"

Neha Pawar

10/14/2020, 10:23 PM

because it’s not able to complete the segment. the consumer consumed 50k rows based on this config and is not able to move forward coz segment creation is failing

murat migdisoglu

10/14/2020, 10:24 PM

batch process works by the way

murat migdisoglu

10/14/2020, 10:24 PM

but its another table(offline) afaik

Neha Pawar

10/14/2020, 10:25 PM

can you share the whole controller and server log

murat migdisoglu

10/14/2020, 10:27 PM

Copy code

Could not build segment
java.lang.NullPointerException: null
        at org.apache.pinot.core.segment.creator.impl.SegmentColumnarIndexCreator.addColumnMetadataInfo(SegmentColumnarIndexCreator.java:535) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.core.segment.creator.impl.SegmentColumnarIndexCreator.writeMetadata(SegmentColumnarIndexCreator.java:489) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.core.segment.creator.impl.SegmentColumnarIndexCreator.seal(SegmentColumnarIndexCreator.java:399) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:240) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:223) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:127) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:742) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:693) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:604) [pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-d87bbc9032c6efe626eb5f9ef1db4de7aa067179]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Pr

murat migdisoglu

10/14/2020, 10:27 PM

I've found an error

murat migdisoglu

10/14/2020, 10:28 PM

but after that, no matter how much data comes to kafka, it does not generate any error

Neha Pawar

10/14/2020, 10:29 PM

like i said before, the consumption is going to stop, if it cannot create the segment.

Neha Pawar

10/14/2020, 10:32 PM

Neha Pawar

10/14/2020, 10:32 PM

i dont know if this was included in the 0.5.0. Checking

Neha Pawar

10/14/2020, 10:33 PM

To unblock, you could try without

aggregateMetrics

, or build from source

Neha Pawar

10/14/2020, 10:37 PM

yup, that fix is not part of 0.5.0. Could you build from source?

murat migdisoglu

10/14/2020, 11:00 PM

You're right It worked without aggregation.

murat migdisoglu

10/14/2020, 11:01 PM

For the sake of poc i'll stop here. But im suprised to have a bug in such a fundemantal feature :(

murat migdisoglu

10/14/2020, 11:01 PM

Thx a lot for your help

Neha Pawar

10/14/2020, 11:14 PM

i’m also surprised 🙂 this feature is being used in some places i believe. So my hunch is that it is the combination of

aggregateMetrics: true

columnMinMaxValueGeneratorMode: ALL

. I have a feeling it may work fine if you remove columnMinMaxValueGenerator. And fwiw, it has been fixed on master and will be available in the next release

Neha Pawar

10/14/2020, 11:16 PM

@Mayank this flag is used at LinkedIn right? How does it work inspite of this: https://github.com/apache/incubator-pinot/pull/5862 ? Is there something specific in this table config that might be triggering this?

Mayank

10/14/2020, 11:25 PM

IIRC, this was a bug that got introduced and we hit the same problem at LinkedIn. I believe that #5862 fixed the issue

👍 1

Mayank

10/14/2020, 11:25 PM

@Neha Pawar ^^

Open in Slack

Previous Next