👋 is there any plan to upgrade Kafka version? I was working on to add a prometheus integration with Pinot and found the yammer-metrics (which got deprecated) still have a hard dependency on kafka. The metrics lib is not a reason for the question, but just curious to understand on the latest Kafka support.

is there any best practice on what are the metrics to watch/ alert for Pinot? any shareable runbook for operating the clusters?

@User Pinot works with its own internal segment format, so data will be converted from other formats. We currently don't have support for ORC/Parquet.

👋 I'm frequently getting these exception

Exception while executing a state transition task clog__5__22__20190226T1034Zjava.lang.reflect.InvocationTargetException
java.lang.IllegalArgumentException: Illegal memory allocation 0 for segment clog__5__22__20190226T1034Z column clog__5__22__20190226T1034Z:clog_json.dict\n\tat com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)\n\tat com.linkedin.pinot.core.io.readerwriter.RealtimeIndexOffHeapMemoryManager.allocate(RealtimeIndexOffHeapMemoryManager.java:74)\n\tat com.linkedin.pinot.core.io.writer.impl.MutableOffHeapByteArrayStore$Buffer.<init>(MutableOffHeapByteArrayStore.java:100)\n\tat com.linkedin.pinot.core.io.writer.impl.MutableOffHeapByteArrayStore$Buffer.<init>(MutableOffHeapByteArrayStore.java:85)\n\tat com.linkedin.pinot.core.io.writer.impl.MutableOffHeapByteArrayStore.expand(MutableOffHeapByteArrayStore.java:196)\n\tat com.linkedin.pinot.core.io.writer.impl.MutableOffHeapByteArrayStore.<init>(MutableOffHeapByteArrayStore.java:183)\n\tat com.linkedin.pinot.core.realtime.impl.dictionary.StringOffHeapMutableDictionary.<init>(StringOffHeapMutableDictionary.java:34)\n\tat com.linkedin.pinot.core.realtime.impl.dictionary.MutableDictionaryFactory.getMutableDictionary(MutableDictionaryFactory.java:41)\n\tat com.linkedin.pinot.core.indexsegment.mutable.MutableSegmentImpl.<init>(MutableSegmentImpl.java:146)\n\tat com.linkedin.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.<init>(LLRealtimeSegmentDataManager.java:1080)\n\tat com.linkedin.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:257)\n\tat com.linkedin.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:123)\n\tat com.linkedin.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164)\n\tat com.linkedin.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeConsumingFromOf

logs shjould indicate what happened here. @User if you can grep for segment name from your logs and post what happned, that will help. Also, if you can please let us know what git hash you are running, that can help us with getting to the right code and checking whether a bug is fixed or not.

@User if you can provide the following info that will help a lot: 1. Zookeeper Segment metadata for the consuming segment : Should be in helix-cluster-name/pinot/PROPERTYSTORE/CONFIGS/SEGMENTS/tableName/segmentName 2. TableConfig Thanks

FYI: if the Kafka message size > buffer size of the Pinot simple consumer, it will not consume anything and not log any exception either. Not sure if that's the case here but somethign to check

It’s being addressed here

@User have you been able to make progress on the issues you were facing? Should we do a shared debugging session if required?

no - pinot bug: its the one @User referred to earlier https://github.com/apache/incubator-pinot/pull/3869

typically max message size is set on the broker (not sure if it can be set per topic)

☝️ yes, that is the recommended approach.

<!here> Big Data Meetup at LinkedIn on "Supercharged OLAP on Big Data" - https://superchargedolap.splashthat.com/. Please join us to hear from a great set of speakers

We added the configuration to test default column feature and be able to turn it off if it does not work. Since it has been there for years, we should always enable it and remove the configuration: https://github.com/apache/incubator-pinot/pull/4106

ok. should be simple to add that. I saw someone in mailing list asking for some few things to get started

@User yes, we have a lot of cool undocumented stuff 😉

like put the condition in the filtering condition?

So after this pr, if you have provided your own custom log4j config, you will need to convert the config to log4j2 format. If you used

pinot-admin

command to start the instances, there will be no difference

@User You can write a custom log4j config and pass it to as part of the jvm config. You can use something like

RollingFileAppender

https://howtodoinjava.com/log4j/log4j-rolling-file-appender/ to avoid the issue

Anyone havin any idea : as per the thirdeye documentation "Import metric from Presto/MySQL" should be available on create alerts. Bit code is commented. ??

Is it possible to have the exact same segments on two different clusters but somehow the data row counts are different?

@User Any ideas for schema definition nested Json while consuming from kafka.

Hi. When trying to run kafka stream and ingest mock dataset from Pinot, I face Query had processing exceptions: ! [{"errorCode":410,"message":"BrokerResourceMissingError"}] Any ideas why? Adding the stack trace to the message bellow

@User It seems that the broker is not setup correctly. Can you tell us a bit more about your current pinot setup?

How are you setting up the table config to consume from your Kafka topic

"stream.kafka.hlc.zk.connect.string": "our_kafka_dns:9092"

this should point to zookeeper

Do we have a documentation of supported data types. Any plans of supporting data types "array of object" ?

Hi. One of the engineers I work with is asking if there is a Pinot whitepaper. She had a look and tried to find one but didn't manage. I'd appreciate a link if you guys know.

Hi, can I have multiple tables, each with one stream/topic attached? Thanks

I don't think we are deprecating it