Hello , im getting the following error while connecting pinot to presto presto:default> SHOW TABLES from pinot.default -> ; Query 20201206_170528_00006_xcrsm failed: java.io.UncheckedIOException: java.net.UnknownHostException: pinot-controller: Name or service not known pinot is running

Hello, we have a channel #C01GBP88VCJ to discuss Pinot performance problems/solutions/tunings. Please feel free to join/contribute.

Running java11 with pinot and it works but I get a lot of classloader exceptions on startup:

java.lang.IllegalArgumentException: object is not an instance of declaring class
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at org.apache.pinot.spi.plugin.PluginClassLoader.<init>(PluginClassLoader.java:50) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
        at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:171) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
        at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
        at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
        at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
        at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
        at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:166) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]

And then the plugin that threw the error loads anyway.

This is for pinot-0.5.0

I think java 11 doesn't support plugin loading stuffs, for that we need to add all the plugins into classpath

hi team

Screen Shot 2020-12-08 at 23.32.01.png

The realtime table used too much old jvm ... but could not be cleaned up Why GC could not clear this memory

I consume 1 realtime table with 450 million rows ... and follow jvm to see that the OLD area is using too much memory, but GC does not clean up and keep this large amount of memory, making the system not responding to many queries

Screen Shot 2020-12-08 at 23.36.52.png

Screen Shot 2020-12-08 at 23.39.43.png

pinot.server.netty.port=8000 pinot.server.adminapi.port=8030 pinot.server.instance.dataDir=/app/pinot/data/index pinot.server.instance.segmentTarDir=/app/pinot/data/segment pinot.server.instance.readMode=mmap pinot.server.instance.realtime.alloc.offheap=true pinot.server.instance.realtime.alloc.offheap.direct=false pinot.server.instance.reload.consumingSegment=true pinot.server.query.executor.timeout=90000 pinot.set.instance.id.to.hostname=true {   "REALTIME": {     "tableName": "bhx_bhx_forecast_forecast_item_REALTIME",     "tableType": "REALTIME",     "segmentsConfig": {       "timeType": "MILLISECONDS",       "retentionTimeUnit": "DAYS",       "retentionTimeValue": "9125",       "segmentPushFrequency": "DAILY",       "segmentPushType": "APPEND",       "replication": "4",       "replicasPerPartition": "4",       "timeColumnName": "_TIMESTAMP",       "schemaName": "bhx_bhx_forecast_forecast_item"     },     "tenants": {       "broker": "DefaultTenant",       "server": "DefaultTenant",       "tagOverrideConfig": {}     },     "tableIndexConfig": {       "streamConfigs": {         "streamType": "kafka",         "stream.kafka.consumer.type": "lowlevel",         "stream.kafka.topic.name": "PINOT.BHX.bhx_forecast.forecast_item",         "stream.kafka.table.tablename": "bhx_forecast.forecast_item",         "stream.kafka.table.part.pattern": "_[0-9]+",         "stream.kafka.cdc.format": "CDC",         "stream.kafka.decoder.class.name": "com.mwg.pinot.realtime.KafkaCDCMessageDecoder",         "stream.kafka.consumer.factory.class.name": "com.mwg.pinot.realtime.KafkaCDCConsumerFactory",         "stream.kafka.broker.list": "datastore-broker01-kafka-ovm-6-769092,datastore broker02 kafka ovm 6 779093,datastore-broker03-kafka-ovm-6-78:9094",         "stream.kafka.consumer.prop.auto.offset.reset": "smallest",         "realtime.segment.flush.threshold.rows": "0",         "realtime.segment.flush.threshold.time": "10m",         "realtime.segment.flush.threshold.segment.size": "5M",         "group.id": "bhx_bhx_forecast.forecast_item-PINOT_INGESTION",         "max.partition.fetch.bytes": "167772160",         "receive.buffer.bytes": "67108864",         "isolation.level": "read_committed",         "max.poll.records": "5000"       },       "enableDefaultStarTree": false,       "enableDynamicStarTreeCreation": false,       "aggregateMetrics": false,       "nullHandlingEnabled": false,       "autoGeneratedInvertedIndex": false,       "createInvertedIndexDuringSegmentGeneration": false,       "loadMode": "MMAP"     },     "metadata": {       "customConfigs": {}     },     "routing": {       "instanceSelectorType": "strictReplicaGroup"     },     "instanceAssignmentConfigMap": {       "CONSUMING": {         "tagPoolConfig": {           "tag": "inventory_REALTIME",           "poolBased": false,           "numPools": 0         },         "replicaGroupPartitionConfig": {           "replicaGroupBased": true,           "numInstances": 0,           "numReplicaGroups": 4,           "numInstancesPerReplicaGroup": 5,           "numPartitions": 0,           "numInstancesPerPartition": 0         }       }     },     "upsertConfig": {       "mode": "FULL"     }   } }

PINOT_JAVA_OPTS=-Xmx180g -Xms16G -Dlog4j2.configurationFile=conf/pinot-admin-log4j2.xml -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=10 -XX:G1MaxNewSizePercent=20 -XX:G1HeapRegionSize=32M -XX:G1Re servePercent=5 -XX:G1HeapWastePercent=2 -XX:G1MixedGCCountTarget=3 -XX:+AlwaysPreTouch -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:G1MixedGCLiveThresholdPercent=35 -XX:G 1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:MaxTenuringThreshold=1 -XX:InitiatingHeapOccupancyPercent=30 -XX:-G1UseAdaptiveIHOP -XX:+UseStringDeduplication -XX:+PerfDisableSharedMem -XX:ParallelGCThreads=12 -XX:ConcGCT hreads=6 -XX:HeapDumpPath=/app/pinot/logs/heapdump_oom_pinot.log -javaagent/app/pinot/jmx prometheus/jmx prometheus javaagent 0.13.0.jar=8081/app/pinot/jmx_prometheus/config.yaml -Xlog:gc*/app/pinot/logs/gc stat pinot.txttime,u ptime,level,tags:filecount=10,filesize=100m

The odd thing is that the segment is stored in the heap instead of the off-heap like configuring pinot.server.instance.realtime.alloc.offheap.direct = true

@dhurandar what's the format at rest?

Parquet now

But we are facing issues in a couple of ways. Presto doesn't support data skipping since in our case dimensions are many (approx 25). Abloom filter based data skipping would help us. We are working on that internally since we also use Presto to query our data lake which is 5 petabytes. The current partition is mostly hourly daily and monthly, but we do need others too like Companies, Countries, and User Segments. Since we now see queries across these dimensions also

Cost is a big deal for us since its a Saas offering

short answer - you can push 100's of TB of data

okay

there are many things you can do to optimize on speed and cost

It would require its own storage as far as in reading and test. in addition to S3

i see

yes

you can use the persistent volume in AWS to host the data

Yes the idea is to use EBS volumes

.

Screen Shot 2020-12-29 at 01.42.39.png,Screen Shot 2020-12-29 at 01.42.44.png,Screen Shot 2020-12-29 at 01.42.51.png