https://pinot.apache.org/ logo
Join Slack
Powered by
# troubleshooting
  • t

    Taran Rishit

    12/06/2020, 5:27 PM
    Hello , im getting the following error while connecting pinot to presto presto:default> SHOW TABLES from pinot.default -> ; Query 20201206_170528_00006_xcrsm failed: java.io.UncheckedIOException: java.net.UnknownHostException: pinot-controller: Name or service not known pinot is running
  • m

    Mayank

    12/07/2020, 6:09 PM
    Hello, we have a channel #C01GBP88VCJ to discuss Pinot performance problems/solutions/tunings. Please feel free to join/contribute.
  • e

    Elon

    12/08/2020, 5:29 AM
    Running java11 with pinot and it works but I get a lot of classloader exceptions on startup:
    Copy code
    java.lang.IllegalArgumentException: object is not an instance of declaring class
            at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
            at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
            at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
            at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
            at org.apache.pinot.spi.plugin.PluginClassLoader.<init>(PluginClassLoader.java:50) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
            at org.apache.pinot.spi.plugin.PluginManager.createClassLoader(PluginManager.java:171) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
            at org.apache.pinot.spi.plugin.PluginManager.load(PluginManager.java:162) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
            at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:137) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
            at org.apache.pinot.spi.plugin.PluginManager.init(PluginManager.java:103) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
            at org.apache.pinot.spi.plugin.PluginManager.<init>(PluginManager.java:84) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
            at org.apache.pinot.spi.plugin.PluginManager.<clinit>(PluginManager.java:46) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
            at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:166) ~[pinot-all-0.5.0-jar-with-dependencies.jar:0.5.0-7efd67a228a2f40139c768d8a55081e5c9ab1ef5]
  • e

    Elon

    12/08/2020, 5:29 AM
    And then the plugin that threw the error loads anyway.
  • e

    Elon

    12/08/2020, 5:29 AM
    This is for pinot-0.5.0
  • x

    Xiang Fu

    12/08/2020, 5:30 AM
    I think java 11 doesn't support plugin loading stuffs, for that we need to add all the plugins into classpath
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:44 PM
    hi team
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:45 PM
    Screen Shot 2020-12-08 at 23.32.01.png
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:45 PM
    The realtime table used too much old jvm ... but could not be cleaned up Why GC could not clear this memory
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:45 PM
    I consume 1 realtime table with 450 million rows ... and follow jvm to see that the OLD area is using too much memory, but GC does not clean up and keep this large amount of memory, making the system not responding to many queries
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:45 PM
    Screen Shot 2020-12-08 at 23.36.52.png
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:46 PM
    Screen Shot 2020-12-08 at 23.39.43.png
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:46 PM
    pinot.server.netty.port=8000 pinot.server.adminapi.port=8030 pinot.server.instance.dataDir=/app/pinot/data/index pinot.server.instance.segmentTarDir=/app/pinot/data/segment pinot.server.instance.readMode=mmap pinot.server.instance.realtime.alloc.offheap=true pinot.server.instance.realtime.alloc.offheap.direct=false pinot.server.instance.reload.consumingSegment=true pinot.server.query.executor.timeout=90000 pinot.set.instance.id.to.hostname=true {   "REALTIME": {     "tableName": "bhx_bhx_forecast_forecast_item_REALTIME",     "tableType": "REALTIME",     "segmentsConfig": {       "timeType": "MILLISECONDS",       "retentionTimeUnit": "DAYS",       "retentionTimeValue": "9125",       "segmentPushFrequency": "DAILY",       "segmentPushType": "APPEND",       "replication": "4",       "replicasPerPartition": "4",       "timeColumnName": "_TIMESTAMP",       "schemaName": "bhx_bhx_forecast_forecast_item"     },     "tenants": {       "broker": "DefaultTenant",       "server": "DefaultTenant",       "tagOverrideConfig": {}     },     "tableIndexConfig": {       "streamConfigs": {         "streamType": "kafka",         "stream.kafka.consumer.type": "lowlevel",         "stream.kafka.topic.name": "PINOT.BHX.bhx_forecast.forecast_item",         "stream.kafka.table.tablename": "bhx_forecast.forecast_item",         "stream.kafka.table.part.pattern": "_[0-9]+",         "stream.kafka.cdc.format": "CDC",         "stream.kafka.decoder.class.name": "com.mwg.pinot.realtime.KafkaCDCMessageDecoder",         "stream.kafka.consumer.factory.class.name": "com.mwg.pinot.realtime.KafkaCDCConsumerFactory",         "stream.kafka.broker.list": "datastore-broker01-kafka-ovm-6-769092,datastore broker02 kafka ovm 6 779093,datastore-broker03-kafka-ovm-6-78:9094",         "stream.kafka.consumer.prop.auto.offset.reset": "smallest",         "realtime.segment.flush.threshold.rows": "0",         "realtime.segment.flush.threshold.time": "10m",         "realtime.segment.flush.threshold.segment.size": "5M",         "group.id": "bhx_bhx_forecast.forecast_item-PINOT_INGESTION",         "max.partition.fetch.bytes": "167772160",         "receive.buffer.bytes": "67108864",         "isolation.level": "read_committed",         "max.poll.records": "5000"       },       "enableDefaultStarTree": false,       "enableDynamicStarTreeCreation": false,       "aggregateMetrics": false,       "nullHandlingEnabled": false,       "autoGeneratedInvertedIndex": false,       "createInvertedIndexDuringSegmentGeneration": false,       "loadMode": "MMAP"     },     "metadata": {       "customConfigs": {}     },     "routing": {       "instanceSelectorType": "strictReplicaGroup"     },     "instanceAssignmentConfigMap": {       "CONSUMING": {         "tagPoolConfig": {           "tag": "inventory_REALTIME",           "poolBased": false,           "numPools": 0         },         "replicaGroupPartitionConfig": {           "replicaGroupBased": true,           "numInstances": 0,           "numReplicaGroups": 4,           "numInstancesPerReplicaGroup": 5,           "numPartitions": 0,           "numInstancesPerPartition": 0         }       }     },     "upsertConfig": {       "mode": "FULL"     }   } }
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:46 PM
    PINOT_JAVA_OPTS=-Xmx180g -Xms16G -Dlog4j2.configurationFile=conf/pinot-admin-log4j2.xml -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=10 -XX:G1MaxNewSizePercent=20 -XX:G1HeapRegionSize=32M -XX:G1Re servePercent=5 -XX:G1HeapWastePercent=2 -XX:G1MixedGCCountTarget=3 -XX:+AlwaysPreTouch -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:G1MixedGCLiveThresholdPercent=35 -XX:G 1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:MaxTenuringThreshold=1 -XX:InitiatingHeapOccupancyPercent=30 -XX:-G1UseAdaptiveIHOP -XX:+UseStringDeduplication -XX:+PerfDisableSharedMem -XX:ParallelGCThreads=12 -XX:ConcGCT hreads=6 -XX:HeapDumpPath=/app/pinot/logs/heapdump_oom_pinot.log -javaagent/app/pinot/jmx prometheus/jmx prometheus javaagent 0.13.0.jar=8081/app/pinot/jmx_prometheus/config.yaml -Xlog:gc*/app/pinot/logs/gc stat pinot.txttime,u ptime,level,tags:filecount=10,filesize=100m
  • l

    lâm nguyễn hoàng

    12/08/2020, 5:49 PM
    The odd thing is that the segment is stored in the heap instead of the off-heap like configuring pinot.server.instance.realtime.alloc.offheap.direct = true
  • m

    Mayank

    12/18/2020, 5:31 PM
    https://apache-pinot.slack.com/archives/CDRCA57FC/p1608311932340000
  • m

    Mayank

    12/18/2020, 5:32 PM
    @dhurandar what's the format at rest?
  • d

    dhurandar

    12/18/2020, 5:33 PM
    Parquet now
  • d

    dhurandar

    12/18/2020, 5:36 PM
    But we are facing issues in a couple of ways. Presto doesn't support data skipping since in our case dimensions are many (approx 25). Abloom filter based data skipping would help us. We are working on that internally since we also use Presto to query our data lake which is 5 petabytes. The current partition is mostly hourly daily and monthly, but we do need others too like Companies, Countries, and User Segments. Since we now see queries across these dimensions also
  • d

    dhurandar

    12/18/2020, 5:37 PM
    Cost is a big deal for us since its a Saas offering
  • k

    Kishore G

    12/18/2020, 5:37 PM
    short answer - you can push 100's of TB of data
    ➕ 1
  • d

    dhurandar

    12/18/2020, 5:38 PM
    okay
  • k

    Kishore G

    12/18/2020, 5:38 PM
    there are many things you can do to optimize on speed and cost
  • d

    dhurandar

    12/18/2020, 5:39 PM
    It would require its own storage as far as in reading and test. in addition to S3
  • d

    dhurandar

    12/18/2020, 5:39 PM
    i see
  • k

    Kishore G

    12/18/2020, 5:39 PM
    yes
  • k

    Kishore G

    12/18/2020, 5:39 PM
    you can use the persistent volume in AWS to host the data
  • d

    dhurandar

    12/18/2020, 5:40 PM
    Yes the idea is to use EBS volumes
  • e

    Elon

    12/24/2020, 6:01 AM
    .
  • l

    lâm nguyễn hoàng

    12/28/2020, 7:09 PM
    Screen Shot 2020-12-29 at 01.42.39.png,Screen Shot 2020-12-29 at 01.42.44.png,Screen Shot 2020-12-29 at 01.42.51.png
1...138139140...166Latest