https://linen.dev logo
Join Slack
Powered by
# troubleshooting
  • s

    Slackbot

    03/16/2023, 8:57 AM
    This message was deleted.
    h
    g
    v
    • 4
    • 6
  • s

    Slackbot

    03/16/2023, 11:11 AM
    This message was deleted.
    g
    • 2
    • 1
  • s

    Slackbot

    03/16/2023, 2:38 PM
    This message was deleted.
    l
    u
    • 3
    • 8
  • s

    Slackbot

    03/16/2023, 10:39 PM
    This message was deleted.
    s
    m
    g
    • 4
    • 14
  • s

    Slackbot

    03/17/2023, 6:28 AM
    This message was deleted.
    b
    e
    a
    • 4
    • 10
  • s

    Slackbot

    03/17/2023, 6:57 AM
    This message was deleted.
    b
    s
    g
    • 4
    • 11
  • s

    Slackbot

    03/17/2023, 8:24 AM
    This message was deleted.
    c
    d
    • 3
    • 3
  • s

    Slackbot

    03/17/2023, 2:24 PM
    This message was deleted.
    r
    g
    • 3
    • 3
  • r

    Renato Santos

    03/17/2023, 6:13 PM
    I'm having issues with (range) compaction, partial_dimension_distribution is failing but keep running forever, then the compaction task start to get null exceptions but keep running also forever in the partial_dimension_distribution, all looks fine until there's this message:
    Copy code
    2023-03-17T18:07:48,636 INFO [parent-monitor-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM shutdown.
    and then the tasks stop but the status is still the same
    Copy code
    2023-03-17T18:07:49,024 INFO [Thread-69] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
    Maybe it's k8s sending a kill signal, I've increase the memory to 12gb per MM, and i'm using
    -Xms512M -Xmx1322M -XX:MaxDirectMemorySize=3g
    for the peon task
  • s

    Slackbot

    03/17/2023, 6:53 PM
    This message was deleted.
    r
    s
    v
    • 4
    • 60
  • s

    Slackbot

    03/20/2023, 12:24 PM
    This message was deleted.
    middleManager.log
    l
    v
    r
    • 4
    • 12
  • s

    Slackbot

    03/20/2023, 1:20 PM
    This message was deleted.
    m
    n
    • 3
    • 7
  • m

    Martin Maqueira

    03/20/2023, 1:21 PM
    Copy code
    {
      "id": "index_kafka_qps_e1e08c491120d95_dkgbobjj",
      "groupId": "index_kafka_qps",
      "type": "index_kafka",
      "createdTime": "2023-03-20T12:12:49.298Z",
      "queueInsertionTime": "1970-01-01T00:00:00.000Z",
      "statusCode": "FAILED",
      "status": "FAILED",
      "runnerStatusCode": "WAITING",
      "duration": -1,
      "location": {
        "host": "172.31.13.78",
        "port": 8103,
        "tlsPort": -1
      },
      "dataSource": "qps",
      "errorMsg": "No task in the corresponding pending completion taskGroup[0] succeeded before completion timeout ela..."
    }
  • m

    Martin Maqueira

    03/20/2023, 1:25 PM
    And then on the logs of the middlemanager i found it this:
  • m

    Martin Maqueira

    03/20/2023, 1:26 PM
    Copy code
    2023-03-20T12:18:05,279 INFO [Thread-70] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/announcements/172.31.13.78:8103]
    2023-03-20T12:18:05,329 INFO [Thread-70] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/segments/172.31.13.78:8103/172.31.13.78:8103_indexer-executor__default_tier_2023-03-20T12:13:04.001Z_c69a5170b7ee404b8bc71df4625354e10]
    2023-03-20T12:18:05,338 INFO [Thread-70] org.apache.druid.curator.announcement.Announcer - Unannouncing [/druid/internal-discovery/PEON/172.31.13.78:8103]
    2023-03-20T12:18:05,346 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
    2023-03-20T12:18:05,373 INFO [Thread-70] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@c6c84fa{HTTP/1.1, (http/1.1)}{0.0.0.0:8103}
    2023-03-20T12:18:05,374 INFO [Thread-70] org.eclipse.jetty.server.session - node0 Stopped scavenging
    2023-03-20T12:18:05,380 INFO [Thread-70] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@5e05a706{/,null,STOPPED}
    2023-03-20T12:18:05,391 INFO [Thread-70] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
    2023-03-20T12:18:05,395 INFO [Thread-70] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='172.31.13.78:8103', hostAndPort='172.31.13.78:8103', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}]
    2023-03-20T12:18:05,395 INFO [Thread-70] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
    2023-03-20T12:18:05,395 INFO [Thread-70] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
    2023-03-20T12:18:05,395 INFO [Thread-70] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_qps_e1e08c491120d95_dkgbobjj].
    2023-03-20T12:18:05,396 INFO [Thread-70] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping forcefully (status: [READING])
    2023-03-20T12:18:05,398 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting.
    org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
    	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:520) ~[?:?]
    	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281) ~[?:?]
    	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[?:?]
    	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1297) ~[?:?]
    	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1238) ~[?:?]
    	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[?:?]
    	at org.apache.druid.indexing.kafka.KafkaRecordSupplier.poll(KafkaRecordSupplier.java:128) ~[?:?]
    	at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.getRecords(IncrementalPublishingKafkaIndexTaskRunner.java:95) ~[?:?]
    	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:612) [druid-indexing-service-0.23.0.jar:0.23.0]
    	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:265) [druid-indexing-service-0.23.0.jar:0.23.0]
    	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:149) [druid-indexing-service-0.23.0.jar:0.23.0]
    	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) [druid-indexing-service-0.23.0.jar:0.23.0]
    	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) [druid-indexing-service-0.23.0.jar:0.23.0]
    	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
    	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
    Caused by: java.lang.InterruptedException
    	... 17 more
  • m

    Martin Maqueira

    03/20/2023, 1:28 PM
    i am running this on EKS AWS. Druid is in Paris and Kafka in ohio,usa
  • m

    Martin Maqueira

    03/20/2023, 1:28 PM
    on kafka logs i am not founding an error...
  • m

    Martin Maqueira

    03/20/2023, 1:40 PM
    @Sergio Ferragut
  • m

    Martin Maqueira

    03/20/2023, 1:41 PM
    @Peter Marshall @Mark Herrera
  • v

    Vijay Narayanan

    03/20/2023, 1:59 PM
    Any errors in overlord log?
  • s

    Slackbot

    03/20/2023, 2:01 PM
    This message was deleted.
    v
    m
    • 3
    • 6
  • s

    Slackbot

    03/20/2023, 2:06 PM
    This message was deleted.
    g
    n
    +2
    • 5
    • 32
  • l

    Luiz Augusto

    03/20/2023, 2:07 PM
    Maybe using slack threads, folks?
  • s

    Slackbot

    03/20/2023, 2:27 PM
    This message was deleted.
    k
    n
    • 3
    • 2
  • s

    Slackbot

    03/20/2023, 7:57 PM
    This message was deleted.
    r
    s
    • 3
    • 9
  • s

    Slackbot

    03/21/2023, 8:47 AM
    This message was deleted.
    a
    b
    r
    • 4
    • 8
  • s

    Slackbot

    03/21/2023, 2:37 PM
    This message was deleted.
    r
    l
    • 3
    • 4
  • s

    Slackbot

    03/22/2023, 12:37 AM
    This message was deleted.
    r
    h
    • 3
    • 2
  • s

    Slackbot

    03/22/2023, 9:53 AM
    This message was deleted.
    a
    m
    • 3
    • 3
  • s

    Slackbot

    03/22/2023, 12:05 PM
    This message was deleted.
    j
    r
    • 3
    • 3
1...252627...53Latest