This message was deleted Apache Druid #general

Join Slack

This message was deleted.

# general

Slackbot

12/26/2022, 6:34 AM

This message was deleted.

Brandon Tan

12/26/2022, 7:20 AM

Hi, 1. What is your version ? 2. This is an

Info/WARN

warning - are the ingestion tasks running ? 3.

connection timed out

- can you check connectivity to your Kafka broker?

Vijay Narayanan

12/26/2022, 7:42 AM

Also check your kafka listener. If you are using ip address for broker but if the advertised listener is returning host names and the dns is not correct then timeout will happen during metadata fetch

Chandra Prakash Meher

12/26/2022, 7:45 AM

@Brandon Tan 1. My Kafka Version:

2.6.1

2. Yes the ingestion tasks are running but without actually consuming any data from the kafka topic. I can see the reported lag for my topic though in the supervisor status 3. The other supervisor tasks run fine, but sometimes cause this issue and have to restart the entire cluster

Chandra Prakash Meher

12/26/2022, 7:46 AM

@Vijay Narayanan My kafka runs on MSK and I am using the DNS directly

Chandra Prakash Meher

12/26/2022, 9:43 AM

Attaching my supervisor status:

Copy code

{
  "dataSource": "conversation-analytics-events-v2",
  "stream": "conversations-analytics-event-v2",
  "partitions": 2,
  "replicas": 1,
  "durationSeconds": 3600,
  "activeTasks": [
    {
      "id": "index_kafka_conversation-analytics-events-v2_896fb93ac9e434b_fommhjii",
      "startingOffsets": {
        "0": 124556,
        "1": 124659
      },
      "startTime": "2022-12-26T09:08:54.117Z",
      "remainingSeconds": 1594,
      "type": "ACTIVE",
      "currentOffsets": {
        "0": 124556,
        "1": 124659
      },
      "lag": {
        "0": 8,
        "1": 7
      }
    }
  ],
  "publishingTasks": [],
  "latestOffsets": {
    "0": 124564,
    "1": 124666
  },
  "minimumLag": {
    "0": 8,
    "1": 7
  },
  "aggregateLag": 15,
  "offsetsLastUpdated": "2022-12-26T09:42:09.846Z",
  "suspended": false,
  "healthy": true,
  "state": "RUNNING",
  "detailedState": "RUNNING",
  "recentErrors": [
    {
      "timestamp": "2022-12-26T09:03:08.759Z",
      "exceptionClass": "java.util.concurrent.ExecutionException",
      "message": "org.apache.druid.indexing.common.IndexTaskClient$TaskNotRunnableException: Aborting request because task [index_kafka_conversation-analytics-events-v2_d36b77f9038cc00_mjlehghd] is not runnable",
      "streamException": false
    }
  ]
}

Brandon Tan

12/26/2022, 10:24 AM

Hi Chandra, I set up

24.0.2

and I wonder if you can provide me with a credentials on accessing your MSK with a dummy datasource. I am not familiar with MSK. If you could provide the 1.

topic

consumerProperties

Thanks !

Vijay Narayanan

12/26/2022, 11:17 AM

@Chandra Prakash Meher what is your advertised listener in listener config?

Vijay Narayanan

12/26/2022, 11:21 AM

Also are you using vpc peering?

Chandra Prakash Meher

12/26/2022, 12:36 PM

@Brandon Tan Its just AWS managed Kafka Services. Can we catchup over a quick huddle and debug instead please?

Chandra Prakash Meher

12/26/2022, 12:37 PM

@Vijay Narayanan I am advertising 8091 port and yes both kafka and my middlemanager/peon are in the same VPC

Chandra Prakash Meher

12/26/2022, 12:46 PM

For reference, attaching my supervisor config;

Copy code

{
  "type": "kafka",
  "spec": {
    "dataSchema": {
      "dataSource": "analytics-conversation-transformed-event",
      "timestampSpec": null,
      "dimensionsSpec": null,
      "metricsSpec": [],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": {
          "type": "none"
        },
        "rollup": true,
        "intervals": []
      },
      "transformSpec": {
        "filter": null,
        "transforms": []
      },
      "parser": {
        "type": "string",
        "parseSpec": {
          "format": "json",
          "timestampSpec": {
            "column": "timestamp",
            "format": "auto"
          },
          "dimensionsSpec": {
            "dimensions": [
              "botId",
              "botReferenceId",
              "recordId",
              "customFields",
              "subject",
              "firstIntent",
              "conversationId",
              "uuid",
              "messageId",
              "exchangeType",
              "env",
              "falsePositive",
              "interactionMode",
              "visitorMessageSource",
              "keywords",
              "classificationStatus",
              "visitorMessageTime",
              "timestamp",
              "channel",
              "locale",
              "supportedLanguage",
              "exchangeCategory",
              "exchangeSubCategory",
              "skillId",
              "intentId",
              "intent",
              "intentCategory",
              "previousIntentId",
              "mappedTopic",
              "mappedCategory",
              "inboundMessageText",
              "botMessage",
              "ticketStatus",
              "currentState",
              "escalationReason",
              "feedback",
              "customerFeedback",
              "agentFeedback",
              "sendStatus",
              "customFieldsJson",
              "shortAnswer",
              "longAnswer",
              "question",
              "nodeType",
              "masterIntentId",
              "kbAnswer",
              "kbAnswerQuestion",
              "kbAnswerShortAnswer",
              "kbAnswerLongAnswer",
              "kbArticles",
              "masterIntent",
              "feedbackRemarks",
              "exchangeDuration",
              "narrowingOption1",
              "narrowingOption3",
              "narrowingOption2",
              "narrowingOption4",
              "narrowingOption6",
              "narrowingOption7",
              "narrowingOption8",
              "narrowingOption9",
              "narrowingOption10",
              "suggestionsOption1",
              "narrowingOption5",
              "suggestionsOption2",
              "suggestionsOption3",
              "suggestionsOption4",
              "suggestionsOption5",
              "suggestionsOption6",
              "suggestionsOption7",
              "suggestionsOption8",
              "suggestionsOption9",
              "suggestionsOption10",
              "menuOption1",
              "menuOption2",
              "menuOption3",
              "menuOption4",
              "menuOption5",
              "menuOption6",
              "menuOption7",
              "switchType",
              "fromIntentId",
              "menuOption8",
              "menuOption9",
              "menuOption10"
            ],
            "dimensionExclusions": [],
            "spatialDimensions": []
          }
        }
      }
    },
    "ioConfig": {
      "topic": "analytics-conversation-transformed-event",
      "inputFormat": null,
      "replicas": 1,
      "taskCount": 1,
      "taskDuration": "PT3600S",
      "consumerProperties": {
        "bootstrap.servers": "host1:9092,host2:9092"
      },
      "pollTimeout": 100000,
      "startDelay": "PT5S",
      "period": "PT30S",
      "useEarliestOffset": false,
      "completionTimeout": "PT1800S",
      "lateMessageRejectionPeriod": null,
      "earlyMessageRejectionPeriod": null,
      "lateMessageRejectionStartDateTime": null,
      "stream": "analytics-conversation-transformed-event",
      "useEarliestSequenceNumber": false,
      "autoscalerConfig": null,
      "type": "kafka"
    },
    "tuningConfig": {
      "type": "kafka",
      "appendableIndexSpec": {
        "type": "onheap"
      },
      "maxRowsInMemory": 500,
      "maxBytesInMemory": 0,
      "skipBytesInMemoryOverheadCheck": false,
      "maxRowsPerSegment": 50000,
      "maxTotalRows": null,
      "intermediatePersistPeriod": "PT2M",
      "maxPendingPersists": 0,
      "indexSpec": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "roaring",
          "compressRunOnSerialization": true
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "reportParseExceptions": false,
      "handoffConditionTimeout": 0,
      "resetOffsetAutomatically": false,
      "segmentWriteOutMediumFactory": null,
      "workerThreads": 10,
      "chatThreads": 10,
      "chatRetries": 8,
      "httpTimeout": "PT10S",
      "shutdownTimeout": "PT80S",
      "offsetFetchPeriod": "PT30S",
      "intermediateHandoffPeriod": "PT10M",
      "logParseExceptions": false,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0,
      "skipSequenceNumberAvailabilityCheck": false,
      "repartitionTransitionDuration": "PT120S"
    }
  }
}

Chandra Prakash Meher

12/26/2022, 12:48 PM

For additional information, I am running my entire druid setup on AWS Elastic Container Services and I am not setting the

druid.host

property

Karan Kumar

12/26/2022, 1:34 PM

This looks to me an coordinator problem which generally run on 8081. (the plaint text version of it). My guess is to check the coordinator logs first and to see whats happening there. If nothing strikes out, check nw connectivity between peons, overlord -> coordinator. This is for the first exception. For the

index_kafka_conversation-analytics-events-v2_d36b77f9038cc00_mjlehghd

not found • Go to the overlord • Grep for

index_kafka_conversation-analytics-events-v2_d36b77f9038cc00_mjlehghd

in the logs and see which MM this task is assinged. • Go to the mm and see whats happening with

index_kafka_conversation-analytics-events-v2_d36b77f9038cc00_mjlehghd

👆 1

Open in Slack

Previous Next