Apache Druid

```2022-09-12T11:07:39,921 ERROR [forking-task-runner-8] org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with code[137] for task: index_kafka_nbr_8322bd4a68e5423_pclejmmh```

```2022-09-12T11:21:45,819 WARN [HttpClient-Netty-Boss-0] org.jboss.netty.channel.SimpleChannelUpstreamHandler - EXCEPTION, please implement org.jboss.netty.handler.codec.http.HttpContentDecompressor.exceptionCaught() for proper handling.
java.net.ConnectException: Connection refused: /172.31.3.118:8102
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_275]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_275]
	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152) ~[netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.6.Final.jar:?]
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.6.Final.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]```

environment_druid (copia).yaml

i am replying to the last message you sent so we can start a thread

this is the real key here:

```2022-09-12T11:07:39,921 ERROR [forking-task-runner-8] org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with code[137] for task: index_kafka_nbr_8322bd4a68e5423_pclejmmh```

exit code 137 is signal 9 a.k.a. SIGKILL — the task JVM was killed abruptly

generally most common reason for this is it tried to use too much memory — so the thing to do would be verify your memory settings &amp; make sure we aren't exceeding limits

most relevant settings are here

```  druid_indexer_runner_javaOptsArray: '["-server", "-Xmx3g", "-Xms256m", "-XX:MaxDirectMemorySize=3g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]'```
and

```druid_indexer_fork_property_druid_processing_buffer_sizeBytes: 256MiB```
and

```druid_worker_capacity: "16"```
and

```  druid_processing_numThreads: "6"
  druid_processing_numMergeBuffers: "2"```

<https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html> has some details about how to interpret these parameters but simple version is:

• each mm will run up to 16 tasks (`druid.worker.capacity`)
• each task will use up to 3 GB of heap (`-Xmx3g`) and ~2.5 GB offheap (6 threads + 2 merge buffers; times 256MB; plus a little extra)
• so if all 16 tasks are running, that's potentially 88 GB of memory usage on the mm
do you have that much memory available on the machine?

&gt; generally most common reason for this is it tried to use too much memory
btw, you can verify this in `dmesg`: it will say something about the oom-killer killing a `java` process

Since you deployed in Kubernetes, did you see if your MM is being restarted?

<@U0344FW86DD> you were completely right! Thanks of u, it  is solved! Thank u so much!

Hey <@U03HF9GLL1X> you may want to cross-post this into <#C030K0Z7S1Z|>

with 2 supervisors everything works ok! with the 3º all start failing

Hey <@U041Z7XSFLK> just to clarify – is this when you’re adding another ingestion task?

And is it safe to presume that `172.31.3.118` is one of your Druid nodes?

Also, are you able to see, on that node itself, the log entries for `index_kafka_nbr_8322bd4a68e5423_pclejmmh` ?  (I _think_ !!! that tasks that start up have logs created but they do not end up in your log store if they fail too early… )

Also I would check that `<http://kafka-int.we-grow.mobi:9094|kafka-int.we-grow.mobi:9094>` is reachable from all the nodes where your Middle Managers are running – and I’d also check there are 3 worker slots available (supervisor + 2 workers)

What is curious is that all your supervisors seem to error when you start up this new one.  Is it safe to assume that this supervisor going to a different datasource?

(It could be a worker capacity thing, I guess – if each one can’t get a slot to do the work)

@ yes Peter. I am adding another ingestion task.

`172.31.3.118`    is another druid node (kubernetes pod)

The logs `index_kafka_nbr_8322bd4a68e5423_pclejmmh`  are disappearing so fast!

yes `<http://kafka-int.we-grow.mobi:9094|kafka-int.we-grow.mobi:9094>`  is reacheable from all druid nodes!

every supervisor is going to a different datasource

OOI when the other tasks start failing, what’s the error that those tasks report?

i also replied here with some thoughts: <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1663028734437029?thread_ts=1662982204.176429&amp;cid=C0309C9L90D>

it is better to put all your stuff in a single slack thread so we can coalesce the conversation :slightly_smiling_face:

I would start with a dig into the individual task logs – perhaps they are waiting to write into deep storage, for example, and don’t do it quickly enough – the logs should give you a clue.

can you share the ingestion spec? I had a similar "issue" when I set the timeout too low, lower than the task run time haha

I heard “10k rows per second per task” – but tbh it really depends what throughput you get for your own data… as well as your own source system’s usual stuff, it’s also about what the task is being asked to do – like rollup or not, datasketches or not, filtering or not, and so on.

Thanks for the info. For the 10k rows per second per task benchmark, do you know what is the row size there?

we did some benchmarks here <https://imply.io/blog/exactly-once-streaming-ingestion/> when first introducing kafka indexing

per task number varied from 15k/s to 160k/s depending on the size of the messages

10k/s/task i think would be a pretty conservative estimate for larger messages (like a few KB, &gt;50 columns)

with a large resultset like that, you could be hitting timeouts leading to truncation of results

see this section of the docs for info on errors in the SQL API: <https://druid.apache.org/docs/latest/querying/sql-api.html#errors>

this part:

&gt; As a caller, it is important that you properly handle response truncation. This is easy for the object and array formats, since truncated responses will be invalid JSON. For the line-oriented formats, you should check the trailer they all include: one blank line at the end of the result set. If you detect a truncated response, either through a JSON parsing error or through a missing trailing newline, you should assume the response was not fully delivered due to an error.

with CSV, a complete API response always includes a trailing blank line — if you don't see that then it means the response got cut off due to error/timeout. you can increase your timeout in that case

What and where are those config which I can change to get the whole query response instead of truncated response <@U0344FW86DD>?
I could not find much from the documentation actually..

i'm just guessing it's timeout-related, which is described here under property `timeout` <https://druid.apache.org/docs/latest/querying/query-context.html>

Ohh. unfortunately that does not work when I pass the `timeout` context parameter with the API call. Still it gets only part of data out of Druid

Even using timeout of 0 we are getting below error -
```AvaticaClientRuntimeException: Remote driver error: QueryTimeoutException: url[<http://10.111.125.60:8083/druid/v2/>] timed out. Error -1 (00000) UNKNOWN

org.apache.druid.query.QueryTimeoutException: url[<http://10.XXX.XXX.XX:8083/druid/v2/>] timed out
	at org.apache.druid.client.JsonParserIterator.timeoutQuery(JsonParserIterator.java:204)
	at org.apache.druid.client.JsonParserIterator.next(JsonParserIterator.java:119)
	at org.apache.druid.java.util.common.guava.BaseSequence.makeYielder(BaseSequence.java:90)
	at org.apache.druid.java.util.common.guava.BaseSequence.access$000(BaseSequence.java:27)
	at org.apache.druid.java.util.common.guava.BaseSequence$1.next(BaseSequence.java:114)
	at org.apache.druid.java.util.common.guava.MergeSequence.makeYielder(MergeSequence.java:1```

I have infact increased the timeout on broker, historical, MM to 20 mins. Still failing.. :disappointed:

i wonder if you are running into one of the other things mentioned on <https://druid.apache.org/docs/latest/configuration/index.html> such as `druid.broker.http.readTimeout` (default PT15M) or `druid.router.http.readTimeout` (default PT8M)

Yeah that does sound like a timeout thing! Try setting longer timeout in query context, like, if 15 mins is OK then set `timeout: 900000` (15 minutes) and raise `druid.router.http.readTimeout = PT15M`

it's normal for realtime tasks to come and go as they cycle, hence the log message being INFO

do you feel this log message is associated with a problem?

no, we are troubleshooting some issue wanted to check on this as we see this more often in broker logs

Might be better to do it upstream in a stream processor like Flink.

there isn't a builtin `row_index` var, but you can do `%` on some other appropriate var you have in the data (which, if it doesn't exist, yeah, you may need to add upstream via flink or similar)

or, if you can pre-computer your topn somehow and using batch ingestion, you can use lookup to replace values not in the lookup cache/map.

If you have some estimate about "rows per minute", you can allow a specific minute? so, instead of `row_index` you can use `CURRENT_TIMESTAMP`  or something
I'm not sure it's possible, but maybe we can use the JS and return some random number based on the minute, like `Math.random(new Date().getTime()/60000)` and if that number is greater than 0.1 (10% per minute) the value would be stored or something like that...

&gt; Those logs are very valuable for my customers to self-help themselves.
:point_up: this is definitely true!

there's technical reasons why it doesn't happen: we don't have an easy way to redirect logs for different tasks to different files when they run in the same JVM

with MM, we do it by capturing stdout of the forked `java` process

one thing that helps: in new SQL-based ingest we have added much more detail to reports (structured JSON) so in many cases logs are not needed for debugging

info like details on job stages, counters, warnings, error messages

that being said, i do agree logs are valuable. if we had a good solution for capturing them that would be nice

&gt; we don't have an easy way to redirect logs for different tasks to different files when they run in the same JVM
could we try sifting log appenders?

maybe? i am not a log4j2 expert so i don't 100% know what is possible

Sorry for this dumb question, but if you have the taskID already, don't you have enough information to spawn a new file logger into the same folder location?

I was going through the log4j documentation and it seems like <https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender> can help with dynamically changing the file to log to. I will try to go over it in detail in a while.

implementation note: an issue is shared thread pools: if you look through the logs for a task that ran on MM you will notice that there are lots of different threads in play, and in the Indexer some of these are shared across tasks. for example, the processing thread pool (for queries on realtime tasks) and the thread pool used for merging and pushing segments (in UnifiedIndexerAppenderatorsManager) are both shared in the Indexer model. this is one of the reasons the Indexer is cool (better resource sharing). but i believe it also means some attention is needed to capture all these logs (or, deciding we're ok with not capturing them)

I created a Github issue to track it: <https://github.com/apache/druid/issues/13082>

To add to <@U03149ALHN3>’s idea, <@U0344FW86DD> - it should be "easyish" to track the log file per indexer task in a data structure accessible by the shared thread pools, no?

the thing that seems tricky to me is the logger running in the thread won't necessarily know what task it's associated with

like, for example, query-handling threads: in this case a query is accepted and handled by a shared QueryResource and QueryProcessingPool; the way it gets the segments from the correct task is through an impl of QuerySegmentWalker

that code doesn't activate until partway through the query handling process, so we wouldn't be able to "know" the task until that activation at the earliest

it could be fine, i suppose: i'm just trying to say that we wouldn't be able to capture _everything_ for reasons like this

we'd certainly be able to capture stuff logged by the main task thread itself

I see. Yeah, at least in my experience, I mostly have been looking at the ingestion related logs. For queries, I mostly try to look at metrics and exceptions.

&gt; we'd certainly be able to capture stuff logged by the main task thread itself
This seems to be the highest value information for the use case we are discussing.  Primarily I use these logs to diagnose failing ingestion.

I just did a few more investigation on this logging issue in the comments. <https://github.com/apache/druid/issues/13082>

What do y’all think?

task-log-coordinator-issued_kill_pmdata_kejdhffb_2022-09-13T04_00_00.000Z_2022-09-13T18_00_00.000Z_2022-09-14T03_12_36.229Z.log

[2022-09-13T04:00:00.000Z/2022-09-13T18:00:00.000Z] was revoked.","message":"Lock for interval [2022-09-13T04:00:00.000Z/2022-09-13T18:00:00.000Z] was revoked.
Are you running streaming ingestion/batch ingestion in that period on the data source ?

usually late date can come in streaming ingestion which creates older segments

is there any way forward to overcome this, it is piling up in deep storage and unused entries are not removed from the Postgres metadata store.

`druid.coordinator.kill.durationToRetain` is set to 60min. We can increase it so that the Kill tasks when started does not have the lock revoked because of late data arriving in your Kafka stream.

Another option is to set `lateMessageRejectionPeriod` in your Kafka stream which might drop the late data arriving. But this is not recommended as it will drop the events completely.

Also I think the Kill Tasks configs are too aggressive,
Whats the issue you are facing on the the postgress metadata store.

Please share your load rules as well.

there's this guide: <https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html|https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html>

beyond that if you have specific questions feel free to ask!

<@U0344FW86DD> Thanks for your reply!
Actually, I've read that link and got overwhelmed by the docs and the maths inside that :man-gesturing-ok:
So just want to check if there is a calculator to provide configurations and make my job easy :grinning:

not that i'm aware of, although it should ideally be built into the software itself, so look for that in the future

you can start from the small/medium/large/x-large configs bundled in the distribution; they match common server sizes

How can I find those configs that you mentioned?

I believe it's <https://github.com/apache/druid/tree/2a039e7e6a4ebe5d02987a3fc74f29311b07ce08/examples/conf/druid/single-server|here>  and here's the <https://druid.apache.org/docs/latest/operations/single-server.html#x-large-64-cpu-512gib-ram-i316xlarge|reference>. Then you have <https://druid.apache.org/docs/latest/tutorials/cluster.html|this> for the migration

in the distribution tarball (the binary download) the files are in `conf/druid/single-server`

Tried running this query on the latest master (whose behavior should be similar to the upcoming 24.0.0 release) and it shows the desired result for me. I will checkout the behavior in 0.22.1 as well.

i tried on 0.23.0 also , seeing same error
SELECT LATEST(channel,10),cityName FROM "wikipedia" WHERE (comment IN( 'added project') ) and (comment NOT IN ('added project' )) GROUP BY cityName

Able to reproduce the issue on a 0.22.1 cluster. Unfortunately I was running into some troubles while trying to run the cluster from IDE (due to some newer artefacts not being cleaned up) therefore I wasn’t able to debug it thoroughly. But from what it seems, the issue is due to incorrect parsing of the SQL query in the broker. Will try to run it on 0.23.0 as well.

I was thinking <https://github.com/apache/druid/pull/11949>, might have fixed it in the later versions, however since this commit is pre 0.23.0 I might be incorrect. Hopefully I am able to debug that one. In any case, once 24.0.0 is out, upgrade to that should help alleviate this problem.

Thanks <@U031AD802MN>
Any idea when 0.24 will be out?

Some fix would help for 0.22 or 0.23

No idea on an exact date, but it shouldn’t be long since AFAIK the release process for the same has started.

A temp workaround can be to run the native version of the query in 0.22.1/0.23.0. I will try to figure out the patch where it might have been fixed.

<https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1663346869505079>

Hi Team
With druid 24.0 also this issue is reproducible however on 0.21.1 this query paases
<@U031AD802MN> can you please let me know what steps you are following to not get the issue

getting the following stacktrace for this query
SELECT LATEST(channel,10),cityName FROM "wikipedia" WHERE (comment IN( 'added project') ) and (comment NOT IN ('added project' )) GROUP BY cityName

2022-09-19T05:17:00,916 WARN [sql[4e54d4ab-e32b-4ec2-88b1-a1b8f7dd505b]] org.apache.druid.sql.http.SqlResource - Failed to handle query: SqlQuery{query='SELECT LATEST(channel,10),cityName FROM "wikipedia" WHERE (comment IN( 'added project') ) and (comment NOT IN ('added project' )) GROUP BY cityName', resultFormat=array, header=true, typesHeader=true, sqlTypesHeader=true, context={sqlOuterLimit=1001, sqlQueryId=4e54d4ab-e32b-4ec2-88b1-a1b8f7dd505b}, parameters=[]}
java.lang.AssertionError: not a literal: $2
at org.apache.calcite.rex.RexLiteral.findValue(RexLiteral.java:1124) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.calcite.rex.RexLiteral.intValue(RexLiteral.java:1099) ~[calcite-core-1.21.0.jar:1.21.0]
at org.apache.druid.sql.calcite.aggregation.builtin.EarliestLatestAnySqlAggregator.toDruidAggregation(EarliestLatestAnySqlAggregator.java:210) ~[druid-sql-24.0.0.jar:24.0.0]
at org.apache.druid.sql.calcite.rule.GroupByRules.translateAggregateCall(GroupByRules.java:130) ~[druid-sql-24.0.0.jar:24.0.0]

:thinking_face: Thanks for trying it out in the latest Druid version.
Last I tried, it did work on the master build. I will test it out again later in the day. ~Can you also please share if you are setting any flags in the query context.~ (They are visible in the ST above)

Screenshot 2022-09-19 at 10.54.55 AM.png

Hi Laksh.. thanks for the quick reply
i am not giving any specific settings in query context.

Updated in git issues also
I think this change is causing the issue

i found this when checking the druid code hope this is helpful
In DruidPlanner.java -&gt; plan()
we have 2 functions
planWithDruidConvention() -&gt; this is for normal tables/datasources
planWithBindableConvention() -&gt; this is for bindable tables i.e. metadata table
&gt; `SELECT LATEST(channel,10),cityName FROM "wikipedia" WHERE (comment IN( 'added project') ) and (comment NOT IN ('added project' )) GROUP BY cityName`
for the above query
In 0.21.1
• the plan fails in planWithDruidConvention() with the follwoing exception and goes to planWithBindableConvention() as a fallback
&gt; org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are not enough rules to produce a node with desired properties: convention=DRUID, sort=[].
In 24.0
• I can see changes in decision path(`if (!bindableTables.isEmpty()) { }`).
• the plan goes to planWithDruidConvention() and it fails , there is no fallback

Screenshot 2022-09-20 at 12.00.51 AM.png

This is the output on my latest debug version. While it’s not exactly 24.0.0, it’s behavior should be similar

Thanks for debugging the issue. IMO planWithDruidConvention should not fail in the first part. I think the issue that we are seeing is in the aggregator and how it picks up the second parameter in the Latest/Earliest function.

<@U031AD802MN> in 24.0 released version.. the issue is reproducible… right?

Do you have some changes in code in your debug environment? Can that be released as a minor 

I don’t have any changes pertaining to this in my env. Let me try to reproduce it in 24.0.0. Wondering if it could be due to a certain config set in my IDE. Are you running any custom configs or are they same as the quickstart?

Same as quick start
Just inserting the wikipedia datasource then running the query

Hey, I didn’t get around to it after our last exchange. Apologies for the late response. Thanks for updating the Github Issue with the findings, will take a look at it in sometime.

Could you please share the replication factor and other retention configs for this datasource?

```{
  "type": "kafka",
  "spec": {
    "dataSchema": {
      "dataSource": "analytics",
      "timestampSpec": {
        "column": "timestamp",
        "format": "millis",
        "missingValue": null
      },
      "dimensionsSpec": {
        "dimensions": [
          {
            "type": "string",
            "name": "bundle",
            "multiValueHandling": "SORTED_ARRAY",
            "createBitmapIndex": true
          }
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "DAY",
        "queryGranularity": "MINUTE",
        "rollup": true,
        "intervals": []
      },
      "transformSpec": {
        "filter": null,
        "transforms": []
      }
    },
    "ioConfig": {
      "topic": "analytics",
      "inputFormat": {
        "type": "json",
        "flattenSpec": {
          "useFieldDiscovery": true,
          "fields": []
        },
        "featureSpec": {},
        "keepNullColumns": true
      },
      "replicas": 1,
      "taskCount": 2,
      "taskDuration": "PT86400S",
      "consumerProperties": {
        "bootstrap.servers": "kafka:9094"
      },
      "autoScalerConfig": null,
      "pollTimeout": 100,
      "startDelay": "PT5S",
      "period": "PT30S",
      "useEarliestOffset": false,
      "completionTimeout": "PT1800S",
      "lateMessageRejectionPeriod": null,
      "earlyMessageRejectionPeriod": null,
      "lateMessageRejectionStartDateTime": null,
      "stream": "analytics",
      "useEarliestSequenceNumber": false
    },
    "tuningConfig": {
      "type": "kafka",
      "appendableIndexSpec": {
        "type": "onheap",
        "preserveExistingMetrics": false
      },
      "maxRowsInMemory": 1000000,
      "maxBytesInMemory": 0,
      "skipBytesInMemoryOverheadCheck": false,
      "maxRowsPerSegment": 5000000,
      "maxTotalRows": null,
      "intermediatePersistPeriod": "PT10M",
      "basePersistDirectory": "/opt/apache-druid-0.17.0/var/tmp/druid-realtime-persist5729124401212538543",
      "maxPendingPersists": 0,
      "indexSpec": {
        "bitmap": {
          "type": "concise"
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "indexSpecForIntermediatePersists": {
        "bitmap": {
          "type": "concise"
        },
        "dimensionCompression": "lz4",
        "metricCompression": "lz4",
        "longEncoding": "longs",
        "segmentLoader": null
      },
      "reportParseExceptions": false,
      "handoffConditionTimeout": 0,
      "resetOffsetAutomatically": false,
      "segmentWriteOutMediumFactory": null,
      "workerThreads": null,
      "chatThreads": null,
      "chatRetries": 8,
      "httpTimeout": "PT10S",
      "shutdownTimeout": "PT80S",
      "offsetFetchPeriod": "PT30S",
      "intermediateHandoffPeriod": "P2147483647D",
      "logParseExceptions": true,
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0,
      "skipSequenceNumberAvailabilityCheck": false,
      "repartitionTransitionDuration": "PT120S"
    }
  },
  "context": null,
  "suspended": false
}```

image.png

But it doesn't seem to have much or any data

Could you please share the supervisor's and tasks' statistics and reports?

```{
  "dataSource": "analytics",
  "stream": "analytics",
  "partitions": 1,
  "replicas": 1,
  "durationSeconds": 86400,
  "activeTasks": [
    {
      "id": "index_kafka_analytics_9e38c45938839da_mbelbbdp",
      "startingOffsets": {
        "0": 82126
      },
      "startTime": "2022-09-14T13:40:11.133Z",
      "remainingSeconds": 12914,
      "type": "ACTIVE",
      "currentOffsets": {
        "0": 82130
      },
      "lag": {
        "0": 0
      }
    }
  ],
  "publishingTasks": [],
  "latestOffsets": {
    "0": 82130
  },
  "minimumLag": {
    "0": 0
  },
  "aggregateLag": 0,
  "offsetsLastUpdated": "2022-09-15T10:04:41.558Z",
  "suspended": false,
  "healthy": true,
  "state": "RUNNING",
  "detailedState": "RUNNING",
  "recentErrors": []
}```

The partition offsets and the segment values match

yes . Datasource was not there until i sent and event to kafka. (but i was there yesterday)

Could you mark as used all segments for the datasource? Perhaps they were marked as unused?

```Perhaps they were marked as unused?```

&gt; Could you mark as used all segments for the datasource?
Does this help? Getting marked as unused is a possible explanation as to why this may have happened

&gt; Could you mark as used all segments for the datasource?
how can this help to the issue? i am not very sure

if dind't manually mark the segments as unused . why does the segments will get marked as unused?

&gt; if dind't manually mark the segments as unused . why does the segments will get marked as unused?
I'm not sure. What I can say is that all the segments becoming unused can cause the datasource to "vanish" from the UI

but when datasource status is unhealthy  the datasource it's "vanish" from UI too. Is this a correct behaviour?  or something else it is happening?

i am not so sure to flag druid as productive with this issue!!!

By datasource status, do you mean "supervisor status"?

Did it have successful tasks with multiple segments published and handed off?

I guess a basic question is: Did it have multiple segments on the historicals at any point in time or is this a very new supervisor which has only had realtime segments with failing tasks?

i think "it is  a very new supervisor which has only had realtime segments with failing tasks"

Thank you for the clarification. "Today I found datasource was not there" made it seem like a very old datasource with a lot of data

hahah! i am sorry for this misunderstanding

ok so the issue is that task are failing before segments are published?

Would you please be able to share why the task was failing?

The log is deleted as soon as task is failing!

`druid.indexer.logs.kill.enabled` is this enabled?

i am not sure because i didn't  specified this on the setup. But it is enabled by default i think

If you didn't, it shouldn't be.  (It is false by default)

Have you changed the task log location from the default value?

```  druid.indexer.logs.type: s3
  druid.indexer.logs.s3Bucket: dsp-druid-indexer-logs
  druid.indexer.logs.s3Prefix: druid/indexing-logs
  druid.indexer.task.restoreTasksOnRestart: "true"```

```  druid.indexer.logs.type: s3
  druid.indexer.logs.s3Bucket: dsp-druid-indexer-logs
  druid.indexer.logs.s3Prefix: druid/indexing-logs
  druid.indexer.task.restoreTasksOnRestart: "true"```
Where have these been added? Could you add them in _common instead of middlemanager if that is the case?

the same setup and all druid components/roles

Also, just to clarify, is this also a new setup?
If yes, have you added and configured the s3 extension before configuring s3-based task long-term logs and possibly deep-storage?

this is a "new setup". I tried on the past. But i replicated this setup on a new work so it means the amazon account is new . May be there is an issue on aws s3.

The s3 issue may benefit from being started in a different thread (since it is a separate issue than the one in the main message)

sure i will check s3 setup . Deepstorage is with same s3 butcket and credentials so this can be the root .

i will change temporary to log locallly . S3 credentials looks ok!

```2022-09-15T12:19:21,461 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting. 
org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException                                                                                                   
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:520) ~[?:?]                                    
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281) ~[?:?]                                                            
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) ~[?:?]                                                            
        at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1297) ~[?:?]                                                                           
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1238) ~[?:?]                                                                                     
        at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1211) ~[?:?]                                                                                     
        at org.apache.druid.indexing.kafka.KafkaRecordSupplier.poll(KafkaRecordSupplier.java:128) ~[?:?]                                                                            
        at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.getRecords(IncrementalPublishingKafkaIndexTaskRunner.java:95) ~[?:?]                           
        at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:612) [druid-indexing-service-0.23.0.jar:0.23.0]    
        at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:265) [druid-indexing-service-0.23.0.jar:0.23.0]            
        at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:149) [druid-indexing-service-0.23.0.jar:0.23.0]                        
        at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) [druid-indexing-service-0.23.0
        at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) [druid-indexing-service-0.23.0
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]                                                                                                   
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]                                                                            
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]                                                                            
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]                                                                                                                      
Caused by: java.lang.InterruptedException                                                                                                                                           
        ... 17 more      ```

Could we have a new thread for this as it is a separate issue, please?

the datasource supervisor was on status "RUNNING"

i am having an error before publishing the segments.

i have kafka in a differente aws region. USA is kafka and Europe the Druid

there is any recomendation about having kafka far away from druid supervisor?

hmm… not that i know of, but maybe there's some way that is escaping me

I wonder if that’s an actual problem, or just a red herring

This just seems like the middle manager shutting down its task management executors. Do you see any other symptom too?

No, it did made me pause and look in a wrong direction when I was trying to figure out why middle manager was not restarting tasks (which was unrelated, and due to my misconfiguration)

Oh, cool. Yeah, some of these logs show up from time to time when services shut down their executors. I suppose we should ensure that they are logged more cleanly and don't raise false alarms.

Please let us know if you encounter any other issue. :slightly_smiling_face:

So, I have less of an issue, and more of a general question about scaling historical and middle manager nodes, I’ll post it on <#C0303FDCZEZ|>

What was your original ingestion spec that you used to separate the dimensions and metrics?

I wonder if this is happening specifically for the time range where you made the cutover

During compaction, Druid constructs a new dimensions and metrics list based on taking the union of what's there in existing segments

During the cutover time range, certain columns would be in both dimensions (older segments) and metrics (newer segments) hence the conflict

It's an interesting situation to be in, since mixing nonrolledup and rolledup data in the same datasource doesn't always make sense (for example, typically you want a `count` column in your rolled-up data so you can sum it at query time, but you don't need this in nonrolledup data)

If this is in fact your situation, then the right thing to do depends on the specifics of what change you made when you switched from nonrolledup to rolledup

In some cases it will be difficult to reconcile and you may need to either create a new datasource, or backfill from the original data if possible

In other cases it will be possible to reconcile with a manual compaction, which you could run to set things straight

Btw, a good way to think about this is: how do you query this data? Generally, the aggregators you use at query time would inform what you want to do with a manual compaction