Apache Druid

check the overlord log for the taskid.  It may have more information

Hi Ian, nothing in the Overlord log for this task...

hmmm, I would think there would be.  Try grep with the ID in the OL log as well as the MM log.

Does the console show a reason for the failure if you click on the task?

Sorry my bad... I grep on the wrong ID... I have that in the OL:
```2023-03-30T13:13:36,603 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] because: [Task is not in knownTaskIds]
2023-03-30T13:13:36,605 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: 6.1.245.106:8091, status 200 OK, response: {"task":"index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc"}
2023-03-30T13:13:36,705 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] because: [Task is not in knownTaskIds]
2023-03-30T13:13:36,707 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to worker: 6.1.245.106:8091, status 200 OK, response: {"task":"index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc"}
2023-03-30T13:13:42,519 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[6.1.245.106:8091] wrote FAILED status for task [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] on [TaskLocation{host='6.1.245.106', port=8101, tlsPort=-1}]
2023-03-30T13:13:42,519 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[6.1.245.106:8091] completed task[index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] with status[FAILED]
2023-03-30T13:13:42,519 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc
2023-03-30T13:13:42,520 ERROR [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Ignoring notification for already-complete task: {class=org.apache.druid.indexing.overlord.TaskQueue, task=index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc}
2023-03-30T13:13:42,521 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] because: [notified status change from task]
2023-03-30T13:13:42,521 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] on worker[6.1.245.106:8091]
2023-03-30T13:13:42,523 WARN [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Unknown task completed: index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc
2023-03-30T13:13:42,523 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task FAILED: index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc (1241680 run duration)
2023-03-30T13:13:42,523 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task[index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] went bye bye.```
`Shutdown [index_kafka_ms_bell_national_kpis_15min_3a971c91bedd138_ijilabcc] because: [Task is not in knownTaskIds]` can be the cause ?

Yes, that is the cause.  It looks like the overlord did not know of the running task and tried to issue a shutdown.  The task had already succeeded so I think it is ok that it was just marked failed by the overlord

Did the overlord restart during this task running or anything like that?

`The task had already succeeded` actually the task ends due to this shutdown (after only 20minutes running)...
and no , no restart

Why the OL wouldn't know the task ? not registered in Metadata/Zookeeper ? Don't know who handle that

The overall reason for this is the overlord instructed the task to be killed due to an issue (unknown task) and the task reports success in executing the kill and doing what it was instructed to do.

I see it mentions curator in the threads so I am guessing it was not found in zookeeper

the overlord did what it should in that it does not want rogue tasks running

OK the list of running tasks is stored in ZooK ?

there is a config that controls this you may want to explore -&gt; <https://druid.apache.org/docs/latest/configuration/index.html#overlord-operations>

`druid.indexer.runner.type`

that is the latest version, not sure what version you are on so it may not be set to that

We're in 24.0 so I guess it is `httpRemote` . How can I check that directly in the system ?

you can check in runtime.properties of overlord

damn...
```druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata```


Ok its only by dafault in 25.0
<https://github.com/apache/druid/pull/13092>

Thanks <@U03TAQRMJPQ>, will try to use httpRemote and see if its works better...

Capture d’écran 2023-03-30 à 15.41.53.png

Capture d’écran 2023-03-30 à 15.40.36.png

Hello Druid folks,
I’m ingesting data from Kafka (with HOUR segment granularity), and every hour I start `index_parallel` task to change partitioning from `numbered` to  `hashed`. Then every night, I start another `index_parallel` task that changes segment granularity from HOUR to DAY.
In my deepstorage, I can see segments with DAY granularity but I can also see segments with HOUR granularity (cf 1st screenshot).
When I run queries to check number of segments per day and per datasource, it does not match with what I’ve got in my deepstorage. (cf 2nd screenshot).
Is this an expected behaviour or am I missing something here?
Thanks a lot

enable this monitor on middle-manger end with this error ..

1) No implementation for org.apache.druid.server.metrics.TaskCountStatsProvider was bound.
while locating org.apache.druid.server.metrics.TaskCountStatsProvider
    for the 1st parameter of org.apache.druid.server.metrics.TaskCountStatsMonitor.&lt;init&gt;(TaskCountStatsMonitor.java:37)
  while locating org.apache.druid.server.metrics.TaskCountStatsMonitor
  at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
  at org.apache.druid.server.metrics.MetricsModule.getMonitorScheduler(MetricsModule.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
  while locating org.apache.druid.java.util.metrics.MonitorScheduler
  at org.apache.druid.server.metrics.MetricsModule.configure(MetricsModule.java:95) (via modules: com.google.inject.util.Modules$OverrideModule -&gt; com.google.inject.util.Modules$OverrideModule -&gt; org.apache.druid.server.metrics.MetricsModule)
  while locating org.apache.druid.java.util.metrics.MonitorScheduler annotated with @com.google.inject.name.Named(value=ForTheEagerness)

how can we know which monitor class can be enabled on which druid component ?

<https://www.druidforum.org/t/unable-to-configure-specific-emiiter/6454>
can you please expose the table in the link on the official documentation ..

and better to add this on each monitor class as a java doc ..

Hi Druid experts

for how many queries are served by druid
I am getting data from here
```sum(druid_query_success_count{cluster="$cluster",component="historical"})```
but from broker i am not getting any data is there any reason
```sum(druid_query_success_count{cluster="$cluster",component="overlord"})```


can you paste the overlord dynamic config here?

```{
    "selectStrategy": {
        "type": "fillCapacity"
    },
    "autoScaler": {
        "envConfig": {
            "numInstances": 1,
            "projectId": "gcp-druid-dev",
            "zoneName": "us-east4-c",
            "managedInstanceGroupName": "druid-middlemanagers"
        },
        "maxNumWorkers": 2,
        "minNumWorkers": 1,
        "type": "gce"
    }
}```

```druid.indexer.autoscale.strategy="gce"
druid.indexer.autoscale.doAutoscale=true
druid.indexer.autoscale.workerVersion=1
druid.indexer.autoscale.originTime=2023-03-30T15:00:00Z
druid.indexer.autoscale.workerIdleTimeout=PT30M```

I had an issue where the extension was not able to delete instance because it was running in a regional MIG

it is trying to delete the instance based on hostname, instead of name.

Also this extension is keep on deleting/creating instances every few secs. I have configured the min instance to 1.

what are druid.indexer.autoscale.provisionPeriod and druid.indexer.autoscale.terminatePeriod? Depending on pending tasks and these configs middle managers will be created and terminated

These are default which is PM1M and PT5M respectively. however cluster don’t have any tasks, even an idle cluster, auto scalar will continuously adding and removing nodes.

Do you see any error in the middle manager logs in those nodes? I am wondering whether the issue is with middle managers not coming up properly

I don’t see anything unusual, middlemanagers are getting registered and shows up in console also when task is submitted, it is able to run it. however, I do see a problem after sometime, tasks goes into pending state which only runs when new node is added back to cluster by auto scalar.

Pretty sure that one is for the `overlord` since it handles tasks

on my deployment - coordinator and overload are bind together ..
can I do something coordinator to have a better visablity if metrics are submited ?

are you seeing other druid metrics in data dog?

it seem that only the merrices added on TaskCountStatsMonitor , not shown on data-dog end ..

can you add TaskCountStatsMonitor to the common runtime properties and check?

if I add a montor on spcific component , it will not overide the common config ?

yours is in coordinator runtime properties?

I am not very sure whether org.apache.druid.server.metrics.TaskCountStatsMonitor should be in coordinator-overlord or middle manager runtime properties.

<https://www.druidforum.org/t/unable-to-configure-specific-emiiter/6454>

I specify it on corrdinator ..as mention above .

but I can try put it on common as well ..

do you know if I can open some logs to see if metric submited to data-dog ?

hmm, im a bit fuzzy on what is currently possible because in druid 26 its going to be a completely different story :sweat_smile:

<https://github.com/apache/druid/pull/13803>

so like i don’t actually recommend using master right now because im getting ready to open a PR sometime soon (like within the next 24 hrs soon because it needs to go into druid 26) that changes a ton about ^ because its backwards incompatible

but also does a totally rad thing to optimize the way the serialization of ^ works to write out only what is necessary for the specific type of column

For context, parquet data can be stored as list (i.e. list of double) but for other types like tsv, csv, etc I can seem to get druid ingestion to make the value into a array.

Context part 2. I want to feed an array into an aggregate for example building a double sketch from an array of double. This is useful for when your data (i.e. exisiting druid table or external table) is already rollup with the values stored as an array.


Use case is for building/changing type of sketch on a table that is already roll up

i feel like i might be aware of some bug with parquet array conversion, but it might be lists of lists that have the problem, let me check

ah its nested arrays,`[[3,4], [5,6]]` currently ends up something like `[{"list":[3,4]},{"list":[5,6]}]`  i think

there was another bug previously too <https://github.com/apache/druid/pull/13294>

expression transform array handling was improved by <https://github.com/apache/druid/pull/13947>

btw, here is the follow-up PR that reworks the first array PR <https://github.com/apache/druid/pull/14014>

image.png

im not sure array casts are wired up in SQL right now, will check

but the native syntax is like `cast(['1.0', '2.0', '3.0'], 'ARRAY&lt;LONG&gt;')`

Can it be use at ingestion time (in a transform)? 

well, i think it can but i think there were some bugs with trying to do array stuff in expression transforms prior to <https://github.com/apache/druid/pull/13947>

they weren’t very well tested because the only type of column that could really hold them before was multi-value strings

Here is MM configuration for emitter and monitors.

```    druid_indexer_fork_property_druid_emitter: prometheus
    druid_indexer_fork_property_druid_emitter_prometheus_strategy: pushgateway
    druid_indexer_fork_property_druid_emitter_prometheus_pushGatewayAddress: ...
    druid_indexer_fork_property_druid_monitoring_monitors:
      '
        [
          "org.apache.druid.java.util.metrics.JvmMonitor", 
          "org.apache.druid.java.util.metrics.JvmCpuMonitor"
        ]
      '```

Hi, the streaming lag metrics are emitted by the Overlord

you may include those monitors as part of `common.runtime.properties`

Hi <@U0319HD8HEC>, <@U04FY7YTU4F> Thanks for the answer.

I am running the overload inside coordinator. Which monitor should I set in the coordinator configuration?

Currently, I did set these monitors for coordinator.

```    druid_monitoring_monitors:
      '
        [
          "org.apache.druid.java.util.metrics.JvmMonitor", 
          "org.apache.druid.java.util.metrics.JvmCpuMonitor",
          "org.apache.druid.server.metrics.TaskCountStatsMonitor"
        ]
      '```

Do you see `task/action/run/time`  as one of the metrics from the overlord? If not, it's possible that your overlord is not emitting metrics

<@U0319HD8HEC> I can’t see the task_action_run_time metric from coordinator prometheus endpoint (:9090)

I will attach the whole metrics.

Should I run overlord separately in case of kubernetes environment?

It'd be great if you could try that. It appears that the metrics here probably belong only to the coordinator and not the overlord

I executed overlord service separately but still not getting the `task/action/run/time` metrics from overlord.

overlord-metrics

Seems kinesis indexing service doesn’t report lag

```{
"0": {
"index_kinesis_track_kinesis_r0_60d0414dfa8c05d_nhpoiddd": {
"movingAverages": {
"buildSegments": {
"5m": {
"processed": 0.33287108369807955,
"unparseable": 0,
"thrownAway": 0,
"processedWithError": 0
},
"15m": {
"processed": 0.6930406200864409,
"unparseable": 0,
"thrownAway": 0,
"processedWithError": 0
},
"1m": {
"processed": 0.00408677143846408,
"unparseable": 0,
"thrownAway": 0,
"processedWithError": 0
}
}
},
"totals": {
"buildSegments": {
"processed": 5,
"processedWithError": 0,
"thrownAway": 0,
"unparseable": 0
}
}
}
}
}```

Can you share a task log with the exception. What Druid version are you on?

We are on v25.0.0
logs here
```2023-04-05T00:55:23,310 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_kafka_private.DATAPLATFORMV2.events_3f76be529bcb647_gcgcihkh",
  "status" : "FAILED",
  "duration" : 3397,
  "errorMsg" : "org.apache.druid.java.util.common.parsers.ParseException: Failed to parse avro message.\n\tat org.apac...",
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}
2023-04-05T00:55:23,315 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2023-04-05T00:55:23,327 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2023-04-05T00:55:23,331 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@2e60d4f4{HTTP/1.1, (http/1.1)}{0.0.0.0:8100}
2023-04-05T00:55:23,332 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging
2023-04-05T00:55:23,333 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@272485a6{/,null,STOPPED}
2023-04-05T00:55:23,336 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2023-04-05T00:55:23,336 INFO [main] org.apache.druid.server.coordination.ZkCoordinator - Stopping ZkCoordinator for [DruidServerMetadata{name='10.40.89.93:8100', hostAndPort='10.40.89.93:8100', hostAndTlsPort='null', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}]
2023-04-05T00:55:23,336 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopping...
2023-04-05T00:55:23,337 INFO [main] org.apache.druid.server.coordination.SegmentLoadDropHandler - Stopped.
2023-04-05T00:55:23,337 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[index_kafka_private.DATAPLATFORMV2.events_3f76be529bcb647_gcgcihkh].
2023-04-05T00:55:23,337 INFO [main] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Stopping forcefully (status: [READING])
2023-04-05T00:55:23,406 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2023-04-05T00:55:23,407 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopping.
2023-04-05T00:55:23,407 INFO [main] org.apache.druid.security.basic.authorization.db.cache.CoordinatorPollingBasicAuthorizerCacheManager - CoordinatorPollingBasicAuthorizerCacheManager is stopped.
2023-04-05T00:55:23,407 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopping.
2023-04-05T00:55:23,407 INFO [main] org.apache.druid.security.basic.authentication.db.cache.CoordinatorPollingBasicAuthenticatorCacheManager - CoordinatorPollingBasicAuthenticatorCacheManager is stopped.
2023-04-05T00:55:23,408 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2023-04-05T00:55:23,513 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10005abdb2e053b
2023-04-05T00:55:23,513 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x10005abdb2e053b closed
2023-04-05T00:55:23,517 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task```

I'm looking into it. I have the same expectation you do. It should skip the parse errors, I need to spend some time in the code to try to understand where it might be going wrong. Can you share your ingestion spec?

here is the spec., pretty much standard except that we use a custom avro decoder that we wrote to handle avro schema embedded in the message rather than inline or registry based.
```{
  "type": "kafka",
  "spec": {
    "ioConfig": {
      "type": "kafka",
      "consumerProperties": {
        ....
      },
      "topic": "topic-abc",
      "inputFormat": {
        "type": "kafka",
        "valueFormat": {
          "type": "avro_stream",
          "avroBytesDecoder": {
            "type": "glo_schema" #this is our custom decoder, where we process schema that is embedded in the message, and throws ParseException based on schema/payload validation
          }
        },
        "flattenSpec": {
          "useFieldDiscovery": true
        },
        "binaryAsString": false
      },
      "appendToExisting": false,
      "useEarliestOffset": true,
      "idleConfig": {
        "enabled": true,
        "inactiveAfterMillis": 300000
      }
    },
    "tuningConfig": {
      "type": "kafka",
      "maxParseExceptions": 2147483647,
      "maxSavedParseExceptions": 0
    },
    "dataSchema": {
      .....
    }
  }
}```

Here's what I found. I'm not sure this is the reason, I think a committer may need to take a look: <https://github.com/apache/druid/blob/58a3acc2c4c5d5d22098e15f7ff25d6e9baa8541/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/StreamChunkParser.java#L148|https://github.com/apache/druid/blob/58a3acc2c4c5d5d22098e15f7ff25d6e9baa8541/inde[…]org/apache/druid/indexing/seekablestream/StreamChunkParser.java>
I'm wondering whether the byteEntityReader.read() is referencing you reader and when/if it throws the ParseException here it would not use the ParseExceptionHandler. Not sure what the solution is, but I think this may be the cause.

The resulting iterator has the parseException logic in it here: <https://github.com/apache/druid/blob/f2ac6cd96edfacca6ba7880d5eb9c532ea0b95f2/indexing-service/src/main/java/org/apache/druid/indexing/common/task/FilteringCloseableInputRowIterator.java#L60|https://github.com/apache/druid/blob/f2ac6cd96edfacca6ba7880d5eb9c532ea0b95f2/inde[…]id/indexing/common/task/FilteringCloseableInputRowIterator.java>
But it won't get there if the byteEntityReader.read() throws the exception.

A more detailed stack trace from the full task log would help us confirm.

<@U030K4UM3H7> as you mentioned, apparently looks like its not hitting the handler.

```2023-04-06T04:38:07,887 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting.
org.apache.druid.java.util.common.parsers.ParseException: Failed to parse avro message from this origin:unknown
	at org.apache.druid.data.input.avro.GlobalityAvroBytesDecoder.parse(GlobalityAvroBytesDecoder.java:111) ~[?:?]
	at org.apache.druid.data.input.avro.AvroStreamReader.intermediateRowIterator(AvroStreamReader.java:70) ~[?:?]
	at org.apache.druid.data.input.IntermediateRowParsingReader.intermediateRowIteratorWithMetadata(IntermediateRowParsingReader.java:231) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.data.input.IntermediateRowParsingReader.read(IntermediateRowParsingReader.java:49) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.data.input.kafkainput.KafkaInputReader.buildBlendedRows(KafkaInputReader.java:158) ~[?:?]
	at org.apache.druid.data.input.kafkainput.KafkaInputReader.read(KafkaInputReader.java:129) ~[?:?]
	at org.apache.druid.segment.transform.TransformingInputEntityReader.read(TransformingInputEntityReader.java:43) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SettableByteEntityReader.read(SettableByteEntityReader.java:70) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.StreamChunkParser.parseWithInputFormat(StreamChunkParser.java:135) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.StreamChunkParser.parse(StreamChunkParser.java:104) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:635) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:266) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.runTask(SeekableStreamIndexTask.java:151) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 36 out of bounds for length 2
	at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:460) ~[?:?]
	at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:284) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) ~[?:?]
	at org.apache.druid.data.input.avro.GlobalityAvroBytesDecoder.parse(GlobalityAvroBytesDecoder.java:95) ~[?:?]
	... 19 more```

task does not immediately shutdown though, but get’s into `StreamAppenderatorDriver` and shuts down after this
```2023-04-06T04:38:07,977 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task.
org.apache.druid.java.util.common.parsers.ParseException: Failed to parse avro message from this origin:unknown
	at org.apache.druid.data.input.avro.GlobalityAvroBytesDecoder.parse(GlobalityAvroBytesDecoder.java:111) ~[?:?]
	at org.apache.druid.data.input.avro.AvroStreamReader.intermediateRowIterator(AvroStreamReader.java:70) ~[?:?]
	at org.apache.druid.data.input.IntermediateRowParsingReader.intermediateRowIteratorWithMetadata(IntermediateRowParsingReader.java:231) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.data.input.IntermediateRowParsingReader.read(IntermediateRowParsingReader.java:49) ~[druid-core-25.0.0.jar:25.0.0]
	at org.apache.druid.data.input.kafkainput.KafkaInputReader.buildBlendedRows(KafkaInputReader.java:158) ~[?:?]
	at org.apache.druid.data.input.kafkainput.KafkaInputReader.read(KafkaInputReader.java:129) ~[?:?]
	at org.apache.druid.segment.transform.TransformingInputEntityReader.read(TransformingInputEntityReader.java:43) ~[druid-processing-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SettableByteEntityReader.read(SettableByteEntityReader.java:70) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.StreamChunkParser.parseWithInputFormat(StreamChunkParser.java:135) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.StreamChunkParser.parse(StreamChunkParser.java:104) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:635) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:266) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.runTask(SeekableStreamIndexTask.java:151) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-25.0.0.jar:25.0.0]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 36 out of bounds for length 2
	at org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:460) ~[?:?]
	at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:284) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247) ~[?:?]
	at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) ~[?:?]
	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) ~[?:?]
	at org.apache.druid.data.input.avro.GlobalityAvroBytesDecoder.parse(GlobalityAvroBytesDecoder.java:95) ~[?:?]
	... 19 more
2023-04-06T04:38:07,984 DEBUG [task-runner-0-priority-0] org.apache.druid.indexing.common.task.AbstractTask - Not pushing task logs and reports from task.
2023-04-06T04:38:07,984 DEBUG [task-runner-0-priority-0] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_private.DATAPLATFORMV2.events_dcfe8936f2ad337_aepmioak] status changed to [FAILED].
2023-04-06T04:38:07,986 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_kafka_private.DATAPLATFORMV2.events_dcfe8936f2ad337_aepmioak",
  "status" : "FAILED",
  "duration" : 3710,
  "errorMsg" : "org.apache.druid.java.util.common.parsers.ParseException: Failed to parse avro message from this ori...",
  "location" : {
    "host" : null,
    "port" : -1,
    "tlsPort" : -1
  }
}```

in my decoder code, I am just throwing
```throw new ParseException(
            null,
            false,
            "Failed to parse avro message from this origin:%s",origin
        );```

Right. So it seems like whenever a byteEntityDecoder is used it will have this behavior. I've created a bug report on this here: <https://github.com/apache/druid/issues/14041>

Not that I know of. But you could use a <https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html#transforms|transformation> with a <https://druid.apache.org/docs/latest/misc/math-expr.html#general-functions|case_searched expression>.

Or an nvl(), if you want a default for missing value.

Data is stored on disk in a columnar format, but logically it's written by the row. So, you can't ingest two different columns for one row independently, if that's what you mean.

also answered in <#C0309C9L90D|>.  (fyi, <#C0303FDCZEZ|> would probably have been the best channel to ask in.)

Screenshot 2023-04-04 at 9.27.29 PM.png

have you tried using the data loader in the web console? The UI will guide you through the process and build the spec correctly. In the last step you can hand edit the spec / copy it to use it from another tool as needed.

Screenshot 2023-04-04 at 9.29.53 PM.png

&gt; use druid deep segment data to re ingest data to same datasource by adding partition columns
I don't think I fully understand this. If you are trying to change the partitioning scheme for a datasource, you can also do this via the auto-compaction config

+1 about the second point.  (What are partition columns?)

We tested with one spec but it is not working 

What are your DR recovery time and recovery point objectives?
Here are a few thoughts on this:
A Druid Cluster can be recreated/restored as long as you have:
• Deep Storage segment files ( typically already highly resilient like <https://aws.amazon.com/blogs/storage/architecting-for-high-availability-on-amazon-s3/|S3 HA> )
• Metadata DB: You'll need to setup your metadata repository for high availability. 
If you lose an entire Druid Cluster you can deploy a brand new cluster using the same Deep Storage and Metadata DB. The segment data will take some time to be downloaded into the new historical nodes. Any streaming tasks will be restarted and will start ingesting from the most recent checkpoint. In order to safeguard the streaming ingestion, the streaming platform should have enough retention to cover the time it takes to re-deploy the Druid cluster. <https://www.druidforum.org/t/re-druid-user-druid-disaster-recovery/6323| Here's another discussion on the subject>.

data_duplicates.png

Hi, could you share your Druid version?
If you're using a version with MSQ, could you please try using SQL based ingestion on the same dataset into a different datasource?

Druid:25.0.0  I am using batch ingestion. Metadata store is postgresql.

25.0.0 does have MSQ / SQL based ingestion. Could you please try it out on the same dataset?

<@U03BJAMQ4H5> The timestamps are not unique. But I guess that should not be the problem.

Is the first row in your screenshot supposed to be the timestamp?

If yes, it appears that the duplicate rows that you shared don't belong to the intervals in the spec

Yes,its timestamp. The interval is adjusted via code.

  datestr=yyyy+"-"+mm+"-"+dd
  spec["spec"]["dataSchema"]["granularitySpec"]["intervals"] = [datestr+"T09:00:00.000000Z/"+datestr+"T17:00:00.000000Z"]

Right, but your screenshot shows that the data has a timestamp in 2018

interval_payload.png

Yes, like I mentioned interval is adjusted via code. You can check payload(pfa) for the task. apologies for confusion.

the issue is likely with the uri. I will scan all the folders in that bucket. Do you have multiple copies of that file or files with identical data? Go folder by folder and ingest each folder sperataely and check if the issue is still there

<@U03BJAMQ4H5> , I use complete path in the ingestion spec. I am not using prefix. This is how uri in the payload looks.

"ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "s3",
        "uris": [
          "<s3://bucket-name/2018/01/01/Future.csv>"
        ],

I would print out all the uris and check. 

Thanks a lot Vijay for your time. if possible, can you point me to a working parquet ingestion spec? I am generating parquet and csv using python pandas dataframe. I will give a shot with parquet format as well.

I think the issue is you are trying to submit a supervisor spec with tyep index_parallel?  Shouldnt it be type kinesis?

Hi <@U03TAQRMJPQ> Here is the spec I submitted

oh! I used the wrong button. I should have submitted the task... not the supervisor.

Was just going to ask how you submitted it

Yes, however have another question. The index_parallel task is constantly failing with this error. Could you help?

``` 2023-04-05T15:04:32,036 ERROR [qtp685452027-113] com.sun.jersey.spi.container.ContainerResponse - A message body writer for Java class com.google.common.collect.SingletonImmutableBiMap, and Java type class com.google.common.collect.Single │
│ The registered message body writers compatible with the MIME media type are:                                                                                                                                                                   │
│ */* -&gt;                                                                                                                                                                                                                                         │
│   com.sun.jersey.core.impl.provider.entity.FormProvider                                                                                                                                                                                        │
│   com.sun.jersey.core.impl.provider.entity.StringProvider                                                                                                                                                                                      │
│   com.sun.jersey.core.impl.provider.entity.ByteArrayProvider                                                                                                                                                                                   │
│   com.sun.jersey.core.impl.provider.entity.FileProvider                                                                                                                                                                                        │
│   com.sun.jersey.core.impl.provider.entity.InputStreamProvider                                                                                                                                                                                 │
│   com.sun.jersey.core.impl.provider.entity.DataSourceProvider                                                                                                                                                                                  │
│   com.sun.jersey.core.impl.provider.entity.XMLJAXBElementProvider$General                                                                                                                                                                      │
│   com.sun.jersey.core.impl.provider.entity.ReaderProvider                                                                                                                                                                                      │
│   com.sun.jersey.core.impl.provider.entity.DocumentProvider                                                                                                                                                                                    │
│   com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider                                                                                                                                                                             │
│   com.sun.jersey.core.impl.provider.entity.SourceProvider$SourceWriter                                                                                                                                                                         │
│   com.sun.jersey.server.impl.template.ViewableMessageBodyWriter                                                                                                                                                                                │
│   com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider                                                                                                                                                                                         │
│   com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider                                                                                                                                                                                       │
│   com.sun.jersey.core.impl.provider.entity.XMLRootElementProvider$General                                                                                                                                                                      │
│   com.sun.jersey.core.impl.provider.entity.XMLListElementProvider$General                                                                                                                                                                      │
│ application/octet-stream -&gt;                                                                                                                                                                                                                    │
│   com.sun.jersey.core.impl.provider.entity.ByteArrayProvider                                                                                                                                                                                   │
│   com.sun.jersey.core.impl.provider.entity.FileProvider                                                                                                                                                                                        │
│   com.sun.jersey.core.impl.provider.entity.InputStreamProvider                                                                                                                                                                                 │
│   com.sun.jersey.core.impl.provider.entity.DataSourceProvider                                                                                                                                                                                  │
│   com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider                                                                                                                                                                             │
│                                                                                                                                                                                                                                                │
│ 2023-04-05T15:04:32,036 ERROR [qtp685452027-113] com.sun.jersey.spi.container.ContainerResponse - Mapped exception to response: 500 (Internal Server Error)                                                                                    │
│ javax.ws.rs.WebApplicationException: com.sun.jersey.api.MessageException: A message body writer for Java class com.google.common.collect.SingletonImmutableBiMap, and Java type class com.google.common.collect.SingletonImmutableBiMap, and M │
│                                                                                                                                                                                                                                                │
│     at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:284) ~[jersey-server-1.19.4.jar:1.19.4]                                                                                                                     │
│     at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510) ~[jersey-server-1.19.4.jar:1.19.4]                                                                                               │
│     at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.4.jar:1.19.4]                                                                                                │
│     at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.4.jar:1.19.4]                                                                                                │
│     at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.4.jar:1.19.4]                                                                                                                    │
│     at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.4.jar:1.19.4]                                                                                                            │
│     at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.4.jar:1.19.4]                                                                                                            │
│     at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]                                                                                                                                       │
│     at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]                                                                                                                      │
│     at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]                                                                                                                          │
│     at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]                                                                                                                            │
│ ```

&gt; Caused by: com.sun.jersey.api.MessageException: A message body writer for Java class com.google.common.collect.SingletonImmutableBiMap, and Java type class com.google.common.collect.SingletonImmutableBiMap, and MIME media type application /octet-stream was not found.                                                                                                                                                                                                                  


hmmm does your spec pass without any formatting errors?

Yes, I was able to parse Parquet file on Druid console.

Update: Only literals are parsed.

How druid handle parquet’s map and struct type? I wanted to convert them into COMPELX&lt;JSON&gt; but seems not working.

I would take a look at the docs on parquet ingestion -&gt; <https://druid.apache.org/docs/latest/ingestion/data-formats.html#parquet>

I thought the transfer spec would work well.

I will read the Parquet Ingestion documentation what you gave to me
and try to test whether I can change the Map and Struct types to Druid’s COMPLEX JSON types with a flatten Spec.

I think there is an error between coordinator and middle manager. Here is coordinator log.

``` <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Failed to assign task[index_parallel_druid_import_test_fkgmjomk_2023-04-05T15:29:25.907Z] to worker[10.100.127.172:8091]. Response Code[500] and Message[]. Retrying...                                  │
│     at org.apache.druid.indexing.overlord.hrtr.WorkerHolder.lambda$assignTask$4(WorkerHolder.java:265) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                                                             │
│     at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-25.0.0.jar:25.0.0]                                                                                                                                 │
│     at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-25.0.0.jar:25.0.0]                                                                                                                                  │
│     at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-25.0.0.jar:25.0.0]                                                                                                                                 │
│     at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-25.0.0.jar:25.0.0]                                                                                                                                 │
│     at org.apache.druid.indexing.overlord.hrtr.WorkerHolder.assignTask(WorkerHolder.java:246) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                                                                      │
│     at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.runTaskOnWorker(HttpRemoteTaskRunner.java:412) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                                                 │
│     at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.pendingTasksExecutionLoop(HttpRemoteTaskRunner.java:1252) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                                      │
│     at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.lambda$startPendingTaskHandling$8(HttpRemoteTaskRunner.java:1139) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                              │
│     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]                                                                                                                                                          │
│     at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]                                                                                                                                                                         │
│     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]                                                                                                                                                  │
│     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]                                                                                                                                                  │
│     at java.lang.Thread.run(Thread.java:829) ~[?:?]                                                                                                                                                                                            │
│ 2023-04-05T15:29:26,893 WARN [hrtr-pending-tasks-runner-0] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 2) in 2,545ms.                                                                                                        │
│ <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Failed to assign task[index_parallel_druid_import_test_fkgmjomk_2023-04-05T15:29:25.907Z] to worker[10.100.127.172:8091]. Response Code[500] and Message[]. Retrying...                                  │
│     at org.apache.druid.indexing.overlord.hrtr.WorkerHolder.lambda$assignTask$4(WorkerHolder.java:265) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                                                             │
│     at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-core-25.0.0.jar:25.0.0]                                                                                                                                 │
│     at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-core-25.0.0.jar:25.0.0]                                                                                                                                  │
│     at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-core-25.0.0.jar:25.0.0]                                                                                                                                 │
│     at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-core-25.0.0.jar:25.0.0]                                                                                                                                 │
│     at org.apache.druid.indexing.overlord.hrtr.WorkerHolder.assignTask(WorkerHolder.java:246) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                                                                      │
│     at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.runTaskOnWorker(HttpRemoteTaskRunner.java:412) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                                                 │
│     at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.pendingTasksExecutionLoop(HttpRemoteTaskRunner.java:1252) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                                      │
│     at org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner.lambda$startPendingTaskHandling$8(HttpRemoteTaskRunner.java:1139) ~[druid-indexing-service-25.0.0.jar:25.0.0]                                                              │
│     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]                                                                                                                                                          │
│     at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]                                                                                                                                                                         │
│     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]                                                                                                                                                  │
│     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]                                                                                                                                                  │
│     at java.lang.Thread.run(Thread.java:829) ~[?:?]                                                                                                                                                                                            │
│ 2023-04-05T15:29:28,918 INFO [org.apache.druid.metadata.SqlSegmentsMetadataManager-Exec--0] org.apache.druid.metadata.SqlSegmentsMetadataManager - Polled and found 24 segments in the database                                                │
│ ```

Parquet files are parsed well even with struct and map types. I tested.

that looks like it cannot talk to the middlemanager. are you able to submit any tasks at all?

Yes, Kinesis Indexing service were submited and works well.

is 10.100.127.172 node healthy?  How many middlemanagers do you have?

I have 1 MM and and it’s healthy.

I am trying to use ‘remote’ option for `druid.indexer.runner.type` instead of `httpRemote`

How many slots does the MM have and can you try httpRemote?

MM has 5 slots.

I will try the `httpRemote` option and will check MM logs too. (I just stopped 1 kinesis indexing service)

That IP does not match what it was trying to submit to?

I restarted the MM kubernetes pod for `remote` option.

Now it’s working well. These are the options I modified

```druid.indexer.runner.type: remote
druid.indexer.storage.type: metadata```

the default option of `druid.indexer.runner.type` httpRemote works well too!
I had to set `druid.indexer.storage.type: metadata` :disappointed:

If it wasn’t for you, I would have stopped the Druid POC thinking it was a Parquet Format problem that couldn’t handle Map and Struct types. Thank <@U03TAQRMJPQ>  very much.

You are very welcome! Have a great day <@U051RJ7VA9W>

You might be able to use <https://druid.apache.org/docs/latest/querying/nested-columns.html|Nested Columns> which have support for AVRO in Druid 25.0. With a nested column, you can dynamically change the nested fields and the <https://druid.apache.org/docs/latest/querying/nested-columns.html#streaming-ingestion|streaming ingestion >will automatically index each field. so if you put all new fields into the nested column, that should resolve this.
Let us know if this helps. There are also improvements coming in Apache Druid 26.0 that enhance automatic column detection in general (not just in nested columns).

Was the disk full or did it hit the configured maxSize? For the first case I believe it crashes.

kk. Ideally, should it crash? it seems… like an unstable behavior.

I guess it could probably be made more resilient to this type of failure but the question would then be what is the proper behavior. The easiest way to mitigate this is to use a separate disk/disks for segment-cache and set maxSize &lt; total disk size. This should make it very unlikely to happen.

oh, ok. That’s easy to do. No big deal.

Have you successfully ingested any data on this cluster? I'm wondering if this is a Deep Storage configuration issue.

This is the first time ingestion after setting up the cluster in K8s and its failing with above error.

The deep storage I am using is a Pure FlashBlade Storage with S3 API.

*DeepStorage Configuration:*
druid.storage.type=s3
druid.storage.bucket=nonprod-druid-deepstorage
druid.storage.baseKey=druid/segments
druid.s3.accessKey=****************************
druid.s3.secretKey=******************************
druid.s3.enabePathStyleAccess=true
druid.s3.endpoint.url=http://&lt;IP_Address

Sounds like a problem on the FlashBlade side in supporting S3 API that Druid is using. Can you see the error on the FlashBlade side? Perhaps there is more detail there.
I'm not sure whether there are Druid config changes that would help make it compatible by changing how Druid uses S3 API. Some possibilities are:
```- druid.storage.disableAcl (this defaults to false) using true would use Object ownership permissions instead of ACL permissions.
- druid.s3.disableChunkedEncoding
- druid.s3.forceGlobalBucketAccessEnabled```
<https://druid.apache.org/docs/latest/development/extensions-core/s3.html#connecting-to-s3-configuration|More details on these parameters here>.

Thanks <@U030K4UM3H7>, I will take a look into these parameters and keep you posted.

Adding the above parameters seem to have worked, I can see the DataSource segments are created in the bucket.

Thanks <@U030K4UM3H7> :slightly_smiling_face:

Screenshot 2023-04-06 at 6.38.41 PM.png

Did you provide 4 paths in your s3 input source ?
So each file is assigned to each worker. Hence you get very less improvement by increasing workers since the slowest stage ie the stage 0 is always capped at 4 workers.

Some more documentation : <https://druid.apache.org/docs/latest/multi-stage-query/concepts.html#read-external-data-with-extern>

Also, if the parquet input column is boolean, it cannot be parsed? That seemed wrong.

the parquet column is boolean and the Druid column is “long” (per documentation). And we got that error above.

Based on the code, it makes sense why the exception is raised:
```      final ColumnDescriptor.Builder builder = ColumnDescriptor.builder();
      ValueType type = metricsValueTypes.get(metric);
      switch (type) {
        case LONG:
          builder.setValueType(ValueType.LONG);
          builder.addSerde(createLongColumnPartSerde(writer, indexSpec));
          break;
        case FLOAT:
          builder.setValueType(ValueType.FLOAT);
          builder.addSerde(createFloatColumnPartSerde(writer, indexSpec));
          break;
        case DOUBLE:
          builder.setValueType(ValueType.DOUBLE);
          builder.addSerde(createDoubleColumnPartSerde(writer, indexSpec));
          break;
        case COMPLEX:
          final String typeName = metricTypeNames.get(metric);
          builder.setValueType(ValueType.COMPLEX);
          builder.addSerde(
              ComplexColumnPartSerde
                  .serializerBuilder()
                  .withTypeName(typeName)
                  .withDelegate(writer)
                  .build()
          );
          break;
        default:
          throw new ISE("Unknown type[%s]", type);
      }```
But what doesn’t make sense is why `type` coming in is Boolean. In our ingestion spec, we specifically said `long`

I think this is a bug on native ingestion spec. If input data is BOOLEAN, it should be converted to LONG based on documentation.

```this_is_our_column_from_parquet=false```

i feel like i have fixed some bugs around boolean handling recently, will check to see if they have been released and it was longer ago than i remember or not

oof, I think the long dimension indexer doesn’t actually handle booleans

I don’t see `ColumnCapabilitiesImpl` containing any mentions of boolean

ah… is that why the docs say that boolean should use long?

the native boolean expressions all deal in longs is probably why

but amusingly you can’t ingest them directly into a long (today)

i merged a thing recently that will be a better way to ingest booleans as longs in druid 26, an ‘auto’ dimension schema <https://github.com/apache/druid/pull/14014> (pardon the terse PR description, haven’t got around to filling it out yet)

its related to a set of work i’ve been doing around making the nested column indexer suitable to use for schema auto discovery

i could also fix the classic long dimension indexer to support booleans since its sort of wack the docs say that and it doesn’t work out of the box

the ‘auto’ schema spits out the most appropriate type of column based on the inputs it processes

with all the features of the nested columns, so the numbers have value indexes and stuff

which hasn’t made its way back to the classic number column indexers

<https://github.com/apache/druid/pull/14069> adds support to the long indexer for booleans

You could stop a historical and clear its segment cache, then restart it.

:+1: That'll test reading from it. You can also do a quick wikipedia ingestion to make sure you can write to it.

yeah just go into the data loader, select the example dataset (wikipedia) and click next next next until you submit the job. Then make sure it works fine and the new datasource appears in the query tab

Looks like it is having trouble finding the overlord leader. How many overlords do you have? Are they restarting or something?

we have 2 overlords . no they are up and running .

but at any given time we will have only one overlord as leader right ?

ok .. let me stop one overlord and see if it helps

3 = highly available you need at least 3 nodes for zookeeper to elect a leader

I guess you could have more then 3 but that is just a waste of money

this is our non prod cluster . will test and if it is working then will add 3 overlords .

if it is non prod (and does not need to be highly available) just do 1

Save $ and also energy, and emit less CO2 :slightly_smiling_face:

Sure, one more qq how can i make sure druid logs rotate every day .?

I thought that was the default... <https://druid.apache.org/docs/latest/configuration/logging.html>

we are using .25 overlord log is not rotating and it is occupying around 80GB in a day

was this cluster set up in an older version of Druid and then upgraded (inheriting the old logging configs)?

we have .22 and upgraded to .25 recently on AWS

i see it is present
```&lt;!-- Rolling Files--&gt;
    &lt;RollingRandomAccessFile name="FileAppender"
                             fileName="${sys:druid.log.path}/${sys:druid.node.type}.log"
                             filePattern="${sys:druid.log.path}/${sys:druid.node.type}.%d{yyyyMMdd}.log"&gt;
      &lt;PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/&gt;
      &lt;Policies&gt;
        &lt;TimeBasedTriggeringPolicy interval="1" modulate="true"/&gt;
      &lt;/Policies&gt;
      &lt;DefaultRolloverStrategy&gt;
        &lt;Delete basePath="${sys:druid.log.path}/" maxDepth="1"&gt;
          &lt;IfFileName glob="*.log" /&gt;
          &lt;IfLastModified age="7d" /&gt;
        &lt;/Delete&gt;
      &lt;/DefaultRolloverStrategy&gt;
    &lt;/RollingRandomAccessFile&gt;```

this is the path `/opt/druid/conf/druid/cluster/_common`

every after having only one overlord , still seeing same issue

hmmm, is there anything interesting in your (now single) overlord log?

don’t see anything . we have shutdown all running tasks as overlord is updating continously . and after that also we are seeing same issue and nothing written into overlord logs

<@U030W50CNAC> Thanks for the help issue is fixed after restarting broker

router sends the request via broker right so it is failing on broker

```<http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Failed to talk to leader node at [/druid/indexer/v1/tasks]. Error code [404], description [Not Found].
        at org.apache.druid.sql.calcite.schema.SystemSchema.getThingsFromLeaderNode(SystemSchema.java:1080) ~[druid-sql-25.0.0.jar:25.0.0]
        at org.apache.druid.sql.calcite.schema.SystemSchema.getTasks(SystemSchema.java:903) ~[druid-sql-25.0.0.jar:25.0.0]
        at org.apache.druid.sql.calcite.schema.SystemSchema.access$600(SystemSchema.java:102) ~[druid-sql-25.0.0.jar:25.0.0]
        at org.apache.druid.sql.calcite.schema.SystemSchema$TasksTable.scan(SystemSchema.java:869) ~[druid-sql-25.0.0.jar:25.0.0]
        at org.apache.calcite.interpreter.TableScanNode.createScannable(TableScanNode.java:116) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.TableScanNode.create(TableScanNode.java:93) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:70) ~[calcite-core-1.21.0.jar:1.21.0]
        at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_292]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_292]
        at org.apache.calcite.util.ReflectUtil.invokeVisitorInternal(ReflectUtil.java:257) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.util.ReflectUtil.invokeVisitor(ReflectUtil.java:214) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.util.ReflectUtil$1.invokeVisitor(ReflectUtil.java:464) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:451) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.rel.SingleRel.childrenAccept(SingleRel.java:72) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:447) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:43) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405) ~[calcite-core-1.21.0.jar:1.21.0]
        at org.apache.calcite.interpreter.Interpreter.&lt;init&gt;(Interpreter.java:88) ~[calcite-core-1.21.0.jar:1.21.0]```

1. &lt;http://d:4080/druid/v2/sql|http://&lt;endpoint&gt;:4080/druid/v2/sql&gt;
2. *Request Method:* POST
3. *Status Code:* 500 Server Error
4. *Remote Address:* [::1]:8000
5. *Referrer Policy:* strict-origin-when-cross-origin


Check coordinator logs to see whats going on there.

Hi <@U0315MQKEHK> it got fixed after restarting broker

You could concat user_id and product_id and sketch that.

Right, This will help but this will add another column  which lead to overhead (storage, ingestion, compute etc)

```select dimension, approx_count_ds_theta(concat(user_id,product_id)) from table group by 1 order by 2 desc limit 5```