Apache Druid

image.png

what version are you on? Segment over-replication was fixed here <https://github.com/apache/druid/pull/13114> in 25.0

<@U031A9D19NU> I’m using v0.22.1. thank for sharing this issue!!

Maybe historicals become unresponsive, so other ones try to load their segments?  Network or zk issue?  Anything in zk logs or coordinator logs?  (Or historical, but I'm guessing that's where you see the message you posted.)

Big question is why #1 happens.  Hopefully there's something in druid or zk logs.

<@U0411DE2SS0> It seems true! I checked coordinator logs and found number of `org.apache.druid.client.BatchServerInventoryView - Server Disappeared [historical]`  there must be announcing problem i guess. thanks for sharing clues

And also `org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for …`  . maybe zookeeper doesn’t work properly sometimes

it seems I solved the issue by setting
druid_emitter_logging_logLevel=error , (I am using datadog integration)
setting this to - druid_emitter_logging_logLevel=info
write into the docker/overlay2 ..
to which location druid_emitter_logging_logLevel are writen ?
also it a good point to improve on documentation for docker setup

how we make sure the logs location of emmiter is on docs .. and that docker setup can lead to diskspace issue ?
can we open jira issue on that ?

Yes, at <https://github.com/apache/druid/issues>.  (Iirc, since docs are at <https://github.com/apache/druid/tree/master/docs> .)

are you talking about segment granularity or query granularity? Is this a rollup? Any logs, specs or screenshots to share?

what is difference between query granularity &amp; segment granularity?

segment granularity is the time range of your segments, each of them might cover a day of data, an hour, a month… a segment should store up to a few million records. You’ll set this in ‘Partition’ tab.

when you’re doing a rollup, you decide how big is going to be the chunks each row will summarize, thats query granularity. You set this in ‘Configure Schema’ tab, usually when you’re doing a rollup.

So you can create a table with hourly segment granularity, stored in segments with daily or weekly granulariy.

So, If I have to keep my data summarized to 1 minute level, I will do it in query granularity. And segment is like physical partition of this data stored.

&gt; So you can create a table with hourly ~segment~ query granularity, stored in segments with daily or weekly granulariy.
just to make sure as I noticed an error

yeah, it makes sense to summarize your raw data into minute-level rows, stored in hourly or daily segments.

so when you run a query for a time range, druid will know which segments might have your data, so no reason to check in every segment.

Going back to original question, reindexing from druid data source into another, I am missing summary of metrics. All are showing as 0

This happens if the original datasource has no rollup.  The reindex doesn't have rollup to rollup, I guess.  Here are some steps I found about getting around it:

Since the original ingestion was done without rollup or defining metrics, reindexing with rollup enabled did not find any metrics to aggregate, which is why it shows null/zero.
*Solution:* In the Re-Indexing spec, the metrics from the original datasource can be specified as part of the inputSource spec, via something like:
``` "inputSource": {
    "type": "druid",
    "metrics": [
      "m1"        &lt;--=============---
    ],
    "dataSource": "orig",
    "interval": "2000/3000"
  }```
If using the web console, the UI presents the option to specify metrics during the "connect" step (default is empty). The "m1" metric can be added thereto generate the correct spec.
After adding the "m1" metric, I'm able to query the reindexed data source and see the expected (non-zero) values for the sum_m1 column.

That would be in the ioConfig section, btw

There was another thread that was reporting something similar. In their case they had just upgraded to 25.0 which defaults to HTTP based communication between processes instead of ZK. This causes more work on the http threads. The solution was to increase http threads to accommodate this work. Just throwing it out there to see if this might help.

note also that a common cause for 137 errors is overcommitting the memory on the server

in Druid 25 (the latest release) you can start druid via the `bin/start-druid` script and it will automatically determine reasonable configs given the available resources on the server

<@U0344FW86DD> We faced this issue due to the znode size reached in the zookeper by default it is 1MB we increased it to 12MB using the max jute buffer and then it got fixed

<@U0344FW86DD> can you please suggest the best documentation or approach for setting up the druid monitoring.
Because we are settuped druid exporter right now but it was emitting huge number of metrics and it was bombarding our prometheus and grafana

<@U0344FW86DD> if you can provide me any design document for implementing the monitoring it will be really helpful

Personally I like using Druid itself to monitor Druid. Druid itself can handle the number of metrics that Druid generates :slightly_smiling_face:

The idea would be use the Kafka emitter, and then set up Druid to read that data back from Kafka. This is more or less what we do at Imply. (We have some extra stuff in the pipeline but it's basically that idea.)

If you want to keep using prometheus/grafana then I believe the other emitters have some ability to control what metrics and dimensions get emitted. You could remove some of those to reduce the load

such as the dimensions map for <https://druid.apache.org/docs/latest/development/extensions-contrib/statsd.html>

<@U0344FW86DD> This is my common configuration file... can you please suggest me any corrections i need to make to get the metric count less.
``` # Extensions
    druid.extensions.directory=/opt/druid/extensions
    druid.extensions.hadoopDependenciesDir=/opt/druid/hadoop-dependencies
    druid.extensions.loadList=["druid-kafka-indexing-service","druid-s3-extensions","postgresql-metadata-storage","druid-datasketches","druid-lookups-cached-global","druid-parquet-extensions","druid-avro-extensions","kafka-emitter"]

    # Zookeeper
    druid.zk.paths.base=/druid
    druid.zk.service.host=${ZOOKEEPER_SERVER}

    # Logging
    druid.emitter=composing
    druid.emitter.logging.logLevel=debug
    druid.startup.logging.logProperties=true

    #http
    # druid.emitter=http
    druid.emitter.http.recipientBaseUrl=<http://druid-http-exporter.cloudops.svc.cluster.local:8080/druid>

    # For PostgreSQL (make sure to additionally include the Postgres extension):
    druid.metadata.storage.type=postgresql
    druid.metadata.storage.connector.connectURI=jdbc:postgresql://${POSTGRES_URL}:${POSTGRES_PORT}/${POSTGRES_DB}
    druid.metadata.storage.connector.user=${POSTGRES_USER}
    druid.metadata.storage.connector.password=${POSTGRES_PASSWORD}

    # Service discovery
    druid.selectors.indexing.serviceName=druid/overlord
    druid.selectors.coordinator.serviceName=druid/coordinator

    # Indexing service logs
    druid.indexer.logs.type=s3
    druid.indexer.logs.s3Bucket=${BUCKET_INDEX}
    druid.indexer.logs.s3Prefix=indexing-logs

    # Monitoring
    druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]

    # Deep Storage
    druid.storage.type=s3
    druid.storage.bucket=${BUCKET_STORAGE}
    druid.storage.baseKey=segments
    druid.storage.useS3aSchema=True

    # Security
    druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]

    # SQL
    druid.sql.enable=true
    #Enable Metrics
    druid.monitoring.emissionPeriod=PT15S```

we are using druid exporter to get the metrics from druid and push it to prometheus

I don't see your full emitter config there; you're using `composing` emitter so there should be some `druid.emitter.composing.emitters` and then emitter-specific config. is it possible something is missing?

Your druid.monitoring.emissionPeriod is set to every 15 seconds. The default is one minute. That would reduce it to 25% for time based metrics.

<@U030K4UM3H7> can we put more than 1 minute as well is it ?

Yes. You can determine the cadence for many of the metrics. Metrics that are query specific will be emitted when the query is processed, so those will not be affected by the `druid.monitoring.emissionPeriod`.

<@U030K4UM3H7> Even after increasing the emissionperiod to 60s still it is emitting of same size

I was thinking even though the druid has metrics it will only emit every one minute... so here after 1 minute druid is emitting all the metrics at once

does metrics count depends on segment count as well ?

it varies by metric, some metrics emit more instances based on individual queries and segments involved. So yes, take a look here: <https://druid.apache.org/docs/latest/operations/metrics.html#historical>
The dimensions associated to each metric define their granularity, and yes in many cases `segment` is a dimension, so the metric will be emitted for each segment involved.

More debugging later. I enabled metadata task storage, so now when I restart the overlord / coordinator, the problem actually persists between restarts. Right now I have max tasks set to 15, but of the tasks that are not SUCCESS, only six tasks are in RUNNING. When the parallel index batch starts, it tries to run tasks and gets back errors `java.lang.IllegalStateException: Too many tasks (max = %,d) [15]`, but I have no idea how there can be too many tasks and not list them.

More debugging. It appears to be a race condition in the KubernetesTaskRunner. Some process in druid is submitting tasks in rapid fashion to this class, which ignores most of it because it keeps its own internal task list in a synchronized map. When the task completes, everything is fine, it gets success, it cleans it all up, it gets removed from all task lists, but then within a second the same job gets resubmitted to this process, since it is no longer in it's task list, it gets put back in, then spawned again. At some point the taskQueueManager notices a process running that shouldn't be and kills it, but KubernetesTaskRunner never gets that memo I guess, it leaves the task in it's internal list. At some point (guessing here), the size of this list exceeds the number of threads it has allocated to run more processes(number of threads == maxTasks) and so it won't spawn new tasks beyond the number it has in its list.

you're asking a performance question: how can you get 6 or 8 tasks to do the work of 12 tasks?

best way to tackle that is to use a profiler to see where the time is going

i suggest attaching a profiler to one of your tasks, seeing what it does and if that leads to any ideas for next steps

feel free to post a profile result here and we can take a look

<@U0344FW86DD> We added new parameters to consumer properties that were not listed by druid but were still supported by Kafka. Will the druid be able to pick them up?

eg: `max.partition.fetch.bytes` `max.poll.records` etc

Druid ignores the consumer-group-related properties and serde-related properties (since we do our own offset management and our own deserialization)

Other properties are respected. So I believe the ones you listed would indeed be respected

small point - stay under 32G.  Even 32765M.  At 32G, internal pointers go from 32-bit to 64-bit, something about that, it's worse unless you get up to somewhere like 40G.  31G or 32765M might be better.  I didn't find the best reference, but you can google `java heap 32G`.  :)

note that in Druid 25 (the latest release) you can start druid via the `bin/start-druid` script and it will automatically determine reasonable configs given the available resources on the server. i would start from that as a baseline

Cause now, I really have no luck with the Indexers :disappointed:

<@U0344FW86DD> `bin/start-druid` how does the script work ? will auto-bootstrap configs ?
Just thinking from k8s side, we still need add in resources depending upon the druid nodetype etc

It looks at the number of processors and amount of memory on the machine, and which set of services are run; then sets the basic configs (like Xmx, direct memory, etc) to reasonable defaults

You can also provide a specific number of processors and memory if you don't want to use everything available on the machine

```k logs -f druid-e2e-cluster-brokers-748ddd54d9-z8pxz  -n druid
python interepreter not found```


seems like it will need an init-container to bootstrap python

Finally figured out  why the Indexers were failing! maps! Had to set
```vm.max_map_count=1048576```


Indexer is generating a lot of intermediate files during ingestion!

ah, this may also be related to `maxColumnsToMerge` — you can read more here <https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1663951074038569?thread_ts=1663936671.637089&amp;cid=C0309C9L90D>

That thread is about direct buffer memory, but merging phase can also use a lot of mmaps

TLDR is that if you use SQL-based batch ingest (via MSQ tasks) then it is tuned automatically and you don't have to worry about it

If you are using other forms of ingest, you can adjust that if you have any further issues

not that i am aware of through configuration. you could get it through a code change of course.

another way of approaching the problem is to set `-XX:HeapDumpOnOutOfMemoryError` and look at what's in the heap when the OOME is there. It will include the Query object corresponding to the query that tripped you up. It will also include the actual objects that tripped you up, which would be useful as well

if you figure out what it is, please let us know so we can ensure it does not occur in the default configuration (when using `bin/start-druid`)

Right, I dig the code a little bit, it seems that in class `SqlLifecycle` we can add the logging in `initialized()` . Otherwise, the logging from `emitLogsAndMetrics` would only be afterward. Similar story applies to native query side in broker rest end point too.

Do you feel this is a feature we can potentially add to the current code?

It could be a useful option, or even, we could _always_ log to the Broker log file at "debug" level. That way, people can see it if they want to by enabling debug logging

thanks. but then 300,000,000 bytes = 300Mb
while the default calculates to 1Gb?
How does providing less space works?

Hmm. It shouldn't…

Do you have `-Xmx16g` in your historical `jvm.config` and `druid.processing.numMergeBuffers = 4` in your historical `runtime.properties`?

hmm let me see if the same thing happens on my machine

I tried those settings and the merging dictionary size was indeed set to 1GB by default

Maybe the settings aren't taking, somehow? Possible that the service is not actually getting launched with `-Xmx16g` and `druid.processing.numMergeBuffers = 4`?

yes, from jvm file
```-Xms2g
-Xmx16g
-XX:MaxDirectMemorySize=14g```
and historical/runtime.properties
```druid.server.http.numThreads=60
druid.processing.buffer.sizeBytes=500MiB
druid.processing.numMergeBuffers=4
druid.processing.numThreads=15```

hmm. strange that i do not see the same behavior. is this something that can be reproduced using druid 25.0.0 from <http://druid.apache.org|druid.apache.org>? if so, please let us know how to repro it and we will figure it out!

How do I figure out the merging dictionary size for my cluster?

Also, http://&lt;druid-broker&gt;:8082/status returns
{
    "version": "0.22.1",
    "modules": ...
Seems i on a primitive version

is the xms coming into play? it starts at 2g

if the JVM never has to increase it may be coming in smaller than the 1GB?  Just a thought

In Druid 25 there is a new `start-druid`<https://druid.apache.org/docs/latest/tutorials/index.html#start-up-druid-services| script> that auto sizes all the services memory configs is that something that alleviates this issue?

"Druid 25" - I am afraid, I cannot check that since my company uses the older version.

"is the xms coming into play" - yes, thats a good point. I am trying to find ways to see how much memory is currently with the dictionary. is there any way to check that?

you could just try setting xms and xmx the same and see if the behavior persists

just wonder if it is looking at the startup heap and determining the size then and not adjusting as the heap grows which would place it lower than 1GB

Yes, I will try setting Xms and Xmx as same. Current /status from Historiical
```"memory": {
   "maxMemory": 15271460864, ~15G Xmx
   "totalMemory": 4903141376, ~4.9G
   "freeMemory": 3141649328,  ~3G
   "usedMemory": 1761492048,  ~1.7G
   "directMemory": 15032385536~15G
}```
seems like only 1.7G is being used and so if the dictionary mem is calculated as per this, it will indeed be way less.

Obs on setting Xms = Xmx, mem usage:
```"memory": {
  "maxMemory": 17015767040,		17G
  "totalMemory": 17015767040,	17G
  "freeMemory": 15408552704,	15G
  "usedMemory": 1607214336,		1.6G
  "directMemory": 15032385536	15G
}```
Still the usage is 1.6G, so I guess I will have to setup "_maxMergingDictionarySize_" via config, which is kind of dicy. i should't set it to 300Mb but to 1Gb then. which will be the max it will ever get - MIN(30%heap, 1Gb)

hmmm the jvm should have started with the 16GB if they were both set to that?

Could it be being calculated off of totalMemory?   4.9 * 0.3 / 4 = 0.367 but I guess that would still be higher than 300MiB...

Ah, I just noticed you pasted something saying your Druid version is 0.22.1

In that version, merging dictionary was hard-coded to 100000000 (i.e. 100MB)

The adaptive dictionary size based on available memory was added in a later version

You should upgrade :slightly_smiling_face: but if you can't, then yeah you should manually set the dictionary size

Thank you all for your input. this is very valuable. I couln't get this info from anywhere. :pray:

that's why we're here in slack :slightly_smiling_face:

Wouldn't this follow from setting up auth for druid in general?  Here are a couple of docs on setting up "basic" auth, and a bit more: <https://druid.apache.org/docs/latest/operations/security-overview.html> , <https://support.imply.io/hc/en-us/articles/360051794093-How-to-Enable-Basic-Auth-in-Druid>

I would suggest using a supported version so either 3.7.1 or 3.8.1. I normally run the stable version 3.7.1 though ZK is pretty stable overall and some are already running it with 3.8.1.

We run all of our clusters on 3.8.1 ZK. No issues at all.

Running on 3.8.x also alleviates you from old Log4J vulnerability headache.

good question!  Maybe it just hasn't gotten the engineering attention yet?  PR sounds like a good idea, if you have use for it!

Thanks! Just asking in case I’m missing something obvious 

I don't think so!  I don't see any such function either.

We did check the code today, and tuples, as opposed to tetha funtions, have no binding.

i think it's just a less-commonly used sketch flavor, so to date SQL bindings had not yet been submitted. we would welcome a PR with SQL bindings! when you raise it, feel free to post it in <#C030CMF6B70|> here too

Voila <https://github.com/apache/druid/pull/13887>

just posted a review: <https://github.com/apache/druid/pull/13887>

<@U0344FW86DD> your point have been addressed. Would be nice if we could close the PR :slightly_smiling_face:

just took another look; the latest LGTM! I kicked off a CI run and I think we'll be good to merge after that passes. (I'll keep an eye on it)

What went wrong? Others in this channel could learn from it as well.

well, no, i am observing minute level truncated timestamp. but then whats the diff b/w minute and five_minute granularity?

my bad, scratch the above., earlier i had modified an existing ingestion task which had MINUTE level gran.
i recreated the ingestion task and this time added FIVE_MINUTE granularity, and i can see timestamps as 3:30, 3:35, 3:40, ...

floor (or trunc) is the right answer. A message from 12:04:59 with FIVE_MINUTE query granularity will be trunc’ed into 12:00:00 record

```String literals should be quoted by single quotation marks.```
<https://druid.apache.org/docs/latest/misc/math-expr.html>

Did it end up working for you with that change? (Just curious)

<@U0344FW86DD> pardon my lateness - yes it worked like a charm! I'm using it to label the different sorts of events that are all submitted into the same table

Hi, the definitions are mentioned in :
<https://druid.apache.org/docs/latest/operations/metrics.html>

<@U04CGQ1P68Z> yes for Coordinator but what metric among these?
<https://druid.apache.org/docs/latest/configuration/index.html#enabling-metrics>

You can setup batch ingestion . Also make sure you have enough worker capacity to handle both streaming and batch ingestion.
is my understanding is correct?
<https://druid.apache.org/docs/latest/ingestion/index.html#batch>

Just understand kafka ingestion is ‘always on’, new records in the topic will be ingested asap.

Batch ingestions work in a different way, you set them when you need to ingest a set of files; or you can use something like Airflow to trigger batch jobs periodically.

Have you tried ‘load data’ tab to run a batch ingestion from S3 or HDFS?

Yes mixed segment granularity like this should work.

What are your early/late message rejection settings for your ingestion tasks?  Can you check to see if you are actively ingesting data and creating segments in all of these time chunks, and that your compaction job isn't colliding with one of them?

Also, what version of Druid are you on?

25.0.0. I don't have any early / late rejection settings at the moment, as I'm still just testing things out. The time chunk that was errant was 2023-02-21, so basically, the current day. When I changed the ingestion to 'month' suddenly it started working. But I had marked as unused and then killed everything from 2023-02-01 on, and double checked there was nothing stray in s3 for that time range. Is it possible the coordinator thought it knew something it didn't and I should have restarted it?

Ingesting into time chunks where there is already an existing segment with partially or fully overlapping that time chunk, shouldn't matter ... the ingestion just starts a new segment in the time chunk you specified, and later on you compact them together.

If your issue was that ingestion was not working, unfortunately I don't know enough about process communications here to be of much help.  If you changed ingestion segment granularity though that means you must have restarted the supervisor, so that could have reset something and hte change in granularity not needed.

Okay well I'll experiment with it some more. I'm sure it is something simple, I just wish the error message it gave back was more explicit.

```ERROR [qtp2050083360-92] org.apache.druid.server.security.PreResponseAuthorizationCheckFilter - Request did not have an authorization check performed, original response status[503].: {class=org.apache.druid.server.security.PreResponseAuthorizationCheckFilter, uri=/druid/indexer/v1/task/partial_index_***_mfbfkkjd_2023-02-15T01%3A15%3A58.865Z/status, method=GET, remoteAddr=10.7.221.38, remoteHost=10.7.221.38, queryId=null}```

This message is in Overlord Log ,, and remote IP is of Data Pod

i see there was bug for similar issue <https://github.com/apache/druid/pull/5579>

<@U0344FW86DD> sorry to come back late.. we are using
```Apache Druid is running version 2023.01.0-iap```

Screenshot 2023-02-23 at 12.44.57 AM.png

Could you be overextending the resources when tasks switch to publishing mode?
What do CPU and memory utilization look like on the nodes when this happens?
What CPU and Memory resources do you have on the nodes?
And how are they split amongst the Peons? In other words, what are the settings for javaOptsArray for the Peons in the Middle Manager config?

My MiddleManager has 8 vcpu and 64 GB RAM.

CPU utilisation is around 70%-85% during simultaneous reading and publishing phase.

Memory utilisation is around 40GB-50GB during simultaneous reading and publishing phase

*Split amongst the Peons :* 
```druid.indexer.runner.javaOpts=-server -Xmx3g -Daws.region={{region}} -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps```
worker.capacity = 7 [1 cpu per worker] per node
Total No of MM = 5 [35 workers in total]

Seems like you are good on resources, the only missing items I see are `-Xms3g` (recommended to be set the same as `-Xmx` ) and `-XX:MaxDirectMemorySize=3g` set this based on
```(druid.processing.numThreads + druid.processing.numMergeBuffers + 1) * druid.processing.buffer.sizeBytes```
What version are you on? I believe you are hitting this issue: <https://github.com/apache/druid/issues/11414>
There are significant improvements on this included in 25.0:
1. <https://github.com/apache/druid/pull/13328> -- Optimize the metadata calls
2. <https://github.com/apache/druid/pull/13334> -- Do not queue more than one run notice at a time.
3. <https://github.com/apache/druid/pull/13354> -- Contacts tasks asynchronously, eliminate chat threads and reduce the load on worker threads.
4. <https://github.com/apache/druid/pull/13369> -- to speed up segment allocation and reduce metadata calls by clubbing the multiple requests together.

What about the network bandwidth. With tasks reading writing how much throughput would be needed on the network? How many GB/s would be required on the ingest to keep up with kafka? 

If each task does about 25GB/hour then 25*15*8/3600=0.8 gbps. If you have a 1 g network then things will slow down once publishing also starts. You should be ok if you have a 10g network

i would also check if you have pending tasks while the publishing is happening

typically during publishing, a new round of tasks is launched

if these get stuck in queue (pending) then you will see lag until the original tasks finish publishing

for this reason you need free slots equal to at least 2x your `taskCount` (so you can have both old/new sets running at once during publish)

try running with druid.indexer.tasklock.batchSegmentAllocation=true in the overlord config

<@U03BJAMQ4H5>, due to some other issues identified, can you help to confirm if the IP address in this line is coordinator?
```2023-02-22T21:25:47,394 WARN [task-runner-0-priority-0] org.apache.druid.indexing.common.IndexTaskClient - submitRequest failed for [<http://10.64.211.239:8100/druid/worker/v1/chat/index_parallel_mulesoft_long_query_d1_2_bmkknked_2023-02-22T17%3A54%3A09.474Z/segment/allocate>]```


you can check the services tab and see what port the corrdinator is running on

to answer your original question: `10.64.211.239:8100` would have been the main `index_parallel` task for your ingest job

the subtask calls `/segment/allocate` on the main task, then the main task hands it a segment ID

it's possible the main task failed for some reason and that's why you saw so many failures on the workers (the main task went away)

if you're having problems with native batch ingest, i suggest trying with SQL-based ingest via MSQ (<https://druid.apache.org/docs/latest/multi-stage-query/>)

It is much easier to debug! many errors don't happen at all due to better design. and the errors that do happen are collected and displayed more cleanly

Thx <@U0344FW86DD> for the explanation. Saved me a lot time to dig the code. The root cause is clear. We have a deployment which killed all the tasks.

ha! that'll do it! :slightly_smiling_face:

Do you mean a usecase that queries that hit two different datasources, like <https://druid.apache.org/docs/latest/querying/sql.html#table-level>? If not could you provide an example of a query?

Yeah - Query similar to the one shared in above link as well:

```{
  "queryType": "groupBy",
  "dataSource":
     {
       "type": "union",
       "dataSources": [
           {
               "type": "query",
                "query": {
                     "type": "groupBy",
                      ...
                 }
            },
            {
               "type": "query",
                "query": {
                     "type": "groupBy",
                      ...
                 }
            },
       ]
    },
  "granularity": "day",
  "dimensions": ["country", "device"],
  "limitSpec": { "type": "default", "limit": 5000, "columns": ["country", "data_transfer"] },
  "filter": {
    "type": "and",
    "fields": [
      { "type": "selector", "dimension": "carrier", "value": "AT&amp;T" },
      { "type": "or", 
        "fields": [
          { "type": "selector", "dimension": "make", "value": "Apple" },
          { "type": "selector", "dimension": "make", "value": "Samsung" }
        ]
      }
    ]
  },
  "aggregations": [
    { "type": "longSum", "name": "total_usage", "fieldName": "user_count" },
    { "type": "doubleSum", "name": "data_transfer", "fieldName": "data_transfer" }
  ],
  "postAggregations": [
    { "type": "arithmetic",
      "name": "avg_usage",
      "fn": "/",
      "fields": [
        { "type": "fieldAccess", "fieldName": "data_transfer" },
        { "type": "fieldAccess", "fieldName": "total_usage" }
      ]
    }
  ],
  "intervals": [ "2012-01-01T00:00:00.000/2012-01-03T00:00:00.000" ],
  "having": {
    "type": "greaterThan",
    "aggregation": "total_usage",
    "value": 100
  }
}```

currently `union` of `query` is not supported. we'll be adding it in a future release

Hey <@U0344FW86DD> - Thanks. Do we’ve any timeline for this?

Or any suggestion on a workaround for now?

No specific timeline although definitely working towards supporting full SQL is important to us. So hopefully not too long

In the meantime, hard to suggest a workaround without understanding more of what you're trying to do. Perhaps a table-level union would work? ie union of two or more tables and then doing your queries on top of that

Is there any workaround in sql way to do table level union with non identical schema datasources ?
This can be done in native queries. But this cannot be saved as `view` . I am using superset to query druid

could be zookeeper, or network issues, eg.  Anything in zookeeper logs at the time?  Router/historical/broker logs?

Perhaps maxParseExceptions is what you are looking for?
<https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-reference.html#:~:text=no%2C%20default%20%3D%3D%20false-,maxParseExceptions,-Integer|https://druid.apache.org/docs/latest/development/extensions-core/kafka-supervisor-reference.html#:~:text=no%2C%20default%[…]ParseExceptions,-Integer>

Another approach is to cleanse your data upstream? Using a tool like ksqlDB you can filter bad records / good records into separate topics.   And then just ingest from the good topic. (and if you want clean up the records in the bad topic)

<@U03NCEXDNJK> My Spec already has the maxParseExceptions field set ... still I am facing the issue ..

<@U04BPJQ6PRD> Thanks for the response ..but for the current pipeline cannot cleanse data upstream..

What version are you using?  Did you see this thread?  <https://www.druidforum.org/t/how-to-tell-druid-to-skip-bad-kafka-records-during-ingestion-with-schema-registry/7186/3>  Looks as if there was a regression that I think was fixed recently.

we are using the recent one ... and have looked for this fix..

some additional thoughts here: <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1677634477234209?thread_ts=1677215359.488029&amp;cid=C0303FDCZEZ>

let's continue with this thread, since it's in the correct channel :slightly_smiling_face:

Have you run the job only once or multiple times. Would you please share the ingestion spec that you are running, Parameters like appendtoexisting, rollup, replication factor could affect the number of rows.