Apache Druid

hi, i am getting the below error during data ingestion to druid. Please let me know how to resolve it.

is it possible you have some fields that have a huge amount of data in them? this is saying you have a single row that cannot fit in 1MB, which is a limit for Druid SQL based ingestion

Hi <@U0344FW86DD>, Yes.. there is a multi-value column which is having more than 1 lakh elements with comma separated. Is there a way to change this configuration to increase the limit from 1MB to 2/3MB?     Thanks..

hi <@U0344FW86DD> it is possible to have column which is a collection which can have many product_ids in them

I wonder what happens if you use SQL based batch ingest? It's generally more reliable, and it also gives you better progress indicators— so if it _did_ get stuck it'd be easier to see where

You probably either don’t have enough RAM or don’t have enough temporary disk space.

This issue was resolved after changing the volume mount, which was EFS to EBS in AWS. The fact that CPU was not going high but memory was being block by JVM gave some clue that it was an IO issue. Also the log also indicated persisting file is taking longer.

I would like to try the SQL based ingestion too as you suggested <@U0344FW86DD>. I didnt see the connect to external icon in my druid UI, druid is version 25. Guessing it is still not default and need some configuration. Will checkout

it uses the same one for deep storage; there currently isn't an option to override it for an `inputSource`

the S3 one does support overriding access and secret key (through `properties`, which is an `S3InputSourceConfig`) — a similar enhancement could be made to the Azure one, if you're interested :slightly_smiling_face:

thanks <@U0344FW86DD>, this would be a nice feature to have. Orgs would want to keep the data in separate places for deep storage and ingestion. I will take a shot at it. had been looking for some beginner-friendly change requests. Maybe this is the one :slightly_smiling_face:

Hmm, I'm not entirely sure. You might need to add a SAN

It is way easier to just terminate certs at Nginx level.

2023-07-12T05:15:19,029 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10004df15760039, timeout of 30000ms exceeded
2023-07-12T05:15:22,573 WARN [NIOWorkerThread-1] org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x10004df1576003a, likely client has closed socket

image.png

You might get some hints in the `coordinator` log

Are you running the quickstart version on a local machine?

How long did you wait after finishing the ingestion? It can take a couple of minutes until the segments show up in the Datasources view.

There is only one piece of data in my JSON file, I mean it shouldn't take long

Yes, these are all the ones I have recently loaded

the reason it takes a while is because each segment has to go through the handover process
• first the peon (ingestion process) builds the segment
• then it is written to deep storage, which in the local quickstart is just a special path within your file system but generally it would be object storage

• then the segment is loaded into the cache of the historical process, which in your quickstart is another local path
• only then is it available for query

on the off chance, could you check the datasources view again to see if the segment is available now?

Any chance there’s a retention policy set? The `kill` job in the task list makes me wonder

No, it hasn't been successfully loaded yet

ok, no - it’s not a retention thing. This looks like the historical cannot load from deep storage

please have a look in the logs - coordinator or historical should give you some clues

Can I see logs in this visualization interface. Or should I go to the installation directory and check the logs

you will need to go into the install tree

```Could you please help me check where the problem is```

I remember it was possible to load in before, but I recently tried to load the data again and found that it didn't work anymore. And I have reinstalled Druid

did java version is changed recently ,
Also looks like java issue . Check appropriate java version as well.
<https://www.druidforum.org/t/druid-user-unable-to-run-ingestion-with-specific-java-version/6136>

```2023-07-12T07:46:51,507 ERROR [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.curator.framework.listen.ListenerContainer - Listener (org.apache.druid.curator.discovery.CuratorDruidLeaderSelector$1@6bb4cc0e) threw an exception
java.lang.ClassFormatError: Illegal field name "org.apache.druid.server.coordinator.DruidCoordinator$this" in class org/apache/druid/server/coordinator/DruidCoordinator$CoordinatorHistoricalManagerRunnable
    at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_333]```
<https://druid.apache.org/docs/latest/tutorials/index.html>

can you  pls try using lower/higher  version of java than current if possible ? is coordinator service is running fine?  did you verified as per
<https://www.druidforum.org/t/druid-user-unable-to-run-ingestion-with-specific-java-version/6136>  .eg java home is set etc

What version of JDK is recommended? I'll give it a try

lower  than this 1.8.0_333  version for now.

or  as per<https://druid.apache.org/docs/latest/tutorials/index.html#requirements>

The Java environment variables I configured in the profile file

does it have the correct JAVA_HOME setting in the shell that you are using to start druid?

AFAIK the settings in /etc/profile and ~/.profile are only used for a login shell

regular shells use something like .bashrc or .zshrc

Do you mean that I have anything else that hasn't been configured, so which file is it

if you type `echo $JAVA_HOME` in your shell, what do you get?

oh no.The path obtained here matches the path I installed and configured

ok. type `. /etc/profile` and then again `echo $JAVA_HOME`

but you want this environment every time you open a shell, not only in the login shell. and it can be really hard to identify the login shell, that’s why I am asking. usually you would have a file named `.bashrc` (if you are using bash), in your home directory, that gets sourced _every time_ you open a shell

if you go to your home and type `ls -la` it will also list those hidden files

I don't think it's a problem with the Java version

I will take a look tomorrow and thank you very much for your help:slightly_smiling_face::relaxed::+1:

Hi <@U030M5A8CVB>, yes you can run those components together although it’s not a recommended production setup.

You can create a custom configuration file in `conf/supervise/cluster`, merging the configurations for query and master server. Then create a copy of one of the start scripts in `bin`, which uses your custom configuration file.

Perfect. Thanks a lot <@U032CA4JJV6> Appreciate. Will try that out. :slightly_smiling_face:

<@U030M5A8CVB> there’s also the new `start-druid` script – that may be a solution, too.

<https://druid.apache.org/docs/latest/tutorials/index.html#start-up-druid-services>

<@U0306F95YR3> Thanks. But it will start all the processes right ? I have a dedicated Data server running in a different machine.

Nope - you can specify the services you want to run

Like:
```bin/start-druid -m=4g -s=zookeeper,broker,router```

Or maybe for all leader processes plus ZK:
```bin/start-druid -m=4g -s=coordinator-overlord,zookeeper,broker,router```

<https://pmio.hashnode.dev/start-druid-the-new-startup-script-for-apache-druid>

That looks good. I will go with this option :slightly_smiling_face: Thanks a lot <@U0306F95YR3>

Well don’t thank me yet. <@U032CA4JJV6> is quite possibly the hero here :smile: :smile:

(Noting that the config file locations change with this script - so just a note of caution… You can see what I did in that blog thing)

Sure. Going through that as we speak :slightly_smiling_face:

One more question <@U0306F95YR3>, how do I run all those services in background ? I tried `nohup bin/start-cluster-data-server &gt;&gt; start-cluster-data-server.log &amp;`   but if I close the EC2 instance, services stopped.

Perhaps a query on Druid that something like:
```SELECT entity_id, TIMESTAMPDIFF( SECOND, MAX(__time), CURRENT_TIMESTAMP) last_seen_seconds_ago
FROM events
WHERE __time &gt; TIME_SHIFT(CURRENT_TIMESTAMP, 'PT1H', - 1)
GROUP BY 1
HAVING last_seen_seconds_ago &gt; N  ```
Run that every N seconds and from the result build message to feed into the events topic that you are ingestion from. This step would need to be done by some other workflow tool like AirFlow or NiFi.

Are there are any availability issues on historicals/MMs?? These missing segment list are part of missing segments response from historicals and middle managers.

can you check this metrics if you are logging it ?
```segment/unavailable/count```

i don't think we are logging this metric but will check

Tried running this, but no luck
```nohup bin/start-druid -s=coordinator-overlord,zookeeper,broker,router &gt; /tmp/logfile 2&gt;&amp;1 &amp; ```


I think you can use `disown` The <https://unix.stackexchange.com/questions/3886/difference-between-nohup-disown-and|first answer in this discussion >is pretty comprehensive.

Thanks a lot <@U030K4UM3H7> I ran the druid services as a daemon process. Steps are as below.
1. Create a druid_data.service file in `/usr/lib/systemd/system/`
```[Unit]
Description=Druid Data Service

[Service]
Type=simple
ExecStart=/home/ec2-user/apache-druid-26.0.0/bin/start-cluster-data-server```
2. Reload the systemd service files: `sudo systemctl daemon-reload`
3. Check that it is working with `sudo systemctl start druid_data`

That sounds much better. :slightly_smiling_face:

It looks like your zookeeper is unhealthy?  I would check the zookeeper logs and make sure it is in a good state

i didn't find any exception or errors from zoo-keeper. any other keywords i may look into?

I updated Apache Druid to version 26.0.0 yesterday. Attempt to load data failed, and I checked the log, which says

无标题

Can you paste the indexer log ?

var/druid/indexing-logs/index_parallel_test_data_kidlbcen_2023-07-12T15:39:38.669Z.log

failed restart doesn't reach this log statement
• Checking whether all servers have been synced at least once yet
```2023-07-13T06:52:12,624 INFO [FilteredHttpServerInventoryView-3] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-58-240.ap-southeast-1.compute.internal:8083/_1689231131789>] synced successfully for the first time.
2023-07-13T06:52:12,631 INFO [FilteredHttpServerInventoryView-0] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-67-166.ap-southeast-1.compute.internal:8083/_1689231131796>] synced successfully for the first time.
2023-07-13T06:52:12,641 INFO [FilteredHttpServerInventoryView-2] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-78-50.ap-southeast-1.compute.internal:8083/_1689231131783>] synced successfully for the first time.
2023-07-13T06:52:12,650 INFO [FilteredHttpServerInventoryView-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-60-200.ap-southeast-1.compute.internal:8083/_1689231131777>] synced successfully for the first time.
2023-07-13T06:52:12,653 INFO [FilteredHttpServerInventoryView-1] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-64-197.ap-southeast-1.compute.internal:8083/_1689231131796>] synced successfully for the first time.
2023-07-13T06:52:12,744 INFO [FilteredHttpServerInventoryView-3] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-76-215.ap-southeast-1.compute.internal:8083/_1689231131797>] synced successfully for the first time.
2023-07-13T06:53:52,078 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.2.5.Final
2023-07-13T06:53:52,702 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-parquet-extensions], jars: parquet-jackson-1.12.0.jar, snappy-java-1.1.8.4.jar, hadoop-hdfs-client-2.8.5.jar, commons-configuration-1.6.jar, htrace-core4-4.0.1-incubating.jar, slf4j-api-1.7.36.jar, parquet-column-1.12.0.jar, zstd-jni-1.5.2-3.jar, avro-1.9.2.jar, jackson-core-asl-1.9.13.jar, parquet-format-structures-1.12.0.jar, parquet-avro-1.12.0.jar, audience-annotations-0.12.0.jar, parquet-common-1.12.0.jar, hadoop-mapreduce-client-core-2.8.5.jar, javax.annotation-api-1.3.2.jar, hadoop-auth-2.8.5.jar, parquet-encoding-1.12.0.jar, druid-parquet-extensions-25.0.0.jar, hadoop-common-2.8.5.jar, hadoop-annotations-2.8.5.jar, parquet-hadoop-1.12.0.jar, jackson-mapper-asl-1.9.13.jar```

Success restart log printing this
• Checking whether all servers have been synced at least once yet
```2023-07-13T06:53:59,108 INFO [FilteredHttpServerInventoryView-0] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-58-240.ap-southeast-1.compute.internal:8083/_1689231238305>] synced successfully for the first time.
2023-07-13T06:53:59,116 INFO [FilteredHttpServerInventoryView-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-78-50.ap-southeast-1.compute.internal:8083/_1689231238301>] synced successfully for the first time.
2023-07-13T06:53:59,129 INFO [FilteredHttpServerInventoryView-3] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-64-197.ap-southeast-1.compute.internal:8083/_1689231238305>] synced successfully for the first time.
2023-07-13T06:53:59,165 INFO [FilteredHttpServerInventoryView-1] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-60-200.ap-southeast-1.compute.internal:8083/_1689231238299>] synced successfully for the first time.
2023-07-13T06:53:59,339 INFO [FilteredHttpServerInventoryView-0] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-67-166.ap-southeast-1.compute.internal:8083/_1689231238310>] synced successfully for the first time.
2023-07-13T06:53:59,354 INFO [FilteredHttpServerInventoryView-4] org.apache.druid.server.coordination.ChangeRequestHttpSyncer - [<http://ip-10-0-76-215.ap-southeast-1.compute.internal:8083/_1689231238311>] synced successfully for the first time.
2023-07-13T06:54:03,453 INFO [FilteredHttpServerInventoryView-2] org.apache.druid.client.HttpServerInventoryView - Checking whether all servers have been synced at least once yet....
2023-07-13T06:54:03,453 INFO [FilteredHttpServerInventoryView-2] org.apache.druid.client.HttpServerInventoryView - All servers have been synced successfully at least once.
2023-07-13T06:54:03,453 INFO [FilteredHttpServerInventoryView-2] org.apache.druid.client.HttpServerInventoryView - Calling SegmentCallback.segmentViewInitialized() for all callbacks.```

<@U05GE6XENF9> - <@U0318CX1NH4> has recently fixed a bug in this area where a bad historical/MM can cause a delay of up to 4 minutes. The fix is in 27 release thats coming soon. How much time do you give for broker to come up?

<@U031A9D19NU> thanks for info.
• we only gave 2 mins at the moment before registering back to ELB. 
• do you think giving 4 mins is safe enough? 
    ◦ i have never tried to wait until 2 mins though. 
• i was expecting broker should be running after few seconds as router can start pretty fast. 

2 mins is definitely less. It does segment discovery and builds table schema. It's usually fast but sometimes can be slow. I would suggest raising up to 10 minutes

i see. for every config changes/deployment for 2 query node will take more than 20 mins in this case :thinking_face:

do you have any recommendation for each node for waiting time? i couldn't find anywhere in druid wiki. I have 3 node at the moment.
• master - (start-cluster-master-no-zk-server)
• query - (start-cluster-query-server)
• data - (start-cluster-data-server)

Is there a way to pass relative time to druid native queries? <@U031A9D19NU>

Thanks Abhishek. Will follow up on the ticket.

Hey <@U031A9D19NU> - We’ve built a plugin for druid and want to use the functions that are defined inside. We’re able to access those functions with native query. How do we do so with SQL query?

Any suggestion on this?

can you give an example and tell me a bit more about this plugin? In general, when you are writing a higher-level interface that calls druid behind the scenes, you can change the implementation to talk in SQL. so whatever thing is translating the function to a druid native query, needs to change

<@U031A9D19NU> - Wanted something like this - <https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1689340426961649>
The response here should be helpful.

how are you running this ingestion spec? it looks like it is expecting a query rather than ingestion?

error.png

Hi <@U03TAQRMJPQ>, Running json spec from druid query editor.

you have to run ingestion via the ingestion tab, there are 3 dots in the bottom menu to submit json ingestion

ingestion_submit.png

submit button is disabled if i try run ingestion via the ingestion tab. attached screenshot. Have i missed something in the json spec ?

that looks like `json` data problem, try deserializing it `cat /tmp/ingestion.json | jq`  or  `cat /tmp/ingestion.json| python -mjson.tool` will tell the error.

If you are on the latest version of Druid, MSQ will help generate better compacted segments out of the box. See the examples here <https://druid.apache.org/docs/latest/multi-stage-query/examples.html>

What time interval does your data scan - and roughly how many rows in total? Compaction will likely help reduce the size of the data stored in Druid in your scenario

<https://druid.apache.org/docs/latest/ingestion/schema-design.html#general-tips-and-best-practices> - These tips are also helpful in defining your ingestion spec to minimize the size of segments.

<@U02VCGJFER5> Thanks for the reply. I have below `granularitySpec`
```"granularitySpec": {
        "queryGranularity": "hour",
        "rollup": true,
        "segmentGranularity": "day"
      },```
Total # of rows 3,723,766 with a 10 days duration.

great, that tells me the best you can do is 10 segments for 10 days instead of the current 39.

seeing that all the data is 85MB, in this scenario, you can likely get away with MONTH partitioning and fit all the data in a single segment

The other big area for segment size improvement I would guess is in the dimension order in your index spec

That sounds good. When you say MONTH partitioning, I can still say- `"queryGranularity": "hour"` as I have some vis to show hourly trend and have an option to choose a date range from.

yes, you can have data rolled up to hour (queryGranularity), but partitioned by month (segmentGranularity)

So I have the below set of dimensions :-
```"dimensions": [
          "IPAddress",-- STRING
          "method", -- STRING
          "uri", -- STRING
          "http_version", -- STRING
          "Status", -- INT
          "Size", -- INT
          "hostname" -- STRING
        ]```

some things that stick out to me - size looks like a metric, not a dimension, so moving that over to a metric will likely improve your compression substantially

the dimension ordering will depend on your data model + how you query the data. Making some guesses - I'd think you may want to often filter + select uri hostname method. I'd stick those 3 in the front of the list and would expect to see some compression from the fact that the same host is likely serving the same sets of uris + methods

<@U02VCGJFER5> Thanks a lot for all the detailed explanation. One interesting thing I observed.
I have around 3M records in s3 with 90MiB size, all are distinct records.
1. *Without roll up* and with monthly granularity, the segment size becomes *97 MiB*. with `"maxRowsPerSegment" : 5000000` . 
2. *With roll up* and with monthly granularity, with 2 metrics i.e total time and total size for the dimensions - `IPAddress, method, uri, http_version, Status, hostname` , with partition by on `method` - the segment size becomes *85 MiB.* But the roll up does not affect the data as I am getting the same record count after roll up. ( Infact  a new col introduced called `count` ). 
3. Observed that without roll up, by default `dynamic` partition type is used where as with roll up, `hashed` partition is used. Is it affecting the segment size ? 
4. If it does not make sense to apply roll up based on the nature of data, if we still apply roll up to reduce the segment size ( as per the above case study ), I think, it will add overhead to the indexer operation. 

Hi Amir, as Suneet mentioned, the order of the dimensions can play a big role is the level of compression. roll up is likely helping by sorting the data even if it isn't reducing the rows, putting same values together helps compression. for 3, your best choice regardless of whether you use rollup or not would be to use `range` <https://druid.apache.org/docs/latest/ingestion/native-batch.html#multi-dimension-range-partitioning|partitioning> on the column(s) you are most likely to filter on and also have those columns be the first ones in the dimensions list. For #4 if you apply range partitioning and like you said rollup isn't really doing any aggregation, you should still see the benefits in compression without rollup.

That makes sense. Thanks a lot for clarification <@U030K4UM3H7>. Will try the range partitioning and update. :slightly_smiling_face:

You can also disable rollup and use range partitioning, since you are not getting any compression from enabling rollup

If you can post your index spec, that will be helpful

Sure, here is my ingestion spec.
```{
  "type": "index_parallel",
  "spec": {
    "ioConfig": {
      "type": "index_parallel",
      "inputSource": {
        "type": "s3",
        "prefixes": [
          "<s3://here/is/my/data/>"
        ]
      },
      "inputFormat": {
        "type": "json"
      }
    },
    "tuningConfig": {
      "type": "index_parallel",
      "partitionsSpec": {
        "type": "hashed",
        "partitionDimensions": [
          "method"
        ]
      },
      "forceGuaranteedRollup": true
    },
    "dataSchema": {
      "dataSource": "tmp_ttyl_2",
      "timestampSpec": {
        "column": "eventtimestamp",
        "format": "dd/MMM/yyyy:HH:mm:ss Z"
      },
      "transformSpec": {},
      "granularitySpec": {
        "queryGranularity": "hour",
        "rollup": true,
        "segmentGranularity": "month"
      },
      "dimensionsSpec": {
        "dimensions": [
          "IPAddress",
          "method",
          "uri",
          "http_version",
          "Status",
          "hostname"
        ]
      },
      "metricsSpec": [
        {
          "name": "count",
          "type": "count"
        },
        {
          "type": "longSum",
          "name": "response_size_bytes",
          "fieldName": "Size"
        },
        {
          "type": "longSum",
          "name": "response_time_ms",
          "fieldName": "Time"
        }
      ]
    }
  }
}```

Hi <@U030M5A8CVB>, sorry for the late follow up...
based on your ingestion spec, the couple of changes we suggested might help, change partitioning to `"type":"range"` and maybe consider adding more partitioning columns which will help dealing with skew (if that is an issue  with `method` ).
If you are not getting any rollup, this means that the combination of dimensions you are using create unique or almost unique combinations, so it does not help.
Alternatively you might consider removing high cardinality dimensions (i.e. IPAddress), if you do not need to specifically filter on them or group by them in your query workload. With rollup enabled, you can create <https://druid.apache.org/docs/latest/tutorials/tutorial-sketches-theta.html#use-theta-sketches-for-fast-approximation-with-set-operations|sketches> from such columns to do distinct counts and/or quantile/distribution statistics while reducing the data significantly.

Thanks a lot <@U030K4UM3H7> Will try that and update!

Can you tell us a bit more about the environment and the ingestion?  How many MMs, worker slots are available? How many tasks are you running on the ingestion spec?

Hi Rohit, there’s likely a mismatch of configuration settings. If you want to use hash or range partitioning, rollup will be perfect. For dynamic partitioning, rollup will be best effort. See <https://druid.apache.org/docs/latest/ingestion/native-batch.html#partitionsspec> for the explanation.

If you want to post your ingestion spec here, I can take a look.

Have you considered upgrading to Druid 26? Since version 0.23, Druid offers multi-dimension range partitioning, which has a lot of advantages and is now used as the default in most batch ingestion scenarios.

<@U032CA4JJV6>: Thanks for resolving this issue, I would like to know more about multi-dimension range partitioning, Can you provide me the link so that I would be able to look into it

Hi <@U05FQ2QTEN5>, the documentation for all partitioning settings in native ingestion is at <https://druid.apache.org/docs/latest/ingestion/native-batch.html#partitionsspec> but I am afraid as a reference, this may not be terribly helpful if you are just getting started with it.

The corresponding piece for SQL based ingestion is <https://druid.apache.org/docs/latest/multi-stage-query/concepts.html#clustering>.

A technical deep dive is at the Imply blog <https://imply.io/blog/multi-dimensional-range-partitioning/>.

A more practical breakdown is in this blog of mine <https://blog.hellmar-becker.de/2022/01/25/partitioning-in-druid-part-3-multi-dimension-range-partitioning/>. I wrote this when range partitioning was brand new, meanwhile it _is_ supported within the Druid console so it should be fairly easy to configure.

<@U032CA4JJV6>: This is very useful, also one more thing I want to ask with respect to 0.17.0, the *order* in which we mention dimensions in the *dimensionsSpec*, does it create *any impact inside the segment when dynamic partitioning is used*?

Hi Rohit, I am pretty sure it does. Inside a segment, rows are ordered first by timestamp, then by the dimensions in the order they are listed in the spec.

Thanks <@U032CA4JJV6>, this pretty useful and you have provided all the links necessary for partitioning.
Thanks a ton :saluting_face:

What volume of data are you expecting this to return? what is the avg size of the segments? Perhaps compaction can help.

For secrets I use this pattern in my config file `druid.s3.secretKey=${env:DRUID_SECRETKEY}`

i am able to fix now using iam role and druid.indexer.logs.disableAcl=true. thanks

You can use <https://docs.imply.io/latest/druid/operations/dynamic-config-provider/|dynamic configuration providers.>

What version of Druid are you using? Can you share the ingestion spec and a sample of the data it is failing on?

error `Could not acquire lock for rowInterval`   suggest there is some other job runnign and locking the interval, check overlord log for more details on the locked job

hey <@U03R5H3B750> below are the logs .

```23-06-30T14:43:48,118 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}
2023-06-30T14:43:48,118 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}
2023-06-30T14:43:48,118 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}
2023-06-30T14:43:48,118 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}
2023-06-30T14:43:48,118 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}
2023-06-30T14:43:48,118 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}
2023-06-30T14:43:48,118 ERROR [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Tier[_default_tier] has no servers! Check your cluster configuration!: {class=org.apache.druid.server.coordinator.rules.LoadRule}
2023-07-12T14:32:16,804 ERROR [qtp1286172885-128] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not acquire lock for rowInterval[2023-06-19T07:00:50.000Z/2023-06-19T07:00:50.001Z], segmentInterval[2023-06-01T00:00:00.000Z/2023-07-01T00:00:00.000Z].
2023-07-14T01:02:47,407 ERROR [qtp1286172885-151] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not acquire lock for rowInterval[2023-06-08T07:25:07.305Z/2023-06-08T07:25:07.306Z], segmentInterval[2023-06-01T00:00:00.000Z/2023-07-01T00:00:00.000Z].
2023-07-14T04:03:19,617 ERROR [qtp1286172885-133] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not acquire lock for rowInterval[2023-06-08T07:25:07.305Z/2023-06-08T07:25:07.306Z], segmentInterval[2023-06-01T00:00:00.000Z/2023-07-01T00:00:00.000Z].
2023-07-14T04:03:44,215 ERROR [qtp1286172885-147] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not acquire lock for rowInterval[2023-06-08T07:25:07.305Z/2023-06-08T07:25:07.306Z], segmentInterval[2023-06-01T00:00:00.000Z/2023-07-01T00:00:00.000Z].
2023-07-15T08:32:07,574 ERROR [qtp1286172885-138] org.apache.druid.indexing.common.actions.SegmentAllocateAction - Could not acquire lock for rowInterval[2023-06-15T07:25:47.983Z/2023-06-15T07:25:47.984Z], segmentInterval[2023-06-01T00:00:00.000Z/2023-07-01T00:00:00.000Z].
2023-07-15T10:42:07,633 ERROR [KafkaSupervisor-eberspacher_gateway_sensor] org.apache.druid.indexing.overlord.hrtr.WorkerHolder - Failed to shutdown task[index_kafka_eberspacher_gateway_sensor_e0298609ba6419b_kdcjkjkg] on worker[10.101.37.237:8091] failed.
2023-07-15T11:44:59,485 ERROR [KafkaSupervisor-eber_gateways_sensors_data] org.apache.druid.indexing.overlord.hrtr.WorkerHolder - Failed to shutdown task[index_kafka_eber_gateways_sensors_data_785568a3990bdc5_dphgoehg] on worker[10.101.37.237:8091] failed.
2023-07-15T19:50:49,353 ERROR [KafkaSupervisor-eber_gateways_sensors_data] org.apache.druid.indexing.overlord.hrtr.WorkerHolder - Failed to shutdown task[index_kafka_eber_gateways_sensors_data_8aef43931580e0d_dcampgih] on worker[10.101.37.237:8091] failed.```

This is coordintaor log. overlord log should have more info on which job was actually having the lock

This is a bit strange as normally real time kafka tasks usually have the highest prioritization.  Bibek is definitely correct that another task is likely acquiring a lock with an equal or higher prioritization.

1689582728777.png

1689582655164.jpg

1689582655183.jpg

1. INSERT INTO test3 PARTITIONED BY (TIME_FLOOR(__time TO DAY)) VALUES ('value1', '2023-07-17T12:34:56Z');

2. INSERT INTO &lt;table name&gt;
&lt; SELECT query &gt;
PARTITIONED BY &lt;time frame&gt;
[ CLUSTERED BY &lt;column list&gt; ]

After studying the document and searching, it seems that there is a unified syntax for the insertion of the droid. I have tried both of them, but they are not effective. What grammar should I use?学

I could be wrong, but I’m pretty sure you cannot do `INSERT` with JDBC

You will need to use the REST API to do inserts <https://druid.apache.org/docs/latest/multi-stage-query/api.html#submit-a-query>

Can only select statements be used on the console using Druid? It shouldn't be possible! I have used the method you mentioned before, but the execution efficiency is too slow, so I am considering directly executing SQL statements

On the console, you can run any queries (SELECT / INSERT / REPLACE) - it will choose the appropriate API to send the request to (see screenshot)
Using the APIs you can send appropriate queries to appropriate APIs: Native queries to `/druid/v2`, SQL queries to `/druid/v2/sql`, MSQ queries to  `/druid/v2/sql/*task*`
Using JDBC you can only send SQL SELECT queries

Hello! Excuse me! Which method performs insertion the fastest? I need 18 seconds to import 1w pieces of data locally in bulk. Is there a faster solution?

we have seen that SQL based batch ingestion can be quite a bit faster than native (json) based, but it depends on your data

also, it depends on the partitioning strategy that you choose - dynamic partitioning is faster at ingestion, but slower at query because it does not optimize the layout of the data (but you can do that later using autocompaction)

if you append data to existing time ranges (in SQL this is an INSERT statement), you always get dynamic partitioning

if you replace data (SQL: REPLACE), you can use range partitioning right away

Can SQL based batch processing only use this SQL format? I want to insert data from other sources, but I don't want to use this method. My understanding is that this method reads data from other tables and writes it to this table

SELECT in the MSQ version uses an EXTERN clause to address data sources that are not Druid tables: <https://druid.apache.org/docs/latest/multi-stage-query/concepts.html#read-external-data-with-extern>