Apache Druid

Druid has the ability to use basic authentication (including the console) - Have you seen <https://druid.apache.org/docs/latest/operations/security-user-auth.html> ?

You’re invited! Registration is now open for Virtual Druid Summit 2022, December 6-7th! Join fellow developers, architects, and other data professionals in learning or expanding your knowledge of #ApacheDruid. Register today! <http://www.druidsummit.org|www.druidsummit.org>.

ok, found this one <https://docs.imply.io/latest/deploy-manually/>

IIRC, the `*.sh` stuff is used for Docker images

this tutorial talks about the `bin/start-*` stuff, which you can use if you're running outside the Docker images: <https://druid.apache.org/docs/latest/tutorials/cluster.html>

thanks for the information <@U0344FW86DD>

Yes, you should be able to by changing the `PatternLayout` within your `log4j2.xml` file:

`&lt;*PatternLayout* pattern="%d{ISO8601} %p [%t] %c - %m%n"/&gt;`

Here's the Druid doc for reference:

<https://druid.apache.org/docs/latest/configuration/logging.html>

and here's the `PatternLayout` doc:

<https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html>

A colleague of mine has finished up a Druid logs course, by the way. Hopefully it will be out of preview soon.

Druid marks offsets when stopped and restart the tasks from the same offset when kafka has been brought up. If the offset which druid wants to start from is not available in kafka then it will fail.

If you want to read from the first available record in topic, you may do a HARD RESET in supervisor and then start it from beggining

but does this duplicate or loss data in druid ?

what is the other option there other than Hard Reset !

Hard Reset will start reading the first available data in the topic and will read continuously further. I believe there is no other option of setting specific offset value, But I am not completely sure of that.

Ok thank you Sahoo,  looks like after hard reset am getting data into druid.  :pray:

We using single server - small configuration deployed of druid database with Kafka stream ingestion.

It was worked fine till date... but recently facing following error =&gt;  in ingestion

      "exceptionClass": "java.util.concurrent.ExecutionException",
      "message": "<http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Exception [Connection refused (Connection refused)] while pausing Task [index_kafka_

While checking the data-source status log &amp; report " request failed with status code 404 "

Actual problem am facing is  Tasks (in the ingestion) is keep going to PENDING status

While it's in PENDING status getting 404 exception

Responded in the actual thread, PENDING status could be because of slots available, you can check the slots available in the services page in middle manager

If there are more tasks running than available slots, the tasks will go into PENDING and wait for any tasks to be completed and slots to be available

will see the logs and kafka connection live status thank you

This is what I did and it worked fine, I don’t know it this is the best way though

Thanks for the reply Julien. What was the metadata you were using ? It is the internal derby DB ?

Aah.. Ohk.
Only part I am more cautious is copying the conf &amp; var folders and what happens with the internal Derby DB.
I think somewhere I read that when we restart the service, it would pull the segments in the var folder from deep storage automatically. but I may be wrong though.

I don’t feel confident enough with my current druid skill to talk on the internal Derby DB subject

Number of jobs  does not have any direct relation with data servers. Number of tasks will depend on “taskCount” defined in the ingestion spec.

We getting GC error even though we have increased the worker count , heap size to the max ram available

The data we ingesting is around 35 million records from mysql to druid we doing

And we can't figure out which data server its refering to bcz thr are in only one task instance

In the druid console ingestion page where it shows the job status as running, There is a “Location” column that should show in which data server the job is running.

yeah i checked the location part it looks like only one data server is getting used how do i make sure all the 3 data servers are getting used for ingestion?

<@U03R5H3B750> between how to write the taskCount in the ingestion spec?

<https://druid.apache.org/docs/24.0.0/operations/basic-cluster-tuning.html#task-count>

yeah like i said i have increased the worker.capacity to 10 in the middleManager runtime.properties file even the druid.indexer.fork.property.druid.processing.numThreads to 10

still it launches only one task instance on the console when i see and it fails due to GC error

<https://druid.apache.org/docs/latest/ingestion/native-batch.html#parallel-task-indexing>

• the `maxNumConcurrentSubTasks` greater than 1 in the `tuningConfig`. Otherwise tasks run sequentially. The `index_parallel` task reads each input file one by one and creates segments by itself.


Number of running tasks :
The `maxNumConcurrentSubTasks` in the `tuningConfig` determines the number of concurrent worker tasks that run in parallel. The supervisor task checks the number of current running worker tasks and creates more if it’s smaller than `maxNumConcurrentSubTasks` regardless of the number of available task slots. This may affect to other ingestion performance. See <https://druid.apache.org/docs/latest/ingestion/native-batch.html#capacity-planning|Capacity planning> section for more details.

ok i'll check this, if you can help with this as well it would be helpful " how do i make sure all the 3 data servers are getting used for ingestion?"

IMG_0486.PNG

Local in-memory cache is the default, but you can set a different type of cache by using `druid.cache.type`:

<https://druid.apache.org/docs/latest/configuration/index.html#cache-configuration>

Can you share a bit about your use case and your particular local cache needs?

Thank you Mark! 

For my use case: we need to store huge amount of data in deep storage, and then would query them through Druid using an API. 

Basically our concern is: do we need to have enough space for historical nodes to be able to store all the data we have in deep storage? If not, what would be the drawback of not having all data on historical node? 

You're welcome, and thanks for sharing more details!

I'm guessing more people will chime in, but here's <https://druid.apache.org/docs/latest/dependencies/deep-storage.html|something to consider>:

&gt; This deep storage infrastructure defines the level of durability of your data, as long as Druid processes can see this storage infrastructure and get at the segments stored on it, you will not lose data no matter how many Druid nodes you lose. If segments disappear from this storage layer, then you will lose whatever data those segments represented.
I know of a couple of instances, though, where people have lost deep storage (:scream:) and queries kept getting served. But those were definitely mishaps.

Thank you Mark! I really appreciate your input. We are very unsure about whether storage space on historical nodes needs to be as big as our entire data stored in deep storage. 

just keep enlarging the maxBytesInRam. But make sure you have enough heap size for your JVM process.

according to druid documentation on maxBytesInMemory  , by default , it is 1/6 of max jvm, as you can see, it is already pretty high here. maxBytesInMemory[1431655765]
```Used in determining when intermediate persists to disk should occur. Normally this is computed internally and user does not need to set it. This value represents number of bytes to aggregate in heap memory before persisting. This is based on a rough estimate of memory usage and not actual usage. The maximum heap memory usage for indexing is maxBytesInMemory * (2 + maxPendingPersists)```

From our experience, if you have &gt; 10GB heap, you can set it to 1/3 max JVM

Clearly your payload size is way bigger than your MM RAM capacity

thanks, I am doing a batch processing here.

As native ingestion requires a bit of magic to tune, I would urge you to try the new SQL based ingestion: <https://druid.apache.org/docs/latest/multi-stage-query/index.html> where in the memory is tuned automatically.

1.4 Gig in memory sounds like a lot to me. Maybe just add more workers ?
and use `useMaxMemoryEstimates=false`

will `useMaxMemoryEstimates=false` cause OOM issue?

It does a better job estimating the memory so chances are reduced

where do I configure this? in task definition?

You can set in the `context` of the task.
Or in the overlord runtime properties.
`druid.indexer.task.default.context={"useMaxMemoryEstimates":false}`

image.png

More storage sounds like a comparison so it's hard to say if it requires more than something else. To clarify there are two types of storage that Druid uses. Deep Storage should be a durable storage that all historicals can read from and middle managers can write to. This allows for data to be shuffled across the historicals for balancing and makes it easy for them to reload the data if something happens to a single instance. The second storage type is the segment cache. This is normally less durable storage directly attached to the historical machine which allows for faster access. This keeps query times lower.

Normally deep storage would be something like S3, GCS, or HDFS. Segment cache is normally something like EBS or locally attached storage.

Do you know why that particular Druid cluster is working, even with the wrong identifier?

It's only "wrong" in the sense that it's not the name I gave the cluster... but I think the point of this identifier is just for Druid to internally reconcile different clusters, no? IOW I believe I can pass any value I want. It just in this case I would like to change the value I'm passing so it's logically consistent with the rest of my env.

I'll keep digging, but I apologize for using the word "wrong" in that way. It was clunky and had the wrong connotation. I was looking at the description in the <https://druid.apache.org/docs/latest/development/extensions-core/kubernetes.html|documentation> and could have chosen a more artful word:

&gt; Unique identifier for this Druid cluster in Kubernetes e.g. us-west-prod-druid.

Oh my gosh, no need to apologize! I really appreciate the help, and I certainly didn't take your question badly. Sorry if my use of quotes made it look like I was irritated or something :slightly_smiling_face:

Druids = most polite and collegial database users :thinking_face:

You are most kind for the follow-up, thank you!

Regarding `druid.discovery.k8s.clusterIdentifier`, it's interesting because I'm not finding more details on GitHub:

<https://github.com/apache/druid/search?q=druid.discovery.k8s.clusterIdentifier>

I think what I'll do is first try changing it in my dev cluster and see if anything breaks there

Sounds good, and thanks for taking the time to document this.

OK, so I think it'll be fine to change this value. It looks like what happens (my clusters are deployed with the <https://github.com/druid-io/druid-operator|Druid Operator>, btw) is that changing the value causes the cluster components to do a rolling restart in k8s, which hypothetically is zero-downtime. So I think I'm in the clear to update the value.

Hello, on the Druid console, you can issue a kill task that will remove unused segments for good.

will this issue kill task for all unused segments?

Once all the “old” unused segments will be removed, the coordinator should be able to keep up the pace to remove your “new” unused segments

Hello <@U03LL3GQSN4>, did you manage to accomplish what you wanted?

not yet Thomas.. will try and update in this thread

i tried this but it was throwing java heap space error when i triggered for entire data source.
So had to trigger kill tasks separately for each datasource and each interval

Or, is there a way to set the number of retires after receiving HTTP Bad Response from s3 deep storage servers? 

Some background: we see this issue whenever there are long ingestion tasks running at the same time. The errors are several: HTTP bad response from &lt;s3_server_address&gt;: Connection timed out. 

And the error messages are:
[task-runner-0-priority-0] org.apache.druid.indexing.common.IndexTaskClient-Bad response HTTP [no response] from [deep_storage_address]; will try again in [PT2S] (body/ exception: [Connection timed out (Connection timed out))

&gt; we see this issue whenever there are long ingestion tasks running at the same time
Just curious: how big are the segments that are successfully written to deep storage?

Hi Mark, the size of the segments that are successfully written actually varies. Sometimes big ones succeed and sometimes small ones succeed. We have a setting of maximum of 60,000 rows.

Thank you. I'm wondering if you might be seeing a <https://druid.apache.org/docs/latest/operations/segment-optimization.html|segment size optimization> problem?

Optimal settings vary by workload, but generally each segment should have around 5M rows and/or be between 300 -700MB. Were you able to test with those recommended segment sizes before settling on a maximum of 60,000 rows?

Hi Mark, thank you very much for the suggestions! I think sometimes we have segment size about 1G, and we have maximum rows per segment of 5M. In this case we can reduce the maximum row per segment and try again. I will report back when I know more. Thanks again!

We have done a lot of work in SQL based ingestion: <https://druid.apache.org/docs/latest/multi-stage-query/reference.html> so that you do not need to set max rows explicitly and the segments are automatically cut based on the data. If you have time, do check that out too.

By any chance, are there gaps (days with no data or something analogous) in the interval specified in this compaction task?

my interval is
`"interval": "2022-11-01T00:00:00.000Z/2022-12-01T00:00:00.000Z",`
and i can see few segments(which are in interval after today's date) have 2 rows
will that cause any issue

and one more thing i can see for all segments in this interval is
`"Partitioning":"numbered_overwrite"` 
`"Partition":32768`
i think this was done when i ran compaction task with `"forceTimeChunkLock": false` is this correct?

I believe that's correct. Partition numbers 32768 or higher are generally from non-time-chunk locking

Btw, if you continue to have any issues with native batch ingest I'd suggest upgrading to 24.0 and going with SQL-based ingestion (<https://druid.apache.org/docs/latest/multi-stage-query/index.html>). It's much nicer :smile:

Hi <@U0344FW86DD>
I recently upgraded to 0.22.1
Ingestion is working fine

Only have issue in compaction task
Can you please help in this issue ?

i haven't seen this particular issue before

if you have steps to reproduce it on a fresh installation, or any other additional details, you could try posting that here or in github at <https://github.com/apache/druid/issues/new/choose>

are you referring to the taskCount in the spec to allow one at a time to run?

Yes that is probably what we need. Thanks Ian!

If there is a document for taskCount that would be awesome. I tried to search online but only found this <https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html>.

Is this for a real time ingestion spec (kafka,kinesis)?

Actually it is for native batch ingestion Ian.

the type of the ingestion is "index_parallel"

you may be looking for index instead of index_parallel -&gt; <https://druid.apache.org/docs/latest/ingestion/native-batch.html>

Oh I see. So we should use "index" instead of "index_parallel." Thanks Ian!

You can also use SQL-based ingestion <https://druid.apache.org/docs/latest/multi-stage-query/index.html> and set the number of tasks to 2 (the minimum)

I'm wondering why you want to run tasks one at a time, btw?

Thanks Gian! I really appreciate your input!

For the reason to run tasks one at a time: we are seeing some issues for Druid to contact its data nodes when multiple index_parallel tasks are running. We are trying to find the root cause, but in the meantime we need a temporary solution. We believe that by queuing up index_parallel tasks in Druid, we can avoid this issue temporarily until we find the root cause.

Would it be possible to buffer up the messages near the source?

If so, one option I could think of is to implement the `SeekableStreamIndexTask` interface provided by Druid to create tasks that pull messages over http from the source. Since ingestion would be best effort only, it would be a lot simpler since we won't have to worry about all the state management that KafkaIndexing tasks have to do to ensure exactly once ingestion etc.

&gt; If so, one option I could think of is to implement the `SeekableStreamIndexTask` interface provided by Druid to create tasks that pull messages over http from the source.
Yes, I'd go as far as saying we could keep a small in-memory buffer / queue of inflight messages. Combined with an aggressive concurrency limit, this should be safe + efficient.

There'd be lots of architectural questions to address, such as how do you manage traffic / load balancer / discovery with multiple tasks on a single hosts.

<https://github.com/druid-io/tranquility|Tranquility> seems to have some way to do this (browsing the docs), but I'm not clear on its relation to the Druid project.

Tranquility was something we also evaluated internally as a means to push data to Druid vs having Druid pull it from somewhere.

The push mode has issues including inability to guarantee exactly once ingestion, data loss in case of exhausted retries or events arriving out of the window. There are use cases like the one you are alluding to that probably don't need these guarantees or features where tranquility might provide an option. However, we need to be cautious here as the project hasn't been maintained in a while (the last build was against Druid 0.12, I believe).

I wrote Tranquility as a way to push rows directly into Druid, but I stopped maintaining it once the Kafka ingestion stuff came out

(Tranquility predated the current exactly-once Kafka integration)

The main reason was that I did not find very many use cases where not needing Kafka outweighed the downsides Samarth mentioned

Thanks. I hadn’t given much thought to duplicates etc. good to know where it’s at.

Imply is hiring :slightly_smiling_face: <https://imply.io/positions?gh_jid=5392979003>

Join operators exist, but will impact performance: <https://docs.imply.io/latest/druid/querying/datasource/#join>
But maintaining two separate datasources, one with the typical dimensions and another with additional ones, could work.

If you have some tool that's routing queries in an intelligent way, you could definitely maintain two data cubes with the same data.

fwiw this functionality in a single datasource is something that is likely to be on the nearish term roadmap. In some discussions we have been calling it ‘projections’, but its the same basic idea of having a few different representations of the data and choosing the most appropriate/efficient one at query time

i don’t have an exact timeline or anything

but the perf improvement should be huge in use cases like this

uh, very cool. I’m looking forward for projections :+1:

No, you can have only one auto-compaction configuration. For your use case, you could try to re-index the older data (where you want YEAR granularity) into a separate datasource.

Thanks. I have manual compaction tasks for different periods in one data source, was curios if we can automate it. Would be good feature.

Yeah, manual compaction would be the other alternative.
Would you mind creating a github issue for this? It would be useful to track the feature request and to see if other users in the community also have similar requirements.

check you coordinator log to see if the datanode is registered, make sure the middlemanager is up and is not blacklisted.  You could check the MM log itself as well.

you may need to increase your task count: check the Services tab in the web console to see how many slots are being used

For streaming ingestion, if nothing else is running, you can set `taskCount` up to half the slots if `replicas` is 1, or 1/4 the slots if `replicas` is 2

For batch ingestion via MSQ you can use all slots if nothing else is running, by setting `maxNumTasks` to the number of slots

I was going to say... SQL based ingestion (via MSQ) makes it super simple to see how many resources you are actually using.... it puts it front and center in the UI:

In my screenshot the first stage only used one worker because it was only reading one file (this tells me I could boost the speed by splitting the input data into more files) but the subsequent stages went to the max allowed workers: 3 (for a total of 4 tasks including controller)

Hey Jose.
First of all I think its pretty cool that you are exploring druid for a class project. Kudos for that.

Okay so from what I can gather `pravega` look like a "service/external" system from which you want to ready data and then index that data inside druid so that you can query.

This kind of system, service is modelled as an input source inside druid
<https://druid.apache.org/docs/latest/ingestion/native-batch-input-sources.html>
They are multiple such input sources like Azure,S3,HDFS,HTTP, local.

If pravega supports http endpoints to read data, then you can use the `httpinputsource`and ingest the data accordingly.
If its supports other RPC calls, then `InputSource.java` needs to be implemented in your extension. Checkout `OssInputSource` has a reference.



Other thing can be pravega, has its own "format" not covered in <https://druid.apache.org/docs/latest/ingestion/data-formats.html>

For that the changes would require you to implement `InputFormat.java` in your extension. You can look at the `KafkaInputFormat` for example.


In both the cases the new implementation needs to be binded in guice.
Checkout `OssInputSourceDruidModule` for binding the input sources and `KafkaIndexTaskModule` for binding the input format.


Now you will have to load the extension in the loadlist of indexer/middlemanager/peon/overlord so that the new extension gets invoked.

Let me know if you need further clarifications on the above points.
Happy to help.

Also if you could add a bit of documentation about `pravega`  it helps us giving pointed answers.

By pravega did you mean this : <https://cncf.pravega.io/>

I think he meant the pravega from the link you shared above. Generally it would be great to include pravega along side stream ingestion sources currently supported like kinesis and kafka.

Hi sorry for the late response, yes I meant Pravega as in the link you sent above, we want to create a connector from this data source to druid. A helpful piece of documentation for pravega is this link:  <https://cncf.pravega.io/blog/2020/09/22/pravega-client-api-101/>
This is the api we are using to fetch the existing data from pravega (we also want to transform this data)

thank you <@U0315MQKEHK> for the information in your message above regarding the input sources. We will investigate that link and also check to see what kind of endpoints Pravega supports, we will reach out for further questions!

<@U0315MQKEHK> , is there information related stream ingestion for Apache Druid? The link sent above regarding the input sources focused on batch ingestion and not stream ingestion which is what we are looking for since Pravega is a data source for streams. Similar to Kafka or Kinesis

Yes. Implementing a stream connector is going to be more involved. It would also depend on what primitives and concepts does Pravega offer. E.g. if they have concepts of offsets and partitions (sequence and shards in Kinesis), then it may be easier to extend.

you can start exploring kafka-indexing-service extension. that, IMO, will be a good starting point

maybe your kafka ingestion spec needs a change to account for that column?

what do you mean with that "account for column"?

I mean, does the column need to be added to the ingestion spec?

i just added the column in  the ingestion spec and the new data has arrived. Thank you. But does that mean I have to do reindex and then add in ingestion spec too for adding new column in datasource?

If you have added the filed in the ingestion spec then there is no need to do re-indexing. The filed will get added for all the ingestion going forward in time from the time you modified the spec. Are you trying to backfill this filed for a time period before you modified the spec?

added an integration to this channel: <https://apachedruidworkspace.slack.com/services/B04C0E9LJEQ|linen.dev>

I am working on some the config stuff, and will post the link to the searchable workspace when it is set up.

For some reason, linen is replacing the names with some random strings like `agreeable-spoon-97825`. I don't see the same happening in flink project. is this a result of some setting?

We use ZGC here and there, for example on some of our non-prod clusters.

I am currently investigating if ZGC is the way to go for Historical because I have this theory that all of our `DruidProcessingModule` threads are hanging because G1GC takes a very long time collecting garbage when our users are abusing Druid with their `LIKE` queries.

ZGC is currently not-generational so some of the GC metrics are missing.

But I currently don’t know if swapping to ZGC will cause query performance degradation or not. I don’t know how to measure it.

do you have druid `query/time` and `jvm/gc/cpu` metrics being collected somewhere?

jvm.gc.cpu is one of the metric that I weirdly no longer have. Even on G1 GC

I do have broker.query.time.{table}.groupBy.{95percentile|median|max|etc}.

you might need to add some additional jvm command line params to get it to work properly

ooo i remember, I got exception on boot due to Sigar library

the list we'll use for 25.0.0 is here <https://github.com/apache/druid/blob/25.0.0/examples/bin/run-java>

last one `--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED` i think is the one you need for full gc metrics

although it doesn't hurt to add all of them

this is what I have so far:
```--add-exports=jdk.management/com.sun.management.internal=ALL-UNNAMED
--add-exports=java.base/jdk.internal.perf=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.management/sun.management=ALL-UNNAMED```
I see that they are quite different. This is list is helpful!

http 504 is a timeout; it's certainly possible there's so many tasks that it takes too long for the call to come back

i'm not sure what version you're using but this was improved somewhat in recent versions

that being said if you have tons of tasks in the relevant time window, then reducing `druid.indexer.storage.recentlyFinishedThreshold` would certainly help with the calls the console makes

i typically set it to `PT24H` or `PT48H` in very busy clusters to minimize the times those API calls take

note that you can retrieve info for a specific older task by API if you know the ID

Hi Gian, thank you very much! That is really helpful! We will reduce the `druid.indexer.storage.recentlyFinishedThreshold` to improve the performance, and use API call to retrieve old task logs.

Quick question: do you know which version contains the improvement?

<@U0344FW86DD> Hi Gian, another quick question: does setting `druid.indexer.storage.recentlyFinishedThreshold=P1W` mean that the sys.tasks table would only contain 1 week of ingestion logs, or would the table still has all the ingestion logs in the past, but just that when querying ingestion logs the time would be restricted to past 1 week? Thanks in advance!

the task metadata would still be there in your metadata store (mysql/postgresql) but not visible in `sys.tasks`

(this is the main thing that speeds up the API calls)

Oh I see. That makes sense now. Thank you Gian!

Hello Druids

The Apache Druid team is proud to announce the release of Apache Druid
24.0.1.
Druid is a high-performance analytics data store for event-driven data.

Apache Druid 24.0.1 is a bug fix release. We have fixed several bugs and the full list can be found <https://github.com/apache/druid/milestone/47?closed=1|here>.

Some of the notable issues that were fixed are as follows:
- Hadoop ingestion with custom extension config was not working
- Overlord doesn't start if there are dangling locks in the system. This required manually cleaning up the metadata store.
- SQL statement objects were leaking memory
Source and binary distributions can be downloaded from:
<https://druid.apache.org/downloads.html>

Release notes are at:
<https://github.com/apache/druid/releases/tag/druid-24.0.1>

A big thank you to all the contributors to this milestone release!

Thank you <@U0318CX1NH4> for driving this release.

druid_comm_member_month_1400x1400.png

<@U0465LEAZD5> Thank you very much :pray:

Yahs <@U031Y2P142E>! Very well deserved. Saw you speak recently and your talk really showed how expert you are :slightly_smiling_face: Thanks for sharing all you know with people :smile:

I haven't, but just thinking through this. If you are just pruning older data should be a matter of <https://druid.apache.org/docs/latest/operations/rule-configuration.html|setting up retention rules>, and running a kill task. You have the data split into time intervals (as folders) in Deep Storage and you have the metadata rows for the sys.segments table. I think that's all you need to backup: copy the segment files into a different bucket/folder and backup the metadata rows.  In order to be able to restore them later, add those rows back into the sys.segments table and put the segment files back into their respective Deep Storage location... it would be nice to automate this.

Interesting <@U030K4UM3H7> let me look at this further, I could experiment with some of the older segments.  Thanks!!

you can certainly back up the files on deep storage; that's the easiest way

check out the `loadSpec` of the `payload` column of each segment in your metadata store (that would be the `druid_segments` table in mysql/postgersql)

it has all the exact paths in deep storage to look for

Thanks <@U0344FW86DD> I will do that.  Looks like I need to take all that data and load it into a relational DB like Hive.  I would imagine that I have to go through a Druid instance to do that...

you can use the dump-segment tool: <https://druid.apache.org/docs/latest/operations/dump-segment.html|https://druid.apache.org/docs/latest/operations/dump-segment.html>

however as that page points out:

&gt;&gt; Note that the dump is not necessarily a full-fidelity translation of the segment. In particular, not all metadata is included, and complex metric values may not be complete.

so watch out for issues with complex fields, if you use any of those (like datasketches)

btw I'm wondering what the purpose of such pruning is? is it mainly a cost thing?

we're planning some new features around being able to query data from deep storage without preloading it on Historicals; designed for use with older or less user data

if that sounds interesting i'd like to understand more about your needs here, as that will help us build the feature in a way that is useful to everyone

Hey <@U0344FW86DD> thanks for the pointer to the tool, I don't have direct access to the file storage but I'll ask our team about this.  We are doing this pruning to reduce the file system footprint of our Druid cluster, and this dataset was identified as one of the larger ones.

Hey <@U0344FW86DD> I think I may go down the route of doing scan queries: <https://druid.apache.org/docs/latest/querying/scan-query.html>. While this means that I'll need to do more than 100,000 of these, I can rate limit them and keep track of offsets so I can restart the process.  This can be fairly easily scripted and run off-hours, without impacting our application server that sits on top of Druid.