This message was deleted.

There are different query load testing tools out there ... one I have heard used frequently is Locust https://pflb.us/blog/load-testing-using-locust/ ... and connect directly to the Druid SQL API to run the tests.

We did this quite a few times. 1. Build the new Druid inside a separate Kubernetes clusters with similar machine spec. 2. Point the new Druid to the same PostgreSQL and S3. 3. Start load testing (or tell your customers to start load testing). 4. When it’s deemed good, flip the top level load balancer to the new Druid. 5. Shutdown the old Druid.

Is it safe to have two druids pointing at the same database and s3?

We had that same question and spent 6 months verifying it. It’s not difficult but it requires some work. Basically, you can split druid into read-path and write-path. 1. Tell coordinator that you want to replicate to both new and old clusters using replication tier Rest API. 2. When creating a new Druid cluster, you don’t need to run Overlord and MiddleManager (they are the write-path). This way writes are safe. 3. Each cluster must have their own ZK. Do not share ZK between clusters. That’s it.

Tempting as that is, I would be afraid druid would try to migrate table schemas or something else dire like that.

This is a legit concern. When Druid creates a new table of columns… there will be no way of going back.

I don't know how "downgradeable" the releases are. If you want to protect your ability to revert to the older release and there is no documented downgrade path, then my first thought is to stand up a new cluster with the new release, populate it by doing deep storage and metadata migrations (HERE and HERE), and set up dual ingestion ... then test and shut down the cluster you don't want to keep.

That may be the right answer for us. Copy the files, copy the db, stand it up, see how it runs, change the ingress to the second druid, decommission first. Particularly easy if you are going s3 -> s3, postgres -> postgres.

Fortunately Druid's storage file architecture makes the migration process relatively easy ... the segment files are simple zip files with smart-coded filenames, and the only permanent direct reference to these files are in the druid_segments table.