This message was deleted.

Sounds bad and very risky. I would recommend you setup new druid cluster, doesnt need to be as big as exsisting at the begining, and migrate datasources one by one. Starts with ingestions, then copy all old data between hadoops, export/import metadata, check if datasource on new cluster have the same data as on old cluster, switch queries to new druid cluster, unload datasource on old cluster, decommision historical from old cluster, shut them and reconfigure to add to new cluster. Repeat.

Yeah... figured it was not an ideal solution, but ive had middlemanagers crash with dozen running tasks on them to just to be restarted on another box... without much impact... So figured id check with you guys if someone actually did that.

Wouldn't you need to restart druid services to read the updated hdfs.xml anyway? I'd think you could copy, update and restart, or do something like Piotr suggested, to avoid downtime.

That’s the plan update the configs, copy everything to new cluster, restart brokers, historical, and overlord nodes and then restart middle managers last. Since historical nodes are the ones responsible to store to deep storage. Since our tasks run in batch ingestion in 1H, I think that should be plenty of time to rotate everything and restart?

I personally think that could work, but I'd suggest at least creating a small test cluster and testing on that first, to check for issues, because I could be wrong.

Historicals dont write to deep storage, only read form it. Handoff process is only for realtime ingestion when creating segment is done on middlemanager. Then MM pushes segment to deep storage, announce segment is ready and coordinator tells historicals to load from deep. When loading od complete, brokers are notified to route queries to historicals instead of MM and MM clean up task

just finished migration.. and wanted to let you guys know how i did it with 0 downtime: Wrote ansible playbook which 1: backed up and copied new hdfs config files. 2: Updated common.runtime.properties and runtime.properties for middleManager to new hdfs path. 3: updated mysql metadata

"update druid_segments set payload = REPLACE(payload, '<hdfs://nameservice1>', '<hdfs://new_nameservice>') where payload like '%<hdfs://nameservice1>%';

4: restarted all services. All set. One note tho.. i didnt restart middleManager right away, and did it one by one.. but if i would do it again, i would restart middleManager also together with all other services, cuz then tasks will restart and will use new hdfs path. Otherwise, the service will keep using the old path still and try to load them to wrong path. (unless you keep the "hdfs://nameservice1" part the same name for new cluster too.. then you dont have to worry about that or updating the metadata.