Apache Druid

I actually have an internal PR for hadoop ingestion that allows reading data from one cluster and write to other cluster. 

arrr… I would love to have that but it won’t work for our use-case because of org-wide policy of not sharing PG access between k8s clusters.

Well, all you need at the end of the day is the list of files from deep storage of Datasource A that you want to ingest. You could possibly execute sql against postgres for cluster 1, build the static path of the spec listing the files, and then submit the spec to do the batch ingest on cluster 2. Of course, it would be nice if Druid could provide that out of the box which shouldn't be too hard to implement, too.

The most unfortunate part is that data blob inside druid_segments, one has to deserialize/serialize the blob just to change the s3 bucket name. I wish those info are stored in real PG columns.

oh! perhaps I can simplify my problem by having the 2 clusters sharing the same s3 bucket. then it’s just pg_dump and pg_restore process.

My need happened to be for setting up a DR cluster for 0 downtime maintenance.