https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
d
Too bad the authn/authz data inside 
druid_config
is stored in Smile format. Why not just store that in plain-text JSON? It’s a bit hard to introspect.
s
Hey @Didip Kerabat, are the authentication settings the same on both clusters? Also, "it should have worked"? 😄 How are you planning to deal with conflicting writes to the metadata DB? My first thought is the server_segments table that holds the segment-to-historical assignments... that can't possibly work without two separate metadata DBs.
d
You were right. 1 of the basicAuth creds were not the same. This is my bad. About metadata db, I am planning on still running the ingestion on the old cluster until the last minute. For a temporary moment, the new cluster will be read-only. After the cut over, I will kill the MM on the old cluster and switch them to the new cluster. And then we will run backfill on the missing dates. That’s the plan so far.
s
Hi @Didip Kerabat are you trying to basically: 1. Migrate from one cluster to another where you will turn off one cluster once the other one is available? 2. Are you trying to keep 2 clusters running in parallel and then after testing decommission the old cluster? If you are doing number 1 then you can use the same deep storage and metadata because you can point the new cluster to the existing deep storage and metadata. You need to make sure that the old cluster is stopped. This is needed because both the old cluster and the new cluster will try to effectively write in the same deep storage directory for the same time frame. You can avoid this by using a different deep storage basekey for the new cluster. The kafka offset is stored in metadata. So, streaming can start in the new cluster from where it left off. For option 2, you would ideally want a new deep storage and a new metadata with all the data copied across. The metadata has paths to the segments stored in the deep storage that you would need to change.
d
It’s option number one, we don’t want to keep the old cluster too long. Yes, the Historical replication is solved by using different replication tier, e.g. 
_default_tier
and 
_default_tier_2
. The ingestion is the tricky bit, as you mentioned. We have to run the ingestion in the old cluster until the very last second. Then we will shutdown the old ingestion, and then run the identical one in the new cluster.
s
Actually the ingestion is the most fun part. As the offset is stored in the metadata storage, as long as you have a long enough kafka retention period, the new cluster will catch up. I am in a meeting currently. I am happy to clarify this more if that helps once I finish the meeting
d
Good point. Thanks for explaining that for Kafka ingestion. Thankfully most of our ingestion is batched, thus we can just run backfill from Airflow.
s
I see ok that makes sense why you want to keep them running till the last moment.
To be honest if most of your ingestion is batched and you can pause your batch ingestion then you can just delete your old cluster, point the deep storage and metadata to the new cluster and then start the batch ingestion. You shouldn’t miss anything.
Is there any edge case that you are thinking about here? Why do you want to keep the old cluster running in parallel for a short period? I am trying to understand the concerns here.
By the way I am thinking about doing a meetup presentation on druid migration where this topic is covered and then do a follow up blog post on this. Do you think this will be helpful?
🎉 2
d
It definitely will be helpful. I’d consider this fairly advanced topic so may not be useful for beginners
True, even batch ingestion can be paused. The query serving is far more important and even there, there could be issues. We are also thinking of on-boarding all of our Druid clusters into the in-house, planet-scale, blue-green load balancer system as part of the same deployment. Just in-case if that move went wrong, I want to make sure that the old cluster is still running with its existing LB setup.
s
Another thought is auto-compaction, i don't think you want that to be happening concurrently on both clusters. Both making different decisions about compaction and making possibly conflicting changes on the metadata repo and deep storage.
d
ho ho ho, damn. I did not think of that. However, the data versioning should save us here, no?
s
But wouldn't there be two different overlords managing the segment allocation, I think they could still collide. You could turn auto-compaction off on the target cluster until you are ready to switch over.
d
good point
s
Thank you for the feedback on the meetup and blog. I will provide a link here once the meetup is scheduled or the blog is published.
As you mentioned above, you are trying to do a few changes and that’s why you want to keep the old cluster around. If that is the case then the safest way for you to do this is to copy your deep storage and metadata storage and run the old cluster and new cluster in parallel. If you do that, then you eliminate the risk of things going wrong during the testing phase. All of your queries and streams can go to both of the clusters. Once you are happy with the new cluster, you simply delete the old cluster. Please note that in the metadata storage druid_segments table, s3 paths are tied to the bucket name. So, you need to ensure that you change the bucket name so that the new cluster finds the right bucket. You can find more information on this if you check the payload in druid_segments table. This is the best way to run 2 clusters in parallel and do production cluster migration with zero downtime. Otherwise, if you want to run 1 cluster at a time, you can simply delete the old cluster and point the new cluster to the deep storage and metadata storage of the old cluster. This will eliminate the risk of 1 cluster overriding the data of the other cluster. If you still want to keep 1 deep storage and metadata storage then you can configure a different base key for the new cluster deep storage and then the new cluster data will essentially go in a different folder. This will ensure that the clusters don’t override each other’s data. You will still need a different metadata store for the 2 different clusters because the tables in the metadata storage store a lot of metadata information that is meant to serve only 1 cluster. For example, the druid_segments table has all the path of the segments in deep storage. It also has a column called used. Druid checks this column to find out which segments are in use and which are not. Both of the clusters will get very confused when they have new data because of this column and the associated payload.
Hi @Didip Kerabat I hope you are well. Did you manage to complete the migration? How was the experience?
d
Not yet, maybe in around 1 month. But in the lab it looks great! Everything worked all the way up to the geo-LB that can swing traffic between the 2 clusters. We just have to ensure the writes only go to 1 cluster.
s
Great 🙂
3 Views
Open in Slack
PreviousNext
Powered by