You need to close and delete the branch, then delete and readd the backport label