CI checks run before a PR can be entered into the merge queue and again while a PR is in the merge queue.
This is the series of events we are dealing with:
1. All CI checks pass on the PR. The PR is now ready to merge.
2. If the "Merge when ready" option on the PR is checked, it will enter the merge queue.
3. In the merge queue the CI checks run again. Due to flaky backend unit tests, issues pulling a docker image, etc, the CI checks which previously succeeded now fail during the merge queue.
4. The PR is removed from the merge queue due to the failure(s).
5. The PR is no longer marked as "Merge when ready".
Step #5 above is the concern. We are dealing with some flaky CI issues, and while we work on fixing them, we'd like for the opposite of #5 to happen. After the PR is removed from the merge queue, it would still be marked as "Merge when ready" and could reenter the merge queue to try again.