This is something that we noticed about the evergreen merge bot that we fixed, so I wanted to flag it.

If the bot that does the merging of a PR is an admin for the repo, github will let it merge, and show the pr as "mergeable" when the user that is asking if it's mergeable is able to merge it. Since the user (aka mongo-cr-bot) asking is an admin, it'll show it as mergeable because the admin is technically able to override branch protection rules and merge anyway.

That means that if for example, the repo is set up to require that a specific test passes before merging, the bot will merge it even if that test didn't pass because it's not checking if it's mergeable for the author of the PR, it's checking if it's mergable for the admin user that's asking.

We circumvented this by only merging it if the mergeable_state returned here is something other than "blocked".