Late last night we had some issues with mongos (unfortunately not clear what went wrong - bouncing fixed it). About an hour later we then had a massive spike in the number of connections from mongos to the primary. This then caused 'too many open connections' to start flooding the primaries logs and connection attempts throughout our application to consistently fail. In effect, our primary was dead.

However, our primary was still telling all the secondaries that it was alive and well so no failover happened.

I think the health checks need to do more than they do. The primary can't just be "alive" it must be "alive and well" - i.e. responding to queries and new connections.