Summary

All the db transactions were suddenly started to fail. Below logs were continuously coming. 

- Timeout waiting for a pooled item after 162 MILLISECONDS
- Too many operations are already waiting for a server. Max number of operations (maxWaitQueueSize) of 500 has been exceeded

From the thread dump, we could see, almost all the AsyncGetter got stuck in TIMED_WAITING state. We took multiple thread dumps in the interval of 5 seconds. In all the thread dumps, observation is same w.r.t AsyncGetter thread.
We recovered the setup by restarting the application process. Post restart, no issues were seen and again we took multiple thread dumps and it looks fine i.e. none of the AsyncGetter thread was in TIMED_WAITING.

Java Mongo client driver version: 3.12.9
Mongo server version: 3.6

How to Reproduce

No definitive steps. Issue is random and inconsistent.

Additional Background

Irrespective of load, this issue is coming. Once issue got hit then all the further db transactions are failing.  Attached the stack trace(AsyncGetter) which have taken during issue and non-issue time.