Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-4851

AsyncGetter thread got stuck and all db transactions were got failed

    • Type: Icon: Question Question
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.12.9
    • Component/s: Connection Management
    • None

      Summary

      All the db transactions were suddenly started to fail. Below logs were continuously coming. 

      - Timeout waiting for a pooled item after 162 MILLISECONDS
      - Too many operations are already waiting for a server. Max number of operations (maxWaitQueueSize) of 500 has been exceeded

      From the thread dump, we could see, almost all the AsyncGetter got stuck in TIMED_WAITING state. We took multiple thread dumps in the interval of 5 seconds. In all the thread dumps, observation is same w.r.t AsyncGetter thread.
      We recovered the setup by restarting the application process. Post restart, no issues were seen and again we took multiple thread dumps and it looks fine i.e. none of the AsyncGetter thread was in TIMED_WAITING.

      Java Mongo client driver version: 3.12.9
      Mongo server version: 3.6

      How to Reproduce

      No definitive steps. Issue is random and inconsistent.

      Additional Background

      Irrespective of load, this issue is coming. Once issue got hit then all the further db transactions are failing.  Attached the stack trace(AsyncGetter) which have taken during issue and non-issue time.

            Assignee:
            jeff.yemin@mongodb.com Jeffrey Yemin
            Reporter:
            senthilkumarc88@gmail.com senthil kumar c
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: