Uploaded image for project: 'Motor'
  1. Motor
  2. MOTOR-625

Change stream waiting cursors can fill up the executor with jobs and block all other queries

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 2.3.1
    • Affects Version/s: 2.3
    • Component/s: None
    • None

      When using a tailable cursor or a change stream, the "next" method might be expected to hang a significant amount of time.

      Motor is simply executing this method on the usual thread pool, which is of finite size.

      This makes it quite quite easy to deadlock an application with something like this:

       

      import asyncio
      import motor.motor_asyncio
      import multiprocessing
      import pymongo.cursor
      
      async def main():
          jobs = multiprocessing.cpu_count() * 5
          client = motor.motor_asyncio.AsyncIOMotorClient()
      
          async def job():
              async for it in client.local.oplog.rs.find(
                  {'stall': True},
                  cursor_type=pymongo.cursor.CursorType.TAILABLE_AWAIT,
              ):
                  print('got', it) # never reached (as expected)
      
          [asyncio.create_task(job()) for _ in range(jobs)]
          await asyncio.sleep(1)
      
          # now the complete motor thread pool is filled with blocking jobs that are waiting for a new document 
         # and this will stall indefinitely
          print('inserting')
          await client.db.col.insert_one({})
          print('ok') # never reached
      
      asyncio.run(main())
      

      This can lead to quite serious load issues in production when you have a large API with a few long running endpoints that rely on change streams.

      I'm not exactly sure how this could be fixed, but I would suggest either documenting this behaviour, or switching this "next" actions to another (infinite?) threadpool.

       

      Thanks!

            Assignee:
            shane.harvey@mongodb.com Shane Harvey
            Reporter:
            arthur.darcet@glose.com Arthur Darcet
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: