Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17751

Make cleanupOrphaned cmd interruptible

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Sharding
    • ALL
    • Sharding 2017-12-04

      Environment:

      OS: CentOS 6.5/6.6
      MongoDB: 2.6.4, compiled with ssl support

      We are running sharded cluster with 6 replica sets and as per http://docs.mongodb.org/v2.6/reference/command/cleanupOrphaned/ we are executing following code to clean orphaned documents from sharded collections:

      var nextKey = {};
      var result;
      
      while  ( nextKey != null ) {
        result = db.runCommand( { cleanupOrphaned: namespace, startingFromKey: nextKey } );
      
        if (result.ok != 1)
           print("Unable to complete at this time: failure or timeout.")
      
        printjson(result);
      
        nextKey = result.stoppedAtKey;
      }
      

      For some collections command works as expected. However, on some collections, command hangs waiting for open cursors. Log lines appearing every minute are given below.

      2015-03-26T12:49:00.180+0100 [conn1736834] rangeDeleter waiting for open cursors in: DATABASE.COLLECTION, min: { _id: MinKey }, max: { _id: 3074457345618258600 }, elapsedSecs: 293637, cursors: [ 1788183098665 1788838028619 ]
      2015-03-26T12:50:00.287+0100 [conn1736834] rangeDeleter waiting for open cursors in: DATABASE.COLLECTION, min: { _id: MinKey }, max: { _id: 3074457345618258600 }, elapsedSecs: 293697, cursors: [ 1788183098665 1788838028619 ]
      

      As can be seen from the logging lines, same cursors are always listed, so is the max key. We are not aware of any other way to stop the command but to switch primary on every replica set. Since there is a lot of collections that are sharded and need to be cleaned of possible orphaned documents, switching primaries whenever clanupOrphaned hangs waiting for open cursors is not a viable solution.

      Is there another way of stopping execution of cleanupOrphaned command? Or is there a way of closing open cursors from server side?

            Assignee:
            kaloian.manassiev@mongodb.com Kaloian Manassiev
            Reporter:
            ljuba.ned@gmail.com Ljuba Nedeljkovic
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: