Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9872

Server deadlock caused by infinite $near & $geoWithin queries on 2dsphere index

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 2.4.2, 2.4.3
    • Component/s: Geo, Querying
    • None
    • Environment:
      CentOS 5.6 x86_64, 2.6.18-238.12.1.el5
    • Linux
    • Hide

      Unfortunately we have been unable to find specific steps to repeat yet. We have tried different servers and different versions of MongoDB.

      Show
      Unfortunately we have been unable to find specific steps to repeat yet. We have tried different servers and different versions of MongoDB.

      We are staging our application for production deployment but are experiencing periodic server deadlocks during testing that we believe is related to our use of the $near or $geoWithin geo queries. By deadlock I mean that all databases on the server cannot be queried or updated. The CPU also spikes to at/near 100%. We cannot list collections for any of the DBs in the instance. The db.currenOp() function returns all statements in process where all are

      {"waitingForLock" : true}

      except for one geo query. We've tried both the $near and $geoWithin operators with the same result.

      We have experienced this problem on both 2.4.2 and 2.4.3 and on separate physical servers.

      Here is the query from db.currentOp().

      {
      "inprog" : [
      {
      "opid" : 130836,
      "active" : true,
      "secs_running" : 3925,
      "op" : "query",
      "ns" : "vintank.geoFenceActivity",
      "query" : {
      "$query" : {
      "loc.geoPt" : {
      "$geoWithin" :

      { "$centerSphere" : [ [ -122.28658676147461, 38.301287089941106 ], 0.0007848061528802386 ] }

      },
      "pubDt" :

      { "$gte" : ISODate("2013-05-30T17:38:25.423Z") }

      },
      "$orderby" :

      { "pubDt" : -1 }

      },
      "client" : "192.168.1.5:44865",
      "desc" : "conn3956",
      "threadId" : "0x47e27940",
      "connectionId" : 3956,
      "locks" :

      { "^" : "r", "^vintank" : "R" }

      ,
      "waitingForLock" : false,
      "numYields" : 0,
      "lockStats" : {
      "timeLockedMicros" : {

      },
      "timeAcquiringMicros" :

      { "r" : NumberLong(2), "w" : NumberLong(0) }

      }
      }
      ]
      }

      Our queries are generated using the Java driver. Below is the relevant Java source for submitting the query (where the query and sort expressions can be determined from the currentOp output above).

      ds.getCollection(GeoFenceActivity.class).find(query).limit(Math.max(limit, 1000)).sort(sort).limit(limit);

      Since our app is still in testing, the actual lat/lng used in the query is always the same. There are currently only 2,282 documents in this collection.

      Here are the indexes on the vintank.geoFenceActivity collection:

      { "v" : 1, "key" :

      { "_id" : 1 }

      , "ns" : "vintank.geoFenceActivity", "name" : "id" }
      { "v" : 1, "key" :

      { "site" : 1, "srcId" : 1 }

      , "unique" : true, "ns" : "vintank.geoFenceActivity", "name" : "site_1_srcId_1" }
      { "v" : 1, "key" :

      { "pubDt" : 1 }

      , "ns" : "vintank.geoFenceActivity", "name" : "pubDt_1", "expireAfterSeconds" : 1209600 }
      { "v" : 1, "key" :

      { "siteType" : 1, "fncPt" : 1, "loc.geoPt" : "2dsphere", "pubDt" : -1 }

      , "ns" : "vintank.geoFenceActivity", "name" : "siteType_1_fncPt_1_loc.geoPt_2dsphere_pubDt_-1" }

      I checked our mongodb log and didn't see anything out of the ordinary.

      This my first bug submission so please let me know if I can get you more information. I've tried to repeat the problem on my local dev system by randomly throwing thousands of queries at the collection but have not been able to reproduce.

            Assignee:
            hari.khalsa@10gen.com hari.khalsa@10gen.com
            Reporter:
            james@vintank.com James Jory
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: