Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-10285

Seg Fault when parsing Point using 2dsphere Indexes

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 2.4.7, 2.5.3
    • Affects Version/s: 2.4.5
    • Component/s: Geo
    • Environment:
      AWS EC2 machine running latest stable build of mongo 2.4.5
    • Linux
    • Hide

      Hard to know, we ran the latest in production for 3 whole days before we saw any problem. We are storing Points (lat, lon) and using 2dspherical index and query ing Mongo using geoNear command.

      We are using the latest mongo java driver 2.9.3

      Show
      Hard to know, we ran the latest in production for 3 whole days before we saw any problem. We are storing Points (lat, lon) and using 2dspherical index and query ing Mongo using geoNear command. We are using the latest mongo java driver 2.9.3

      Issue Status as of October 22nd, 2013

      ISSUE SUMMARY
      In very rare and intermittent cases, queries that use a 2dsphere index with legacy coordinate pairs crashed the server. The crash occurred when the server converted a point from lat / long to radians but floating-point inaccuracy resulted in values that were slightly out of bounds.

      USER IMPACT
      Rare segfaults.

      SOLUTION
      After converting a legacy coordinate pair from lat / long to radians and attempting to normalize it (i.e., wrapping its lat and long to valid radian values), the server now checks if the point is valid. If not, it uasserts with "coords invalid after normalization" instead of crashing.

      WORKAROUNDS
      None.

      PATCHES
      Production release v2.4.7 contains the fix for this issue, and production release v2.6.0 will contain the fix as well.

      Our production system crashed hard yesterday afternoon with what looks like a Segmentation Fault parsing a Point. It took out all 6 Nodes including the master.

      We unfortunately don't have the query (lat,lon) that cause the issue. Here is the stack trace on master.

      There seem to be a lot of GEO code changes in this release https://jira.mongodb.org/browse/SERVER-8349 and https://github.com/mongodb/mongo/commit/ba239918c950c254056bf589a943a5e88fd4144c

      We are rolling back to previous version before all these changes were made.

      Jul 22 00:08:06 ip-10-38-67-89 mongod.27017[8632]: Backtrace:#0120xdd9e31 0x6d0d09 0x7fe2f0f01920 0x7fe2f0f018a5 0x7fe2f0f03085 0xea2752 0x992d5f 0x994d9d 0x9a7769 0x98a933 0x8d4f0a 0x8d7042 0x8d80b2 0xa7e220 0xa82aec 0x9f6919 0x9f7e43 0x6e8b68 0xdc659e 0x7fe2f1c10851 #012 /opt/mongodb/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xdd9e31]#012 /opt/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0d09]#012 /lib64/libc.so.6(+0x32920) [0x7fe2f0f01920]#012 /lib64/libc.so.6(gsignal+0x35) [0x7fe2f0f018a5]#012 /lib64/libc.so.6(abort+0x175) [0x7fe2f0f03085]#012 /opt/mongodb/bin/mongod(_ZNK8S2LatLng7ToPointEv+0xf2) [0xea2752]#012 /opt/mongodb/bin/mongod(_ZN5mongo9GeoParser10parsePointERKNS_7BSONObjEP7Vector3IdE+0x17f) [0x992d5f]#012 /opt/mongodb/bin/mongod(_ZN5mongo9NearQuery16parseFromGeoNearERKNS_7BSONObjEd+0x45d) [0x994d9d]#012 /opt/mongodb/bin/mongod(_ZN5mongo18run2DSphereGeoNearERKNS_12IndexDetailsERNS_7BSONObjERKNS_16GeoNearArgumentsERSsRNS_14BSONObjBuilderE+0x179) [0x9a7769]#012 /opt/mongodb/bin/mongod(_ZN5mongo16Geo2dFindNearCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x773) [0x98a933]#012 /opt/mongodb/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0x8d4f0a]#012 /opt/mongodb/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xc02) [0x8d7042]#012 /opt/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x5f2) [0x8d80b2]#012 /opt/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x40) [0xa7e220]#012 /opt/mongodb/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xd7c) [0xa82aec]#012 /opt/mongodb/bin/mongod() [0x9f6919]#012 /opt/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x383) [0x9f7e43]#012 /opt/mongodb/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21Abs
      

            Assignee:
            hari.khalsa@10gen.com hari.khalsa@10gen.com
            Reporter:
            grealish Adrian Grealish
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: