Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-7913

Invalid BSON in db.system.profile

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.2.1
    • Component/s: Logging
    • None
    • Environment:
      Linux - Ubuntu Lucid
      Linux db1e.songza.com 2.6.32-348-ec2 #54-Ubuntu SMP Fri Sep 7 11:21:33 UTC 2012 x86_64 GNU/Linux
    • ALL

      We are getting what appear to be invalid bson documents in the system profile collection, which is making it rather difficult to analyze the results.

      After enabling the profiler, these invalid documents will appear after a few minutes of run time. Dropping the capped collection and recreating it seems to have no effect as the invalid documents will be reinserted after a few minutes time. So far we have only seen this issue on our primary node. The secondary machines in our replica set do not appear to be affected. I don't recall ever getting this sort of error when we were running mongo 2.0. This is the first time since our upgrade to 2.2.1 that we have turned on the profiler.

      Attempts to iterate through db.system.profile in the mongo shell result in the following assertion:

      Wed Dec 12 16:16:59 Assertion: 10320:BSONElement: bad type -128
      0x6073f1 0x5d1aa9 0x4b0d98 0x5c17a6 0x6b3f35 0x6b6a2c 0x69be0a 0x6aab74 0x695ebe 0x6aa5ac 0x69c025 0x6aa13f 0x668e46 0x668ec2 0x66a2ce 0x5cbcc4 0x4a47a5 0x4a67e6 0x7feb2ccf9c4d 0x49f669 
       mongo(_ZN5mongo15printStackTraceERSo+0x21) [0x6073f1]
       mongo(_ZN5mongo11msgassertedEiPKc+0x99) [0x5d1aa9]
       mongo(_ZNK5mongo11BSONElement4sizeEv+0x1d8) [0x4b0d98]
       mongo(_ZN5mongo16resolveBSONFieldEP9JSContextP8JSObjectljPS3_+0x146) [0x5c17a6]
       mongo(js_LookupPropertyWithFlags+0x3f5) [0x6b3f35]
       mongo(js_GetProperty+0x7c) [0x6b6a2c]
       mongo(js_Interpret+0x10ea) [0x69be0a]
       mongo(js_Invoke+0x9d4) [0x6aab74]
       mongo() [0x695ebe]
       mongo(js_Invoke+0x40c) [0x6aa5ac]
       mongo(js_Interpret+0x1305) [0x69c025]
       mongo(js_Execute+0x36f) [0x6aa13f]
       mongo(JS_EvaluateUCScriptForPrincipals+0x66) [0x668e46]
       mongo(JS_EvaluateUCScript+0x22) [0x668ec2]
       mongo(JS_EvaluateScript+0x6e) [0x66a2ce]
       mongo(_ZN5mongo7SMScope4execERKNS_10StringDataERKSsbbbi+0x144) [0x5cbcc4]
       mongo(_Z5_mainiPPc+0x2455) [0x4a47a5]
       mongo(main+0x26) [0x4a67e6]
       /lib/libc.so.6(__libc_start_main+0xfd) [0x7feb2ccf9c4d]
       mongo(__gxx_personality_v0+0x2a1) [0x49f669]
      Error: BSONElement: bad type -128
      

      Running db.system.profile.validate(

      {full: true}

      ) results in the following output:

      {
      	"ns" : "songza.system.profile",
      	"capped" : true,
      	"max" : 2147483647,
      	"firstExtent" : "98:50708000 ns:songza.system.profile",
      	"lastExtent" : "98:50708000 ns:songza.system.profile",
      	"extentCount" : 1,
      	"extents" : [
      		{
      			"loc" : "98:50708000",
      			"xnext" : "null",
      			"xprev" : "null",
      			"nsdiag" : "songza.system.profile",
      			"size" : 4001792,
      			"firstRecord" : "98:507080b0",
      			"lastRecord" : "98:50872684"
      		}
      	],
      	"datasize" : 1454728,
      	"nrecords" : 1872,
      	"lastExtentSize" : 4001792,
      	"padding" : 1,
      	"firstExtentDetails" : {
      		"loc" : "98:50708000",
      		"xnext" : "null",
      		"xprev" : "null",
      		"nsdiag" : "songza.system.profile",
      		"size" : 4001792,
      		"firstRecord" : "98:507080b0",
      		"lastRecord" : "98:50872684"
      	},
      	"cappedOutOfOrder" : 0,
      	"objectsFound" : 1872,
      	"invalidObjects" : 3,
      	"bytesWithHeaders" : 1484680,
      	"bytesWithoutHeaders" : 1454728,
      	"deletedCount" : 1,
      	"deletedSize" : 2516936,
      	"nIndexes" : 0,
      	"keysPerIndex" : {
      		
      	},
      	"valid" : false,
      	"errors" : [
      		"invalid bson object detected (see logs for more info)"
      	],
      	"advice" : "ns corrupt, requires repair",
      	"ok" : 1
      }
      

      During the validation the following log lines are emitted:

      Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id
      Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id
      Wed Dec 12 16:56:42 [conn1740365] Invalid bson detected in songza.system.profile and couldn't find _id
      

      I've also attached a copy of the affected collection to this jira.

        1. system.profile.bson.bz2
          268 kB
          Michael Henson

            Assignee:
            tad Tad Marshall
            Reporter:
            michael@songza.com Michael Henson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: