Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-6975

$push idempotency issues with version 2.2.0

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: 2.2.0
    • Component/s: Replication
    • None
    • Environment:
      ubuntu/3.0.0-17
    • Linux

      We first got this crash during initial sync of a secondary node, ended up downgrading to 2.0 because the sync could not resume, and upgrading to 2.2 once it completed. Now after three days of uptime we have the crash again:

      Sat Sep  8 17:53:33 [repl writer worker 7] ERROR: writer worker caught exception: E11000 duplicate key error index: [collection].$_id_  dup key: { : ObjectId('50287481463c0b83d82f8dea') } on: { ts: Timestamp 1347152002000|57, h: -1992302823673752270, op: "u", ns: "[collection]", o2: { _id: ObjectId('50287481463c0b83d82f8dea'), tu: { $size: 6 } }, o: { $set: { a: ObjectId('5026b145003c0b2238a9ddf6') }, $push: { tu: ObjectId('0000000000000000002721eb') }, $set: { tuc: 7 } } }
Sat Sep  8 17:53:33 [repl writer worker 7]   Fatal Assertion 16360
0xade6e1 0x802e03 0x64f77d 0x77d3dd 0x7c3659 0x7f69a5e88efc 0x7f69a521a59d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0x802e03]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12d) [0x64f77d]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0x77d3dd]
 /usr/bin/mongod() [0x7c3659]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f69a5e88efc]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f69a521a59d]
Sat Sep  8 17:53:33 [repl writer worker 7]

***aborting after fassert() failure


Sat Sep  8 17:53:33 Got signal: 6 (Aborted).

Sat Sep  8 17:53:33 Backtrace:
0xade6e1 0x5582d9 0x7f69a516d420 0x7f69a516d3a5 0x7f69a5170b0b 0x802e3e 0x64f77d 0x77d3dd 0x7c3659 0x7f69a5e88efc 0x7f69a521a59d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9]
 /lib/x86_64-linux-gnu/libc.so.6(+0x36420) [0x7f69a516d420]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f69a516d3a5]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f69a5170b0b]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0x802e3e]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12d) [0x64f77d]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0x77d3dd]
 /usr/bin/mongod() [0x7c3659]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f69a5e88efc]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f69a521a59d]

      And it crashes on startup with the same exception:

      at Sep  8 17:57:08 [rsStart] replSet STARTUP2
Sat Sep  8 17:57:08 [rsHealthPoll] replSet member 192.168.1.40:27017 is up
Sat Sep  8 17:57:08 [rsHealthPoll] replSet member 192.168.1.40:27017 is now in state PRIMARY
Sat Sep  8 17:57:08 [rsHealthPoll] replSet member 192.168.1.20:27017 is up
Sat Sep  8 17:57:08 [rsHealthPoll] replSet member 192.168.1.20:27017 is now in state ARBITER
Sat Sep  8 17:57:09 [rsSync] replSet SECONDARY
Sat Sep  8 17:57:13 [rsBackgroundSync] replSet syncing to: 192.168.1.40:27017
Sat Sep  8 17:57:13 [repl writer worker 8] ERROR: writer worker caught exception: E11000 duplicate key error index: [collection].$_id_  dup key: { : ObjectId('50287481463c0b83d82f8dea') } on: { ts: Timestamp 1347152002000|57, h: -1992302823673752270, op: "u", ns: "[collection]", o2: { _id: ObjectId('50287481463c0b83d82f8dea'), tu: { $size: 6 } }, o: { $set: { a: ObjectId('5026b145003c0b2238a9ddf6') }, $push: { tu: ObjectId('0000000000000000002721eb') }, $set: { tuc: 7 } } }
Sat Sep  8 17:57:13 [repl writer worker 8]   Fatal Assertion 16360
0xade6e1 0x802e03 0x64f77d 0x77d3dd 0x7c3659 0x7f993759cefc 0x7f993692e59d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0x802e03]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12d) [0x64f77d]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0x77d3dd]
 /usr/bin/mongod() [0x7c3659]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f993759cefc]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f993692e59d]
Sat Sep  8 17:57:13 [repl writer worker 8]

***aborting after fassert() failure


Sat Sep  8 17:57:13 Got signal: 6 (Aborted).

Sat Sep  8 17:57:13 Backtrace:
0xade6e1 0x5582d9 0x7f9936881420 0x7f99368813a5 0x7f9936884b0b 0x802e3e 0x64f77d 0x77d3dd 0x7c3659 0x7f993759cefc 0x7f993692e59d
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1]
 /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9]
 /lib/x86_64-linux-gnu/libc.so.6(+0x36420) [0x7f9936881420]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f99368813a5]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f9936884b0b]
 /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0x802e3e]
 /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12d) [0x64f77d]
 /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0x77d3dd]
 /usr/bin/mongod() [0x7c3659]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f993759cefc]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f993692e59d]

            Assignee:
            milkie@mongodb.com Eric Milkie
            Reporter:
            braden Braden Evans
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: