Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9828

MONGO replica set is down due to duplicate key issue on one of the databases

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: Replication
    • None
    • Environment:
      Linux mongodb302p.stag.ch3.s.com 2.6.32-358.0.1.el6.x86_64 #1 SMP Wed Feb 27 06:06:45 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
      MONGO 2.4.3
    • Linux

      One of the replica set is down due to following issue and not able to start it and appreciate all your suggestions for it.

      Wed May 29 14:58:57 [rsSync] replSet still syncing, not yet to minValid optime 519f99e7:3
      Wed May 29 14:58:57 [initandlisten] connection accepted from 10.235.46.86:47796 #49 (46 connections now open)
      Wed May 29 14:58:57 [initandlisten] connection accepted from 10.235.79.64:50349 #50 (47 connections now open)
      Wed May 29 14:58:58 [initandlisten] connection accepted from 10.235.71.190:26662 #51 (48 connections now open)
      Wed May 29 14:58:58 [initandlisten] connection accepted from 10.235.71.190:26663 #52 (49 connections now open)
      Wed May 29 14:58:58 [conn52] authenticate db: local

      { authenticate: 1, nonce: "19ab8a9ce924f728", user: "__system", key: "394bc2730622d6df6e5fe8883c3fdf35" }

      Wed May 29 14:58:59 [conn52] command admin.$cmd command: { replSetGetStatus: 1, $auth: { local:

      { __system: 2 }

      } } ntoreturn:1 keyUpdates:0 reslen:717 1332ms
      Wed May 29 14:58:59 [rsSyncNotifier] replset setting oplog notifier to mongodb301p.stag.ch3.s.com:10001
      Wed May 29 14:58:59 [repl writer worker 1] replication update of non-mod failed: { ts: Timestamp 1369414119000|3, h: 407750364380487357, op: "u", ns: "sywl_site_qa.customerLocation", o2:

      { _id: ObjectId('51841e09e4b01f3f7f47cc3f') }

      , o: { _id: ObjectId('51841e09e4b01f3f7f47cc3f'), _class: "com.shc.ecom.sywlocal.services.customer.entity.UserLocationBean", userId: "cardtest17@gmail.com", locationName: "-1", longitude: "GAkAGgBXAg5X", latitude: "AQMWBAdXCQBU", isDefaultLocation: false, kmartUnitNumber: "3914", kmartLocationDistance: "7.782068777401162", kMartAddress:

      { address1: "156 South Gary Ave", address2: "", city: "Bloomingdale", state: "IL", zipCode: "60108" }

      } }
      Wed May 29 14:58:59 [repl writer worker 1] Fatal Assertion 16359
      0xb07561 0xacc8b3 0x9aba19 0xadab5d 0xb4d3d9 0x35dca07851 0x35dc2e890d
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
      /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xacc8b3]
      /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x79) [0x9aba19]
      /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
      /usr/bin/mongod() [0xb4d3d9]
      /lib64/libpthread.so.0() [0x35dca07851]
      /lib64/libc.so.6(clone+0x6d) [0x35dc2e890d]
      Wed May 29 14:58:59 [repl writer worker 1]

      ***aborting after fassert() failure

      Wed May 29 14:58:59 Got signal: 6 (Aborted).

      Wed May 29 14:58:59 Backtrace:
      0xb07561 0x5598c9 0x35dc232920 0x35dc2328a5 0x35dc234085 0xacc8ee 0x9aba19 0xadab5d 0xb4d3d9 0x35dca07851 0x35dc2e890d
      /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
      /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5598c9]
      /lib64/libc.so.6() [0x35dc232920]
      /lib64/libc.so.6(gsignal+0x35) [0x35dc2328a5]
      /lib64/libc.so.6(abort+0x175) [0x35dc234085]
      /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xacc8ee]
      /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x79) [0x9aba19]
      /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
      /usr/bin/mongod() [0xb4d3d9]
      /lib64/libpthread.so.0() [0x35dca07851]
      /lib64/libc.so.6(clone+0x6d) [0x35dc2e890d]

      Please let me know if you need more information on it.

      Thanks,
      Venkat

            Assignee:
            Unassigned Unassigned
            Reporter:
            vreddim Venkat Reddimachu
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: