Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-17487

cloner dropDups removes _id entries belonging to other records

    • Fully Compatible
    • ALL
    • Hide

      -replica set
      -insert 2M doc
      -run mix traffic (insert/deletion, update) and re-sync one member

      • all wiredTiger as storageEngine
      • multiple member replica set
      Show
      -replica set -insert 2M doc -run mix traffic (insert/deletion, update) and re-sync one member all wiredTiger as storageEngine multiple member replica set

      after re-sync is done, the _id index count, secondary index count and data table count could be different.

              "index" : 2000000,
              "secondary_index" : 2047102,
              "table" : 2047102
      

      and log file show (with Dan's patch for more printout)

      2015-03-06T01:05:28.851+0000 I REPL     [rsSync] ******
      2015-03-06T01:05:28.851+0000 I REPL     [rsSync] creating replication oplog of size: 4094MB...
      2015-03-06T01:05:28.853+0000 I STORAGE  [rsSync] Starting WiredTigerRecordStoreThread local.oplog.rs
      2015-03-06T01:05:28.870+0000 I REPL     [rsSync] ******
      2015-03-06T01:05:28.870+0000 I REPL     [rsSync] initial sync pending
      2015-03-06T01:05:28.870+0000 I REPL     [ReplicationExecutor] syncing from: 10.1.1.107:27017
      2015-03-06T01:05:28.874+0000 I REPL     [rsSync] initial sync drop all databases
      2015-03-06T01:05:28.874+0000 I STORAGE  [rsSync] dropAllDatabasesExceptLocal 1
      2015-03-06T01:05:28.874+0000 I REPL     [rsSync] initial sync clone all databases
      2015-03-06T01:05:28.875+0000 I REPL     [rsSync] initial sync cloning db: sbtest
      2015-03-06T01:05:42.605+0000 I INDEX    [rsSync] build index on: sbtest.sbtest1 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "sbtest.sbtest1" }
      2015-03-06T01:05:42.605+0000 I INDEX    [rsSync]         building index using bulk method
      2015-03-06T01:05:47.969+0000 I INDEX    [rsSync] build index done.  scanned 2047128 total records. 5 secs
      2015-03-06T01:05:47.969+0000 I STORAGE  [rsSync] Deleting: RecordId(1955618)
      2015-03-06T01:05:47.969+0000 I STORAGE  [rsSync] Deleting: RecordId(1955661)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1955710)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1955778)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1955802)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1955996)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1956089)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1956102)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1956127)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1956187)
      2015-03-06T01:05:47.970+0000 I STORAGE  [rsSync] Deleting: RecordId(1956305)
      ....
      
      2015-03-06T01:05:49.275+0000 I STORAGE  [rsSync] Deleting: RecordId(2047128)
      2015-03-06T01:05:49.275+0000 I STORAGE  [rsSync] index build dropped: 47128 dups
      2015-03-06T01:05:49.449+0000 I REPL     [rsSync] initial sync data copy, starting syncup
      2015-03-06T01:05:49.449+0000 I REPL     [rsSync] oplog sync 1 of 3
      
      • not sure why there is dup for _id
      • dropped dups are not removed from data table and secondary index tree
      • saw one case when mixed traffic stopped, re-sync still have issue with dups and out of sync, but cannot reproduce it now.

        1. cloner_dups.js
          2 kB
        2. SERVER-17487.patch
          1 kB

            Assignee:
            mathias@mongodb.com Mathias Stearn
            Reporter:
            rui.zhang Rui Zhang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: