Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32562

Importing local database into standalone server causes update to replset.minvalid

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Minor - P4 Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: Replication
    • None
    • ALL
    • Repl 2018-01-29

      TL;DR
      When importing the local database into a standalone server, the document inside of the replset.minvalid collection is modified with an updated timestamp. Since the server is a standalone server and not a member of a replica set, this should not occur. This unexpected behavior made the process described below a bit more difficult to work through.

      Background

      I wanted to move a large database (1TB) to a new replica set without having any significant application downtime.

      Only one of my applications targeted the specific database in the replica set which was 1TB in size. The total size of all of the databases in the replica set was 6TB.

      The Initial Plan

      The plan was to add two secondary nodes to the existing replica set, allow the initial sync to populate the new nodes and then remove the new nodes from the replica set to create a new one with all of the data in tact. The removal of the new nodes and spinning up a new replica set (with an arbiter as the third node) could be done in minutes vs the many hours it would take for a backup/restore. Once the new replica set was up the application that targeted the 1TB database would be switched to the new replica set. At that point, he would drop the 1TB database in the original replica set and drop the 5TB databases in the new replica set.

      My requirement was that he would have at least 2 available data bearing nodes of each replica set so he would not lose high availability.

      The Steps

      1. add 2 new nodes to the existing replica set and wait for an initial sync to complete
      2. remove the 2 new members from the replica set
      3. start each node as a standalone and remove the local database from each
      4. shutdown one of the nodes and restart with the new --replSet name. This node will become the primary.
      5. run rs.initiate() on this node. We now have a single node replica set with all of the data.
      6. export the local database from the new replica set with the mongodump command e.g.
        mongodump --port primary-node-port-number --db local
      7. import the local database dump into the second node using the mongorestore command e.g. mongorestore --port second-node-port-number --db local dump/local
      8. shut down the second node and restart with the new --replSet name
      9. run the rs.add() command from your primary (created in step #5) to add the second node to the replica set
      10. an initial sync process will be initiated but since the oplog shows both to be at the same point, the second node should transition to SECONDARY almost immediately.

      The Issues

      1. mongodump of the local database does not include the collection system.replset. This is a known issue recorded in TOOLS-1387. Although I think that should be reconsidered it is not the topic of this SERVER ticket.
      1. The replset.minvalid collection is updated by the server with a new timestamp that will not match any timestamp in the primary's oplog.

      Workarounds

      1. In order to workaround the issue of system.replset being missed, I tried to dump that collection separately using:

        mongodump --port port-number --db local --collection system.replset

      1. In order to reverse the effect of the replset.minvalid collection being changed, I run:

        use local
        db.replset.minvalid.update({}, {$set: {ts:Timestamp(0,0)}})

      Summary

      The steps above allowed me to achieve my goals. The fact that the server modified data in the local database when the local database was imported into a standalone server was unexpected and should be fixed albeit as a low priority.

      Note: a script to reproduce this issue is attached

            Assignee:
            spencer@mongodb.com Spencer Brody (Inactive)
            Reporter:
            arnie.listhaus@mongodb.com Arnie Listhaus
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: