Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18908

Secondaries unable to keep up with primary under WiredTiger

    • Fully Compatible
    • ALL
    • Quint Iteration 7, QuInt A (10/12/15), QuInt C (11/23/15), Repl 2016-08-29, Repl 2016-09-19, Repl 2016-10-10, Repl 2016-10-31

      • hardware: 24 CPUs, 64 GB memory, SSD (all mongods and clients on same machine)
      • start 3-member replica set with following options:
        mongod --oplogSize 50 --storageEngine wiredTiger --nojournal --replSet ...
        

        Note: repros with journal also, ran without journal to rule out that as a cause.

      • Simple small document insert workload: 5-16 threads (number doesn't matter to the repro) each inserting small documents {_id:..., x:0} in batches of 10k

      Replica lag grows unbounded as secondaries process ops at maybe 50-80% the rate of the primary.

      Some stats of note:

      primary

      secondary

      • op rate on secondary is maybe half that on primary
      • ops in flight (i.e. active queues) is much less even on the secondary, although that isn't reflected in the reported op rates
      • secondary is executing far more search near calls, about one per document, vs what appears to be about one every 100 documents on primary

      Will get stack traces.

        1. insert-3.0.4.patch
          3 kB
        2. insert-3.1.4.patch
          2 kB
        3. lag-02.html
          991 kB
        4. lag-03.html
          496 kB
        5. lag-04.html
          769 kB
        6. lagC-3.0.4.patch
          18 kB
        7. lagC-3.1.4.patch
          17 kB
        8. lagD-3.0.4.patch
          20 kB
        9. log.png
          log.png
          60 kB
        10. pri.png
          pri.png
          100 kB
        11. search_near.png
          search_near.png
          237 kB
        12. sec.png
          sec.png
          104 kB

            Assignee:
            mathias@mongodb.com Mathias Stearn
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            11 Vote for this issue
            Watchers:
            53 Start watching this issue

              Created:
              Updated:
              Resolved: