Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-18908

Secondaries unable to keep up with primary under WiredTiger

XMLWordPrintableJSON

    • Fully Compatible
    • ALL
    • Quint Iteration 7, QuInt A (10/12/15), QuInt C (11/23/15), Repl 2016-08-29, Repl 2016-09-19, Repl 2016-10-10, Repl 2016-10-31

      • hardware: 24 CPUs, 64 GB memory, SSD (all mongods and clients on same machine)
      • start 3-member replica set with following options: 
        mongod --oplogSize 50 --storageEngine wiredTiger --nojournal --replSet ...

        Note: repros with journal also, ran without journal to rule out that as a cause.

      • Simple small document insert workload: 5-16 threads (number doesn't matter to the repro) each inserting small documents {_id:..., x:0} in batches of 10k

      Replica lag grows unbounded as secondaries process ops at maybe 50-80% the rate of the primary.

      Some stats of note:

      primary

      secondary

      • op rate on secondary is maybe half that on primary
      • ops in flight (i.e. active queues) is much less even on the secondary, although that isn't reflected in the reported op rates
      • secondary is executing far more search near calls, about one per document, vs what appears to be about one every 100 documents on primary

      Will get stack traces.

        1. pri.png
          pri.png
          100 kB
        2. sec.png
          sec.png
          104 kB
        3. search_near.png
          search_near.png
          237 kB
        4. log.png
          log.png
          60 kB
        5. lag-02.html
          991 kB
        6. lag-03.html
          496 kB
        7. lag-04.html
          769 kB
        8. insert-3.0.4.patch
          3 kB
        9. insert-3.1.4.patch
          2 kB
        10. lagC-3.0.4.patch
          18 kB
        11. lagC-3.1.4.patch
          17 kB
        12. lagD-3.0.4.patch
          20 kB

            Assignee:
            mathias@mongodb.com Mathias Stearn
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            11 Vote for this issue
            Watchers:
            53 Start watching this issue

              Created:
              Updated:
              Resolved: