Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32465

Abnormally high (100%) CPU on secondary instance

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.2.12
    • Component/s: None
    • None
    • ALL

      We experienced the following on our production replica set: 3 secondaries 1 primary, MongoDB 3.2.12 + WiredTiger.

      1. All secondaries saw a spike in disk usage (about 4-5x base load, we see something like this about twice per day randomly)
      2. One of the secondary's mongod CPU usage shot up to 100%. At this time the replica set continued to route queries to the affected instance, but the instance did not process them.
      3. We killed all queries on the affected instance via console, the CPU remained at 100% it continued to NOT process queries.
      4. Finally we removed the affected process from the replica set and again we could process all queries.

      Here is a CPU chart of the 4 instances. As you can see CPU of one of the instances hit 100%. (As it was not responding to queries, our app failed and stopped sending new queries, so other instances went to low CPU)

      Here is a chart showing the disk usage spike on the affected instance which happened before the high CPU.

      Unfortunately we did not get a core dump of the affected process.

        1. 2017122709_mongod.log.gz
          2.93 MB
        2. diagnostic_data_04db.tar.gz
          94.34 MB
        3. diagnostic_data_05f4.tar.gz
          101.23 MB
        4. diagnostic_data_0ddb.tar.gz
          105.26 MB
        5. diagnostic_data_0f6a.tar.gz
          102.81 MB
        6. image-2017-12-27-19-58-29-795.png
          image-2017-12-27-19-58-29-795.png
          42 kB
        7. image-2017-12-27-19-58-33-746.png
          image-2017-12-27-19-58-33-746.png
          145 kB
        8. mongoparsed.log
          26.43 MB
        9. Screen Shot 2017-12-27 at 16.25.29.png
          Screen Shot 2017-12-27 at 16.25.29.png
          276 kB
        10. Screen Shot 2017-12-27 at 16.25.56.png
          Screen Shot 2017-12-27 at 16.25.56.png
          336 kB
        11. Screen Shot 2017-12-27 at 16.26.52.png
          Screen Shot 2017-12-27 at 16.26.52.png
          471 kB
        12. Screen Shot 2017-12-27 at 16.27.22.png
          Screen Shot 2017-12-27 at 16.27.22.png
          617 kB

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            johnnyshields Johnny Shields
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: