Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16914

15-second stall associated with "sched: RT throttling activated" under WiredTiger

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.0.1
    • Affects Version/s: 2.8.0-rc5
    • Component/s: WiredTiger
    • None
    • Fully Compatible
    • ALL

      • heavy mixed workload
      • Ubuntu 14.04.1 LTS, 3.13.0-32-generic
      • VMware, 6 cores

      5-second pause in db ops was seen following this message in syslog:

      Jan 16 14:39:15 ubuntu kernel: [20023.738805] [sched_delayed] sched: RT throttling activated
      

      • For a second or so from A to B no samples were reported by the external monitoring processes (a mongo shell process calling serverStatus and a python process monitoring system stats.)
      • An extremely high context switch rate was reported at A.
      • At B the monitoring processes resumed.
      • At around B (to within syslog resolution of 1 second) the above message appeared in syslog.
      • Starting at B for about 5 seconds db op rate dropped to 0
      • However not all activity was blocked: evictions appeared to be continuing, serverStatus was being processed.
      • "slots selected for switching that were unavailable" was high from B to C.

      It appears that some behavior involving extreme CPU utilization at high priority from A to B, possibly involving context switches, caused the kernel to suspend those threads for 5 seconds. The threads involved with eviction apparently weren't suspended.

        1. throttle-pre9-15.png
          throttle-pre9-15.png
          215 kB
        2. throttle-pre9-10-gdbmon.html
          6.33 MB
        3. throttle-pre9-10.png
          throttle-pre9-10.png
          174 kB
        4. throttle-pre9-10.html
          479 kB
        5. throttled-15s.png
          throttled-15s.png
          146 kB
        6. throttled.png
          throttled.png
          295 kB
        7. Screenshot 2015-02-16 12.45.11.png
          Screenshot 2015-02-16 12.45.11.png
          650 kB
        8. repro.html
          2.07 MB
        9. fixed.png
          fixed.png
          172 kB

            Assignee:
            sue.loverso@mongodb.com Susan LoVerso
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: