Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-3345

Improve rwlock scaling

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT2.9.3, 3.2.14, 3.4.5, 3.5.9
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage 2017-06-19

      WiredTiger can't use POSIX rwlocks or other system equivalents because our resources are connected to a session handle that can be passed between threads. In other words, we may lock a resource (e.g., when opening a cursor) as part of an operation in one thread, then that session can be used in a different thread when the resource is unlocked. Common read/write lock interfaces reserve the right to use thread-local storage internally (meaning the unlock has to happen in the same thread as the lock), regardless of whether all implementations need it.

      Our existing rwlock implementation is subject to collapse when there is contention from many threads for a lock. There are two reasons: (1) when there is contention, threads yield and spin rather than waiting for a notification, and (2) the strict fairness in the current implementation means that there is only one thread that can enter the lock at a time, so we rely on the scheduler choosing the single "next" thread, even though all it can see are a whole lot of threads spinning.

      In this ticket, we added a workload to compare with POSIX rwlocks (even though we can't use the directly), and attempt to tune spinning and backoff behavior to improve scalability.

      The workload is a very simple stress test of read / write locks that runs in three modes:

      • NC is no contention, where all requests are read / shared locks and should never block;
      • LC is low contention, where for each thread, 1 in 10K requests are for write / exclusive locks; and
      • HC is high contention, where 1 in 100 requests are for write locks (so with over 100 threads, there is usually at least one write lock request active).

      We compare "develop" (the existing WiredTiger implementation) with pthread rwlocks and a tuned version.

        1. overflow-130k-read.png
          overflow-130k-read.png
          28 kB
        2. rwlock-tests-overview.png
          rwlock-tests-overview.png
          140 kB
        3. rwlock-tests-zoom.png
          rwlock-tests-zoom.png
          185 kB
        4. rwlock-tuning-initial.png
          rwlock-tuning-initial.png
          126 kB
        5. rwlock-tuning-results-final.png
          rwlock-tuning-results-final.png
          113 kB

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: