Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5840

Fix cache stuck/timeout issue in test/csuite/test_wt4333_handle_locks

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None

      We originally diagnosed a potential deadlock scenario in eviction and verify code paths in WT-5785. After the fix, the test is now failing with cache stuck/timeout errors on scenarios with large number of worker threads:

      test_wt4333_handle_locks: 60 seconds, cache_cursors=false, 8 workers, 1 files
      	worker 91403, worker_busy 0, verify 0, verify_busy 14252740
      test_wt4333_handle_locks: 60 seconds, cache_cursors=false, 600 workers, 750 files
      [1584062714:842422][33859:0x7f430ffff700], eviction-server: __evict_server, 440: Cache stuck for too long, giving up: Connection timed out
      =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      transaction state dump
      current ID: 14215295
      last running ID: 14215295
      metadata_pinned ID: 14215295
      oldest ID: 14215295
      durable timestamp: (0, 0)
      oldest timestamp: (0, 0)
      pinned timestamp: (0, 0)
      stable timestamp: (0, 0)
      has_durable_timestamp: no
      has_oldest_timestamp: no
      has_pinned_timestamp: no
      has_stable_timestamp: no
      oldest_is_pinned: no
      stable_is_pinned: no
      checkpoint running: no
      checkpoint generation: 2
      checkpoint pinned ID: 0
      checkpoint txn ID: 0
      session count: 611
      Transaction state of active sessions:
      =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      cache dump
      cache full: no
      cache clean check: no (20.662%)
      cache dirty check: yes (20.000%)
      
      ......
      
      
      cache dump: total found: 1057MB vs tracked inuse 1057MB
      total dirty bytes: 949MB
      [1584062714:911813][33859:0x7f430ffff700], eviction-server: __wt_evict_thread_run, 315: cache eviction thread error: Connection timed out
      [1584062714:911827][33859:0x7f430ffff700], eviction-server: __wt_panic, 490: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1584062714:911833][33859:0x7f430ffff700], eviction-server: __wt_abort, 28: aborting WiredTiger library 

            Assignee:
            haseeb.bokhari@mongodb.com Haseeb Bokhari (Inactive)
            Reporter:
            haseeb.bokhari@mongodb.com Haseeb Bokhari (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: