Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-13248

Cache stuck with dirty eviction [wiredtiger-mongo-v6.0 @ d49ad6fe]

    • Type: Icon: Build Failure Build Failure
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Cache and Eviction
    • StorEng - 2024-10-15

      long-test on rhel80

      Host: i-03d261719a47dfddf
      Project: wiredtiger-mongo-v6.0
      Commit: d49ad6fe

      Task Logs:

      long-test task_log

      Logs:

      [1720446912:862601][3362:0x7fdf61659700], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 493: Cache stuck for too long, giving up: Connection timed out
      
      

      logs

      long-test task_log

      Logs:

      0x22c9510:transaction state dump
      0x22c9510:current ID: 40411517
      0x22c9510:last running ID: 40411517
      0x22c9510:metadata_pinned ID: 40411517
      0x22c9510:oldest ID: 40411517
      0x22c9510:durable timestamp: (0, 39100740)
      0x22c9510:oldest timestamp: (0, 38876413)
      0x22c9510:pinned timestamp: (0, 38876413)
      0x22c9510:stable timestamp: (0, 38937847)
      0x22c9510:has_durable_timestamp: yes
      0x22c9510:has_oldest_timestamp: yes
      0x22c9510:has_pinned_timestamp: yes
      0x22c9510:has_stable_timestamp: yes
      0x22c9510:oldest_is_pinned: yes
      0x22c9510:stable_is_pinned: no
      0x22c9510:checkpoint running: no
      0x22c9510:checkpoint generation: 7
      0x22c9510:checkpoint pinned ID: 0
      0x22c9510:checkpoint txn ID: 0
      0x22c9510:session count: 22
      0x22c9510:Transaction state of active sessions:
      0x22c9510:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x22c9510:cache dump
      0x22c9510:cache full: no
      0x22c9510:cache clean check: no (29.357%)
      0x22c9510:cache dirty check: yes (27.090%)
      0x22c9510:cache updates check: yes (24.641%)
      0x22c9510:file:T00003.wt(<live>):
      0x22c9510:internal: 1 pages, 2.89 KB, 0/1 clean/dirty pages, 0.00/2.89 clean / dirty KB, 2.89 KB max page, 2.89 KB max dirty page
      0x22c9510:leaf: 2 pages, 385.27 KB, 0/2 clean/dirty pages, 0.00 /385.27 /3.13 clean/dirty/updates KB, 194.49 KB max page, 194.49 KB max dirty page
      0x22c9510:file:F00002(<live>):
      0x22c9510:internal: 1 pages, 60.61 KB, 0/1 clean/dirty pages, 0.00/60.61 clean / dirty KB, 60.61 KB max page, 60.61 KB max dirty page
      0x22c9510:leaf: 48 pages, 3790.58 KB, 47/1 clean/dirty pages, 1708.88 /2081.70 /2081.61 clean/dirty/updates KB, 2081.70 KB max page, 2081.70 KB max dirty page
      0x22c9510:file:T00001.wt(<live>):
      0x22c9510:internal: 1 pages, 25.17 KB, 0/1 clean/dirty pages, 0.00/25.17 clean / dirty KB, 25.17 KB max page, 25.17 KB max dirty page
      0x22cb010:[1720446912:862728][3362:0x7fdf5f655700], t, file:T00001.wt, WT_SESSION.rollback_transaction: [WT_VERB_EVICT][DEBUG]: page 0x7fdf3c165170 (column-store fixed-length leaf)
      0x22c9510:leaf: 23 pages, 48570.57 KB, 22/1 clean/dirty pages, 2235.57 /46334.99 /42306.15 clean/dirty/updates KB, 46334.99 KB max page, 46334.99 KB max dirty page
      0x22c9510:file:WiredTigerHS.wt(<live>):
      0x22c9510:internal: 1 pages, 34.79 KB, 0/1 clean/dirty pages, 0.00/34.79 clean / dirty KB, 34.79 KB max page, 34.79 KB max dirty page
      0x22c9510:leaf: 2 pages, 14.36 KB, 2/0 clean/dirty pages, 14.36 /0.00 /0.00 clean/dirty/updates KB, 9.35 KB max page, 0.00 KB max dirty page
      0x22c9510:file:WiredTiger.wt(<live>):
      0x22c9510:internal: 1 pages, 0.60 KB, 1/0 clean/dirty pages, 0.60/0.00 clean / dirty KB, 0.60 KB max page, 0.00 KB max dirty page
      0x22c9510:leaf: 0 pages
      0x22c9510:cache dump: total found: 55.78 MB vs tracked inuse 51.65 MB
      0x22c9510:total dirty bytes: 47.78 MB vs tracked dirty 47.78 MB
      

      logs

      long-test task_log

      Logs:

      [1720446912:862757][3362:0x7fdf61659700], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 344: cache eviction thread error: Connection timed out
      [1720446912:862760][3362:0x7fdf61659700], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 344: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1720446912:862763][3362:0x7fdf61659700], t, eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library
      

      logs

      long-test task_log

      Logs:

      #0  0x00007fdf6529d93f in raise () from /lib64/libc.so.6
      #0  0x00007fdf6529d93f in raise () from /lib64/libc.so.6
      #1  0x00007fdf65287c95 in abort () from /lib64/libc.so.6
      #2  0x00007fdf65e10a6d in __wt_abort (session=session@entry=0x22c9510) at /data/mci/19e6094bb8a4d3e1afe5cbf3f8aba1aa/wiredtiger/src/os_common/os_abort.c:30
      #3  0x00007fdf65e82d43 in __wt_panic_func (session=session@entry=0x22c9510, error=error@entry=110, func=func@entry=0x7fdf65ee0300 <__PRETTY_FUNCTION__.47305> "__wt_evict_thread_run", line=line@entry=344, category=category@entry=WT_VERB_DEFAULT, fmt=fmt@entry=0x7fdf65ee00f6 "cache eviction thread error") at /data/mci/19e6094bb8a4d3e1afe5cbf3f8aba1aa/wiredtiger/src/support/err.c:550
      #4  0x00007fdf65dd18e8 in __wt_evict_thread_run (session=0x22c9510, thread=0x2346d30) at /data/mci/19e6094bb8a4d3e1afe5cbf3f8aba1aa/wiredtiger/src/evict/evict_lru.c:344
      #5  0x00007fdf65e96a2f in __thread_run (arg=0x2346d30) at /data/mci/19e6094bb8a4d3e1afe5cbf3f8aba1aa/wiredtiger/src/support/thread_group.c:31
      #6  0x00007fdf65a3f2de in start_thread () from /lib64/libpthread.so.0
      #7  0x00007fdf65362a63 in clone () from /lib64/libc.so.6
      

      logs

            Assignee:
            backlog-server-storage-engines [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: