Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-9796

Add debugging for logging getting stuck

    • Type: Icon: Build Failure Build Failure
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT11.2.0
    • Affects Version/s: None
    • Component/s: None
    • 8
    • Storage Engines - 2022-11-28
    • v5.0

      recovery-stress-test-1 on ubuntu2004-stress-tests

      Host: i-0eac9e64c4a2ad4dc
      Project: wiredtiger
      Commit: 42ee8909

      Task Logs:

      recovery-stress-test-1 task_log

      Logs:

      Verbose Error message
      [1661231527:736274][38202:0x7f9a80ff9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __evict_server, 473: Cache stuck for too long, giving up: Connection timed out
      
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7307
      
      Cache Stuck Failures
      transaction state dump
      current ID: 56041
      last running ID: 55650
      metadata_pinned ID: 20506
      oldest ID: 54407
      durable timestamp: (0, 54980)
      oldest timestamp: (0, 16915)
      pinned timestamp: (0, 16915)
      stable timestamp: (0, 16915)
      has_durable_timestamp: yes
      has_oldest_timestamp: yes
      has_pinned_timestamp: yes
      has_stable_timestamp: yes
      oldest_is_pinned: yes
      stable_is_pinned: yes
      checkpoint running: yes
      checkpoint generation: 2
      checkpoint pinned ID: 0
      checkpoint txn ID: 0
      session count: 194
      Transaction state of active sessions:
      ID: 55650, pinned ID: 54407, metadata pinned ID: 20506, name: EMPTY
      transaction id: 55650, mod count: 1, snap min: 20506, snap max: 55637, snapshot count: 147, commit_timestamp: (0, 54624), durable_timestamp: (0, 54624), first_commit_timestamp: (0, 54623), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 54623), read_timestamp: (0, 0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x0000301c, isolation: WT_ISO_SNAPSHOT
      ID: 56002, pinned ID: 54407, metadata pinned ID: 0, name: EMPTY
      transaction id: 56002, mod count: 1, snap min: 54407, snap max: 55994, snapshot count: 154, commit_timestamp: (0, 54964), durable_timestamp: (0, 54964), first_commit_timestamp: (0, 54963), prepare_timestamp: (0, 0), pinned_durable_timestamp: (0, 54963), read_timestamp: (0, 0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x0000301c, isolation: WT_ISO_SNAPSHOT
      =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      cache dump
      cache full: no
      cache clean check: yes (95.010%)
      cache dirty check: no (89.760%)
      cache updates check: no (72.071%)
      file:oplog.wt(<live>):
      internal: 1 pages, 30.79 KB, 1/0 clean/dirty pages, 30.79/0.00 clean / dirty KB, 30.79 KB max page, 0.00 KB max dirty page
      leaf: 174 pages, 15317.83 KB, 35/139 clean/dirty pages, 593.03 /14724.79 /11753.77 clean/dirty/updates KB, 206.28 KB max page, 206.28 KB max dirty page
      file:local.wt(<live>):
      internal: 1 pages, 33.43 KB, 1/0 clean/dirty pages, 33.43/0.00 clean / dirty KB, 33.43 KB max page, 0.00 KB max dirty page
      leaf: 189 pages, 15806.64 KB, 46/143 clean/dirty pages, 815.72 /14990.92 /11776.90 clean/dirty/updates KB, 206.24 KB max page, 206.24 KB max dirty page
      file:shadow.wt(<live>):
      internal: 1 pages, 36.03 KB, 1/0 clean/dirty pages, 36.03/0.00 clean / dirty KB, 36.03 KB max page, 0.00 KB max dirty page
      leaf: 204 pages, 15437.29 KB, 57/147 clean/dirty pages, 920.93 /14516.37 /11799.14 clean/dirty/updates KB, 208.92 KB max page, 208.92 KB max dirty page
      file:collection.wt(<live>):
      internal: 1 pages, 36.59 KB, 1/0 clean/dirty pages, 36.59/0.00 clean / dirty KB, 36.59 KB max page, 0.00 KB max dirty page
      leaf: 207 pages, 15439.76 KB, 58/149 clean/dirty pages, 949.01 /14490.75 /11803.69 clean/dirty/updates KB, 203.53 KB max page, 203.53 KB max dirty page
      file:WiredTigerHS.wt(<live>) eviction disabled at open:
      internal: 1 pages, 0.27 KB, 1/0 clean/dirty pages, 0.27/0.00 clean / dirty KB, 0.27 KB max page, 0.00 KB max dirty page
      leaf: 1 pages, 0.09 KB, 1/0 clean/dirty pages, 0.09 /0.00 /0.00 clean/dirty/updates KB, 0.09 KB max page, 0.00 KB max dirty page
      file:WiredTiger.wt(<live>):
      internal: 1 pages, 0.56 KB, 1/0 clean/dirty pages, 0.56/0.00 clean / dirty KB, 0.56 KB max page, 0.00 KB max dirty page
      leaf: 1 pages, 16.82 KB, 1/0 clean/dirty pages, 16.82 /0.00 /16.73 clean/dirty/updates KB, 16.82 KB max page, 0.00 KB max dirty page
      cache dump: total found: 65.56 MB vs tracked inuse 60.70 MB
      total dirty bytes: 57.35 MB vs tracked dirty 57.35 MB
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7423
      
      Verbose Error message
      [1661231527:737070][38202:0x7f9a80ff9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: cache eviction thread error: Connection timed out
      [1661231527:737081][38202:0x7f9a80ff9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_evict_thread_run, 324: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1661231527:737091][38202:0x7f9a80ff9700], eviction-server: [WT_VERB_DEFAULT][ERROR]: __wt_abort, 28: aborting WiredTiger library
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7475
      
      C/CPP Suite Failure message
      test_timestamp_abort: FAILED: Child process 38202 abnormally exited: Invalid argument
      
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7481
      
      Failure keywords
      test_timestamp_abort: process aborting
      WiredTiger Error: __wt_abort, 28: aborting WiredTiger library
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7482
      
      Evergreen YAML Script Failure
      Command failed: shell script encountered problem: exit code 134
      
      Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_ubuntu2004_stress_tests_recovery_stress_test_1_42ee89096f8306dbdcbd1a9416c308cf2e0ccf7e_22_08_23_03_32_11/0?type=T#L7485
      

      logs

        1. WT-9796-stack-trace.txt
          299 kB
        2. image-2022-11-14-17-48-28-767.png
          image-2022-11-14-17-48-28-767.png
          25 kB
        3. image-2022-11-16-12-46-55-254.png
          image-2022-11-16-12-46-55-254.png
          42 kB

            Assignee:
            mick.graham@mongodb.com Mick Graham
            Reporter:
            Xgen-BuildBaron-User xgen-buildbaron-user
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: