Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-5275

Cache stuck full of both clean and dirty pages

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • 3
    • Storage Engines 2020-01-27

      After resolving WT-5082, we started to see cache stuck failure full of both clean and dirty pages, on the PPC machine.

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-ppc/22554/

      ++ nice catchsegv ./t -1 -c ../../../test/format/CONFIG.stress file_type=row leaf_page_max=9 internal_page_max=9 key_min=256 value_min=256
      t: process 4632 running
      [1575010581:770153][4632:0x3fff87bef1b0], t, eviction-server: __evict_server, 440: Cache stuck for too long, giving up: Connection timed out
      0x3fff8bc23030:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x3fff8bc23030:transaction state dump
      0x3fff8bc23030:current ID: 3768
      0x3fff8bc23030:last running ID: 2505
      0x3fff8bc23030:metadata_pinned ID: 2468
      0x3fff8bc23030:oldest ID: 2468
      0x3fff8bc23030:durable timestamp: (0,0)
      0x3fff8bc23030:oldest timestamp: (0,0)
      0x3fff8bc23030:pinned timestamp: (0,0)
      0x3fff8bc23030:stable timestamp: (0,0)
      0x3fff8bc23030:has_durable_timestamp: no
      0x3fff8bc23030:has_oldest_timestamp: no
      0x3fff8bc23030:has_pinned_timestamp: no
      0x3fff8bc23030:has_stable_timestamp: no
      0x3fff8bc23030:oldest_is_pinned: no
      0x3fff8bc23030:stable_is_pinned: no
      0x3fff8bc23030:checkpoint running: yes
      0x3fff8bc23030:checkpoint generation: 10
      0x3fff8bc23030:checkpoint pinned ID: 2505
      0x3fff8bc23030:checkpoint txn ID: 3767
      0x3fff8bc23030:oldest named snapshot ID: 0
      0x3fff8bc23030:session count: 26
      0x3fff8bc23030:Transaction state of active sessions:
      0x3fff8bc23030:ID: 2505, pinned ID: 2468, metadata pinned ID: 2468, name: WT_SESSION.truncate
      0x3fff8bc23030:transaction id: 2505, mod count: 511922, snap min: 2468, snap max: 2497, snapshot count: 9, commit_timestamp: (0,0), durable_timestamp: (0,0), first_commit_timestamp: (0,0), prepare_timestamp: (0,0), read_timestamp: (0,0), checkpoint LSN: [0][0], full checkpoint: false, rollback reason: , flags: 0x01002004, isolation: WT_ISO_READ_UNCOMMITTED
      0x3fff8bc23030:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x3fff8bc23030:cache dump
      0x3fff8bc23030:cache full: yes
      0x3fff8bc23030:cache clean check: yes (1338.921%)
      0x3fff8bc23030:cache dirty check: yes (784.491%)
      0x3fff8bc23030:file:wt.wt(<live>):
      0x3fff8bc23030:internal: 32164 pages, 400MB, 27129/5035 clean/dirty pages, 337/63 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x3fff8bc23030:leaf: 505133 pages, 566MB, 0/505133 clean/dirty pages, 0/566 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x3fff8bc23030:file:WiredTigerLAS.wt(<live>):
      0x3fff8bc23030:internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x3fff8bc23030:leaf: 0 pages
      0x3fff8bc23030:file:WiredTiger.wt(<live>):
      0x3fff8bc23030:internal: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x3fff8bc23030:leaf: 1 pages, 0MB, 0/1 clean/dirty pages, 0/0 clean/dirty MB, 0MB max page, 0MB max dirty page
      0x3fff8bc23030:cache dump: total found: 1044MB vs tracked inuse 1044MB
      0x3fff8bc23030:total dirty bytes: 629MB
      [1575010582:478884][4632:0x3fff87bef1b0], t, eviction-server: __wt_evict_thread_run, 315: cache eviction thread error: Connection timed out
      [1575010582:478920][4632:0x3fff87bef1b0], t, eviction-server: __wt_panic, 490: the process must exit and restart: WT_PANIC: WiredTiger library panic
      [1575010582:478932][4632:0x3fff87bef1b0], t, eviction-server: __wt_abort, 28: aborting WiredTiger library 

      The configuration:

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      alter=0
      assert_commit_timestamp=0
      assert_read_timestamp=0
      auto_throttle=1
      backups=0
      bitcnt=3
      bloom=0
      bloom_bit_count=30
      bloom_hash_count=12
      bloom_oldest=0
      cache=78
      cache_minimum=20
      checkpoints=on
      checkpoint_log_size=171
      checkpoint_wait=72
      checksum=uncompressed
      chunk_size=7
      compaction=0
      compression=snappy
      data_extend=0
      data_source=table
      delete_pct=4
      dictionary=0
      direct_io=0
      encryption=none
      evict_max=1
      file_type=row-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      independent_thread_rng=1
      in_memory=0
      insert_pct=40
      internal_key_truncation=1
      internal_page_max=9
      isolation=random
      key_gap=18
      key_max=256
      key_min=256
      leaf_page_max=9
      leak_memory=0
      logging=1
      logging_archive=1
      logging_compression=lz4
      logging_file_max=173202
      logging_prealloc=1
      long_running_txn=0
      lsm_worker_threads=4
      memory_page_max=3
      merge_max=20
      mmap=1
      modify_pct=36
      ops=0
      prefix_compression=0
      prefix_compression_min=0
      prepare=0
      quiet=1
      random_cursor=0
      read_pct=19
      rebalance=1
      repeat_data_pct=36
      reverse=0
      rows=1000000
      runs=1
      salvage=1
      split_pct=94
      statistics=0
      statistics_server=0
      threads=13
      timer=4
      timing_stress_aggressive_sweep=0
      timing_stress_checkpoint=0
      timing_stress_lookaside_sweep=0
      timing_stress_split_1=0
      timing_stress_split_2=0
      timing_stress_split_3=0
      timing_stress_split_4=0
      timing_stress_split_5=0
      timing_stress_split_6=0
      timing_stress_split_7=0
      timing_stress_split_8=0
      transaction_timestamps=0
      transaction-frequency=85
      truncate=1
      value_max=3174
      value_min=256
      verify=1
      wiredtiger_config=
      write_pct=1
      ############################################ 

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            luke.chen@mongodb.com Luke Chen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: