Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-4114

format failure with timestamps disabled, stalling eviction

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • 2
    • Storage Engines 2020-01-27

      Recently we saw a new type of stress failure with timestamps disabled in test/format periodically pinned cache full. Please note it's different from WT-3799 which has timestamps enabled. 

      An example of the failure: 

      [1528431602:911610][22942:0x7f1cba42f700], t, eviction-server: Cache stuck for too long, giving up: Connection timed out
      t: process 22942
      0x7f1cbee37888:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x7f1cbee37888:transaction state dump
      0x7f1cbee37888:current ID: 4932345
      0x7f1cbee37888:last running ID: 4932345
      0x7f1cbee37888:oldest ID: 4932345
      0x7f1cbee37888:commit timestamp: 0
      0x7f1cbee37888:oldest timestamp: 0
      0x7f1cbee37888:pinned timestamp: 0
      0x7f1cbee37888:stable timestamp: 0
      0x7f1cbee37888:has_commit_timestamp: no
      0x7f1cbee37888:has_oldest_timestamp: no
      0x7f1cbee37888:has_pinned_timestamp: no
      0x7f1cbee37888:has_stable_timestamp: no
      0x7f1cbee37888:oldest_is_pinned: no
      0x7f1cbee37888:stable_is_pinned: no
      0x7f1cbee37888:checkpoint running: yes
      0x7f1cbee37888:checkpoint generation: 10
      0x7f1cbee37888:checkpoint pinned ID: 4931347
      0x7f1cbee37888:checkpoint txn ID: 4931463
      0x7f1cbee37888:oldest named snapshot ID: 0
      0x7f1cbee37888:session count: 49
      0x7f1cbee37888:Transaction state of active sessions:
      0x7f1cbee37888:=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
      0x7f1cbee37888:cache dump
      0x7f1cbee37888:cache full: no
      0x7f1cbee37888:cache clean check: no (27.433%)
      0x7f1cbee37888:cache dirty check: yes (23.236%)
      [1528431602:911734][22942:0x7f1cba42f700], t, file:wt.wt, eviction-server: __tree_walk_internal, 312: btree->root.page != NULL
      [1528431602:911741][22942:0x7f1cba42f700], t, file:wt.wt, eviction-server: aborting WiredTiger library
      *** Aborted 

      The configuration:

      ############################################
      #  RUN PARAMETERS
      ############################################
      abort=0
      alter=0
      auto_throttle=0
      backups=0
      bitcnt=3
      bloom=1
      bloom_bit_count=54
      bloom_hash_count=9
      bloom_oldest=0
      cache=99
      cache_minimum=20
      checkpoints=on
      checkpoint_log_size=187
      checkpoint_wait=79
      checksum=uncompressed
      chunk_size=6
      compaction=0
      compression=snappy
      data_extend=0
      data_source=table
      delete_pct=3
      dictionary=1
      direct_io=0
      encryption=none
      evict_max=0
      file_type=row-store
      firstfit=0
      huffman_key=0
      huffman_value=0
      independent_thread_rng=0
      in_memory=0
      insert_pct=60
      internal_key_truncation=1
      internal_page_max=11
      isolation=random
      key_gap=6
      key_max=104
      key_min=21
      leaf_page_max=16
      leak_memory=0
      logging=0
      logging_archive=1
      logging_compression=none
      logging_file_max=311371
      logging_prealloc=1
      long_running_txn=0
      lsm_worker_threads=3
      merge_max=6
      mmap=1
      modify_pct=4
      ops=0
      prefix_compression=1
      prefix_compression_min=1
      prepare=0
      quiet=1
      read_pct=1
      rebalance=1
      repeat_data_pct=41
      reverse=0
      rows=1000000
      runs=1
      salvage=1
      split_pct=64
      statistics=0
      statistics_server=0
      threads=31
      timer=4
      timing_stress_checkpoint=0
      timing_stress_lookaside_sweep=0
      timing_stress_split_1=0
      timing_stress_split_2=0
      timing_stress_split_3=1
      timing_stress_split_4=0
      timing_stress_split_5=0
      timing_stress_split_6=0
      timing_stress_split_7=0
      transaction_timestamps=0
      transaction-frequency=34
      truncate=1
      value_max=2669
      value_min=16
      verify=1
      wiredtiger_config=
      write_pct=32
      ############################################ 

      The original failure:

      http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress/66564/

       

            Assignee:
            keith.bostic@mongodb.com Keith Bostic (Inactive)
            Reporter:
            luke.chen@mongodb.com Luke Chen
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: