Loading...

Type: Build Failure
Resolution: Fixed
Priority: Critical - P2
Fix Version/s: WT11.3.0, 8.0.0-rc0, 7.3.2
Affects Version/s: None
Component/s: None
Labels:

Assigned Teams:

Storage Engines
Evergreen Project:
Failure Type:
- task-timed-out
Failing Buildvariants:
Failing Tasks:
First Failing Revision:
- b28742aed07482b945164b755ebc3967d5e03851
Linked BFG List:
https://buildbaron.corp.mongodb.com/ui/#/bf/WT-9820
Count of Linked BFGs (Last 30 days):
13

Story Points:
8
Sprint:
Storage Engines - 2022-11-14, 2023-06-27 Lord of the Sprints, 2024-02-20_A_near-death_puffin
Backport Requested:

v7.3

mick.graham@mongodb.com has identified what he suspects is a deadlock over write locking between eviction and checkpoint.
Next steps:

Investigate and validate the finding.
- Note there have not been any build failures for over 30 days, so this might not be worth investigating.
Determine how to reproduce.
Review current code to see if code on develop prevents this deadlock.
- Can the fix be back ported?
- If the current code does obviously prevent this deadlock the reproducer should be run against develop.

spinlock-gcc-test on rhel80

Host: i-048311c0d67e76841
Project: wiredtiger-mongo-v4.4
Commit: b28742ae

Task Logs:

task spinlock-gcc-test

spinlock-gcc-test task_log

Logs:

6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/lang/python'
Making check in test/utility
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/utility'
make[1]: Nothing to be done for 'check'.
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/utility'
Making check in examples/c
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
make  check-TESTS
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
PASS: ex_access
PASS: ex_call_center
PASS: ex_col_store
PASS: ex_config_parse
PASS: ex_all
PASS: ex_cursor
PASS: ex_data_source
PASS: ex_event_handler
PASS: ex_encrypt
PASS: ex_extending
PASS: ex_extractor
PASS: ex_file_system
PASS: ex_hello
PASS: ex_pack
PASS: ex_log
PASS: ex_process
PASS: ex_schema
PASS: ex_smoke
PASS: ex_stat
PASS: ex_thread
PASS: ex_backup
PASS: ex_backup_block
============================================================================
Testsuite summary for WiredTiger 10.0.2
============================================================================
# TOTAL: 22
# PASS:  22
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/examples/c'
Making check in test/bloom
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
make  check-TESTS
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
PASS: t
============================================================================
Testsuite summary for WiredTiger 10.0.2
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/bloom'
Making check in test/checkpoint
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
make  check-TESTS
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
PASS: smoke.sh
============================================================================
Testsuite summary for WiredTiger 10.0.2
============================================================================
# TOTAL: 1
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/checkpoint'
Making check in test/cppsuite
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
make  check-TESTS
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
PASS: csuite_style_example_test
PASS: run
============================================================================
Testsuite summary for WiredTiger 10.0.2
============================================================================
# TOTAL: 2
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
make[3]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
make[2]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
make[1]: Leaving directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/cppsuite'
Making check in test/csuite
make[1]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
make  check-TESTS
make[2]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
make[3]: Entering directory '/data/mci/e153fedf99f6d6e943d4b7d7353e326a/wiredtiger/build_posix/test/csuite'
PASS: incr_backup/smoke.sh
PASS: random_directio/smoke.sh
PASS: test_scope
PASS: tiered_abort/smoke.sh
PASS: test_rwlock
PASS: truncated_log/smoke.sh
PASS: test_wt1965_col_efficiency
PASS: test_wt2403_lsm_workload
PASS: random_abort/smoke.sh
PASS: test_wt2246_col_append
PASS: wt2535_insert_race/smoke.sh
PASS: wt2447_join_main_table/smoke.sh
PASS: test_wt2695_checksum
PASS: test_wt2592_join_schema
PASS: schema_abort/smoke.sh
PASS: test_wt2719_reconfig
PASS: test_wt2834_join_bloom_fix
PASS: test_wt2323_join_visibility
PASS: test_wt2999_join_extractor
PASS: test_wt3120_filesys
PASS: test_wt3135_search_near_collator
PASS: test_wt3184_dup_index_collator
PASS: wt2853_perf/smoke.sh
PASS: test_wt3363_checkpoint_op_races
PASS: test_wt3874_pad_byte_collator
PASS: wt4105_large_doc_small_upd/smoke.sh
PASS: test_wt4117_checksum
PASS: test_wt4156_metadata_salvage
PASS: test_wt3338_partial_update
PASS: test_wt4699_json
PASS: test_wt4803_history_store_abort
PASS: test_wt4891_meta_ckptlist_get_alloc
PASS: wt6185_modify_ts/smoke.sh
PASS: wt6616_checkpoint_oldest_ts/smoke.sh
PASS: test_wt4333_handle_locks
PASS: test_wt8057_compact_stress
PASS: test_wt7989_compact_checkpoint
PASS: wt2909_checkpoint_integrity/smoke.sh
Command stopped early: context canceled
fi
Running task-timeout commands.
Running command 'shell.exec' in "run wt hang analyzer" (step 1 of 1)
# Dump core (-c) and debugger outputs (-o)
wt_hang_analyzer_option="-c -o file -o stdout"
echo "Calling the wt hang analyzer ..."
PATH="/opt/mongodbtoolchain/gdb/bin:$PATH" /opt/mongodbtoolchain/v3/bin/python3 ../test/wt_hang_analyzer/wt_hang_analyzer.py $wt_hang_analyzer_option
Calling the wt hang analyzer ...
Python Version: 3.9.2 (default, Dec 17 2021, 20:59:40)
[GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]
OS: Linux-4.18.0-80.1.2.el8_0.x86_64-x86_64-with-glibc2.28
Cannot determine Linux distro since Python is too old
Current User: 1000
Cannot determine Unix Current Login
Getting list of processes using /usr/bin/ps
['/usr/bin/ps', '--version']
ps from procps-ng 3.3.15
['/usr/bin/ps', '-eo', 'pid,args']
Found 4 interesting processes [(705, 'platform-python'), (1154, 'platform-python'), (6353, 'test_timestamp_abort'), (6356, 'test_timestamp_abort')]
Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing platform-python process with PID 705
Dumping core to dump_platform-python.705.core
['/opt/mongodbtoolchain/gdb/bin/gdb', '--version']
/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gdb-v3.de9/share/gdb/python/gdb/command/prompt.py:48: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if self.value is not '':
/opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gdb-v3.de9/share/gdb/python/gdb/command/prompt.py:60: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if self.value is not '':
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
['/opt/mongodbtoolchain/gdb/bin/gdb', '--quiet', '--nx', '-ex', 'set interactive-mode off', '-ex', 'set print thread-events off', '-ex', 'file platform-python', '-ex', 'attach 705', '-ex', 'info sharedlibrary', '-ex', 'info threads', '-ex', 'thread apply all bt', '-ex', 'set python print-stack full', '-ex', 'set scheduler-locking on', '-ex', 'gcore dump_platform-python.705.core', '-ex', 'set confirm off', '-ex', 'quit']
platform-python: No such file or directory.
Attaching to process 705
ptrace: Operation not permitted.
No shared libraries loaded at this time.
No threads.
Target 'None' cannot support this command.
You can't do that without a process to debug.
Done analyzing platform-python process with PID 705
Debugger /opt/mongodbtoolchain/gdb/bin/gdb, analyzing platform-python process with PID 1154
Dumping core to dump_platform-python.1154.core
['/opt/mongodbtoolchain/gdb/bin/gdb', '--version']
GNU gdb (GDB) 8.3.1
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the ex
Link: https://evergreen.mongodb.com/task_log_raw/wiredtiger_mongo_v4.4_rhel80_spinlock_gcc_test_b28742aed07482b945164b755ebc3967d5e03851_22_08_23_23_12_04/0?type=T#L2213

logs

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

wt9820-stacktrace.txt
23 kB
Nov 15 2022 04:23:41 AM UTC

duplicates

WT-12320 failed: csuite-long-running on ubuntu2004-msan [wiredtiger @ afe0789c]

Closed

WT-12335 Failed: csuite-timestamp-abort-test-s3 on ! Ubuntu 20.04 [WiredTiger (develop) @ 583b4102]

Closed

is related to

WT-11585 test_timestamp_abort timed out: ftruncate-test on rhel80 (wiredtiger-mongo-v5.0)

Closed

related to

WT-12806 Timeout: waited 41 seconds for checkpoint file creation in csuite-timestamp-abort-test-s3

Open

WT-9796 Add debugging for logging getting stuck

Closed

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates