Loading...

Type: Bug
Resolution: Gone away
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Storage
Labels:
None

Operating System:
ALL
Backport Requested:

v5.1
Steps To Reproduce:
Hide

Use DSI locally to setup a perf cluster:

On the iwc node with the following command line:

(cd src/genny && ./src/workloads/contrib/historystore/eMRCf_runner.sh -g 8 -n)
Show
Use DSI locally to setup a perf cluster: On the iwc node with the following command line: (cd src/genny && ./src/workloads/contrib/historystore/eMRCf_runner.sh -g 8 -n)
Sprint:
Storage - Ra 2021-09-20, Storage - Ra 2021-10-04, Storage - Ra 2021-10-18
Story Points:
3

Executing the eMRCf_runner.sh tests with more than 7 growth iterations and enableMajorityReadConcern set to true results in a SIGAbort when shutting down the primary.

{"t":{"$date":"2021-01-10T00:14:48.681+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"conn239","msg":"WiredTiger error","attr":{"error":12,"message":"[1610237688:672274][43957:0x7ff5de9fd700], file:index-39-3882910392635076206.wt, txn rollback_to_stable: __wt_calloc, 52: memory allocation of 2848 bytes failed: Cannot allocate memory"}}
{"t":{"$date":"2021-01-10T00:14:48.695+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"conn239","msg":"WiredTiger error","attr":{"error":12,"message":"[1610237688:695405][43957:0x7ff5de9fd700], WT_CONNECTION.close: __wt_calloc, 52: memory allocation of 264400 bytes failed: Cannot allocate memory"}}
{"t":{"$date":"2021-01-10T00:14:48.697+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"conn239","msg":"WiredTiger error","attr":{"error":12,"message":"[1610237688:696255][43957:0x7ff5de9fd700], WT_CONNECTION.close: __conn_close, 1035: failure during close, disabling further writes: Cannot allocate memory"}}
{"t":{"$date":"2021-01-10T00:14:49.054+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"thread355","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1610237689:54366][43957:0x7ff60225f700], log-wrlsn-server: __log_wrlsn_server, 871: log wrlsn server error: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2021-01-10T00:14:49.057+00:00"},"s":"F",  "c":"-",        "id":23089,   "ctx":"thread355","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":537}}
{"t":{"$date":"2021-01-10T00:14:49.057+00:00"},"s":"F",  "c":"-",        "id":23090,   "ctx":"thread355","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2021-01-10T00:14:49.059+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"thread355","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2021-01-10T00:14:49.238+00:00"},"s":"E",  "c":"CONTROL",  "id":31430,   "ctx":"thread355","msg":"Error collecting stack trace","attr":{"error":"unw_get_proc_name(564559009DFA): unspecified (general) error\nunw_get_proc_name(56455900B1F9): unspecified (general) error\nunw_get_proc_name(564559006DF6): unspecified (general) error\nunw_get_proc_name(7FF60AE1A7E0): unspecified (general) error\nunw_get_proc_name(7FF60AA91B20): unspecified (general) error\nunw_get_proc_name(7FF60AA92FC8): unspecified (general) error\nunw_get_proc_name(5645568CCE5A): unspecified (general) error\nunw_get_proc_name(5645565CA91F): unspecified (general) error\nunw_get_proc_name(5645573CF68B): unspecified (general) error\nunw_get_proc_name(5645565E81BE): unspecified (general) error\nunw_get_proc_name(5645565D376B): unspecified (general) error\nunw_get_proc_name(7FF60AE1040B): unspecified (general) error\nunw_get_proc_name(7FF60AB4AF9F): unspecified (general) error\nunw_get_proc_name(564559009DFA): unspecified (general) error\nunw_get_proc_name(56455900B1F9): unspecified (general) error\nunw_get_proc_name(564559006DF6): unspecified (general) error\nunw_get_proc_name(7FF60AE1A7E0): unspecified (general) error\nunw_get_proc_name(7FF60AA91B20): unspecified (general) error\nunw_get_proc_name(7FF60AA92FC8): unspecified (general) error\nunw_get_proc_name(5645568CCE5A): unspecified (general) error\nunw_get_proc_name(5645565CA91F): unspecified (general) error\nunw_get_proc_name(5645573CF68B): unspecified (general) error\nunw_get_proc_name(5645565E81BE): unspecified (general) error\nunw_get_proc_name(5645565D376B): unspecified (general) error\nunw_get_proc_name(7FF60AE1040B): unspecified (general) error\nunw_get_proc_name(7FF60AB4AF9F): unspecified (general) error\n"}}
{"t":{"$date":"2021-01-10T00:14:49.238+00:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"thread355","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"564559009DFA","b":"56455568F000","o":"397ADFA"},{"a":"56455900B1F9","b":"56455568F000","o":"397C1F9","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"564559006DF6","b":"56455568F000","o":"3977DF6","s":"abruptQuit","s+":"66"},{"a":"7FF60AE1A7E0","b":"7FF60AE09000","o":"117E0"},{"a":"7FF60AA91B20","b":"7FF60AA5E000","o":"33B20","s":"gsignal","s+":"110"},{"a":"7FF60AA92FC8","b":"7FF60AA5E000","o":"34FC8","s":"abort","s+":"148"},{"a":"5645568CCE5A","b":"56455568F000","o":"123DE5A","s":"_ZN5mongo35fassertFailedWithStatusWithLocationEiRKNS_6StatusEPKcj","s+":"0"},{"a":"5645565CA91F","b":"56455568F000","o":"F3B91F"},{"a":"5645573CF68B","b":"56455568F000","o":"1D4068B"},{"a":"5645565E81BE","b":"56455568F000","o":"F591BE","s":"__wt_panic_func","s+":"B3"},{"a":"5645565D376B","b":"56455568F000","o":"F4476B"},{"a":"7FF60AE1040B","b":"7FF60AE09000","o":"740B"},{"a":"7FF60AB4AF9F","b":"7FF60AA5E000","o":"ECF9F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.9.0-alpha-1214-g54e8728","gitVersion":"54e87285f802a49456c02c80cb0ddb0fbb54c88a","compiledModules":["enterprise"],"uname":{"sysname":"Linux","release":"4.14.165-131.185.amzn2.x86_64","version":"#1 SMP Wed Jan 15 14:19:56 UTC 2020","machine":"x86_64"},"somap":[{"b":"56455568F000","elfType":3,"buildId":"7FEA37A074A30C9B75AC461167B7598F4E171408"},{"b":"7FF60AE09000","path":"/lib64/libpthread.so.0","elfType":3,"buildId":"5E12AD1B2983169B48CE4C7051454E8CA38D6EF0"},{"b":"7FF60AA5E000","path":"/lib64/libc.so.6","elfType":3,"buildId":"DAFC6796B1B45E202959E8872844CABEE8A20A32"}]}}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"564559009DFA","b":"56455568F000","o":"397ADFA"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"56455900B1F9","b":"56455568F000","o":"397C1F9","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"564559006DF6","b":"56455568F000","o":"3977DF6","s":"abruptQuit","s+":"66"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AE1A7E0","b":"7FF60AE09000","o":"117E0"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AA91B20","b":"7FF60AA5E000","o":"33B20","s":"gsignal","s+":"110"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AA92FC8","b":"7FF60AA5E000","o":"34FC8","s":"abort","s+":"148"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645568CCE5A","b":"56455568F000","o":"123DE5A","s":"_ZN5mongo35fassertFailedWithStatusWithLocationEiRKNS_6StatusEPKcj","s+":"0"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645565CA91F","b":"56455568F000","o":"F3B91F"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645573CF68B","b":"56455568F000","o":"1D4068B"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645565E81BE","b":"56455568F000","o":"F591BE","s":"__wt_panic_func","s+":"B3"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"5645565D376B","b":"56455568F000","o":"F4476B"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AE1040B","b":"7FF60AE09000","o":"740B"}}}
{"t":{"$date":"2021-01-10T00:14:49.239+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread355","msg":"Frame","attr":{"frame":{"a":"7FF60AB4AF9F","b":"7FF60AA5E000","o":"ECF9F","s":"clone","s+":"3F"}}}

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

Screen Shot 2021-10-13 at 5.22.32 pm.png
43 kB
Oct 13 2021 06:26:13 AM UTC
Screen Shot 2021-10-13 at 4.12.46 pm.png
34 kB
Oct 13 2021 05:16:11 AM UTC
Screen Shot 2021-10-13 at 4.11.49 pm.png
127 kB
Oct 13 2021 05:16:05 AM UTC
mem.png
133 kB
Feb 10 2021 07:45:17 PM UTC

is depended on by

WT-6924 Queue history store pages for urgent eviction when cache pressure is high

Closed

is related to

WT-6300 Delay setting leak memory flag until after the final checkpoint in shutdown

Closed

related to

SERVER-54472 Collect additional FTDC metrics during shutdown for replica sets

Closed

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates