2017-02-28T22:47:37.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-02-28T23:32:18.972+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-02-28T23:32:18.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-02-28T23:58:37.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-02-28T23:58:37.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T00:40:33.972+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T00:40:33.972+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T01:41:23.975+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T01:41:23.976+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T02:12:43.975+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T02:12:43.976+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T04:19:43.972+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T04:19:43.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T04:38:18.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T04:38:18.979+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T04:50:38.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T04:50:38.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T04:52:28.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T04:52:28.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T06:23:07.981+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T06:23:07.981+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T06:56:23.976+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T06:56:23.976+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T07:27:33.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T07:27:33.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T07:29:58.972+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T07:29:58.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T12:34:58.976+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T12:34:58.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T13:15:03.972+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T13:15:03.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T13:41:29.513+0000 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends 2017-03-01T13:41:29.513+0000 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... 2017-03-01T13:41:29.513+0000 I NETWORK [signalProcessingThread] closing listening socket: 7 2017-03-01T13:41:29.513+0000 I NETWORK [signalProcessingThread] closing listening socket: 8 2017-03-01T13:41:29.513+0000 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27019.sock 2017-03-01T13:41:29.513+0000 I NETWORK [signalProcessingThread] shutdown: going to flush diaglog... 2017-03-01T13:41:29.513+0000 I REPL [signalProcessingThread] shutting down replication subsystems 2017-03-01T13:41:29.513+0000 I REPL [signalProcessingThread] Stopping replication reporter thread 2017-03-01T13:41:29.513+0000 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to 10.0.2.28:27017: CallbackCanceled: Reporter no longer valid 2017-03-01T13:41:29.513+0000 I REPL [signalProcessingThread] Stopping replication fetcher thread 2017-03-01T13:41:29.514+0000 I REPL [signalProcessingThread] Stopping replication applier thread 2017-03-01T13:41:29.515+0000 I REPL [signalProcessingThread] Stopping replication storage threads 2017-03-01T13:41:29.515+0000 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture 2017-03-01T13:41:29.517+0000 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down 2017-03-01T13:41:29.591+0000 I STORAGE [signalProcessingThread] shutdown: removing fs lock... 2017-03-01T13:41:29.591+0000 I CONTROL [signalProcessingThread] now exiting 2017-03-01T13:41:29.591+0000 I CONTROL [signalProcessingThread] shutting down with code:0 2017-03-01T13:41:29.591+0000 I CONTROL [initandlisten] shutting down with code:0 2017-03-01T13:45:15.763+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] MongoDB starting : pid=9280 port=27019 dbpath=/mongo/mongo3/data 64-bit host=ip-10-0-2-28 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] db version v3.4.2 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] git version: 3f76e40c105fc223b3e5aac3e20dcd026b83b38b 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] modules: none 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] build environment: 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] distmod: amazon 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] distarch: x86_64 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] target_arch: x86_64 2017-03-01T13:45:15.769+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod3.conf", net: { port: 27019 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod3.pid" }, replication: { replSetName: "thinkmap" }, storage: { dbPath: "/mongo/mongo3/data", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/mongo/mongo3/log/mongod.log" } } 2017-03-01T13:45:15.789+0000 I - [initandlisten] Detected data files in /mongo/mongo3/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2017-03-01T13:45:15.789+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1465M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2017-03-01T13:45:16.038+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2017-03-01T13:45:16.039+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 6797518 records totaling to 5146663866 bytes 2017-03-01T13:45:16.039+0000 I STORAGE [initandlisten] Sampling from the oplog between Jan 25 18:38:50:9e1 and Mar 1 13:41:28:1 to determine where to place markers for truncation 2017-03-01T13:45:16.039+0000 I STORAGE [initandlisten] Taking 1007 samples and assuming that each section of oplog contains approximately 67473 records totaling to 51086418 bytes 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:39:25:dcd 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:40:08:5c7 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:41:06:2a0 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:42:00:1e5a 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:42:39:aa6 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:43:29:10d 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:32:58:91 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:35:23:1983 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:35:54:aa4 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:36:32:cdf 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:37:12:9c8 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:37:53:e90 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:38:34:3cb 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:39:16:c10 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:39:58:14e4 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:40:40:145e 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:07:2ba3 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:09:3205 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:12:20df 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:16:1a01 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:18:d0f 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:23:5d3 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:25:1bcd 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:30:d16 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:31:1004 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:34:d25 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:39:8be 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:43:6bc 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:47:7bf 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:49:1bad 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 21:54:42:1133 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 28 05:49:54:1 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:49:25:2ff 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:49:57:197 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:50:23:12a 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:50:56:311 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:51:28:2b5 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:52:00:427 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:52:34:112 2017-03-01T13:45:16.618+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:53:08:1ad 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:53:42:c 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:54:14:23e 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:54:49:98 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:43:19:1e8 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:44:15:91 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:45:04:292 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:45:58:fc 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:46:47:2c7 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:47:41:352 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:48:31:110 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:49:32:125 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:50:24:32 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:51:21:165 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:52:15:22 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 16:45:51:31 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 17:37:16:22 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 18:28:26:22 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 19:17:13:29 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:52:adf 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:53:83cb 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:55:d3c 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:56:19ad 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:57:2a59 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:58:44ac 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:59:3ec8 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:01:1bdb 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:04:462 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:05:c17 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:06:266b 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 10 09:50:03:1 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 20 21:08:53:1 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:53:4493 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:54:6e5b 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:56:10 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:58:cd9 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:48:10:124b 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:48:14:8250 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:12:7ac1 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:14:7a1 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:15:13b4 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:16:293c 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:17:462b 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:19:9c3 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:21:1e66 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:23:144e 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:25:a88 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:26:3ee8 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:55:53dd 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:56:3f1a 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:57:6476 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:59:c0d 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:00:f7b 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:01:2589 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:03:8e5 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:06:41f 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:07:4492 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:06:4885 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:45:24fd 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:46:2c38 2017-03-01T13:45:16.619+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:47:5d88 2017-03-01T13:45:17.040+0000 I CONTROL [initandlisten] 2017-03-01T13:45:17.040+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2017-03-01T13:45:17.040+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2017-03-01T13:45:17.040+0000 I CONTROL [initandlisten] 2017-03-01T13:45:17.150+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/mongo/mongo3/data/diagnostic.data' 2017-03-01T13:45:17.151+0000 I NETWORK [thread1] waiting for connections on port 27019 2017-03-01T13:45:17.152+0000 I REPL [replExecDBWorker-1] New replica set config in use: { _id: "thinkmap", version: 67868, protocolVersion: 1, members: [ { _id: 3, host: "10.0.2.28:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 30.0, tags: { use: "general" }, slaveDelay: 0, votes: 1 }, { _id: 4, host: "10.0.2.28:27018", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 20.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 5, host: "10.0.2.28:27019", arbiterOnly: false, buildIndexes: true, hidden: true, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2017-03-01T13:45:17.152+0000 I REPL [replExecDBWorker-1] This node is 10.0.2.28:27019 in the config 2017-03-01T13:45:17.152+0000 I REPL [replExecDBWorker-1] transition to STARTUP2 2017-03-01T13:45:17.152+0000 I REPL [replExecDBWorker-1] Starting replication storage threads 2017-03-01T13:45:17.152+0000 I REPL [replExecDBWorker-1] Starting replication fetcher thread 2017-03-01T13:45:17.152+0000 I REPL [replExecDBWorker-1] Starting replication applier thread 2017-03-01T13:45:17.152+0000 I REPL [replExecDBWorker-1] Starting replication reporter thread 2017-03-01T13:45:17.152+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27017 2017-03-01T13:45:17.152+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27018 2017-03-01T13:45:17.153+0000 I REPL [rsSync] transition to RECOVERING 2017-03-01T13:45:17.153+0000 I REPL [rsSync] transition to SECONDARY 2017-03-01T13:45:17.154+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27018 2017-03-01T13:45:17.154+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27018 is now in state SECONDARY 2017-03-01T13:45:17.154+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27017 2017-03-01T13:45:17.154+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27017 is now in state SECONDARY 2017-03-01T13:45:17.855+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:50898 #1 (1 connection now open) 2017-03-01T13:45:17.855+0000 I NETWORK [conn1] received client metadata from 10.0.2.28:50898 conn1: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T13:45:20.630+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:50900 #2 (2 connections now open) 2017-03-01T13:45:20.630+0000 I NETWORK [conn2] received client metadata from 10.0.2.28:50900 conn2: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T13:45:23.154+0000 I REPL [rsBackgroundSync] sync source candidate: 10.0.2.28:27018 2017-03-01T13:45:23.154+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T13:45:23.155+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T13:45:23.155+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T13:45:23.156+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T13:45:23.158+0000 I REPL [replication-0] Choosing new sync source because our current sync source, 10.0.2.28:27018, has an OpTime ({ ts: Timestamp 1488375708000|1, t: 56 }) which is not ahead of ours ({ ts: Timestamp 1488375708000|1, t: 56 }), it does not have a sync source, and it's not the primary (sync source does not know the primary) 2017-03-01T13:45:23.158+0000 I REPL [replication-0] Canceling oplog query because we have to choose a new sync source. Current source: 10.0.2.28:27018, OpTime { ts: Timestamp 0|0, t: -1 }, its sync source index:-1 2017-03-01T13:45:23.158+0000 W REPL [rsBackgroundSync] Fetcher stopped querying remote oplog with error: InvalidSyncSource: sync source 10.0.2.28:27018 (last visible optime: { ts: Timestamp 0|0, t: -1 }; config version: 67868; sync source index: -1; primary index: -1) is no longer valid 2017-03-01T13:45:23.158+0000 I REPL [rsBackgroundSync] sync source candidate: 10.0.2.28:27017 2017-03-01T13:45:23.158+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T13:45:23.158+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T13:45:23.158+0000 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to 10.0.2.28:27018: InvalidSyncSource: Sync source was cleared. Was 10.0.2.28:27018 2017-03-01T13:45:23.158+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27017 2017-03-01T13:45:23.159+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27017 2017-03-01T13:45:23.161+0000 I REPL [replication-0] Choosing new sync source because our current sync source, 10.0.2.28:27017, has an OpTime ({ ts: Timestamp 1488375718000|1, t: 56 }) which is not ahead of ours ({ ts: Timestamp 1488375718000|1, t: 56 }), it does not have a sync source, and it's not the primary (sync source does not know the primary) 2017-03-01T13:45:23.161+0000 I REPL [replication-0] Canceling oplog query because we have to choose a new sync source. Current source: 10.0.2.28:27017, OpTime { ts: Timestamp 0|0, t: -1 }, its sync source index:-1 2017-03-01T13:45:23.161+0000 W REPL [rsBackgroundSync] Fetcher stopped querying remote oplog with error: InvalidSyncSource: sync source 10.0.2.28:27017 (last visible optime: { ts: Timestamp 0|0, t: -1 }; config version: 67868; sync source index: -1; primary index: -1) is no longer valid 2017-03-01T13:45:23.161+0000 I REPL [rsBackgroundSync] could not find member to sync from 2017-03-01T13:45:23.161+0000 I REPL [SyncSourceFeedback] SyncSourceFeedback error sending update to 10.0.2.28:27017: InvalidSyncSource: Sync source was cleared. Was 10.0.2.28:27017 2017-03-01T13:45:24.153+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:50916 #3 (3 connections now open) 2017-03-01T13:45:24.153+0000 I NETWORK [conn3] received client metadata from 10.0.2.28:50916 conn3: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T13:45:28.162+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27018 is now in state PRIMARY 2017-03-01T13:45:28.162+0000 I REPL [rsBackgroundSync] sync source candidate: 10.0.2.28:27018 2017-03-01T13:45:35.634+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:50926 #4 (4 connections now open) 2017-03-01T13:45:35.634+0000 I NETWORK [conn4] received client metadata from 10.0.2.28:50926 conn4: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T13:45:37.163+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27018 is now in state SECONDARY 2017-03-01T13:45:37.163+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27017 is now in state PRIMARY 2017-03-01T13:46:24.153+0000 I - [conn1] end connection 10.0.2.28:50898 (4 connections now open) 2017-03-01T13:46:35.634+0000 I - [conn2] end connection 10.0.2.28:50900 (3 connections now open) 2017-03-01T13:51:08.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T13:51:08.979+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T13:53:38.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T13:53:38.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T13:57:07.984+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T13:57:07.984+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:05:43.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:05:43.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:15:38.980+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:15:38.980+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:18:08.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:18:08.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:27:53.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:27:53.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:31:43.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:31:43.979+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:35:38.975+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:35:38.976+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:38:03.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:38:03.975+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:40:23.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:40:23.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:45:58.975+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:45:58.976+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:49:33.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:49:33.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:50:37.984+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:50:37.984+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:53:18.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:53:18.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:54:28.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:54:28.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:55:38.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:55:38.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:57:18.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:57:18.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T14:59:13.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T14:59:13.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:02:37.983+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:02:37.983+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:18:13.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:18:13.975+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:19:43.979+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:19:43.979+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:22:07.980+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:22:07.981+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:23:33.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:23:33.975+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:28:28.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:28:28.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:29:33.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:29:33.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:33:08.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:33:08.973+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:35:23.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:35:23.978+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:36:48.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:36:48.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:43:38.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:43:38.977+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:53:18.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T15:53:18.974+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T15:55:00.534+0000 E STORAGE [repl writer worker 8] WiredTiger error (0) [1488383700:534159][9280:0x7f8edc3d3700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 45056: block header checksum of 3605474371 doesn't match expected checksum of 3806882032 2017-03-01T15:55:00.534+0000 E STORAGE [repl writer worker 8] WiredTiger error (0) [1488383700:534218][9280:0x7f8edc3d3700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: index-14-4040745961588520825.wt: encountered an illegal file format or internal value 2017-03-01T15:55:00.534+0000 E STORAGE [repl writer worker 8] WiredTiger error (-31804) [1488383700:534226][9280:0x7f8edc3d3700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic 2017-03-01T15:55:00.534+0000 I - [repl writer worker 8] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2017-03-01T15:55:00.534+0000 I - [repl writer worker 8] ***aborting after fassert() failure 2017-03-01T15:55:00.570+0000 F - [repl writer worker 8] Got signal: 6 (Aborted). 0x556c38260c81 0x556c3825fd79 0x556c3826025d 0x7f8ef8ba4100 0x7f8ef88085f7 0x7f8ef8809ce8 0x556c374e8c4d 0x556c37f6f346 0x556c374f2f30 0x556c374f3024 0x556c374f327c 0x556c38b6313f 0x556c38b6368b 0x556c38b5fd9d 0x556c38b64867 0x556c38b81fb6 0x556c38bb863b 0x556c38c4545d 0x556c38c45b58 0x556c38c4606c 0x556c38bc8421 0x556c38c3ae90 0x556c38be25db 0x556c38c3ac44 0x556c38c3b455 0x556c37f68877 0x556c37f6661c 0x556c37f40ef7 0x556c37944487 0x556c37873cda 0x556c37885703 0x556c378b9533 0x556c37885703 0x556c37b886ba 0x556c37b88fdb 0x556c37b8910d 0x556c379e0635 0x556c37c5ee2a 0x556c37d41584 0x556c37d38002 0x556c37d3b00e 0x556c37d3ba6d 0x556c37d3bb7d 0x556c37d3fcf0 0x556c37d41253 0x556c37d3684f 0x556c37d37d4c 0x556c381d7d65 0x556c381d8890 0x556c381d9439 0x556c38cd5ba0 0x7f8ef8b9cdc5 0x7f8ef88c9c9d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"556C36CD7000","o":"1589C81","s":"_ZN5mongo15printStackTraceERSo"},{"b":"556C36CD7000","o":"1588D79"},{"b":"556C36CD7000","o":"158925D"},{"b":"7F8EF8B95000","o":"F100"},{"b":"7F8EF87D3000","o":"355F7","s":"gsignal"},{"b":"7F8EF87D3000","o":"36CE8","s":"abort"},{"b":"556C36CD7000","o":"811C4D","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"556C36CD7000","o":"1298346"},{"b":"556C36CD7000","o":"81BF30"},{"b":"556C36CD7000","o":"81C024","s":"__wt_err"},{"b":"556C36CD7000","o":"81C27C","s":"__wt_panic"},{"b":"556C36CD7000","o":"1E8C13F"},{"b":"556C36CD7000","o":"1E8C68B"},{"b":"556C36CD7000","o":"1E88D9D"},{"b":"556C36CD7000","o":"1E8D867"},{"b":"556C36CD7000","o":"1EAAFB6"},{"b":"556C36CD7000","o":"1EE163B"},{"b":"556C36CD7000","o":"1F6E45D"},{"b":"556C36CD7000","o":"1F6EB58"},{"b":"556C36CD7000","o":"1F6F06C"},{"b":"556C36CD7000","o":"1EF1421"},{"b":"556C36CD7000","o":"1F63E90"},{"b":"556C36CD7000","o":"1F0B5DB"},{"b":"556C36CD7000","o":"1F63C44"},{"b":"556C36CD7000","o":"1F64455"},{"b":"556C36CD7000","o":"1291877","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},{"b":"556C36CD7000","o":"128F61C","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},{"b":"556C36CD7000","o":"1269EF7","s":"_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb"},{"b":"556C36CD7000","o":"C6D487","s":"_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE"},{"b":"556C36CD7000","o":"B9CCDA","s":"_ZN5mongo11IDHackStage6doWorkEPm"},{"b":"556C36CD7000","o":"BAE703","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"556C36CD7000","o":"BE2533","s":"_ZN5mongo11UpdateStage6doWorkEPm"},{"b":"556C36CD7000","o":"BAE703","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"556C36CD7000","o":"EB16BA","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"556C36CD7000","o":"EB1FDB","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"556C36CD7000","o":"EB210D","s":"_ZN5mongo12PlanExecutor11executePlanEv"},{"b":"556C36CD7000","o":"D09635","s":"_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestE"},{"b":"556C36CD7000","o":"F87E2A","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE"},{"b":"556C36CD7000","o":"106A584","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_"},{"b":"556C36CD7000","o":"1061002"},{"b":"556C36CD7000","o":"106400E","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_"},{"b":"556C36CD7000","o":"1064A6D","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"556C36CD7000","o":"1064B7D"},{"b":"556C36CD7000","o":"1068CF0","s":"_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE"},{"b":"556C36CD7000","o":"106A253","s":"_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE"},{"b":"556C36CD7000","o":"105F84F"},{"b":"556C36CD7000","o":"1060D4C"},{"b":"556C36CD7000","o":"1500D65","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"556C36CD7000","o":"1501890","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"556C36CD7000","o":"1502439","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"556C36CD7000","o":"1FFEBA0"},{"b":"7F8EF8B95000","o":"7DC5"},{"b":"7F8EF87D3000","o":"F6C9D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.23-31.54.amzn1.x86_64", "version" : "#1 SMP Tue Oct 18 22:02:09 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "556C36CD7000", "elfType" : 3, "buildId" : "887DF4C9600B8B003BDC24A15D8373ED1E3843E6" }, { "b" : "7FFE9D51B000", "elfType" : 3, "buildId" : "3C621354FA6866C1B7DBFFDA88CE59560C001BE1" }, { "b" : "7F8EF9ABB000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6AF827B6FD200DFDFE70B2BC8D66BBC9881E8817" }, { "b" : "7F8EF96D5000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "83F15DBCD0653F417E98354BC1EED6F96A758367" }, { "b" : "7F8EF94CD000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "2B3151901240D9E854E18E6D0B181C4D580ABA9C" }, { "b" : "7F8EF92C9000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "6335077ACD51527BE9F2F18451A88E2B7350C5B6" }, { "b" : "7F8EF8FC7000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6E343508D15886FE83C438DF4560CE40BEB64B56" }, { "b" : "7F8EF8DB1000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "3FD5F89DE59E124AB1419A0BD16775B4096E84FD" }, { "b" : "7F8EF8B95000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "0836319AA81CDFE97DA2666963F62DE6A2A61346" }, { "b" : "7F8EF87D3000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "5D38A77E8D79E98D717281031C39B9A341323BD1" }, { "b" : "7F8EF9D28000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "93D931BA041229929E5F099514B20E36A70BD651" }, { "b" : "7F8EF8587000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E203354E7F907ACC8C3028FE465541B666DCFBA0" }, { "b" : "7F8EF82A2000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D769C8FFAF8772FDA55031ABF2E167DF2207E378" }, { "b" : "7F8EF809F000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7F8EF7E6D000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6C2243D37143F7FD1E16ED1F6CE4D7F16C2D7EF1" }, { "b" : "7F8EF7C57000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F8EF7A48000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "744272FEAAABCE629AB9E11FAA4A96AEBE8BC2B4" }, { "b" : "7F8EF7845000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F8EF762B000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "1285F9516FFCF13FC00BD135C5634AF2EB16C80B" }, { "b" : "7F8EF740A000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x556c38260c81] mongod(+0x1588D79) [0x556c3825fd79] mongod(+0x158925D) [0x556c3826025d] libpthread.so.0(+0xF100) [0x7f8ef8ba4100] libc.so.6(gsignal+0x37) [0x7f8ef88085f7] libc.so.6(abort+0x148) [0x7f8ef8809ce8] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x556c374e8c4d] mongod(+0x1298346) [0x556c37f6f346] mongod(+0x81BF30) [0x556c374f2f30] mongod(__wt_err+0x9D) [0x556c374f3024] mongod(__wt_panic+0x24) [0x556c374f327c] mongod(+0x1E8C13F) [0x556c38b6313f] mongod(+0x1E8C68B) [0x556c38b6368b] mongod(+0x1E88D9D) [0x556c38b5fd9d] mongod(+0x1E8D867) [0x556c38b64867] mongod(+0x1EAAFB6) [0x556c38b81fb6] mongod(+0x1EE163B) [0x556c38bb863b] mongod(+0x1F6E45D) [0x556c38c4545d] mongod(+0x1F6EB58) [0x556c38c45b58] mongod(+0x1F6F06C) [0x556c38c4606c] mongod(+0x1EF1421) [0x556c38bc8421] mongod(+0x1F63E90) [0x556c38c3ae90] mongod(+0x1F0B5DB) [0x556c38be25db] mongod(+0x1F63C44) [0x556c38c3ac44] mongod(+0x1F64455) [0x556c38c3b455] mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0x77) [0x556c37f68877] mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x556c37f6661c] mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb+0x157) [0x556c37f40ef7] mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0xF7) [0x556c37944487] mongod(_ZN5mongo11IDHackStage6doWorkEPm+0x14A) [0x556c37873cda] mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x556c37885703] mongod(_ZN5mongo11UpdateStage6doWorkEPm+0x3A3) [0x556c378b9533] mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x556c37885703] mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x19A) [0x556c37b886ba] mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x556c37b88fdb] mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x6D) [0x556c37b8910d] mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestE+0x175) [0x556c379e0635] mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE+0x86A) [0x556c37c5ee2a] mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_+0x74) [0x556c37d41584] mongod(+0x1061002) [0x556c37d38002] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_+0x51E) [0x556c37d3b00e] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xFD) [0x556c37d3ba6d] mongod(+0x1064B7D) [0x556c37d3bb7d] mongod(_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE+0x1C0) [0x556c37d3fcf0] mongod(_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE+0x73) [0x556c37d41253] mongod(+0x105F84F) [0x556c37d3684f] mongod(+0x1060D4C) [0x556c37d37d4c] mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x135) [0x556c381d7d65] mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xC0) [0x556c381d8890] mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x149) [0x556c381d9439] mongod(+0x1FFEBA0) [0x556c38cd5ba0] libpthread.so.0(+0x7DC5) [0x7f8ef8b9cdc5] libc.so.6(clone+0x6D) [0x7f8ef88c9c9d] ----- END BACKTRACE ----- 2017-03-01T16:39:10.979+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] MongoDB starting : pid=9783 port=27019 dbpath=/mongo/mongo3/data 64-bit host=ip-10-0-2-28 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] db version v3.4.2 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] git version: 3f76e40c105fc223b3e5aac3e20dcd026b83b38b 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] modules: none 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] build environment: 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] distmod: amazon 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] distarch: x86_64 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] target_arch: x86_64 2017-03-01T16:39:10.985+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod3.conf", net: { port: 27019 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod3.pid" }, replication: { replSetName: "thinkmap" }, storage: { dbPath: "/mongo/mongo3/data", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/mongo/mongo3/log/mongod.log" } } 2017-03-01T16:39:10.985+0000 W - [initandlisten] Detected unclean shutdown - /mongo/mongo3/data/mongod.lock is not empty. 2017-03-01T16:39:11.006+0000 I - [initandlisten] Detected data files in /mongo/mongo3/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2017-03-01T16:39:11.006+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2017-03-01T16:39:11.006+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1465M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2017-03-01T16:39:11.261+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2017-03-01T16:39:11.261+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 6799413 records totaling to 5147161877 bytes 2017-03-01T16:39:11.261+0000 I STORAGE [initandlisten] Sampling from the oplog between Jan 25 18:38:50:9e1 and Mar 1 15:55:00:17 to determine where to place markers for truncation 2017-03-01T16:39:11.261+0000 I STORAGE [initandlisten] Taking 1007 samples and assuming that each section of oplog contains approximately 67486 records totaling to 51086963 bytes 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:39:29:3b8 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:40:09:cd1 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:41:16:9c7 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:42:01:a6b 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:42:43:8d1 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:43:29:146c 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:33:01:c0c 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:35:24:1172 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:35:59:1b82 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:36:33:bc0 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:37:18:60b 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:37:54:10c4 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:38:38:14f8 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:39:17:ccb 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:40:03:306 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:40:41:f35 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:07:3b15 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:09:338a 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:13:9ce 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:17:1bf9 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:23:492 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:28:26d 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:30:3e04 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:34:c77 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:41:de 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:46:bd7 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:49:1a5e 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 21:56:58:c22 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 13:29:25:2b9 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:49:56:3e8 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:50:32:2e7 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:51:06:27c 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:51:38:87 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:52:12:1ab 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:52:45:fa 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:53:18:209 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:53:52:112 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:54:26:10b 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:54:58:f0 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:43:38:8d 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:44:29:188 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:45:22:7b 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:46:12:28c 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:47:04:28c 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:47:54:297 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:48:51:3cc 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:49:45:3bc 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:50:38:3ae 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:51:38:353 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 16:11:02:3d 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 16:59:43:4e 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 17:54:51:1b 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 18:42:20:39 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 19:29:44:53 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:53:bb5 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:54:1ae2 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:55:2b7f 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:56:36e7 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:57:487a 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:59:302 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:00:1c52 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:03:148a 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:04:1ca8 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:05:2877 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:07:240 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 13 13:05:18:1 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:52:2247 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:53:7d70 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:55:12e0 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:56:54c2 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:58:7c21 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:48:11:148a7 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:12:1be 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:13:1790 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:14:2721 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:15:33f5 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:16:4999 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:18:1572 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:19:29c3 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:22:f1d 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:23:287e 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:25:3d2 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:26:2288 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:58:7629 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:55:6ad0 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:56:3804 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:57:4876 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:58:2097 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:59:1bb4 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:00:8b3 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:01:ae9 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:02:4d2 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:03:1a06 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:04:23c1 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:07:246d 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:05:14e3 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:06:a142 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:45:30ab 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:46:4d3a 2017-03-01T16:39:11.878+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 23:52:33:1 2017-03-01T16:39:12.288+0000 I CONTROL [initandlisten] 2017-03-01T16:39:12.288+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2017-03-01T16:39:12.288+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2017-03-01T16:39:12.288+0000 I CONTROL [initandlisten] 2017-03-01T16:39:12.399+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/mongo/mongo3/data/diagnostic.data' 2017-03-01T16:39:12.400+0000 I NETWORK [thread1] waiting for connections on port 27019 2017-03-01T16:39:12.402+0000 I REPL [replExecDBWorker-1] New replica set config in use: { _id: "thinkmap", version: 67868, protocolVersion: 1, members: [ { _id: 3, host: "10.0.2.28:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 30.0, tags: { use: "general" }, slaveDelay: 0, votes: 1 }, { _id: 4, host: "10.0.2.28:27018", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 20.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 5, host: "10.0.2.28:27019", arbiterOnly: false, buildIndexes: true, hidden: true, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2017-03-01T16:39:12.402+0000 I REPL [replExecDBWorker-1] This node is 10.0.2.28:27019 in the config 2017-03-01T16:39:12.402+0000 I REPL [replExecDBWorker-1] transition to STARTUP2 2017-03-01T16:39:12.402+0000 I REPL [replExecDBWorker-1] Starting replication storage threads 2017-03-01T16:39:12.402+0000 I REPL [replExecDBWorker-1] Starting replication fetcher thread 2017-03-01T16:39:12.402+0000 I REPL [replExecDBWorker-1] Starting replication applier thread 2017-03-01T16:39:12.402+0000 I REPL [replExecDBWorker-1] Starting replication reporter thread 2017-03-01T16:39:12.402+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27017 2017-03-01T16:39:12.402+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27018 2017-03-01T16:39:12.402+0000 I REPL [rsSync] transition to RECOVERING 2017-03-01T16:39:12.403+0000 I REPL [rsSync] transition to SECONDARY 2017-03-01T16:39:12.403+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27017 2017-03-01T16:39:12.403+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27018 2017-03-01T16:39:12.404+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27017 is now in state PRIMARY 2017-03-01T16:39:12.404+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27018 is now in state SECONDARY 2017-03-01T16:39:13.023+0000 I FTDC [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK 2017-03-01T16:39:13.591+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:38706 #1 (1 connection now open) 2017-03-01T16:39:13.592+0000 I NETWORK [conn1] received client metadata from 10.0.2.28:38706 conn1: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T16:39:14.329+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:38708 #2 (2 connections now open) 2017-03-01T16:39:14.329+0000 I NETWORK [conn2] received client metadata from 10.0.2.28:38708 conn2: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T16:39:17.382+0000 I - [conn1] end connection 10.0.2.28:38706 (2 connections now open) 2017-03-01T16:39:17.404+0000 I ASIO [ReplicationExecutor] dropping unhealthy pooled connection to 10.0.2.28:27018 2017-03-01T16:39:17.404+0000 I ASIO [ReplicationExecutor] after drop, pool was empty, going to spawn some connections 2017-03-01T16:39:17.404+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27018 2017-03-01T16:39:17.404+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 10.0.2.28:27018 - HostUnreachable: Connection refused 2017-03-01T16:39:17.404+0000 I REPL [ReplicationExecutor] Error in heartbeat request to 10.0.2.28:27018; HostUnreachable: Connection refused 2017-03-01T16:39:17.404+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27018 2017-03-01T16:39:17.404+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 10.0.2.28:27018 - HostUnreachable: Connection refused 2017-03-01T16:39:17.404+0000 I REPL [ReplicationExecutor] Error in heartbeat request to 10.0.2.28:27018; HostUnreachable: Connection refused 2017-03-01T16:39:17.404+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27018 2017-03-01T16:39:17.404+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to 10.0.2.28:27018 - HostUnreachable: Connection refused 2017-03-01T16:39:17.404+0000 I REPL [ReplicationExecutor] Error in heartbeat request to 10.0.2.28:27018; HostUnreachable: Connection refused 2017-03-01T16:39:19.034+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:38724 #3 (2 connections now open) 2017-03-01T16:39:19.035+0000 I - [conn3] end connection 10.0.2.28:38724 (2 connections now open) 2017-03-01T16:39:19.036+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:38728 #4 (2 connections now open) 2017-03-01T16:39:19.037+0000 I NETWORK [conn4] received client metadata from 10.0.2.28:38728 conn4: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T16:39:22.404+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27018 2017-03-01T16:39:22.404+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27018 2017-03-01T16:39:22.404+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27018 is now in state SECONDARY 2017-03-01T16:39:22.405+0000 I REPL [rsBackgroundSync] sync source candidate: 10.0.2.28:27018 2017-03-01T16:39:22.405+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T16:39:22.406+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T16:39:22.406+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T16:39:22.406+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T16:39:22.408+0000 E STORAGE [repl writer worker 7] WiredTiger error (0) [1488386362:408202][9783:0x7fde825da700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 45056: block header checksum of 3605474371 doesn't match expected checksum of 3806882032 2017-03-01T16:39:22.408+0000 E STORAGE [repl writer worker 7] WiredTiger error (0) [1488386362:408228][9783:0x7fde825da700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: index-14-4040745961588520825.wt: encountered an illegal file format or internal value 2017-03-01T16:39:22.408+0000 E STORAGE [repl writer worker 7] WiredTiger error (-31804) [1488386362:408245][9783:0x7fde825da700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic 2017-03-01T16:39:22.408+0000 I - [repl writer worker 7] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2017-03-01T16:39:22.408+0000 I - [repl writer worker 7] ***aborting after fassert() failure 2017-03-01T16:39:22.408+0000 I - [repl writer worker 15] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.408+0000 I - [repl writer worker 15] ***aborting after fassert() failure 2017-03-01T16:39:22.408+0000 I - [repl writer worker 10] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.408+0000 I - [repl writer worker 10] ***aborting after fassert() failure 2017-03-01T16:39:22.408+0000 I - [repl writer worker 9] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.408+0000 I - [repl writer worker 9] ***aborting after fassert() failure 2017-03-01T16:39:22.409+0000 I - [repl writer worker 2] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.409+0000 I - [repl writer worker 2] ***aborting after fassert() failure 2017-03-01T16:39:22.409+0000 I - [repl writer worker 3] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.410+0000 I - [repl writer worker 3] ***aborting after fassert() failure 2017-03-01T16:39:22.410+0000 I - [repl writer worker 4] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.410+0000 I - [repl writer worker 4] ***aborting after fassert() failure 2017-03-01T16:39:22.410+0000 I - [repl writer worker 6] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.410+0000 I - [repl writer worker 6] ***aborting after fassert() failure 2017-03-01T16:39:22.410+0000 I - [repl writer worker 1] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.410+0000 I - [repl writer worker 1] ***aborting after fassert() failure 2017-03-01T16:39:22.410+0000 I - [repl writer worker 0] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.410+0000 I - [repl writer worker 0] ***aborting after fassert() failure 2017-03-01T16:39:22.408+0000 I - [repl writer worker 13] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.410+0000 I - [repl writer worker 13] ***aborting after fassert() failure 2017-03-01T16:39:22.410+0000 I - [repl writer worker 5] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:39:22.410+0000 I - [repl writer worker 5] ***aborting after fassert() failure 2017-03-01T16:39:22.437+0000 F - [repl writer worker 7] Got signal: 6 (Aborted). 0x55c80e9fdc81 0x55c80e9fcd79 0x55c80e9fd25d 0x7fde9e5aa100 0x7fde9e20e5f7 0x7fde9e20fce8 0x55c80dc85c4d 0x55c80e70c346 0x55c80dc8ff30 0x55c80dc90024 0x55c80dc9027c 0x55c80f30013f 0x55c80f30068b 0x55c80f2fcd9d 0x55c80f301867 0x55c80f31efb6 0x55c80f35563b 0x55c80f3e245d 0x55c80f3e2b58 0x55c80f3e306c 0x55c80f365421 0x55c80f3d7e90 0x55c80f37f5db 0x55c80f3d7c44 0x55c80f3d8455 0x55c80e705877 0x55c80e70361c 0x55c80e6ddef7 0x55c80e0e1487 0x55c80e010cda 0x55c80e022703 0x55c80e056533 0x55c80e022703 0x55c80e3256ba 0x55c80e325fdb 0x55c80e32610d 0x55c80e17d635 0x55c80e3fbe2a 0x55c80e4de584 0x55c80e4d5002 0x55c80e4d800e 0x55c80e4d8a6d 0x55c80e4d8b7d 0x55c80e4dccf0 0x55c80e4de253 0x55c80e4d384f 0x55c80e4d4d4c 0x55c80e974d65 0x55c80e975890 0x55c80e976439 0x55c80f472ba0 0x7fde9e5a2dc5 0x7fde9e2cfc9d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55C80D474000","o":"1589C81","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55C80D474000","o":"1588D79"},{"b":"55C80D474000","o":"158925D"},{"b":"7FDE9E59B000","o":"F100"},{"b":"7FDE9E1D9000","o":"355F7","s":"gsignal"},{"b":"7FDE9E1D9000","o":"36CE8","s":"abort"},{"b":"55C80D474000","o":"811C4D","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55C80D474000","o":"1298346"},{"b":"55C80D474000","o":"81BF30"},{"b":"55C80D474000","o":"81C024","s":"__wt_err"},{"b":"55C80D474000","o":"81C27C","s":"__wt_panic"},{"b":"55C80D474000","o":"1E8C13F"},{"b":"55C80D474000","o":"1E8C68B"},{"b":"55C80D474000","o":"1E88D9D"},{"b":"55C80D474000","o":"1E8D867"},{"b":"55C80D474000","o":"1EAAFB6"},{"b":"55C80D474000","o":"1EE163B"},{"b":"55C80D474000","o":"1F6E45D"},{"b":"55C80D474000","o":"1F6EB58"},{"b":"55C80D474000","o":"1F6F06C"},{"b":"55C80D474000","o":"1EF1421"},{"b":"55C80D474000","o":"1F63E90"},{"b":"55C80D474000","o":"1F0B5DB"},{"b":"55C80D474000","o":"1F63C44"},{"b":"55C80D474000","o":"1F64455"},{"b":"55C80D474000","o":"1291877","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},{"b":"55C80D474000","o":"128F61C","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},{"b":"55C80D474000","o":"1269EF7","s":"_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb"},{"b":"55C80D474000","o":"C6D487","s":"_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE"},{"b":"55C80D474000","o":"B9CCDA","s":"_ZN5mongo11IDHackStage6doWorkEPm"},{"b":"55C80D474000","o":"BAE703","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"55C80D474000","o":"BE2533","s":"_ZN5mongo11UpdateStage6doWorkEPm"},{"b":"55C80D474000","o":"BAE703","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"55C80D474000","o":"EB16BA","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"55C80D474000","o":"EB1FDB","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"55C80D474000","o":"EB210D","s":"_ZN5mongo12PlanExecutor11executePlanEv"},{"b":"55C80D474000","o":"D09635","s":"_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestE"},{"b":"55C80D474000","o":"F87E2A","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE"},{"b":"55C80D474000","o":"106A584","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_"},{"b":"55C80D474000","o":"1061002"},{"b":"55C80D474000","o":"106400E","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_"},{"b":"55C80D474000","o":"1064A6D","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"55C80D474000","o":"1064B7D"},{"b":"55C80D474000","o":"1068CF0","s":"_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE"},{"b":"55C80D474000","o":"106A253","s":"_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE"},{"b":"55C80D474000","o":"105F84F"},{"b":"55C80D474000","o":"1060D4C"},{"b":"55C80D474000","o":"1500D65","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"55C80D474000","o":"1501890","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"55C80D474000","o":"1502439","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"55C80D474000","o":"1FFEBA0"},{"b":"7FDE9E59B000","o":"7DC5"},{"b":"7FDE9E1D9000","o":"F6C9D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.23-31.54.amzn1.x86_64", "version" : "#1 SMP Tue Oct 18 22:02:09 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "55C80D474000", "elfType" : 3, "buildId" : "887DF4C9600B8B003BDC24A15D8373ED1E3843E6" }, { "b" : "7FFDE7DF8000", "elfType" : 3, "buildId" : "3C621354FA6866C1B7DBFFDA88CE59560C001BE1" }, { "b" : "7FDE9F4C1000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6AF827B6FD200DFDFE70B2BC8D66BBC9881E8817" }, { "b" : "7FDE9F0DB000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "83F15DBCD0653F417E98354BC1EED6F96A758367" }, { "b" : "7FDE9EED3000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "2B3151901240D9E854E18E6D0B181C4D580ABA9C" }, { "b" : "7FDE9ECCF000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "6335077ACD51527BE9F2F18451A88E2B7350C5B6" }, { "b" : "7FDE9E9CD000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6E343508D15886FE83C438DF4560CE40BEB64B56" }, { "b" : "7FDE9E7B7000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "3FD5F89DE59E124AB1419A0BD16775B4096E84FD" }, { "b" : "7FDE9E59B000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "0836319AA81CDFE97DA2666963F62DE6A2A61346" }, { "b" : "7FDE9E1D9000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "5D38A77E8D79E98D717281031C39B9A341323BD1" }, { "b" : "7FDE9F72E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "93D931BA041229929E5F099514B20E36A70BD651" }, { "b" : "7FDE9DF8D000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E203354E7F907ACC8C3028FE465541B666DCFBA0" }, { "b" : "7FDE9DCA8000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D769C8FFAF8772FDA55031ABF2E167DF2207E378" }, { "b" : "7FDE9DAA5000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7FDE9D873000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6C2243D37143F7FD1E16ED1F6CE4D7F16C2D7EF1" }, { "b" : "7FDE9D65D000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7FDE9D44E000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "744272FEAAABCE629AB9E11FAA4A96AEBE8BC2B4" }, { "b" : "7FDE9D24B000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7FDE9D031000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "1285F9516FFCF13FC00BD135C5634AF2EB16C80B" }, { "b" : "7FDE9CE10000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55c80e9fdc81] mongod(+0x1588D79) [0x55c80e9fcd79] mongod(+0x158925D) [0x55c80e9fd25d] libpthread.so.0(+0xF100) [0x7fde9e5aa100] libc.so.6(gsignal+0x37) [0x7fde9e20e5f7] libc.so.6(abort+0x148) [0x7fde9e20fce8] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55c80dc85c4d] mongod(+0x1298346) [0x55c80e70c346] mongod(+0x81BF30) [0x55c80dc8ff30] mongod(__wt_err+0x9D) [0x55c80dc90024] mongod(__wt_panic+0x24) [0x55c80dc9027c] mongod(+0x1E8C13F) [0x55c80f30013f] mongod(+0x1E8C68B) [0x55c80f30068b] mongod(+0x1E88D9D) [0x55c80f2fcd9d] mongod(+0x1E8D867) [0x55c80f301867] mongod(+0x1EAAFB6) [0x55c80f31efb6] mongod(+0x1EE163B) [0x55c80f35563b] mongod(+0x1F6E45D) [0x55c80f3e245d] mongod(+0x1F6EB58) [0x55c80f3e2b58] mongod(+0x1F6F06C) [0x55c80f3e306c] mongod(+0x1EF1421) [0x55c80f365421] mongod(+0x1F63E90) [0x55c80f3d7e90] mongod(+0x1F0B5DB) [0x55c80f37f5db] mongod(+0x1F63C44) [0x55c80f3d7c44] mongod(+0x1F64455) [0x55c80f3d8455] mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0x77) [0x55c80e705877] mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x55c80e70361c] mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb+0x157) [0x55c80e6ddef7] mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0xF7) [0x55c80e0e1487] mongod(_ZN5mongo11IDHackStage6doWorkEPm+0x14A) [0x55c80e010cda] mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x55c80e022703] mongod(_ZN5mongo11UpdateStage6doWorkEPm+0x3A3) [0x55c80e056533] mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x55c80e022703] mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x19A) [0x55c80e3256ba] mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x55c80e325fdb] mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x6D) [0x55c80e32610d] mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestE+0x175) [0x55c80e17d635] mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE+0x86A) [0x55c80e3fbe2a] mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_+0x74) [0x55c80e4de584] mongod(+0x1061002) [0x55c80e4d5002] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_+0x51E) [0x55c80e4d800e] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xFD) [0x55c80e4d8a6d] mongod(+0x1064B7D) [0x55c80e4d8b7d] mongod(_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE+0x1C0) [0x55c80e4dccf0] mongod(_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE+0x73) [0x55c80e4de253] mongod(+0x105F84F) [0x55c80e4d384f] mongod(+0x1060D4C) [0x55c80e4d4d4c] mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x135) [0x55c80e974d65] mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xC0) [0x55c80e975890] mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x149) [0x55c80e976439] mongod(+0x1FFEBA0) [0x55c80f472ba0] libpthread.so.0(+0x7DC5) [0x7fde9e5a2dc5] libc.so.6(clone+0x6D) [0x7fde9e2cfc9d] ----- END BACKTRACE ----- 2017-03-01T16:47:37.902+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] MongoDB starting : pid=10186 port=27019 dbpath=/mongo/mongo3/data 64-bit host=ip-10-0-2-28 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] db version v3.4.2 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] git version: 3f76e40c105fc223b3e5aac3e20dcd026b83b38b 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] modules: none 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] build environment: 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] distmod: amazon 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] distarch: x86_64 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] target_arch: x86_64 2017-03-01T16:47:37.908+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod3.conf", net: { port: 27019 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod3.pid" }, replication: { replSetName: "thinkmap" }, storage: { dbPath: "/mongo/mongo3/data", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/mongo/mongo3/log/mongod.log" } } 2017-03-01T16:47:37.908+0000 W - [initandlisten] Detected unclean shutdown - /mongo/mongo3/data/mongod.lock is not empty. 2017-03-01T16:47:37.929+0000 I - [initandlisten] Detected data files in /mongo/mongo3/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2017-03-01T16:47:37.929+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2017-03-01T16:47:37.929+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1465M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2017-03-01T16:47:38.122+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2017-03-01T16:47:38.122+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 6799413 records totaling to 5147161877 bytes 2017-03-01T16:47:38.122+0000 I STORAGE [initandlisten] Sampling from the oplog between Jan 25 18:38:50:9e1 and Mar 1 15:55:00:17 to determine where to place markers for truncation 2017-03-01T16:47:38.122+0000 I STORAGE [initandlisten] Taking 1007 samples and assuming that each section of oplog contains approximately 67486 records totaling to 51086963 bytes 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:39:25:59f 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:40:08:86a 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:41:05:c67 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:42:00:20fc 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:42:39:293 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:43:29:3ec 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:32:57:11c0 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:35:24:169 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:35:54:2f6 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:36:32:f4e 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:37:12:1a4 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:37:53:113a 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:38:33:1156 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:39:16:e7b 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:39:58:ca9 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:40:40:1784 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:07:2a7b 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:09:3250 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:12:204c 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:16:1a44 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:18:c7f 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:23:5ff 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:25:1ad5 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:30:2221 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:33:1d9b 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:39:82f 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:43:1b38 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:48:1108 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 21:54:42:942 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 13:29:05:808 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:49:45:4b 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:50:23:89 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:50:58:92 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:51:28:2f3 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:52:02:250 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:52:35:220 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:53:08:1d8 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:53:43:210 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:54:15:272 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:54:49:bd 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:43:23:211 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:44:16:29b 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:45:04:2e7 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:46:00:156 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:46:49:1cb 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:47:41:37a 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:48:32:361 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:49:34:9b 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:50:24:5a 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:51:24:253 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:52:15:11c 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 16:46:01:4 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 17:39:45:26 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 18:30:12:39 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 19:17:20:3f 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:52:12d6 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:54:1d0 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:55:d52 2017-03-01T16:47:38.734+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:56:1d43 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:57:2cfb 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:58:44ed 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:00:234 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:01:1f8c 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:04:48f 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:05:108d 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:06:295f 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 10 10:25:38:1 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 21 09:38:13:1 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:53:4b6f 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:54:6ee3 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:56:8c1 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:58:199f 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:48:10:15db 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:48:14:bf1e 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:12:81db 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:14:7f9 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:15:1905 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:16:2cc3 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:17:4660 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:19:ead 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:21:21d0 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:23:1493 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:25:ede 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:26:424e 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:55:542d 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:56:327a 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:57:41e8 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:58:1b04 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:59:15bf 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:00:2df 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:01:4e8 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:01:4383 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:03:13aa 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:04:1d48 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:07:1d9d 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 13:37:58:1 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:06:7fd6 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:45:1dc1 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:46:3977 2017-03-01T16:47:38.735+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:47:526a 2017-03-01T16:47:39.151+0000 I CONTROL [initandlisten] 2017-03-01T16:47:39.151+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2017-03-01T16:47:39.151+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2017-03-01T16:47:39.151+0000 I CONTROL [initandlisten] 2017-03-01T16:47:39.261+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/mongo/mongo3/data/diagnostic.data' 2017-03-01T16:47:39.263+0000 I NETWORK [thread1] waiting for connections on port 27019 2017-03-01T16:47:39.264+0000 I REPL [replExecDBWorker-1] New replica set config in use: { _id: "thinkmap", version: 67868, protocolVersion: 1, members: [ { _id: 3, host: "10.0.2.28:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 30.0, tags: { use: "general" }, slaveDelay: 0, votes: 1 }, { _id: 4, host: "10.0.2.28:27018", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 20.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 5, host: "10.0.2.28:27019", arbiterOnly: false, buildIndexes: true, hidden: true, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2017-03-01T16:47:39.264+0000 I REPL [replExecDBWorker-1] This node is 10.0.2.28:27019 in the config 2017-03-01T16:47:39.264+0000 I REPL [replExecDBWorker-1] transition to STARTUP2 2017-03-01T16:47:39.264+0000 I REPL [replExecDBWorker-1] Starting replication storage threads 2017-03-01T16:47:39.264+0000 I REPL [replExecDBWorker-1] Starting replication fetcher thread 2017-03-01T16:47:39.264+0000 I REPL [replExecDBWorker-1] Starting replication applier thread 2017-03-01T16:47:39.264+0000 I REPL [replExecDBWorker-1] Starting replication reporter thread 2017-03-01T16:47:39.265+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27017 2017-03-01T16:47:39.265+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27018 2017-03-01T16:47:39.265+0000 I REPL [rsSync] transition to RECOVERING 2017-03-01T16:47:39.265+0000 I REPL [rsSync] transition to SECONDARY 2017-03-01T16:47:39.266+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27018 2017-03-01T16:47:39.266+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27018 is now in state SECONDARY 2017-03-01T16:47:39.266+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27017 2017-03-01T16:47:39.266+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27017 is now in state PRIMARY 2017-03-01T16:47:40.602+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:41680 #1 (1 connection now open) 2017-03-01T16:47:40.602+0000 I NETWORK [conn1] received client metadata from 10.0.2.28:41680 conn1: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T16:47:41.201+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:41682 #2 (2 connections now open) 2017-03-01T16:47:41.201+0000 I NETWORK [conn2] received client metadata from 10.0.2.28:41682 conn2: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T16:47:45.266+0000 I REPL [rsBackgroundSync] sync source candidate: 10.0.2.28:27018 2017-03-01T16:47:45.266+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T16:47:45.267+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T16:47:45.267+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T16:47:45.267+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T16:47:45.269+0000 E STORAGE [repl writer worker 6] WiredTiger error (0) [1488386865:269527][10186:0x7f3ae803b700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 45056: block header checksum of 3605474371 doesn't match expected checksum of 3806882032 2017-03-01T16:47:45.269+0000 E STORAGE [repl writer worker 6] WiredTiger error (0) [1488386865:269552][10186:0x7f3ae803b700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: index-14-4040745961588520825.wt: encountered an illegal file format or internal value 2017-03-01T16:47:45.269+0000 E STORAGE [repl writer worker 6] WiredTiger error (-31804) [1488386865:269561][10186:0x7f3ae803b700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic 2017-03-01T16:47:45.269+0000 I - [repl writer worker 6] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2017-03-01T16:47:45.269+0000 I - [repl writer worker 6] ***aborting after fassert() failure 2017-03-01T16:47:45.269+0000 I - [repl writer worker 10] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:47:45.269+0000 I - [repl writer worker 10] ***aborting after fassert() failure 2017-03-01T16:47:45.269+0000 I - [repl writer worker 8] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:47:45.269+0000 I - [repl writer worker 8] ***aborting after fassert() failure 2017-03-01T16:47:45.269+0000 I - [repl writer worker 0] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:47:45.269+0000 I - [repl writer worker 0] ***aborting after fassert() failure 2017-03-01T16:47:45.269+0000 I - [repl writer worker 9] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:47:45.269+0000 I - [repl writer worker 9] ***aborting after fassert() failure 2017-03-01T16:47:45.269+0000 I - [repl writer worker 12] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:47:45.269+0000 I - [repl writer worker 12] ***aborting after fassert() failure 2017-03-01T16:47:45.269+0000 I - [repl writer worker 7] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:47:45.269+0000 I - [repl writer worker 7] ***aborting after fassert() failure 2017-03-01T16:47:45.269+0000 I - [repl writer worker 5] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:47:45.269+0000 I - [repl writer worker 5] ***aborting after fassert() failure 2017-03-01T16:47:45.269+0000 I - [repl writer worker 11] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T16:47:45.269+0000 I - [repl writer worker 11] ***aborting after fassert() failure 2017-03-01T16:47:45.299+0000 F - [repl writer worker 6] Got signal: 6 (Aborted). 0x55747235ac81 0x557472359d79 0x55747235a25d 0x7f3b0380a100 0x7f3b0346e5f7 0x7f3b0346fce8 0x5574715e2c4d 0x557472069346 0x5574715ecf30 0x5574715ed024 0x5574715ed27c 0x557472c5d13f 0x557472c5d68b 0x557472c59d9d 0x557472c5e867 0x557472c7bfb6 0x557472cb263b 0x557472d3f45d 0x557472d3fb58 0x557472d4006c 0x557472cc2421 0x557472d34e90 0x557472cdc5db 0x557472d34c44 0x557472d35455 0x557472062877 0x55747206061c 0x55747203aef7 0x557471a3e487 0x55747196dcda 0x55747197f703 0x5574719b3533 0x55747197f703 0x557471c826ba 0x557471c82fdb 0x557471c8310d 0x557471ada635 0x557471d58e2a 0x557471e3b584 0x557471e32002 0x557471e3500e 0x557471e35a6d 0x557471e35b7d 0x557471e39cf0 0x557471e3b253 0x557471e3084f 0x557471e31d4c 0x5574722d1d65 0x5574722d2890 0x5574722d3439 0x557472dcfba0 0x7f3b03802dc5 0x7f3b0352fc9d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"557470DD1000","o":"1589C81","s":"_ZN5mongo15printStackTraceERSo"},{"b":"557470DD1000","o":"1588D79"},{"b":"557470DD1000","o":"158925D"},{"b":"7F3B037FB000","o":"F100"},{"b":"7F3B03439000","o":"355F7","s":"gsignal"},{"b":"7F3B03439000","o":"36CE8","s":"abort"},{"b":"557470DD1000","o":"811C4D","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"557470DD1000","o":"1298346"},{"b":"557470DD1000","o":"81BF30"},{"b":"557470DD1000","o":"81C024","s":"__wt_err"},{"b":"557470DD1000","o":"81C27C","s":"__wt_panic"},{"b":"557470DD1000","o":"1E8C13F"},{"b":"557470DD1000","o":"1E8C68B"},{"b":"557470DD1000","o":"1E88D9D"},{"b":"557470DD1000","o":"1E8D867"},{"b":"557470DD1000","o":"1EAAFB6"},{"b":"557470DD1000","o":"1EE163B"},{"b":"557470DD1000","o":"1F6E45D"},{"b":"557470DD1000","o":"1F6EB58"},{"b":"557470DD1000","o":"1F6F06C"},{"b":"557470DD1000","o":"1EF1421"},{"b":"557470DD1000","o":"1F63E90"},{"b":"557470DD1000","o":"1F0B5DB"},{"b":"557470DD1000","o":"1F63C44"},{"b":"557470DD1000","o":"1F64455"},{"b":"557470DD1000","o":"1291877","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},{"b":"557470DD1000","o":"128F61C","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},{"b":"557470DD1000","o":"1269EF7","s":"_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb"},{"b":"557470DD1000","o":"C6D487","s":"_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE"},{"b":"557470DD1000","o":"B9CCDA","s":"_ZN5mongo11IDHackStage6doWorkEPm"},{"b":"557470DD1000","o":"BAE703","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"557470DD1000","o":"BE2533","s":"_ZN5mongo11UpdateStage6doWorkEPm"},{"b":"557470DD1000","o":"BAE703","s":"_ZN5mongo9PlanStage4workEPm"},{"b":"557470DD1000","o":"EB16BA","s":"_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE"},{"b":"557470DD1000","o":"EB1FDB","s":"_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE"},{"b":"557470DD1000","o":"EB210D","s":"_ZN5mongo12PlanExecutor11executePlanEv"},{"b":"557470DD1000","o":"D09635","s":"_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestE"},{"b":"557470DD1000","o":"F87E2A","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE"},{"b":"557470DD1000","o":"106A584","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_"},{"b":"557470DD1000","o":"1061002"},{"b":"557470DD1000","o":"106400E","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_"},{"b":"557470DD1000","o":"1064A6D","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"557470DD1000","o":"1064B7D"},{"b":"557470DD1000","o":"1068CF0","s":"_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE"},{"b":"557470DD1000","o":"106A253","s":"_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE"},{"b":"557470DD1000","o":"105F84F"},{"b":"557470DD1000","o":"1060D4C"},{"b":"557470DD1000","o":"1500D65","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"557470DD1000","o":"1501890","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"557470DD1000","o":"1502439","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"557470DD1000","o":"1FFEBA0"},{"b":"7F3B037FB000","o":"7DC5"},{"b":"7F3B03439000","o":"F6C9D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.23-31.54.amzn1.x86_64", "version" : "#1 SMP Tue Oct 18 22:02:09 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "557470DD1000", "elfType" : 3, "buildId" : "887DF4C9600B8B003BDC24A15D8373ED1E3843E6" }, { "b" : "7FFD89179000", "elfType" : 3, "buildId" : "3C621354FA6866C1B7DBFFDA88CE59560C001BE1" }, { "b" : "7F3B04721000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6AF827B6FD200DFDFE70B2BC8D66BBC9881E8817" }, { "b" : "7F3B0433B000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "83F15DBCD0653F417E98354BC1EED6F96A758367" }, { "b" : "7F3B04133000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "2B3151901240D9E854E18E6D0B181C4D580ABA9C" }, { "b" : "7F3B03F2F000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "6335077ACD51527BE9F2F18451A88E2B7350C5B6" }, { "b" : "7F3B03C2D000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6E343508D15886FE83C438DF4560CE40BEB64B56" }, { "b" : "7F3B03A17000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "3FD5F89DE59E124AB1419A0BD16775B4096E84FD" }, { "b" : "7F3B037FB000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "0836319AA81CDFE97DA2666963F62DE6A2A61346" }, { "b" : "7F3B03439000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "5D38A77E8D79E98D717281031C39B9A341323BD1" }, { "b" : "7F3B0498E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "93D931BA041229929E5F099514B20E36A70BD651" }, { "b" : "7F3B031ED000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E203354E7F907ACC8C3028FE465541B666DCFBA0" }, { "b" : "7F3B02F08000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D769C8FFAF8772FDA55031ABF2E167DF2207E378" }, { "b" : "7F3B02D05000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7F3B02AD3000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6C2243D37143F7FD1E16ED1F6CE4D7F16C2D7EF1" }, { "b" : "7F3B028BD000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F3B026AE000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "744272FEAAABCE629AB9E11FAA4A96AEBE8BC2B4" }, { "b" : "7F3B024AB000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F3B02291000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "1285F9516FFCF13FC00BD135C5634AF2EB16C80B" }, { "b" : "7F3B02070000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55747235ac81] mongod(+0x1588D79) [0x557472359d79] mongod(+0x158925D) [0x55747235a25d] libpthread.so.0(+0xF100) [0x7f3b0380a100] libc.so.6(gsignal+0x37) [0x7f3b0346e5f7] libc.so.6(abort+0x148) [0x7f3b0346fce8] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x5574715e2c4d] mongod(+0x1298346) [0x557472069346] mongod(+0x81BF30) [0x5574715ecf30] mongod(__wt_err+0x9D) [0x5574715ed024] mongod(__wt_panic+0x24) [0x5574715ed27c] mongod(+0x1E8C13F) [0x557472c5d13f] mongod(+0x1E8C68B) [0x557472c5d68b] mongod(+0x1E88D9D) [0x557472c59d9d] mongod(+0x1E8D867) [0x557472c5e867] mongod(+0x1EAAFB6) [0x557472c7bfb6] mongod(+0x1EE163B) [0x557472cb263b] mongod(+0x1F6E45D) [0x557472d3f45d] mongod(+0x1F6EB58) [0x557472d3fb58] mongod(+0x1F6F06C) [0x557472d4006c] mongod(+0x1EF1421) [0x557472cc2421] mongod(+0x1F63E90) [0x557472d34e90] mongod(+0x1F0B5DB) [0x557472cdc5db] mongod(+0x1F63C44) [0x557472d34c44] mongod(+0x1F64455) [0x557472d35455] mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0x77) [0x557472062877] mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x55747206061c] mongod(_ZNK5mongo21WiredTigerIndexUnique9newCursorEPNS_16OperationContextEb+0x157) [0x55747203aef7] mongod(_ZNK5mongo17IndexAccessMethod10findSingleEPNS_16OperationContextERKNS_7BSONObjE+0xF7) [0x557471a3e487] mongod(_ZN5mongo11IDHackStage6doWorkEPm+0x14A) [0x55747196dcda] mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x55747197f703] mongod(_ZN5mongo11UpdateStage6doWorkEPm+0x3A3) [0x5574719b3533] mongod(_ZN5mongo9PlanStage4workEPm+0x63) [0x55747197f703] mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x19A) [0x557471c826ba] mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x4B) [0x557471c82fdb] mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x6D) [0x557471c8310d] mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestE+0x175) [0x557471ada635] mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE+0x86A) [0x557471d58e2a] mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_+0x74) [0x557471e3b584] mongod(+0x1061002) [0x557471e32002] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_+0x51E) [0x557471e3500e] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xFD) [0x557471e35a6d] mongod(+0x1064B7D) [0x557471e35b7d] mongod(_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE+0x1C0) [0x557471e39cf0] mongod(_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE+0x73) [0x557471e3b253] mongod(+0x105F84F) [0x557471e3084f] mongod(+0x1060D4C) [0x557471e31d4c] mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x135) [0x5574722d1d65] mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xC0) [0x5574722d2890] mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x149) [0x5574722d3439] mongod(+0x1FFEBA0) [0x557472dcfba0] libpthread.so.0(+0x7DC5) [0x7f3b03802dc5] libc.so.6(clone+0x6D) [0x7f3b0352fc9d] ----- END BACKTRACE ----- 2017-03-01T17:10:05.818+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] MongoDB starting : pid=10340 port=27019 dbpath=/mongo/mongo3/data 64-bit host=ip-10-0-2-28 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] db version v3.4.2 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] git version: 3f76e40c105fc223b3e5aac3e20dcd026b83b38b 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] modules: none 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] build environment: 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] distmod: amazon 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] distarch: x86_64 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] target_arch: x86_64 2017-03-01T17:10:05.824+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod3.conf", net: { port: 27019 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod3.pid" }, replication: { replSetName: "thinkmap" }, storage: { dbPath: "/mongo/mongo3/data", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, logRotate: "reopen", path: "/mongo/mongo3/log/mongod.log" } } 2017-03-01T17:10:05.824+0000 W - [initandlisten] Detected unclean shutdown - /mongo/mongo3/data/mongod.lock is not empty. 2017-03-01T17:10:05.845+0000 I - [initandlisten] Detected data files in /mongo/mongo3/data created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2017-03-01T17:10:05.845+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2017-03-01T17:10:05.845+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1465M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2017-03-01T17:10:06.048+0000 I STORAGE [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs 2017-03-01T17:10:06.048+0000 I STORAGE [initandlisten] The size storer reports that the oplog contains 6799413 records totaling to 5147161877 bytes 2017-03-01T17:10:06.048+0000 I STORAGE [initandlisten] Sampling from the oplog between Jan 25 18:38:50:9e1 and Mar 1 15:55:00:17 to determine where to place markers for truncation 2017-03-01T17:10:06.048+0000 I STORAGE [initandlisten] Taking 1007 samples and assuming that each section of oplog contains approximately 67486 records totaling to 51086963 bytes 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:39:25:59f 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:40:08:86a 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:41:05:c67 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:42:00:20fc 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:42:39:293 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 18:43:29:3ec 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:32:57:11c0 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:35:24:169 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:35:54:2f6 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:36:32:f4e 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:37:12:1a4 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:37:53:113a 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:38:33:1156 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:39:16:e7b 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:39:58:ca9 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 25 20:40:40:1784 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:07:2a7b 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:09:3250 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:12:204c 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:16:1a44 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:18:c7f 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:23:5ff 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:25:1ad5 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:30:2221 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:33:1d9b 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:39:82f 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:43:1b38 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 13:55:48:1108 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 26 21:54:42:942 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 13:29:05:808 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:49:45:4b 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:50:23:89 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:50:58:92 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:51:28:2f3 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:52:02:250 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:52:35:220 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:53:08:1d8 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:53:43:210 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:54:15:272 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 20:54:49:bd 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:43:23:211 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:44:16:29b 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:45:04:2e7 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:46:00:156 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:46:49:1cb 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:47:41:37a 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:48:32:361 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:49:34:9b 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:50:24:5a 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:51:24:253 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 30 21:52:15:11c 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 16:46:01:4 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 17:39:45:26 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 18:30:12:39 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Jan 31 19:17:20:3f 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:52:12d6 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:54:1d0 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:55:d52 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:56:1d43 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:57:2cfb 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:25:58:44ed 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:00:234 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:01:1f8c 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:04:48f 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:05:108d 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 6 22:26:06:295f 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 10 10:25:38:1 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 21 09:38:13:1 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:53:4b6f 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:54:6ee3 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:56:8c1 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:47:58:199f 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:48:10:15db 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:48:14:bf1e 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:12:81db 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:14:7f9 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:15:1905 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:16:2cc3 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:17:4660 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:19:ead 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:21:21d0 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:23:1493 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:25:ede 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:49:26:424e 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:55:542d 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:56:327a 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:57:41e8 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:58:1b04 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:50:59:15bf 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:00:2df 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:01:4e8 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:01:4383 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:03:13aa 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:04:1d48 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 22 21:51:07:1d9d 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 13:37:58:1 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:06:7fd6 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:45:1dc1 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:46:3977 2017-03-01T17:10:06.151+0000 I STORAGE [initandlisten] Placing a marker at optime Feb 23 21:27:47:526a 2017-03-01T17:10:06.567+0000 I CONTROL [initandlisten] 2017-03-01T17:10:06.567+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2017-03-01T17:10:06.567+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2017-03-01T17:10:06.568+0000 I CONTROL [initandlisten] 2017-03-01T17:10:06.677+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/mongo/mongo3/data/diagnostic.data' 2017-03-01T17:10:06.679+0000 I NETWORK [thread1] waiting for connections on port 27019 2017-03-01T17:10:06.680+0000 I REPL [replExecDBWorker-0] New replica set config in use: { _id: "thinkmap", version: 67868, protocolVersion: 1, members: [ { _id: 3, host: "10.0.2.28:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 30.0, tags: { use: "general" }, slaveDelay: 0, votes: 1 }, { _id: 4, host: "10.0.2.28:27018", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 20.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 5, host: "10.0.2.28:27019", arbiterOnly: false, buildIndexes: true, hidden: true, priority: 0.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 } } } 2017-03-01T17:10:06.680+0000 I REPL [replExecDBWorker-0] This node is 10.0.2.28:27019 in the config 2017-03-01T17:10:06.680+0000 I REPL [replExecDBWorker-0] transition to STARTUP2 2017-03-01T17:10:06.680+0000 I REPL [replExecDBWorker-0] Starting replication storage threads 2017-03-01T17:10:06.680+0000 I REPL [replExecDBWorker-0] Starting replication fetcher thread 2017-03-01T17:10:06.680+0000 I REPL [replExecDBWorker-0] Starting replication applier thread 2017-03-01T17:10:06.680+0000 I REPL [replExecDBWorker-0] Starting replication reporter thread 2017-03-01T17:10:06.680+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27017 2017-03-01T17:10:06.680+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to 10.0.2.28:27018 2017-03-01T17:10:06.680+0000 I REPL [rsSync] transition to RECOVERING 2017-03-01T17:10:06.680+0000 I REPL [rsSync] transition to SECONDARY 2017-03-01T17:10:06.681+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27018 2017-03-01T17:10:06.681+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27018 is now in state SECONDARY 2017-03-01T17:10:06.681+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Successfully connected to 10.0.2.28:27017 2017-03-01T17:10:06.681+0000 I REPL [ReplicationExecutor] Member 10.0.2.28:27017 is now in state PRIMARY 2017-03-01T17:10:07.104+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:49746 #1 (1 connection now open) 2017-03-01T17:10:07.104+0000 I NETWORK [conn1] received client metadata from 10.0.2.28:49746 conn1: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T17:10:07.731+0000 I NETWORK [thread1] connection accepted from 10.0.2.28:49748 #2 (2 connections now open) 2017-03-01T17:10:07.731+0000 I NETWORK [conn2] received client metadata from 10.0.2.28:49748 conn2: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.2" }, os: { type: "Linux", name: "Amazon Linux AMI release 2016.09", architecture: "x86_64", version: "Kernel 4.4.23-31.54.amzn1.x86_64" } } 2017-03-01T17:10:12.681+0000 I REPL [rsBackgroundSync] sync source candidate: 10.0.2.28:27018 2017-03-01T17:10:12.682+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T17:10:12.682+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T17:10:12.682+0000 I ASIO [NetworkInterfaceASIO-RS-0] Connecting to 10.0.2.28:27018 2017-03-01T17:10:12.683+0000 I ASIO [NetworkInterfaceASIO-RS-0] Successfully connected to 10.0.2.28:27018 2017-03-01T17:10:12.684+0000 E STORAGE [repl writer worker 14] WiredTiger error (0) [1488388212:684821][10340:0x7f12ffe85700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 45056: block header checksum of 3605474371 doesn't match expected checksum of 3806882032 2017-03-01T17:10:12.684+0000 E STORAGE [repl writer worker 14] WiredTiger error (0) [1488388212:684850][10340:0x7f12ffe85700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: index-14-4040745961588520825.wt: encountered an illegal file format or internal value 2017-03-01T17:10:12.684+0000 E STORAGE [repl writer worker 14] WiredTiger error (-31804) [1488388212:684857][10340:0x7f12ffe85700], file:index-14-4040745961588520825.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic 2017-03-01T17:10:12.684+0000 I - [repl writer worker 14] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2017-03-01T17:10:12.684+0000 I - [repl writer worker 1] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.684+0000 I - [repl writer worker 8] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.684+0000 I - [repl writer worker 8] ***aborting after fassert() failure 2017-03-01T17:10:12.684+0000 I - [repl writer worker 0] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.684+0000 I - [repl writer worker 0] ***aborting after fassert() failure 2017-03-01T17:10:12.684+0000 I - [repl writer worker 9] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.684+0000 I - [repl writer worker 9] ***aborting after fassert() failure 2017-03-01T17:10:12.685+0000 I - [repl writer worker 11] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.685+0000 I - [repl writer worker 11] ***aborting after fassert() failure 2017-03-01T17:10:12.685+0000 I - [repl writer worker 13] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.685+0000 I - [repl writer worker 13] ***aborting after fassert() failure 2017-03-01T17:10:12.685+0000 I - [repl writer worker 15] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.685+0000 I - [repl writer worker 15] ***aborting after fassert() failure 2017-03-01T17:10:12.685+0000 I - [repl writer worker 7] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.685+0000 I - [repl writer worker 7] ***aborting after fassert() failure 2017-03-01T17:10:12.685+0000 I - [repl writer worker 5] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.685+0000 I - [repl writer worker 5] ***aborting after fassert() failure 2017-03-01T17:10:12.685+0000 I - [repl writer worker 3] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.685+0000 I - [repl writer worker 3] ***aborting after fassert() failure 2017-03-01T17:10:12.684+0000 I - [repl writer worker 1] ***aborting after fassert() failure 2017-03-01T17:10:12.685+0000 I - [repl writer worker 12] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.686+0000 I - [repl writer worker 12] ***aborting after fassert() failure 2017-03-01T17:10:12.684+0000 I - [repl writer worker 14] ***aborting after fassert() failure 2017-03-01T17:10:12.686+0000 I - [repl writer worker 10] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64 2017-03-01T17:10:12.686+0000 I - [repl writer worker 10] ***aborting after fassert() failure 2017-03-01T17:10:12.703+0000 F - [repl writer worker 8] Got signal: 6 (Aborted). 0x5594a00e0c81 0x5594a00dfd79 0x5594a00e025d 0x7f131f65c100 0x7f131f2c05f7 0x7f131f2c1ce8 0x55949f368c4d 0x55949fdefab6 0x55949fde88eb 0x55949fde661c 0x55949fdc3c9e 0x55949f7c3baa 0x55949f5694ca 0x55949f56976d 0x55949f569843 0x55949f5413c1 0x55949f541626 0x55949f541b93 0x55949fae0417 0x55949fbc1584 0x55949fbb8002 0x55949fbbb00e 0x55949fbbba6d 0x55949fbbbb7d 0x55949fbbfcf0 0x55949fbc1253 0x55949fbb684f 0x55949fbb7d4c 0x5594a0057d65 0x5594a0058890 0x5594a0059439 0x5594a0b55ba0 0x7f131f654dc5 0x7f131f381c9d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55949EB57000","o":"1589C81","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55949EB57000","o":"1588D79"},{"b":"55949EB57000","o":"158925D"},{"b":"7F131F64D000","o":"F100"},{"b":"7F131F28B000","o":"355F7","s":"gsignal"},{"b":"7F131F28B000","o":"36CE8","s":"abort"},{"b":"55949EB57000","o":"811C4D","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55949EB57000","o":"1298AB6","s":"_ZN5mongo17wtRCToStatus_slowEiPKc"},{"b":"55949EB57000","o":"12918EB","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},{"b":"55949EB57000","o":"128F61C","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},{"b":"55949EB57000","o":"126CC9E","s":"_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"55949EB57000","o":"C6CBAA","s":"_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl"},{"b":"55949EB57000","o":"A124CA","s":"_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EEPl"},{"b":"55949EB57000","o":"A1276D","s":"_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EEPl"},{"b":"55949EB57000","o":"A12843","s":"_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EEPl"},{"b":"55949EB57000","o":"9EA3C1","s":"_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bPNS_7OpDebugE"},{"b":"55949EB57000","o":"9EA626","s":"_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_PNS_7OpDebugEbb"},{"b":"55949EB57000","o":"9EAB93","s":"_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEPNS_7OpDebugEbb"},{"b":"55949EB57000","o":"F89417","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE"},{"b":"55949EB57000","o":"106A584","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_"},{"b":"55949EB57000","o":"1061002"},{"b":"55949EB57000","o":"106400E","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_"},{"b":"55949EB57000","o":"1064A6D","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"55949EB57000","o":"1064B7D"},{"b":"55949EB57000","o":"1068CF0","s":"_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE"},{"b":"55949EB57000","o":"106A253","s":"_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE"},{"b":"55949EB57000","o":"105F84F"},{"b":"55949EB57000","o":"1060D4C"},{"b":"55949EB57000","o":"1500D65","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"55949EB57000","o":"1501890","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"55949EB57000","o":"1502439","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"55949EB57000","o":"1FFEBA0"},{"b":"7F131F64D000","o":"7DC5"},{"b":"7F131F28B000","o":"F6C9D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.2", "gitVersion" : "3f76e40c105fc223b3e5aac3e20dcd026b83b38b", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.23-31.54.amzn1.x86_64", "version" : "#1 SMP Tue Oct 18 22:02:09 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "b" : "55949EB57000", "elfType" : 3, "buildId" : "887DF4C9600B8B003BDC24A15D8373ED1E3843E6" }, { "b" : "7FFEB0257000", "elfType" : 3, "buildId" : "3C621354FA6866C1B7DBFFDA88CE59560C001BE1" }, { "b" : "7F1320573000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6AF827B6FD200DFDFE70B2BC8D66BBC9881E8817" }, { "b" : "7F132018D000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "83F15DBCD0653F417E98354BC1EED6F96A758367" }, { "b" : "7F131FF85000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "2B3151901240D9E854E18E6D0B181C4D580ABA9C" }, { "b" : "7F131FD81000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "6335077ACD51527BE9F2F18451A88E2B7350C5B6" }, { "b" : "7F131FA7F000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "6E343508D15886FE83C438DF4560CE40BEB64B56" }, { "b" : "7F131F869000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "3FD5F89DE59E124AB1419A0BD16775B4096E84FD" }, { "b" : "7F131F64D000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "0836319AA81CDFE97DA2666963F62DE6A2A61346" }, { "b" : "7F131F28B000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "5D38A77E8D79E98D717281031C39B9A341323BD1" }, { "b" : "7F13207E0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "93D931BA041229929E5F099514B20E36A70BD651" }, { "b" : "7F131F03F000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E203354E7F907ACC8C3028FE465541B666DCFBA0" }, { "b" : "7F131ED5A000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "D769C8FFAF8772FDA55031ABF2E167DF2207E378" }, { "b" : "7F131EB57000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "5C01209C5AE1B1714F19B07EB58F2A1274B69DC8" }, { "b" : "7F131E925000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6C2243D37143F7FD1E16ED1F6CE4D7F16C2D7EF1" }, { "b" : "7F131E70F000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F131E500000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "744272FEAAABCE629AB9E11FAA4A96AEBE8BC2B4" }, { "b" : "7F131E2FD000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F131E0E3000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "1285F9516FFCF13FC00BD135C5634AF2EB16C80B" }, { "b" : "7F131DEC2000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5594a00e0c81] mongod(+0x1588D79) [0x5594a00dfd79] mongod(+0x158925D) [0x5594a00e025d] libpthread.so.0(+0xF100) [0x7f131f65c100] libc.so.6(gsignal+0x37) [0x7f131f2c05f7] libc.so.6(abort+0x148) [0x7f131f2c1ce8] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55949f368c4d] mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x396) [0x55949fdefab6] mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0xEB) [0x55949fde88eb] mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x55949fde661c] mongod(_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb+0xCE) [0x55949fdc3c9e] mongod(_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl+0x20A) [0x55949f7c3baa] mongod(_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EEPl+0xDA) [0x55949f5694ca] mongod(_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EEPl+0x1FD) [0x55949f56976d] mongod(_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EEPl+0x73) [0x55949f569843] mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bPNS_7OpDebugE+0x4F1) [0x55949f5413c1] mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_PNS_7OpDebugEbb+0x176) [0x55949f541626] mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEPNS_7OpDebugEbb+0x83) [0x55949f541b93] mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE+0x1E57) [0x55949fae0417] mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_+0x74) [0x55949fbc1584] mongod(+0x1061002) [0x55949fbb8002] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_+0x51E) [0x55949fbbb00e] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xFD) [0x55949fbbba6d] mongod(+0x1064B7D) [0x55949fbbbb7d] mongod(_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE+0x1C0) [0x55949fbbfcf0] mongod(_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE+0x73) [0x55949fbc1253] mongod(+0x105F84F) [0x55949fbb684f] mongod(+0x1060D4C) [0x55949fbb7d4c] mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x135) [0x5594a0057d65] mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xC0) [0x5594a0058890] mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x149) [0x5594a0059439] mongod(+0x1FFEBA0) [0x5594a0b55ba0] libpthread.so.0(+0x7DC5) [0x7f131f654dc5] libc.so.6(clone+0x6D) [0x7f131f381c9d] ----- END BACKTRACE -----