2019-08-09T18:49:49.180+0800 I CONTROL [main] ***** SERVER RESTARTED ***** 2019-08-09T18:49:49.185+0800 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none' 2019-08-09T18:49:49.201+0800 I CONTROL [initandlisten] MongoDB starting : pid=8527 port=62801 dbpath=/tmxbase/mongodb_cluster/mongodata/shard01 64-bit host=sh.tmxmall.mongodb06 2019-08-09T18:49:49.201+0800 I CONTROL [initandlisten] db version v4.0.9 2019-08-09T18:49:49.201+0800 I CONTROL [initandlisten] git version: fc525e2d9b0e4bceff5c2201457e564362909765 2019-08-09T18:49:49.201+0800 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016 2019-08-09T18:49:49.201+0800 I CONTROL [initandlisten] allocator: tcmalloc 2019-08-09T18:49:49.201+0800 I CONTROL [initandlisten] modules: none 2019-08-09T18:49:49.201+0800 I CONTROL [initandlisten] build environment: 2019-08-09T18:49:49.202+0800 I CONTROL [initandlisten] distmod: ubuntu1604 2019-08-09T18:49:49.202+0800 I CONTROL [initandlisten] distarch: x86_64 2019-08-09T18:49:49.202+0800 I CONTROL [initandlisten] target_arch: x86_64 2019-08-09T18:49:49.202+0800 I CONTROL [initandlisten] options: { config: "/home/work/software/mongodb_cluster/shard01.conf", net: { bindIp: "0.0.0.0", port: 62801 }, processManagement: { fork: true, pidFilePath: "/tmxbase/mongodb_cluster/mongodata/shard01/shard.pid" }, replication: { replSetName: "prodShard1" }, security: { authorization: "enabled", keyFile: "/home/work/software/mongodb_cluster/keyfile" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "/tmxbase/mongodb_cluster/mongodata/shard01", journal: { enabled: true }, wiredTiger: { engineConfig: { cacheSizeGB: 2.5 } } }, systemLog: { destination: "file", logAppend: true, path: "/tmxbase/mongodb_cluster/mongodata/shard01/shard.log" } } 2019-08-09T18:49:49.202+0800 W STORAGE [initandlisten] Detected unclean shutdown - /tmxbase/mongodb_cluster/mongodata/shard01/mongod.lock is not empty. 2019-08-09T18:49:49.202+0800 I STORAGE [initandlisten] Detected data files in /tmxbase/mongodb_cluster/mongodata/shard01 created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2019-08-09T18:49:49.202+0800 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2019-08-09T18:49:49.202+0800 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=2560M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2019-08-09T18:49:50.184+0800 I STORAGE [initandlisten] WiredTiger message [1565347790:184501][8527:0x7fb3ad5bba80], txn-recover: Main recovery loop: starting at 6054/256 to 6055/256 2019-08-09T18:49:50.185+0800 I STORAGE [initandlisten] WiredTiger message [1565347790:185537][8527:0x7fb3ad5bba80], txn-recover: Recovering log 6054 through 6055 2019-08-09T18:49:50.285+0800 I STORAGE [initandlisten] WiredTiger message [1565347790:285146][8527:0x7fb3ad5bba80], file:index-1-9163800971290245725.wt, txn-recover: Recovering log 6055 through 6055 2019-08-09T18:49:50.361+0800 I STORAGE [initandlisten] WiredTiger message [1565347790:361570][8527:0x7fb3ad5bba80], file:index-1-9163800971290245725.wt, txn-recover: Set global recovery timestamp: 5d4d1ea800000003 2019-08-09T18:49:50.398+0800 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(1565335208, 3) 2019-08-09T18:49:50.398+0800 I STORAGE [initandlisten] Triggering the first stable checkpoint. Initial Data: Timestamp(1565335208, 3) PrevStable: Timestamp(0, 0) CurrStable: Timestamp(1565335208, 3) 2019-08-09T18:49:55.714+0800 I STORAGE [initandlisten] Starting OplogTruncaterThread local.oplog.rs 2019-08-09T18:49:55.714+0800 I STORAGE [initandlisten] The size storer reports that the oplog contains 99056030 records totaling to 26699912912 bytes 2019-08-09T18:49:55.714+0800 I STORAGE [initandlisten] Sampling from the oplog between Aug 9 02:34:21:1033 and Aug 9 15:24:52:11 to determine where to place markers for truncation 2019-08-09T18:49:55.714+0800 I STORAGE [initandlisten] Taking 998 samples and assuming that each section of oplog contains approximately 992415 records totaling to 267499051 bytes 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 02:40:17:2898 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 02:48:11:2235 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 02:55:51:2592 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:00:39:1269 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:08:18:2416 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:16:00:1907 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:20:49:1702 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:28:33:2383 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:36:15:3060 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:41:05:1129 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:48:48:1073 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 03:56:33:946 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:01:24:1670 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:09:06:2422 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:16:47:1694 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:21:41:890 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:29:25:1386 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:37:06:1139 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:41:52:2063 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:49:30:3221 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 04:57:11:2675 2019-08-09T18:49:57.931+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:01:58:653 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:09:40:1562 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:17:22:255 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:22:08:3381 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:29:47:1296 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:37:34:356 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:42:20:1795 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:50:02:752 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 05:57:45:2318 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:02:35:3176 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:10:21:659 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:18:05:1343 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:22:53:1169 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:30:37:3123 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:38:21:791 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:43:10:1891 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:50:55:3356 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 06:58:44:2062 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:03:39:2038 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:11:30:1972 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:19:19:1523 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:24:10:2987 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:30:31:1233 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:37:05:3011 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:42:01:2491 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:49:58:1141 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 07:57:59:2397 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 08:03:03:2753 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 08:10:56:2468 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 08:19:00:652 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 08:27:20:1335 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 08:35:42:2001 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 08:44:05:2621 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 08:52:21:849 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 09:00:46:2109 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 09:08:59:2985 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 09:17:31:1127 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 09:26:09:3079 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 09:34:49:2967 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 09:43:38:1286 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 09:51:46:1744 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 10:00:34:1190 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 10:09:27:1403 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 10:18:14:1878 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 10:27:04:1114 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 10:35:46:1889 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 10:44:27:2741 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 10:52:57:3034 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 11:01:55:2962 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 11:10:54:2902 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 11:19:46:352 2019-08-09T18:49:57.932+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 11:28:33:2549 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 11:37:14:2869 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 11:45:46:2990 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 11:54:17:883 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 12:03:07:49 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 12:11:41:743 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 12:20:23:1130 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 12:28:58:1507 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 12:37:32:1422 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 12:46:07:1746 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 12:54:21:2792 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 13:02:42:428 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 13:11:13:1740 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 13:19:42:1543 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 13:28:17:624 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 13:36:53:1952 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 13:45:04:1243 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 13:53:56:325 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 14:02:56:2667 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 14:11:41:1584 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 14:20:36:1992 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 14:29:15:307 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 14:37:57:898 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 14:46:22:744 2019-08-09T18:49:57.933+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 14:55:17:1701 2019-08-09T18:49:57.934+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 15:04:03:1574 2019-08-09T18:49:57.934+0800 I STORAGE [initandlisten] Placing a marker at optime Aug 9 15:12:32:2356 2019-08-09T18:49:58.834+0800 I CONTROL [initandlisten] 2019-08-09T18:49:58.834+0800 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 31270 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files. 2019-08-09T18:50:07.131+0800 I SHARDING [initandlisten] initializing sharding state with: { shardName: "prodShard1", clusterId: ObjectId('5d297651884c1c914fcc5d55'), configsvrConnectionString: "prodConfigs/10.105.196.140:21002,10.105.196.140:21003,10.105.21.185:21001" } 2019-08-09T18:50:07.133+0800 I NETWORK [initandlisten] Starting new replica set monitor for prodConfigs/10.105.196.140:21002,10.105.196.140:21003,10.105.21.185:21001 2019-08-09T18:50:07.133+0800 I SHARDING [initandlisten] Finished initializing sharding components for secondary node. 2019-08-09T18:50:07.149+0800 I NETWORK [shard registry reload] Successfully connected to 10.105.21.185:21001 (1 connections now open to 10.105.21.185:21001 with a 5 second timeout) 2019-08-09T18:50:07.149+0800 I SHARDING [thread1] creating distributed lock ping thread for process sh.tmxmall.mongodb06:62801:1565347807:-7768788710968022110 (sleeping for 30000ms) 2019-08-09T18:50:07.149+0800 I ASIO [ShardRegistry] Connecting to 10.105.21.185:21001 2019-08-09T18:50:07.149+0800 I ASIO [ShardRegistry] Connecting to 10.105.21.185:21001 2019-08-09T18:50:07.152+0800 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to 10.105.196.140:21002 (1 connections now open to 10.105.196.140:21002 with a 5 second timeout) 2019-08-09T18:50:07.153+0800 I NETWORK [initandlisten] Successfully connected to 10.105.196.140:21003 (1 connections now open to 10.105.196.140:21003 with a 5 second timeout) 2019-08-09T18:50:07.154+0800 I ASIO [ShardRegistry] Connecting to 10.105.196.140:21003 2019-08-09T18:50:07.158+0800 I ASIO [ShardRegistry] Connecting to 10.105.21.185:21001 2019-08-09T18:50:07.159+0800 I NETWORK [shard registry reload] Starting new replica set monitor for prodShard0/10.105.196.140:62802,10.105.21.185:62801 2019-08-09T18:50:07.159+0800 I NETWORK [shard registry reload] Starting new replica set monitor for prodShard1/10.105.196.140:62801,10.105.21.185:62802 2019-08-09T18:50:07.162+0800 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory '/tmxbase/mongodb_cluster/mongodata/shard01/diagnostic.data' 2019-08-09T18:50:07.170+0800 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document 2019-08-09T18:50:07.175+0800 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to 10.105.196.140:62802 (1 connections now open to 10.105.196.140:62802 with a 5 second timeout) 2019-08-09T18:50:07.188+0800 I REPL [initandlisten] Rollback ID is 2 2019-08-09T18:50:07.189+0800 I REPL [initandlisten] Recovering from stable timestamp: Timestamp(1565335208, 3) (top of oplog: { ts: Timestamp(1565335492, 11), t: 7 }, appliedThrough: { ts: Timestamp(0, 0), t: -1 }, TruncateAfter: Timestamp(0, 0)) 2019-08-09T18:50:07.189+0800 I REPL [initandlisten] Starting recovery oplog application at the stable timestamp: Timestamp(1565335208, 3) 2019-08-09T18:50:07.189+0800 I REPL [initandlisten] Replaying stored operations from { : Timestamp(1565335208, 3) } (exclusive) to { : Timestamp(1565335492, 11) } (inclusive). 2019-08-09T18:50:07.191+0800 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to 10.105.21.185:62801 (1 connections now open to 10.105.21.185:62801 with a 5 second timeout) 2019-08-09T18:50:07.211+0800 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to 10.105.21.185:62802 (1 connections now open to 10.105.21.185:62802 with a 5 second timeout) 2019-08-09T18:50:07.416+0800 F REPL [repl writer worker 5] writer worker caught exception: DuplicateKey: E11000 duplicate key error collection: config.cache.chunks.pr_tmxbase.toffs_6.chunks index: _id_ dup key: { : { files_id: -8792944871971349461 } } on: { op: "i", ns: "config.cache.chunks.pr_tmxbase.toffs_6.chunks", ui: UUID("03c48aff-f98b-4de3-bb08-3248ab7b588f"), o: { _id: { files_id: -8792944871971349461 }, max: { files_id: -8700326615160610030 }, shard: "prodShard1", lastmod: Timestamp(98, 0), history: [ { validAfter: Timestamp(1564563326, 26), shard: "prodShard1" } ] }, ts: Timestamp(1565335484, 137), t: 7, h: -6045188019592284924, v: 2, wall: new Date(1565335484859) } 2019-08-09T18:50:07.622+0800 F REPL [initandlisten] Failed to apply batch of operations. Number of operations in batch: 4229. First operation: { op: "n", ns: "", o: { msg: "new primary" }, ts: Timestamp(1565335398, 2), t: 7, h: -153190637679831346, v: 2, wall: new Date(1565335398497) }. Last operation: { op: "i", ns: "pr_tmxbase.stfs_2.chunks", ui: UUID("78d50f72-a565-4a31-be18-2354f730743f"), o: { _id: ObjectId('5d4d1fc44ef55e38430807eb'), files_id: "3f05ae47e29a0704c7407a8c3bdf9df4", n: 23, data: BinData(0, 693D22687474703A2F2F7777772E77332E6F72672F323030312F584D4C536368656D612D696E7374616E63652220786D6C6E733A7873643D22687474703A2F2F7777772E7733...) }, ts: Timestamp(1565335492, 11), t: 7, h: -791270771233656130, v: 2, wall: new Date(1565335492057) }. Oplog application failed in writer thread 11: DuplicateKey: E11000 duplicate key error collection: config.cache.chunks.pr_tmxbase.toffs_6.chunks index: _id_ dup key: { : { files_id: -8792944871971349461 } } 2019-08-09T18:50:07.624+0800 F REPL [initandlisten] Caught exception during replication recovery: DuplicateKey: E11000 duplicate key error collection: config.cache.chunks.pr_tmxbase.toffs_6.chunks index: _id_ dup key: { : { files_id: -8792944871971349461 } } 2019-08-09T18:50:07.625+0800 F - [initandlisten] terminate() called. An exception is active; attempting to gather more information 2019-08-09T18:50:07.634+0800 F - [initandlisten] DBException::toString(): DuplicateKey: E11000 duplicate key error collection: config.cache.chunks.pr_tmxbase.toffs_6.chunks index: _id_ dup key: { : { files_id: -8792944871971349461 } } Actual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)11000, mongo::AssertionException> 0x5634a5f505e1 0x5634a5f4ffc5 0x5634a6044a46 0x5634a6044a91 0x5634a4841c3f 0x5634a487bf1c 0x5634a487c7e7 0x5634a45be20c 0x5634a45c09ff 0x5634a45522e9 0x7fb3ab98e830 0x5634a45bb3d9 ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"5634A3B34000","o":"241C5E1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"5634A3B34000","o":"241BFC5"},{"b":"5634A3B34000","o":"2510A46","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"5634A3B34000","o":"2510A91"},{"b":"5634A3B34000","o":"D0DC3F","s":"_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE"},{"b":"5634A3B34000","o":"D47F1C","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE"},{"b":"5634A3B34000","o":"D487E7","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE"},{"b":"5634A3B34000","o":"A8A20C"},{"b":"5634A3B34000","o":"A8C9FF","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"5634A3B34000","o":"A1E2E9","s":"main"},{"b":"7FB3AB96E000","o":"20830","s":"__libc_start_main"},{"b":"5634A3B34000","o":"A873D9","s":"_start"}],"processInfo":{ "mongodbVersion" : "4.0.9", "gitVersion" : "fc525e2d9b0e4bceff5c2201457e564362909765", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-91-generic", "version" : "#114-Ubuntu SMP Tue Aug 8 11:56:56 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "5634A3B34000", "elfType" : 3, "buildId" : "D363637D8649A86A028650E1AE31DD0CCF80723C" }, { "b" : "7FFFA4FE6000", "elfType" : 3, "buildId" : "735D8C7F1005D2D0E443E551BEA48FD5091211DF" }, { "b" : "7FB3AD148000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "5C1A06A89F89E1ADAAA507BC5580C0A7931B0AB2" }, { "b" : "7FB3ACF2D000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, { "b" : "7FB3ACAE9000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "B467957BF7E767DD5C0CA92213C2DAF0182765D2" }, { "b" : "7FB3AC880000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "5A9B9023B6E9ABCEE21DADDB95EF096F505E9E67" }, { "b" : "7FB3AC67C000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7FB3AC474000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7FB3AC16B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB85DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7FB3ABF55000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FB3ABD38000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7FB3AB96E000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7FB3AD3B7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" }, { "b" : "7FB3AB73B000", "path" : "/usr/lib/x86_64-linux-gnu/libidn.so.11", "elfType" : 3, "buildId" : "F3945D210B52CC973C619CCD08C4394BBC33E07F" }, { "b" : "7FB3AB51F000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "8D1CC1204D6B6D33BD1D2C5A2A0516A2234322CF" }, { "b" : "7FB3AB2D5000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "54EB3E674DB58B5BAEF009CBC4AB02CCF5386CA7" }, { "b" : "7FB3AB0C6000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "213C3A6756470FF961EE7FD63BAC177BC6B72A49" }, { "b" : "7FB3AAE75000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "698C15E7BD2CD96618583CADC6EBCE07B293CEB2" }, { "b" : "7FB3AAC5B000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "8D9BD4CE26E45EF16075C67D5F5EEAFD8B562832" }, { "b" : "7FB3AA92B000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "213041D960C027C8C451F53B9B3143A901322015" }, { "b" : "7FB3AA6F8000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B11678F560199547DCF726384EA39153EE0DFABF" }, { "b" : "7FB3AA4C2000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "D6B36C5A463EE0FA84FDD6D5FD3F7726EDB90D54" }, { "b" : "7FB3AA242000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "7B3533D5998D20EE1A1BE3F87789B69041E7F620" }, { "b" : "7FB3A9F70000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "D60D5A759577E5C477314520EF3D13BDBAED5111" }, { "b" : "7FB3A9D41000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "09FA442048F4AA6E701F0B504FEB688F09618E30" }, { "b" : "7FB3A9B3D000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "1E16CB57F699E215A2A8D4EFEF90883BC749B12D" }, { "b" : "7FB3A9932000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "00DFFD025B1AEBFD619FAE9DE01610BA6F209367" }, { "b" : "7FB3A9717000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "96BCC7EB28D81B1469EED6F24FC083CBD58577BC" }, { "b" : "7FB3A94D6000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "1FE877BE52A424D0636AFD4D35BB330E41D6E0F3" }, { "b" : "7FB3A9271000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "2F170DCF36786FE91EB82D547A42B2DEE2E9DF81" }, { "b" : "7FB3A905E000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "A6EE7754728D688B04095ACB34F1A071F99ED6C3" }, { "b" : "7FB3A8E5A000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "3364D4BF2113C4E8D17EF533867ECC99A53413D6" }, { "b" : "7FB3A8C51000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "73A8EADBC85860662B24850E71D4AFBE22C33359" }, { "b" : "7FB3A89C7000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "59E742306A4EA2872E061ECCE92F35FADDA75357" }, { "b" : "7FB3A8725000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "E5C159E415406AE79D21056D752BA949C408B5B1" }, { "b" : "7FB3A84F2000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "7D15576E1F096614D360784E4A01A1F5FAF908C9" }, { "b" : "7FB3A82DC000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "481DB33C28D88E43DA6BED65E1A7599407D4D818" }, { "b" : "7FB3A80D4000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9D9C958F1F4894AFEF6AECD90D1C430EA29AC34F" }, { "b" : "7FB3A7EAB000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "57E25072866B2D30CF02EBE7AE623B84F96FA700" }, { "b" : "7FB3A7C9C000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "F6F1B4E9F89B716C4A0BA5819BDFFAF4A13EFB91" }, { "b" : "7FB3A7A51000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "C60082E3BB78D0D42868D9B359B89BF66CE5A1A7" }, { "b" : "7FB3A777C000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "D9782BA023CAEC26B15D8676E3A5D07B55E121EF" }, { "b" : "7FB3A7544000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "7BDD51353D50310FFA1587E4AA01B40ABE32D582" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x5634a5f505e1] mongod(+0x241BFC5) [0x5634a5f4ffc5] mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x5634a6044a46] mongod(+0x2510A91) [0x5634a6044a91] mongod(_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextEN5boost8optionalINS_9TimestampEEE+0xC7F) [0x5634a4841c3f] mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE+0x54C) [0x5634a487bf1c] mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE+0x237) [0x5634a487c7e7] mongod(+0xA8A20C) [0x5634a45be20c] mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0xEDF) [0x5634a45c09ff] mongod(main+0x9) [0x5634a45522e9] libc.so.6(__libc_start_main+0xF0) [0x7fb3ab98e830] mongod(_start+0x29) [0x5634a45bb3d9] ----- END BACKTRACE -----