-
Type: Question
-
Resolution: Incomplete
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.4.24
-
Component/s: Index Maintenance, WiredTiger
-
None
MongoDB Version: 3.4.24
MongoDB hosted on Linux server was abruptly shut down due to the over-utilization of memory.
Initiated the mongodb repair using: sudo mongod -f /etc/mongodrepair.conf --repair
Repair Logs
2020-07-04T17:17:07.441+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$ 2020-07-04T17:17:07.448+0000 I INDEX [initandlisten] build index on: ZionsBank.summary properties: { v: 1, key: { totalVolume: -1 } , name: "totalV$ lume_-1", ns: "ZionsBank.summary", background: true } 2020-07-04T17:17:07.448+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$ 2020-07-04T17:17:07.456+0000 I INDEX [initandlisten] build index on: ZionsBank.summary properties: { v: 1, key: { ts: -1 } , name: "ts_-1", ns: "Zi$ nsBank.summary", background: true } 2020-07-04T17:17:07.456+0000 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 50 megabytes of RA$ 2020-07-04T17:17:08.673+0000 I - [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database.cpp 195 2020-07-04T17:17:08.673+0000 I - [initandlisten] ***aborting after invariant() failure 2020-07-04T17:17:08.717+0000 F - [initandlisten] Got signal: 6 (Aborted).
Restart Logs
2020-07-04T17:39:14.476+0000 I CONTROL [main] ***** SERVER RESTARTED ***** 2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] MongoDB starting : pid=20485 port=27017 dbpath=/home/db324 64-bit host=ip-*--**-* 2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] db version v3.4.24 2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] allocator: tcmalloc 2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] modules: none 2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] build environment: 2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] distarch: x86_64 2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] target_arch: x86_64 2020-07-04T17:39:14.480+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "-.-.-.-", port: 27017 } , replication: $ oplogSizeMB: 10240, replSetName: "rs1" }, storage: { dbPath: "/home/db324", directoryPerDB: true, engine: "wiredTiger", journal: { enabled$ true } , wiredTiger: { engineConfig: { cacheSizeGB: 108.0 } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.lo$ " } } 2020-07-04T17:39:14.480+0000 W - [initandlisten] Detected unclean shutdown - /home/db324/mongod.lock is not empty. 2020-07-04T17:39:14.499+0000 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] 2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] ** WARNING: The configured WiredTiger cache size is more than 80% of available RAM. 2020-07-04T17:39:14.499+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=110592M,session_max=20000,eviction=(threads_min=4,t$ reads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000)$ checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),verbose=(recovery_progress), 2020-07-04T17:39:14.667+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:667272][20485:0x7fdbcb287580], txn-recover: Main recovery loop$ starting at 73368/128 2020-07-04T17:39:14.667+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:667951][20485:0x7fdbcb287580], txn-recover: Recovering log 733$ 8 through 73369 2020-07-04T17:39:14.733+0000 I STORAGE [initandlisten] WiredTiger message [1593884354:733044][20485:0x7fdbcb287580], txn-recover: Recovering log 733$ 9 through 73369 2020-07-04T17:39:15.164+0000 E STORAGE [initandlisten] WiredTiger error (-31802) [1593884355:164908][20485:0x7fdbcb287580], [file:ZionsBank/collectio$|file://zionsbank/collectio$] -56-3854974571131417844.wt, WT_SESSION.open_cursor: /home/db324/ZionsBank/collection-56-3854974571131417844.wt: handle-read: pread: failed $ to read 4096 bytes at offset 28672: WT_ERROR: non-specific WiredTiger error 2020-07-04T17:39:15.164+0000 I - [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredT$ ger error at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 113 2020-07-04T17:39:15.165+0000 I - [initandlisten] ***aborting after invariant() failure
{{}}
Is there a way to repair/recover the last part of the DB? or
Is there a way to ignore the broken DB? or
Is it possible to cut out the whole 2.4TB data sans the last error db and create a new MongoDB instance with 2.4TB?
{{}}
I would greatly appreciate the help.
Thanks in Advance
{{}}