-
Type: Bug
-
Resolution: Duplicate
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.4.10
-
Component/s: Index Maintenance, WiredTiger
-
Environment:Ubuntu 16.04 Server
-
ALL
mongod 3.4.10 on Ubuntu 16.04 in a replica set with 3 nodes. The primary was building an index and the index build stopped:
Dec 18 22:05:57 hostname mongod.27017[1250]: [conn5562] Index Build (background): 4027500/10026608 40% Dec 18 22:06:00 hostname mongod.27017[1250]: [conn5562] Index Build (background): 4062300/10026608 40% Dec 18 22:06:03 hostname mongod.27017[1250]: [conn5562] Index Build (background): 4096100/10026608 40% Dec 18 22:06:06 hostname mongod.27017[1250]: [conn5562] Index Build (background): 4127300/10026608 41% Dec 18 22:06:09 hostname mongod.27017[1250]: [conn5562] Index Build (background): 4152100/10026608 41% Dec 18 22:06:12 hostname mongod.27017[1250]: [conn5562] Index Build (background): 4189100/10026608 41% Dec 18 22:06:15 hostname mongod.27017[1250]: [conn5562] Index Build (background): 4241600/10026608 42%
At this point it stalled, and no more "Index Build" lines were printed until I restarted mongod several hours later. From here on I could connect to the admin database as the root database user, but I couldn't connect as any other user - the connection would just hang. Logs below in case they're relevant (I suspect they're not).
Dec 18 22:06:38 hostname monit[1400]: 'mongo_index_size' '/usr/local/bin/mongo_index_size.sh' failed with exit status (1) -- Mongo indexes are consuming 14900 MB, higher than threshold of 12034 MB Dec 18 22:06:38 hostname mongod.27017[1250]: [thread1] connection accepted from 127.0.0.1:45244 #5605 (66 connections now open) Dec 18 22:06:38 hostname mongod.27017[1250]: [thread1] connection accepted from 127.0.0.1:45246 #5606 (67 connections now open) Dec 18 22:06:38 hostname mongod.27017[1250]: [conn5606] received client metadata from 127.0.0.1:45246 conn5606: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } } Dec 18 22:06:38 hostname mongod.27017[1250]: [conn5605] received client metadata from 127.0.0.1:45244 conn5605: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "Ubuntu", architecture: "x86_64", version: "16.04" } }
Backtrace attached.
- duplicates
-
SERVER-21307 Replicated DDL (catalog) operation during background index build blocks reads/writes
- Closed
- is related to
-
SERVER-32652 mongod killed by OOM while one secondary lagging
- Closed