After we run a mapreduce job which updates thousends of records the primary mongodb server becomes inaccessable. It was not able to connect via PHP webnode or local mongo shell. In a short time the server reached his connection limit (in normal operation we have around 10/s; after the mapreduce job they step up to > 13000; the PHP webnodes use non-persistent conncetions; see lx03_mongostat_cutted.txt). the 13000 connection where full established but idle (see attachment overview_mongod_after_midnight.html).

Our first action was to shutdown the php webserver nodes. connections jumps back to 10 and the system becomes accessable again.

Second action was to shutdown the secondary and start the map reduce job again. everything run smooths seeming without probems. During the mapreduce job used ram increased steadily (see munin graphs). When the job was finished we start the secondary again. From here everything works as expected running the operations from oplog. After a short sleep we saw in the morning there was a connection jump again to 1000. So I decieded to stop and start the current primary and let the secondary take over to get a clean state again.

The attachments containing mongostats, munin graphs, mongodb.logs and the home view from mongodb internal webserver. The munin graphs contains some leaks where the primary was inaccessable to gather data. (server lx03 is primary, lx04 is secondary)

In the past we had the situation once or twice per month probably from a cron job starting another mapreduce operation. But until yesterday we couldn't track it down.