Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-3705

mongos not closing connections, filling up disk with logs

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.0.0-rc1
    • Affects Version/s: 1.8.2
    • Component/s: None
    • None
    • ALL

      One of our mongos started spewing out this message:

      Mon Aug 29 00:40:36 [mongosMain] Listener: accept() returns -1 errno:24 Too many open files
      

      it wrote 6.5 gigabytes of identical lines (same timestamp and all), which filled up the root partition of the server. All queries running through that mongos started failing at the same time, which led to the application failing.

      However, mongos is still running, and holding 300 TCP connections open (according to lsof).

      From what I can see it started about an hour earlier with mongos not being able to connect to the cluster, which goes on until it can connect to all the nodes except one, and then just minutes before it starts spewing out the messages about too many open files it manages to connect to the last one too. Then it writes the same message until the disk runs out.

            Assignee:
            Unassigned Unassigned
            Reporter:
            iconara Theo Hultberg
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: