We multiple web crawlers, dumping data into single node of mongod running on windows 2008. All crawlers are .net 3.5 on windows 2008 machines as well. I routinely get this error:

System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Initially when the db size was smaller, it happened less frequently, but now was the size of the db grows (currently only at ~13gb), it is happening at a rate of once every 3-4 seconds and it slows everything down. This happens even if I turn off ALL crawlers, and only leave 1 instance running.

I should note that the crawler is multithreaded. The thread pool on the crawler is usually kept between 32-64 and alot of data are flushed into mongod in batches of 200-500 items at a time, every 10-30 seconds (depending on collections).

Attached are the full mongod log file, along with 1 of the service error logs (so you can see the full errors). I've also attached the mongo connection wrapper that we are using and a few cs file of interest so you can see how data are dumped into mongo.

We also notice when used with driver v1.1, the error is a little less frequent than in v1.2. For the driver, we are setting the time out at 10s for requests.