If a server is specified in a replica set seed list goes down and is brought up again with multiple clients (php-fpm workers in this case) trying to connect and write to it, the server segfaults:
Sun Aug 22 07:50:12 MongoDB starting : pid=3225 port=27018 dbpath=/web/HYPE/mongodb 64-bit Sun Aug 22 07:50:12 db version v1.6.1, pdfile version 4.5
Sun Aug 22 07:50:12 git version: c5f5f9a4f3b515dfd5272d373093fd4fd58c95d9Sun Aug 22 07:50:12 sys info: Linux domU-12-31-39-06-79-A1 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 200
9 x86_64 BOOST_LIB_VERSION=1_41Sun Aug 22 07:50:12 [initandlisten] waiting for connections on port 27018
Sun Aug 22 07:50:12 [websvr] web admin interface listening on port 28018Sun Aug 22 07:50:12 [startReplSets] replSet can't get local.system.replset config from self or any seed (yet)
Sun Aug 22 07:50:13 [conn2] Caught Assertion in insert , continuingSun Aug 22 07:50:13 [conn2] insert exception 10058 not master 0ms
Sun Aug 22 07:50:14 [conn4] Caught Assertion in insert , continuingSun Aug 22 07:50:14 [conn4] insert exception 10058 not master 0ms
Sun Aug 22 07:50:15 [conn5] Caught Assertion in insert , continuingSun Aug 22 07:50:15 [conn5] insert exception 10058 not master 0ms
Sun Aug 22 07:50:17 Got signal: 11 (Segmentation fault).
Sun Aug 22 07:50:17 Backtrace:0x813619 0x7fbade1d7af0 0x5fb81e 0x6faf8a 0x6feb44 0x8154db 0x829370 0x7fbadecda9ca 0x7fbade28a69d
mongod(_ZN5mongo10abruptQuitEi+0x399) [0x813619] /lib/libc.so.6(+0x33af0) [0x7fbade1d7af0]
mongod(ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1+0xf8e) [0x5fb81e] mongod() [0x6faf8a]
mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_8SockAddrE+0x14d4) [0x6feb44] mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0x31b) [0x8154db]
mongod(thread_proxy+0x80) [0x829370] /lib/libpthread.so.0(+0x69ca) [0x7fbadecda9ca]
/lib/libc.so.6(clone+0x6d) [0x7fbade28a69d]
Sun Aug 22 07:50:17 dbexit:
(etc)
The only way to add members to the set is to change the client config across all servers, or temporarily halt the processes. This makes using rs in production pretty much impossible.