Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-8766

2 node 1 arb, 2.2.2 RS SSL, upgrade 2 nodes to 2.4.0-rc1, CRLFile, call rs.status() on shell connected to last 2.2.2 node, segfault

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Critical - P2 Critical - P2
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • ALL

      3 AWS micro instances.
      2 nodes, 1 arb 2.2.2

      ./bin/mongod --dbpath ./data --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslPEMKeyPassword mongo --replSet rs1 

      ./bin/mongod --dbpath ./data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslPEMKeyPassword mongo --replSet rs1 --smallfiles

      ./bin/mongod --dbpath ./data/arb/ --port 30000 --replSet rs1 --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem  --sslPEMKeyPassword mongo --smallfiles

      mongo shell connected to ip-10-227-21-162

      rs1:SECONDARY> rs.status()
{
	"set" : "rs1",
	"date" : ISODate("2013-02-27T17:30:21Z"),
	"myState" : 2,
	"syncingTo" : "ip-10-48-238-94:27017",
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-227-21-162:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 66,
			"optime" : {
				"t" : 1361966564000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
			"errmsg" : "syncing to: ip-10-48-238-94:27017",
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-48-238-94:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 33,
			"optime" : {
				"t" : 1361966564000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
			"lastHeartbeat" : ISODate("2013-02-27T17:30:20Z"),
			"pingMs" : 1
		},
		{
			"_id" : 2,
			"name" : "ip-10-48-238-94:30000",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 33,
			"lastHeartbeat" : ISODate("2013-02-27T17:30:20Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}
rs1:SECONDARY>

      upgrade node ip-10-48-238-94, use --sslCRLFile

      ./bin/mongod --dbpath ../mongodb-linux-x86_64-subscription-ubuntu1104-2.2.2/data/ --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem --sslCAFile ../cacert.pem --replSet rs1 --smallfiles --sslCRLFile ../crl.pem

      same shell shows status

      rs1:PRIMARY> rs.status()
{
	"set" : "rs1",
	"date" : ISODate("2013-02-27T17:30:35Z"),
	"myState" : 1,
	"members" : [
		{
			"_id" : 0,
			"name" : "ip-10-227-21-162:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 80,
			"optime" : {
				"t" : 1361966564000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
			"self" : true
		},
		{
			"_id" : 1,
			"name" : "ip-10-48-238-94:27017",
			"health" : 0,
			"state" : 8,
			"stateStr" : "(not reachable/healthy)",
			"uptime" : 0,
			"optime" : {
				"t" : 1361966564000,
				"i" : 1
			},
			"optimeDate" : ISODate("2013-02-27T12:02:44Z"),
			"lastHeartbeat" : ISODate("2013-02-27T17:30:22Z"),
			"pingMs" : 0,
			"errmsg" : "DBClientBase::findN: transport error: ip-10-48-238-94:27017 ns: admin.$cmd query: { replSetHeartbeat: \"rs1\", v: 3, pv: 1, checkEmpty: false, from: \"ip-10-227-21-162:27017\", $auth: {} }"
		},
		{
			"_id" : 2,
			"name" : "ip-10-48-238-94:30000",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 47,
			"lastHeartbeat" : ISODate("2013-02-27T17:30:34Z"),
			"pingMs" : 0
		}
	],
	"ok" : 1
}

      upgrade arbiter, ip-10-48-238-94

      ./bin/mongod --dbpath ../mongodb-linux-x86_64-subscription-ubuntu1104-2.2.2/data/arb/ --port 30000 --replSet rs1 --sslOnNormalPorts --sslPEMKeyFile ../gregorFreeBSD.pem  --sslPEMKeyPassword mongo --smallfiles --sslCAFile ../cacert.pem --sslCRLFile ../crl.pem

      call rs.status() on same shell

      rs1:PRIMARY> rs.status()
Wed Feb 27 17:31:28.726 DBClientCursor::init call() failed
Wed Feb 27 17:31:28.727 JavaScript execution failed: Error: error doing query: failed at src/mongo/shell/query.js:L78
Wed Feb 27 17:31:28.728 trying reconnect to ip-10-227-21-162:27017
Wed Feb 27 17:31:28.732 reconnect ip-10-227-21-162:27017 ok
rs1:SECONDARY>

      SIGSEGV at ip-10-227-21-162 (node not yet upgraded)

      Wed Feb 27 17:31:02 [rsMgr] replSet SECONDARY
Wed Feb 27 17:31:02 [rsMgr] replSet closing client sockets after relinquishing primary
Wed Feb 27 17:31:03 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:04 [conn11] SocketException handling request, closing client connection: 9001 socket exception [2] server [10.48.238.94:37838] 
Wed Feb 27 17:31:04 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:05 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:06 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:07 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:07 [rsMgr] replSet can't see a majority, will not try to elect self
Wed Feb 27 17:31:09 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:09 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:11 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:11 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:13 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:13 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:13 [conn13] end connection 10.48.238.94:37845 (3 connections now open)
Wed Feb 27 17:31:13 [initandlisten] connection accepted from 10.48.238.94:37852 #16 (4 connections now open)
Wed Feb 27 17:31:15 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:15 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:17 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:17 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:19 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:19 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:21 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:21 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:23 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:23 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:25 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:25 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:27 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:27 [rsHealthPoll] DBClientCursor::init call() failed
Wed Feb 27 17:31:27 Invalid access at address: 0xf8 from thread: conn6

Wed Feb 27 17:31:27 Got signal: 11 (Segmentation fault).

Wed Feb 27 17:31:27 [initandlisten] connection accepted from 10.226.133.10:32776 #17 (5 connections now open)
Wed Feb 27 17:31:28 Backtrace:
0xa2a6d3 0x580264 0x580658 0x7fd31e031cb0 0x7fd31ddf1e1c 0x7fd31ddf2c95 0x7fd31ddf031c 0xa16099 0xa1956b 0xa0e584 0xa115ff 0x7fd31e029e9a 0x7fd31c2e8cbd 
 ./bin/mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xa2a6d3]
 ./bin/mongod(_ZN5mongo10abruptQuitEi+0x3e4) [0x580264]
 ./bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x268) [0x580658]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fd31e031cb0]
 /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read_n+0x19c) [0x7fd31ddf1e1c]
 /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read_bytes+0x3f5) [0x7fd31ddf2c95]
 /lib/x86_64-linux-gnu/libssl.so.0.9.8(ssl3_read+0x5c) [0x7fd31ddf031c]
 ./bin/mongod(_ZN5mongo6Socket11unsafe_recvEPci+0x9) [0xa16099]
 ./bin/mongod(_ZN5mongo6Socket4recvEPci+0x2b) [0xa1956b]
 ./bin/mongod(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0x84) [0xa0e584]
 ./bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x41f) [0xa115ff]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fd31e029e9a]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd31c2e8cbd]

            Assignee:
            Unassigned Unassigned
            Reporter:
            gregor Gregor Macadam
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: