Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-20728

mongod hang during 3-node replSet performance test with 3.0.7-pre (SHA: 180db3d2)

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 3.0.7
    • Affects Version/s: 3.0.7
    • Component/s: WiredTiger
    • Fully Compatible
    • ALL

      mongod hang during EVG 3.0.7-pre patch testing

      Setup/Test Details

      Observation

      • mongod hang during YCSB 50 read/50 update test
      • CPU usage on the primary is low
      • shell will hang if do insert/find
      • looked at pending request, lots of “aw” from mongostat
        [ec2-user@ip-10-2-0-100 ~]$ ./mongodb-linux-x86_64-3.0.7-pre-/bin/mongostat
        insert query update delete getmore command % dirty % used flushes vsize   res qr|qw ar|aw netIn netOut conn set repl     time
            *0    *0     *0     *0       0     5|0    35.7   40.8       0 13.2G 12.6G   1|7  1|58  414b    63k   76 rs0  PRI 22:17:29
            *0    *0     *0     *0       0     6|0    35.7   40.8       0 13.2G 12.6G   1|7  1|58  472b    63k   76 rs0  PRI 22:17:30
            *0    *0     *0     *0       0     5|0    35.7   40.8       0 13.2G 12.6G   1|6  1|59  414b    63k   76 rs0  PRI 22:17:31
        
      • output from YCSB
         [2015/10/01 18:00:24.904]  100 sec: 7926293 operations; 88118.59 current ops/sec; [UPDATE AverageLatency(us)=1036.89] [READ AverageLatency(us)=406.35]
         [2015/10/01 18:00:34.897]  110 sec: 8724240 operations; 79794.7 current ops/sec; [UPDATE AverageLatency(us)=1158.45] [READ AverageLatency(us)=432.6]
         [2015/10/01 18:00:44.897]  120 sec: 9584935 operations; 86060.89 current ops/sec; [UPDATE AverageLatency(us)=1062.74] [READ AverageLatency(us)=416.89]
         [2015/10/01 18:00:54.898]  130 sec: 10387888 operations; 80287.27 current ops/sec; [UPDATE AverageLatency(us)=1138.84] [READ AverageLatency(us)=445.68]
         [2015/10/01 18:01:04.898]  140 sec: 11229191 operations; 84130.3 current ops/sec; [UPDATE AverageLatency(us)=1097.95] [READ AverageLatency(us)=413.37]
         [2015/10/01 18:01:14.899]  150 sec: 12017060 operations; 78779.02 current ops/sec; [UPDATE AverageLatency(us)=1155.19] [READ AverageLatency(us)=462.05]
         [2015/10/01 18:01:24.899]  160 sec: 12087119 operations; 7005.9 current ops/sec; [UPDATE AverageLatency(us)=1683.19] [READ AverageLatency(us)=535.9]
         [2015/10/01 18:01:34.899]  170 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:01:44.899]  180 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:01:54.899]  190 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:02:04.899]  200 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:02:14.899]  210 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:02:24.899]  220 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:02:34.900]  230 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:02:44.900]  240 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:02:54.900]  250 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:03:04.901]  260 sec: 12087119 operations; 0 current ops/sec;
         [2015/10/01 18:03:14.901]  270 sec: 12087119 operations; 0 current ops/sec;
        
        

        1. stack-trace.log
          246 kB
          Rui Zhang

            Assignee:
            michael.cahill@mongodb.com Michael Cahill (Inactive)
            Reporter:
            rui.zhang Rui Zhang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: