-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
Affects Version/s: 2.8.0-rc5
-
Component/s: WiredTiger
-
Fully Compatible
-
ALL
Under heavy insert or update load the oplog can grow much larger than the configured size, possibly indefinitely. I've triggered this with a few different workloads, sometimes reproducibly and sometimes not. 100 threads of the following with a configured oplog size of 50 MB in a 1-node replica set seems to do it fairly reliably:
function repro(t) { db.c.drop() db.c.insert({_id:t, i:0}) big = '' for (var i=0; i<100000; i++) big += 'x' count = 1000000 every = 100 for (var i=0; i<count; ) { var bulk = db.c.initializeOrderedBulkOp(); for (var j=0; j<every; j++, i++) bulk.insert({x:big}) try { bulk.execute(); } catch(e) { print(t, 'OOPS') } if (t==1 && i%100==0) print('MB', db.getSiblingDB('local').oplog.rs.stats(1024*1024).size) } }
Oplog growth as reported by the above code for two different runs, in both cases going well past the configured 50 MB size:
first run:
MB 421
MB 335
MB 1381
MB 2260
second run:
MB 263
MB 989
MB 1387
MB 2417
In both cases the cache size was 5GB, but virtual memory grew to about 8GB, and mongod was killed by the OOM killer.
I think the issue is that there's no back-pressure in capped collection inserts to throttle inserts to prevent inserts from outrunning deletes.
- duplicates
-
SERVER-16868 Significant slowdown in capped collection performance when cap reached
- Closed
- is duplicated by
-
SERVER-16868 Significant slowdown in capped collection performance when cap reached
- Closed
- is related to
-
SERVER-17033 Improve performance for bulk insert into WT with under oplog back pressure
- Closed