From: Geert Bosch <geert@mongodb.com>
Subject: Design for thread-safe MMAPv1 Extent Manager (EM)
Date: October 10, 2014 at 4:19:01 PM EDT
To: locks-and-docs <locks-and-docs@10gen.com>

[TL,DR: Use lock manager lock for EM changes, preallocate _files to allow lock-free reads]

***

In order to enable collection level locking for MMAPv1, the EM,
which is per-database, must handle concurrent read/write access 
by operations on different collections.

It is not possible to just protect the EM using a mutex, as operations
may require rollback. Because we do rollback at the memory write level,
it is not possible to rollback one operation without affecting other
concurrent operations.

So, non-const calls on the EM must use a lock that is held until the 
write operation commits (end of the outermost WUOW). The simplest is
to just use locks managed by our lock manager and a new ResourceType,
RESOURCE_EXTENT. 

The locking order (for MMAPv1) will be:

 RESOURCE_DATABASE -> RESOURCE_COLLECTION -> RESOURCE_EXTENT

As long as the extent is the last resource obtained, it should never
be involved in deadlocks.

One issue is that if we'd require recordForV1() and associated methods
to acquire locks as well, that would be too expensive and unnecessarily
serialize everything. As we essentially only require access to the _files
vector, and we only append to the vector without modifying existing entries, 
automatic reallocation means we can't just read without synchronization.

Given the maximum possible number of files (16K), we only need 8 bytes per 
file * 16K files = 128K bytes of memory per database to accommodate the max 
number of files. Even for a large deployment with 1000 databases, the 128M 
of virtual memory should not really be significant: the working set doesn't 
increase.

This will be simple to implement and efficient.

 -Geert