Resmoke currently runs all concurrent jobs with little to no isolation between them. In some cases, this leads to jobs interfering with each other. When they do, it is often difficult to quickly understand what happened, since debugging such issues often requires noticing a correlation between multiple log files.

We should consider what isolation guarantees resmoke could provide that would reduce this category of issues. 

Some initial thoughts (feel free to add if you have ideas):

• Run each job (or test) in its own working directory.
• Track subprocesses explicitly, and only operate on its own subprocesses. This is done in some places already at the resmoke-process level. It could be extended in scope and specific to indiviual jobs/tests.. https://github.com/mongodb/mongo/blob/master/buildscripts/resmokelib/core/process.py#L96

It is worth noting that Bazel's test behavior provide a lot of this type of guarantees. This ticket may just become a no-op if integration tests are run via bazel.