In non-mmf mode monoperf deletes a file with constant non-configurable name mongoperf_testfile_tmp, creates it again, and then in each thread performs random 4k write followed by fdatasync.

On ext4 fdatasync caused inode mutex to be taken. So, whether I give mongoperf 1 thread or many because of this mutex I always get the same throughput.
It took me a while to figure out why this was the case. I think users should be warned about this.

Another issue is that the natural solution to this is to launch N 1-threaded instances of this program. But because the file name is not customizable they end up writing to the same file. But there is yet another catch. Because each instance first deletes the file and creates it again the there will be 64 different inodes created for the same file name and only one is real. Nonetheless the effect of this mess is that 64 instances can freely write to different files and collectively reach much large IO throughput.
All of this is extremely confusing.

The utility needs to expose file name via config parameter so that the user can make intelligent choices of that name in each instance. Also, deleting the file in the beginning even if it exists - I would expose another option to control that.