Prior to MongoDB 3.6 (WT 3.0), WiredTiger used to keep a separate cache in each session of the tables used by that session.
In workloads with many tables and many sessions, the memory overhead of this per-session table cache can be very large. This was fixed in WT-3476, but that change cannot be backported. Find a simpler approach to reducing the overhead that is possible to backport (e.g., limit the cache to some small number of tables).
Note that the solution must be tested against the changes for SERVER-33122, which will provide the option to configure how cursors are cached in MongoDB.