Summary:
Salvaging a row-store file with a large number of overflow keys (or regular keys if there are enough pages), will run the cache out of memory in an unevictable page, and if it takes long enough, eviction can time out. Make sure the root page being built never becomes a threat to eviction.

The cache stalled in a sanitizer run:

http://build.wiredtiger.com:8080/job/wiredtiger-test-format-stress-sanitizer/25881/console

This is interesting because it's a reasonably large cache (180MB), but all the space is pinned in an object open for exclusive access (verify? rebalance?), and that shouldn't happen.

We didn't get a core dump, so the only path here is likely trying to reproduce it based on the CONFIG.