The Document class has a cache to speed up accessing fields on large documents that can increase the value of getApproximateSize as fields are accessed. The outer window function code greedily calls fillCache() on input documents so that their field caches are stable and remove() calls subtracts the same amount of memory as the corresponding add() call removed. However these accumulators still run into the issue since they generate a new Document Value when they calculate sortKeys and it starts out with an empty cache. remove() searches through the internal map structure to remove the value provided, so by the time we do the memory calculation the sortKey's field cache is already fully populated, thus causing remove() calls to subtract more memory than add() added.