This was a suggestion from Henrik during code review:

I think the server always store the data under min/max in the same order as they are stored under data. I see that we create two temporary maps to map the things together. Not sure how much we care but I believe it is possible to do this in the order they appear in BSON (plus a fallback if they do not). Would save you some allocations per bucket.