We currently calculate the number of partitions in the query stats store initially to be 16MB each. If there are less partitions than the number of cores we want to make sure that we can parallelize as much as possible, so we increase the number of partitions by 400, making them 40KB each. This works because the approximate size for a query stats entry is 4KB, and so there should be room to spare.

Nonetheless, a problem arises when the query stats store is resized during runtime, as we cannot at this point change the number of partitions, so the only thing that happens is they all become smaller. If the change is drastic enough we might get partitions small enough that not even a single entry can be recorded, which is undesired behavior. This has happened already in 3 separate BFs (linked below).

If we don't want to change partition algorithm, we might want to limit the cache resizing or at least prompt a warning of some sort.

There's also two TODOs in query_stats_concurrent.js to adjust values after any changes made in this ticket.

Calculation spreadsheet for reference.