-
Type: Improvement
-
Resolution: Done
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Sharding
-
None
If you start with a sharded cluster - don't pre split manually, and then pour in data as fast as you can the balancer struggles to keep up and you end up pouring all data into one shard.
Whilst chunk migration is deliberately slow normally we should change the way chunk splitting happens early on to get all shards into play at the first possible opportunity, possibly by splitting an migrating 'tiny' chunks - at 256K rather than 64MB - growing the chunk size later once we have all shards in play
We should also optionally prioritise chunk migration over insertions where we have a very large imbalance in chunk numbers for example many empty shards.