-
Type: Improvement
-
Resolution: Gone away
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Sharding
-
Catalog and Routing
-
Sharding EMEA 2023-10-16, Sharding EMEA 2023-10-30, CAR Team 2023-11-13, CAR Team 2023-11-27, CAR Team 2023-12-11, CAR Team 2023-12-25, CAR Team 2024-01-08, CAR Team 2024-01-22, CAR Team 2024-02-05, CAR Team 2024-02-19
-
(copied to CRM)
The sharding balancer currently issues listDatabases against every single shard in order to access the totalSize value. This value is used for ensuring that a shard's storage maxSize is not exceeded for users which have that value set.
The listDatabases call is quite heavy, especially for nodes with large number of databases/collections since it will fstat every single file under the instance.
There are a number of optimizations we can make in order to make this statistics gathering less expensive in the presence of maxSize (listed in order of preference):
- Add a parameter to listDatabases to allow it to return cached data size instead of every time {{fstat}}ing all the files
- Issue the listDatabases call in parallel against all shards so different shards' execution overlaps
- Cache the per-shard statistics so that they are not collected on every single round/moveChunk invocation
- Collect the per-shard statistics asynchronously so that multiple concurrent moveChunk requests can benefit
- depends on
-
SERVER-84187 Regression in balancer round storage statistics retrieval
- Closed
- is related to
-
SERVER-30060 Make the balancer gather storage statistics only for shards which have `maxSize` set
- Closed