The background key generator thread is what generates signature keys for cluster time validation. The lifetime of the key generator is like this:
- Unconditionally enabled when a node initializes as a shard
- Disabled on step-down
- Enabled on step-up
- Disabled on FCV downgrade from 3.6 to 3.4
The following problems exist with these transitions:
- Because of (4) above, an FCV change sequence from 3.6 -> 3.4 -> 3.6 will not re-enable the key generator, so it will not generate new keys and will cause the router to fail starting-up
- This is not a major problem, because keys typically last for months and if a stall happens on router start-up, this can be worked around by stepping down the config server primary
- Because of (1) above, a secondary replica set node will end up with the key generator running
- This is mitigated because when that key generator tries to insert a new key, it will fail with a NotMaster error