Right now there are many rules that developers have to keep in mind when it comes to making feature flag checks safe with concurrent setFCV commands:

• Each thread must only check the feature flag once
  • Risk: Otherwise the first feature flag check could return one result while a subsequent check returns a different result. This could cause the feature to partially execute in an unexpected way
• If the feature affects what could be written to disk, we must be holding the global lock in a mode conflicting with S when doing feature flag checks
  • Risk: Otherwise there is a race condition possible where the feature flag check returns one result and then the setFCV operation fully completes (either an upgrade or a downgrade). This would mean that the original operation could execute in an FCV that should not be allowed
• If the feature affects what could be written to disk, we should not execute feature flag checks on secondaries performing oplog application. The oplog entry itself should tell the secondary how to execute the operation.
  • Risk: If a secondary can make its own decision on what the FCV dictates the feature should be, it can race with setFCV and make a different decision than the primary, causing its data to diverge from the primary’s data.

These rules can be difficult to get correct, which can cause tricky bugs. We should try to rethink a way to not require developers to need to know as much about feature flags and setFCV to be able to write proper feature flag checks.