We recently upgraded our MongoDB cluster from version 4.4 to 7.x, and since the upgrade, we have been experiencing significant write performance issues. The write operations, which previously completed in a timely manner, are now taking considerably longer. This is severely impacting our application's performance and user experience. FCV set to 6.0 and after downgrade to 5.x issue resloved. 

We are looking for on the root cause of the performance degradation. While we found a related issue on the community page https://www.mongodb.com/community/forums/t/performance-drop-after-upgrade-6-0-10-7-0-1/246712/4, which mentions a fix in version 6.0.11, our performance issues persist even after applying this update

Details:

• MongoDB Version: Upgraded from 4.4 to 7.x
• Cluster Configuration:
  • Number of nodes: 5
  • Replica Set: Yes
  • Sharding: [No]
  • Deployment Environment: [e.g., On-premise, AWS, etc.] AWS
• Symptoms:
  • Write latency has increased significantly across all nodes.
  • Logs indicate several warnings related to writeConcernMajorityJournalDefault and cache evictions.
  • Schema locks are more frequent, causing contention and delays in write operations.
  • To enhance performance during high-write scenarios, we adjusted the Read and Write concern level from 'majority' to '1'. By setting setDefaultRWConcern and defaultWriteConcern to '1', we eliminated the need to wait for acknowledgments from most nodes. This change significantly reduces write latency, especially during peak traffic, as only the primary node needs to confirm the write operation

• • And also observed schma locks on the logs too.

Steps Taken:

• Reviewed and adjusted writeConcern settings, but the issue persists.
• Analyzed mongostat and mongotop outputs, which confirm the degraded write performance.
• Restarted nodes to check for any temporary relief, but the issue remains.
• No significant issues were present in the logs before the upgrade.