Our setup is capable of running calls at 50-60k TPS and during this some of our replica sets were going into recovery mode and they couldn't sync with the primary. I know that I can recover my system by stopping mongo, deleting data sets and then starting it back on will sync with primary as per the guide https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/

We saw the below error message on our logs 

Resync is needed for secondary member SESSION-SET12:set01j2:vm01:27737 this member is lagging behind by 14747 seconds from the primary

Recovering system is not an issue but would like to know why some of our replica sets are going into recovery mode in the first place and request you to kindly assist

Mongo version
[root@vm01 ~]# rpm -qa | grep mongo
mongodb-org-3.6.17-1.el8.x86_64
mongodb-org-mongos-3.6.17-1.el8.x86_64
mongodb-org-tools-3.6.17-1.el8.x86_64
mongodb-org-server-3.6.17-1.el8.x86_64
mongodb-org-shell-3.6.17-1.el8.x86_64a

Thanks and Regards,

Azhar