Hi Mongo DB Tools Support Team, 

We recently (i.e. in 08/2020) upgraded a single-node MongoDB cluster running version 3.4 to a 3-node (primary, secondary, and arbiter) cluster running 3.6. As such, we are in need of running on-demand DB refreshes from one environment to another and are using the mongodump and mongorestore utilities with the arguments mentioned below.

Some important notes:

• The MongoDB cluster runs in Kubernetes and was deployed using the following Helm chart: https://github.com/bitnami/charts/tree/master/bitnami/mongodb
• The mongodump and mongorestore commands run inside a Docker container running Mongo version 4.4 and connect to the MongoDB cluster "externally".
• The variables are passed using environment variables at runtime.
• The backup/restore size all of the required collections is about 20GB.

# Backup
mongodump \
 --host "${SOURCE_DB_HOST}" \
 --port "${DB_PORT}" \
 --username "${SOURCE_DB_UN}" \
 --password "${SOURCE_DB_PW}" \
 --db "${DB_NAME}" \
 --authenticationDatabase "${DB_NAME}" \
 --out "${OUTPUT_DIR}" \
 --quiet

# Restore
mongorestore \
 --drop "${OUTPUT_DIR}/${DB_NAME}" \
 --host "${DESTINATION_DB_HOST}" \
 --port "${DB_PORT}" \
 --username "${DESTINATION_DB_UN}" \
 --password "${DESTINATION_DB_PW}" \
 --db "${DB_NAME}" \
 --authenticationDatabase "${DB_NAME}"

The backup runs successfully but the restore fails every time on the final/largest collection of size 15.5GB (hence I've marked this a P2). Each failure results in one of the following errors:

1. "Failed: <redacted>: error restoring from <redacted>: (NotMaster) not master".
2. "incomplete read of message header: EOF".

As a debugging step, we've tried the solutions in the following articles to no avail:

https://stackoverflow.com/questions/34142673/mongo3-0-restore-fails-with-error-failed-restore-error-insertion-error-eof

https://jira.mongodb.org/browse/TOOLS-939

https://medium.com/synapseinc/tricks-to-restore-large-mongo-collections-on-small-machine-bc5c27c9076b

https://groups.google.com/g/mongodb-user/c/Ms2MXCajreg/m/OsW6ixcSCgAJ?pli=1

(Created by me and adding it as a reference) https://jira.mongodb.org/browse/TOOLS-2666

Additionally, I have confirmed that the restore only fails on the large collection by removing it from the restore which results in a consecutive success. Finally, I have also tried tweaking the mongorestore command arguments but the process continues to exit with the one of the two errors mentioned above.

What can we try next to resolve this issue? Let me know if you need more information to guide us in the right direction and I'll do my best to provide it for you.