-
Type: Improvement
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
Fully Compatible
-
v3.6
-
TIG 2018-03-26
-
19
We've occasionally seen NetworkTimeout and ServerSelectionTimeoutError exceptions when attempting to insert the canary document as a result of mongod having crashed or hung. If the mongod process is hung, it'd be beneficial to trigger the hang analyzer script. Given how the powertest.py script writes to its stdout and stderr, we should be able to set timeout_secs for the "run powercycle test" function in each of the powercycle* Evergreen tasks and have that correspond to the time limit for a single powercycle loop.
We should raise the "serverSelectionTimeoutMS" and "socketTimeoutMS" parameters used when constructing a pymongo.MongoClient instance to be well above whatever we use for the timeout_secs value to ensure the script doesn't exit before Evergreen has a chance to trigger the hang analyzer.
# Set the serverSelectionTimeoutMS & socketTimeoutMS to 10 minutes mongo_args = { "serverSelectionTimeoutMS": 600000, "socketTimeoutMS": 600000 }
- causes
-
SERVER-34412 powercycle* tasks set timeout_secs in wrong location of task definition
- Closed
- has to be done after
-
SERVER-33691 Remote EC2 instances should have a log disk
- Closed
- related to
-
SERVER-34318 Set timeout for canary insert to 1 hour in powertest.py
- Closed