-
Type: New Feature
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
None
-
Fully Compatible
-
TIG 2017-10-23
The changes from SERVER-23111 introduced a serial_run.yml test suite to run tests in a jstests/serial_run/ directory with --jobs=1. This is undesirable because (1) moving tests to run in the serial_run.yml test suite may cause coverage from test reuse in different configurations to be lost and (2) moving tests to the serial_run.yml test suite disrupts the ability for the directory structure to convey the functional area covered by those tests. As an alternative, it would be useful if we could set the following properties on a resmokelib.testing.Suite instance
- fail_fast - whether to continue on test failure or not
- include_tags - whether to run certain kinds of tests or not (e.g. unreliable, resource intensive, etc.)
- num_jobs - the amount of concurrency to run certain kinds of tests with
- num_repeats - the number of repetitions to run certain kinds of tests with
- report_failure_status - whether to mark tests as "fail" or "silentfail"
to capture the settings needed to run (or not run) unreliable, resource intensive tests. We could then have another script invoke resmoke.py for each test suite to run a version that
- includes all unreliable AND resource intensive tests
- includes all unreliable AND non-resource intensive tests
- includes all non-unreliable AND resource intensive tests
- includes all non-unreliable AND non-resource intensive tests
- causes
-
SERVER-33075 Evergreen tasks with tests that have only "silentfail" status are still marked as failure
- Closed
- is related to
-
SERVER-43802 Create ReplSetTest and ShardingTest control tests
- Closed
- related to
-
SERVER-31579 Tag specific sharding tests as being resource intensive
- Closed