Summary
Add tiered storage python tests to PR testing.
Motivation
I recently added tiered-storage-extensions-test to a couple of test variants. This task runs the python tiered storage tests on all the storage sources (for now local store and S3 store). The task requires a special wiredtiger compilation, with S3 enabled. S3 compilation takes significant time as it has to download and build the AWS SDK each time. Hence S3 is not built by default. tiered-storage-extensions-test builds WiredTiger with S3 enabled and then runs the tiered storage tests on both local store and S3. The local store is built by default and already tested by pull request as part of running python tests.
This leaves us a possibility that a tiered storage change will pass on the local store, be PR tested and checked-in, and possibly fail later on S3 store. Hence I will like to add S3 testing to the pull request tests as well.
Note:
Do consider the costs involved in interacting with S3. We should reduce the tests included in the PR to a small set that gives confidence that there is no major breakage, while keeping costs acceptable.
- Is this issue urgent?
Somewhat, as we do more tiered storage changes, the possibility to break something in S3 increases. It will be better to catch the failures early on, before a check-in into develop.
Acceptance Criteria (Definition of Done)
PR testing builds the S3 extension and runs some of the S3 tests.
[Optional] Suggested Solution
A simple solution is to add tiered-storage-extensions-test to PR testing as such. The test at the moment takes 17 mins, and doesn't depend on any other task. So it should finish well before other tasks if it starts to execute in time. But since the test will be running local_store tests too, which are already covered under regular python unit testing, we can potentially reduce time by running only the S3 scenarios somehow. This could be an enhancement to the naive solution - to pass an argument to the task to run a smaller set of the tests to reduce the run time.