-
Type: Improvement
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Not Applicable
-
Storage Engines
-
5
-
StorEng - Defined Pipeline
Summary
Estimate costs for running various tests
Motivation
- Ability to run various tiered storage tests backed by dir_store and have an idea of how much they cost per run.
- Is this issue urgent?
Not urgent. Appropriate for new hire, intern or skunk works.
Acceptance Criteria (Definition of Done)
The "dir_store" extension should be modified so that it has an option to collect statistics on bytes transferred (read/write), total bytes stored, etc. We should have a mechanism to show these when the storage source is closed (that is, when the connection is closed). We can apply a simple dollar calculation using numbers from https://aws.amazon.com/s3/pricing/?nc=sn&loc=4 . Perhaps use "standard pricing" for simplicity.
Suggested Implementation
S3_store uses the msg_printf callback to transmit these to the caller. Unfortunately, this puts the burden on the individual python tests (or test/format or test_tiered_abort, etc.) to ignore these kinds of messages, but report or collate them at the right time. Part of the issue is that many of these tests will open/close connections many times, and so will want to total the statistics at the end of the entire run. Perhaps better is to have a log file location passed as argument to the extension, the log file is appended to and the entire "cost log" can be collated/added at the end of a run, by a separate program. (rather than having the adding up done separately by each test program). For purposes of this ticket, just printing them to a shared log file would be enough.