How Many Performance Scenarios?
It’s a simple question, but the answer is that “it depends” (upon the context).
To start to answer the question we need to understand something about the specific risks for your project/product:
- What would the impact of performance issues be?
- How likely are these issues to occur?
- What is the risk appetite?
- How much performance risk do you want to mitigate? (i.e. how much time do you want to spend writing, running, analysing, and maintaining scripts)
Understanding Performance Risk
To start understanding and building a model there are more questions to help:
- How complex is your application?
- What is the overall volume of transactions?
- What is the size and “shape” of the data?
- How will the system grow in 1, 2, 5 years?
Some “Rules of Thumb”:
- More than 2
- Fewer than 20 “simple” scenarios
- Fewer than 10 “complex” scenarios
The rational for these are to try to balance the effort and benefit. This also assumes that the performance scripts you will create, like most1, are quite brittle and will likely require a lot of re-work if the application changes frequently.
- this depends on the application under test, tools used, time allowed and the people writing and maintaining the test scripts ↩