QA could have caught but missed on MS03
With multiple passes of regression the attention to detail for long-standing functionality is difficult to maintain. Some of the minor things missed by QA were modifications of appearance or the lack of data in a particular field/column.
Immediate To-Do:
- Developers to provide more insight to QA for back-end changes (complexity, risk and potential impact to other areas of functionality)
- Developers to include time for reviewing QA test plans as part of estimates
- Developers to perform code reviews on all code changes prior to the qa cycle
Long term:
- Automated Regression Testing (QA + Dev)
Can’t Test On MS03
Due to configuration & system differences between QA and stage/live there are some scenarios that cannot be reliably tested in the QA environment. These are:
? Financial Transactions (CP) - actual interactions with payment providers
Immediate To-Do:
- QA: Ask Business Tech team about sandbox environments (how do they test?)
- QA: Allocate appropriate amount of time for testing these scenarios on Stage and prioritize these tests first.
Long term:
- Ensure a sandbox environment is provided with the new e-commerce vendor; if not, stub out the functionality.
? Not enough data in stage – performance issues not discovered
Immediate To-Do:
- Dev & DBA: Run all new queries by the DBA for performance review
- QA & Tech Ops: Determine the amount of data required in the qa db for ineffective queries to be humanly apparent.
- QA: Add acceptable response times to test cases, execute these test cases during the Stage launch
- PM & CSD: After QA signoff, conduct 1-2 hours of ‘live-test’ usage of the new tools from the Stage environment prior to the production launch *** NOTE: This will require email to be turned ON in the Stage environment.
- Tech Ops & Dev: Perform benchmarks of system performance and actively monitor during testing (both db and webservers)
Long term:
- QA: Automated load testing
- Tech Ops: Provide a testing environment with ‘equivalent’ performance & amount of data. (Full mirror vs ‘equivalent’ to be defined).
? Some States are not easy to create
o Membership Grace
o Date Ranges (unviewed tickets)
o Inactivated Accounts
o Deleted Accounts
Immediate To-Do:
- QA: Add the list of known accounts in specific states to the test plans
- QA & Dev: Create the database scripts to put accounts into specific states
OR
- QA & Dev: Create queries that will help QA find the accounts that ARE in specific states
- QA: Create accounts in static states in prod so they are reliably copied over
Potential Longer Term Solutions:
- DEV & QA: Write an import tool
? There is no DB Cleanup/refresh procedures
- QA & Tech Ops: Create a test environment strategy that is easy for QA to refresh the db for a new pass of testing.
? Some test scenarios that require VW integration tests
Immediate To-Do:
- QA: Identify which tests include VW tests, Prioritize these as first for regression on Stage
Potential Longer Term Solutions:
- Tech Ops & QA: Integrate support testing with production test environments
- QA: Determine how to verify data without the virtual world (i.e. use SQL to examine data storage)
Stage Issues (Fixes resulted in issues)
As a result of our system differences between QA and stage/live there were quite a few issues that were discovered for the first time on the stage environment. The lack of process regarding how to handle these issues resulted in a number of ‘rushed’ fixes which resulted in further issues.
Immediate To-Do:
- PM: Schedule more time for the ‘Stage’ phase between QA signoff and Production launch
- New Process : Implement the following process for issues discovered on Stage:
o Blocker & Critical issues to be discussed immediate following the morning scrum.
o Major, Minor and Trivial issues reviewed after the complete first pass
o Information required for a decision on all issues:
§ How is it impacting the user?
§ What is the complexity of the fix?
§ What is the Level of Effort for the fix?
§ What is the Risk to other functionality of the fix?
§ What is the amount of re-testing required for the fix?
Live Issues
Once QA signed off on the Stage environment, the Live launch had to be rolled back three times due to performance usability issues. Furthermore, the production launch was compromised due to a tech-ops/DCAT ticket that had been created in October but not completed by the December launch.
? Load & Performance (see above related section)
? External Dependencies
Immediate To-Do:
- Developers: Create the release page for the next version as soon as code is branched for a release candidate and work continues on the trunk
- Developers: Add ‘release dependencies’ to the release page (i.e. techops-711 must be complete before a deployment to Stage)
- Developers: Always create external dependencies as blockers
- PM & Tech Ops: Ensure all blockers & release dependencies are resolved prior to deployment.