Expediting Testing Without Sacrificing Quality

In a time when faster to market is a common theme in most tech companies, traditional manual testing becomes time consuming. Of course we want to always give our customers the best product and experience that we can but how do we accomplish this quickly? With daily deployments, we needed to take a hard look at our quality program. Here is how we are able to expedite testing while keeping the quality as high as possible.

Manual Testing

One of the very first fundamental changes we needed to make was the manner in which test cases were being written. Typically, software testing professionals write tests according to their own writing styles. This can become a problem when other testers need to execute the tests as the tests do not always include information that would be needed for someone who does not have the same knowledge the author has. To alleviate this issue, we implemented a standardized test case writing template that mandates the following items:

• A sensible clear title
• A short summary which explains why this test is needed and what it is to accomplish
• Prerequisites for any setup needed to complete the test case
• Clear and concise test steps
• Priority of the test at hand
• Automation Status (will or won't automate)
• Components & Labels

As we implemented the standardized test case template, we also taught our team the beauty of consolidation. It is not uncommon to include other test cases in the prerequisite section of a test case. For example, if there is already an existing test case that has steps needed for initial testing, the test case can be linked into the prerequisite section with instructions to execute certain steps. This speeds up testing as the author of the test at hand does not need to repeat steps that already exist in another test case.

Test planning is very important in any testing, especially when working on multiple projects. To make it easy for anyone leading the test effort or anyone working on it, we also implemented a standardized test plan template that mandates the following items:

• Test Coverage sections include
  • Functionality, UI/Usability, Navigation, and Error Handling Testing
• List of supported OS/Browsers for Compatibility Testing
• Automation section so we can ensure coverage when the feature is released
• Performance Testing to include Load Testing
• Localization Testing to make sure all new strings can be localized
• Sign off from Dev and PM project leads (this is a requirement to ensure correct scope of testing and provides the entire team knowledge of what is being tested)

Having test case and test plan templates standardized allows for a much better understanding of each test and its scope. This naturally leads to quicker execution of our tests as each test contains all information needed for someone without the product knowledge. The other added bonus of standardized test case writing is that our automation team (Software Engineers in Test) also have a clear understanding and can then automate the tests at a faster pace.

Automated Testing

Faster testing also requires the need for automated testing. Deciding to embed our automation team members (Software Engineers in Test) into our SCRUM teams, working side-by-side with manual QA Engineers and Software Engineers, is a fantastic way to keep up with new features.

Regression, smoke and sanity testing takes place in our production environment. Our automated regression suite runs nightly, our smoke tests run immediately after each deployment in all environments, and our sanity tests run every 15 minutes to ensure production health.

Manual and Automated Testing Together

The reality in software testing is we will never eliminate manual testing. So, how do we get faster? We needed to incorporate the ability to utilize automated tests while also running manual tests concurrently. This can be accomplished by first having a test plan that includes the full scope of testing, organized by priority of each test, including automated tests. Having the ability to kick off automated smoke tests and/or regression tests in all environments while also running through manual tests speeds up the testing process. It is also imperative to make time for exploratory testing and having a bug bash, all essential to the quality.

Remember faster is not always better but we can be smarter in our testing approach such as automating high risk test cases to cut down on the time needed to get releases into Production.

R&D Working Together

With our larger features, we implement a stabilization period to limit code churn, we call this War Room. When the SCRUM process ends and we are feature complete, the team as a whole works together to get the product tested and ready for production deployment. War Room has the single focus of getting the product stabilized and deployed on time.

Lessons Learned

Before implementing these changes in our quality program, each team was very siloed, including the QA engineering teams. It was not uncommon for each QA engineer to have completely different processes and test case writing styles. Coupling those issues along with not having any automated tests, all of these factors slowed the testing process thereby affecting our ability to deploy daily and/or on-time.

Taking a hard realistic look at the deficiencies we had was not easy but was necessary. Implementing standardized test case writing allowed for quicker automation. Working closely with our automation team as well as our development teams had the added benefit of faster feedback loops and allowed us to fix and/or implement processes quicker. With a full team approach, we were able to accelerate and modernize our testing.

We're Hiring

Have something to contribute? Passionate about purpose-driven, highly-productive software development? Send us an application! KnowBe4 Engineering is always looking for more talented engineers just like you! Check our open positions on our jobs page - https://www.knowbe4.com/careers.