How KnowBe4 solved the "It Works on My Machine" problem with a new approach to provisioning test environments. Before we dive in to KnowBe4's workflows, let's go on a little trip through the history of test environments.

Step in to a Time Machine...

Your alarm sounds. You hop out of bed before dawn, put on your business attire, and begin the hour drive to your office in the city. After all, you are a software developer, and you have to look your best while sitting in your office, alone, writing code. But today was a special day: the new piece of software your team had been writing for the past 12-months was scheduled to deploy to production for the first time. It had been well-tested, you thought, across these varying (static) acceptance-testing environments, and nearly every employee involved would be in an all-day (or longer) meeting to coordinate the deployment of this brand new piece of software.

You arrive 5-minutes early, but your manager has bad news - a bug was discovered the day before, and that meant your project would have to wait until next month to deploy. With a short sigh, you make your way to your office. You're not surprised, though; this marks the second month in a row where your software didn't make it out. Oh well, third time's the charm I guess...

Software Development in the Cloud-Based Era

The present-day software-development landscape is completely different than it was 30, 10, or even 5 years ago. The days of massive code deployments are being quickly phased out, and are instead replaced with small, iterative changes. Rather than all-day deployment meetings, software ships from the comfort of a developer's own home without so much as a zoom, often without many other parties even being aware of the changes. Yet in many cases test environments remain the same - static, provisioned, shared, and ripe for opportunity to fail.

...test environments remain the same - static, provisioned, shared, and ripe for opportunity to fail.

And the more of these static test environments that exist, the more likely they are to have problems and the less representative they are of a proper test environment. Being static, the chance of drift between the test environment, say staging, and the production environment is high. Clasically, these environments were managed by an IT team and kept up-to-date by hand. In modern development, they may be provisioned by a template, but without proper gating and planning even those templates can introduce drift.

These environments are often shared, too. This can be a bottleneck, or result in partially complete deployments by two developers on separate teams testing against the same environment simultaneously. This quickly frustrates everyone involved and results in an entirely familiar problem - both developers' code worked when they tested it locally, but neither's worked once it was deployed to staging.

Finally, it was once common for a piece of software to have a laundry list of dependencies, and missing or improperly versioning just one item could completely change the way an environment behaves. Modern software repositories often include a robust way to define their dependencies, but in some industries it is still common for the test infrastructure to be provisioned without such rigidity. Whether due to staleness of data, a missing library, or another form of inconsistency, it can be just as hard as ever to be confident that software which worked in a test environment will work the same way in production.

it can be just as hard as ever to be confident that software which worked in a test environment will work the same way in production

The Ideal Test Environment

So if the classic solution to test environments doesn't work for modern software development, what should the ideal test environment look like? Does testing in a remote environment have value at all, or should we ship straight to production after testing locally? At KnowBe4, test environments still play a large role in our Software Development LifeCycle, and are used in a way that empowers developers and test engineers to iterate quickly while ensuring a quality product.

That being said, here are some of the requirements of an ideal test environment:

• The test environment should be as identical to production as possible. This includes:
  • Infrastructure
  • Instrumentation
  • Data
  • Deployment strategy
• The test environment should be isolated to a single small feature to eliminate unexpected behavior caused by unrelated changes, particularly from other engineers.
• The engineer should be able to perform intricate integration tests that function and produce real results just as they would in production.
• An engineer should be able to deploy to this environment as-needed, and deployments should complete quickly.
• If necessary, multiple engineers should be able to collaborate and share the environment, in order to facilitate projects with multiple contributors.
• Any seeded data should be designed to cover a very wide variety of scenarios, and the seeds should be able to be reset.
• The test environment should be isolated from other environments, such that an issue with one environment does not compromise the integrity of another environment.
• Other stakeholders, such as QA Engineers, Product Managers, or even the Sales team should be able to access the ideal test environment to test or demo new or changed functionality.

KnowBe4's Evolving Solution

At KnowBe4 we have aimed to find a solution that covers as many of the criteria from the ideal test environment as possible, and the result was On-Demand development environments that can be created and destroyed as-needed. Each engineer can have many ondemand environments, and each environment is comprised of as many (or few) services as the developer needs to be able to build and test the feature being developed.

These environments require specific planning and design in order to satisfy the ideal requirements listed above. First, the underlying infrastructure must be built in a way that it is as close to a mirror of the production environment as possible. At KnowBe4 we solve this by using Terraform, and describing our infrastructure with modules that allow us to deploy repeatable infrastructure with all of the same components in all environments. We take advantage of Terraform's workspace feature which enables us to change small, specific features (for example, reducing the memory allocated to a lambda function) on a per-environment basis without materially changing the overall design of the infrastructure.

A Typical On-Demand Workflow

By having access to create a unique test environment, an engineer now has access to a new workflow - one that enables collaboration points on the developer's timeline rather than being forced to solve or deal with the issues that arise when using shared test environments. The typical workflow of an engineer using On-Demands is as follows:

• You are assigned various tasks (in a sprint, using waterfall, or otherwise)
• Each task is evaluated and worked on in your local development environment
• After completing local and unit testing, the code associated with the task is pushed to a feature branch
• The feature branch for the given service is deployed to an On-Demand, along with any other services needed to complete integration testing (whether automated or manual)
• The deployments of the various services happen in parallel (where possible), in order to facilitate a rapid deployment.
• Basic test data is seeded as part of this deployment process, and additional specific seed data can be added if necessary.
• Upon identifying bugs or other issues, the engineer deploys iterative changes to a specific service within the ondemand environment.
• Upon satisfactory completion of testing, the On-Demand environment is terminated, its resources torn down (using terraform destroy), and the feature branch is merged in to master, triggering a production deployment.

Check Back for More Soon!

In the next parts of this series, we will dive in with specific examples of our testing strategy, our infrastructure-as-code, our deployment strategies, and the internal application that orchestrates these On-Demand environments.

We're Hiring!

Have something to contribute? Passionate about purpose-driven, highly-productive software development? Send us an application! KnowBe4 Engineering is always looking for more talented engineers just like you! Check our open positions on our careers page - www.knowbe4.com/careers.