Enter your email address to follow our blog with all the latest software engineering tricks, tips, and updates from the R&D team at KnowBe4!
By: Matt Duren
Published: 21 Mar 2022
Last Updated: 12 Aug 2022
How KnowBe4 solved the "It Works on My Machine" problem with a new approach to provisioning test environments. Before we dive in to KnowBe4's workflows, let's go on a little trip through the history of test environments.
Your alarm sounds. You hop out of bed before dawn, put on your business attire, and begin the hour drive to your office in the city. After all, you are a software developer, and you have to look your best while sitting in your office, alone, writing code. But today was a special day: the new piece of software your team had been writing for the past 12-months was scheduled to deploy to production for the first time. It had been well-tested, you thought, across these varying (static) acceptance-testing environments, and nearly every employee involved would be in an all-day (or longer) meeting to coordinate the deployment of this brand new piece of software.
You arrive 5-minutes early, but your manager has bad news - a bug was discovered the day before, and that meant your project would have to wait until next month to deploy. With a short sigh, you make your way to your office. You're not surprised, though; this marks the second month in a row where your software didn't make it out. Oh well, third time's the charm I guess...
The present-day software-development landscape is completely different than it was 30, 10, or even 5 years ago. The days of massive code deployments are being quickly phased out, and are instead replaced with small, iterative changes. Rather than all-day deployment meetings, software ships from the comfort of a developer's own home without so much as a zoom, often without many other parties even being aware of the changes. Yet in many cases test environments remain the same - static, provisioned, shared, and ripe for opportunity to fail.
...test environments remain the same - static, provisioned, shared, and ripe for opportunity to fail.
And the more of these static test environments that exist, the more likely they are to have problems and the less representative they are of a proper test environment. Being static, the chance of drift between the test environment, say
staging, and the
production environment is high. Clasically, these environments were managed by an IT team and kept up-to-date by hand. In modern development, they may be provisioned by a template, but without proper gating and planning even those templates can introduce drift.
These environments are often shared, too. This can be a bottleneck, or result in partially complete deployments by two developers on separate teams testing against the same environment simultaneously. This quickly frustrates everyone involved and results in an entirely familiar problem - both developers' code worked when they tested it locally, but neither's worked once it was deployed to staging.
Finally, it was once common for a piece of software to have a laundry list of dependencies, and missing or improperly versioning just one item could completely change the way an environment behaves. Modern software repositories often include a robust way to define their dependencies, but in some industries it is still common for the test infrastructure to be provisioned without such rigidity. Whether due to staleness of data, a missing library, or another form of inconsistency, it can be just as hard as ever to be confident that software which worked in a test environment will work the same way in production.
it can be just as hard as ever to be confident that software which worked in a test environment will work the same way in production
So if the classic solution to test environments doesn't work for modern software development, what should the ideal test environment look like? Does testing in a remote environment have value at all, or should we ship straight to production after testing locally? At KnowBe4, test environments still play a large role in our Software Development LifeCycle, and are used in a way that empowers developers and test engineers to iterate quickly while ensuring a quality product.
That being said, here are some of the requirements of an ideal test environment:
At KnowBe4 we have aimed to find a solution that covers as many of the criteria from the ideal test environment as possible, and the result was On-Demand development environments that can be created and destroyed as-needed. Each engineer can have many ondemand environments, and each environment is comprised of as many (or few) services as the developer needs to be able to build and test the feature being developed.
These environments require specific planning and design in order to satisfy the ideal requirements listed above. First, the underlying infrastructure must be built in a way that it is as close to a mirror of the production environment as possible. At KnowBe4 we solve this by using Terraform, and describing our infrastructure with modules that allow us to deploy repeatable infrastructure with all of the same components in all environments. We take advantage of Terraform's workspace feature which enables us to change small, specific features (for example, reducing the memory allocated to a lambda function) on a per-environment basis without materially changing the overall design of the infrastructure.
By having access to create your a unique test environment, an engineer now has access to a new workflow - one that enables collaboration points on the developer's timeline rather than being forced to solve or deal with the issues that arise when using shared test environments. The typical workflow of an engineer using On-Demands is as follows:
terraform destroy), and the feature branch is merged in to master, triggering a production deployment.
In the next parts of this series, we will dive in with specific examples of our testing strategy, our infrastructure-as-code, our deployment strategies, and the internal application that orchestrates these On-Demand environments.
Have something to contribute? Passionate about purpose-driven, highly-productive software development? Send us an application! KnowBe4 Engineering is always looking for more talented engineers just like you! Check our open positions on our careers page - www.knowbe4.com/careers.