Minimum setup for efficient DevOps — Part 2 — proper pre-prod environments
This is Part 2 of a series on DevOps. We are still building the foundation. The building is not yet visible, but trust me, the foundation is badly needed, plow though the boring and mundane.
Critical part of deploying changes is ensuring quality. We will deal with the process in Part 3 and in Part 5, here we will set necessary more foundation.
If you happen to have a set of automated tests, that allow you to skip manual testing any any kind, I congratulate you and promise to support your bid for your place in the IT-hall-of-fame. However, chances are you need some manual testing. Efforts move solely to automated unit tests can be commanded but this article is for the many who cannot wait until the utopia is realized(1), and have to support changes to their systems now.
So, we assume you need manual testing of some kind, and for that, you need a proper set of environments to deploy your code to. The justification for what I claim is needed in this part, will comes in Part 3. Part 3 describes the process that uses what we build here. None of what I propose is new, but sadly, few organizations do this right.
I want to reiterate important fact. I am not describing here an ideal modern DevOps. I am describing a minimum transition for a traditional team, to a highly efficient setup that will serve as a launch point for further changes and optimizations.
The pre-prod environments
You will need at least 2 properly setup pre-production environments. The environments go by many names; but names are not critical. However, how you will use these two environments, is critical. This article describes the environments and some critical characteristic, Part 3 describes the process of using them
The minimum pre-prod environments
At min, you will need a “QA” and “Staging” environments
QA
This will be a place where you release your code/application/system and where manual testing can take place (and automated, non-unit testing). Basically, this is where your QA team performs 95% of their testing. How the code will be released, what kind of testing to do, is a subject of another part. For now suffices to say, QA is a fully functional , separate and dedicated environment for the QA team to perform their testing.
Staging
“Staging” environment (often called other names too like UAT) is for deployment process testing. Sanity-testing will be performed there by the QA team to determine is the deployment process worked.
—
Characteristic of the environments
Staging should as close as possible a copy of production, in all respects. The code of the applications should be identical. Configuration of the application should also have as few changes as possible. Data should be identical minus masking any sensitive data that cannot be exposed to none-production environments. Configuration of the servers that the system runs on should be as close to identical as possible.
There must an easy way of refreshing staging from production to sync up staging with the latest code, data and configuration of production. The process of doing this must be automated to the maximum.
Staging will not be a place where actual functional testing of new features or regression testing is performed. All testing will be performed on the QA environment. Staging will be used only to test the release process. QA is for testing the changes to the application. Staging is for testing the release process of the release of changes to the application. This is a critical distinction, which will make more sense in part 3. Once the application tested on QA environment is considered working, the code is released to staging following the exact same process as it will be eventually released to production. A sanity check on Staging will be used to confirm that the release was successful. The scope of the sanity check will very, and will be discussed in other parts.
Staging should be treated almost like production. Access should be limited. This is to ensure that no manual fixes or interventions are allowed in this environment. If a bug is found on staging, it can never, ever be fixes directly on staging. The bug must be fixed and tested on QA. We’ll elaborate on that in in Part 3.
QA environment, like staging, should be as close to production as possible. The closer QA is to production, the higher the chance that testing on QA will reveal all bugs. However, typically, due cost considerations, QA maybe more different from production that staging is from production — perhaps not the entire dataset maybe available. Often this is to save a few hundred dollars a year, which in my opinion is a really bad barging. Try to keep QA as close of a copy of prod, as possible.
Easy and automated way of refreshing QA from production is also needed, and since it will be executed more often that a refresh of staging, should be relatively fast. Couple of hours max is a good rule of thumb. This is another reason why sometimes QA is not an exact copy of production data. Err on the side of closeness, rather than optimizing for speed of refresh or cost of the environment.
2 more pre-prod environments
You can add two more environments to this mix — DEV and User Acceptance Testing (UAT) . Development is a pre-QA environment that developers can use to test their code a bit earlier. Automatic deployments should be setup there. UAT is often a synonym for Staging, but can be a separate environment especially if you want external clients or user to test changed before they are made live. This environment works great especially when pointing to production Database, then you have a sort of A/B testing setup.
Multiple QA and Staging environments
In this process, you will likely find yourself having to test separate code bases. For this reason, be prepared to be able to create not just 1 QA or Staging environment, but many.
Footnotes
(1) It is possible to build system that rely purely on automated unit tests, however, for most legacy systems, trying to retrofit the systems with unit test to replace manual testing, is an utopic vision. By all means try to get as far as you can, but manual testing is likely to be needed for the foreseeable future
