The 2019 Drupal Community DevOps Survey shows where DevOps adoption is succeeding and where it needs improvement.
In the first part of this article, we explained why DevOps is the most important strategy for modern technology organizations. To learn more about how DevOps is being used, we launched the 2019 Drupal Community DevOps Survey, which is collecting data until we present the results at DrupalCon on April 10. The survey asks Drupal community members to assess their teams across 24 DevOps dimensions using a Likert scale.
Each dimension in the survey is scored, averaged across all respondents, and then rolled up into an overall score. Each DevOps dimension is also scored and can be considered individually for the value it has to the team and how the team might start or improve on it. Even though the survey is still in progress as we’re writing this, the results (which are updated as responses come in) already seem to be stabilizing across the Drupal community.
The Drupal community has been floating somewhere in the mid-60s on its use of DevOps practices. While the real value is for a team to track its high-level improvement over time, it could be understood superficially as meaning the Drupal community feels like it is consistent with DevOps practices a bit more than some of the time.
A quick-win opportunity: Releasing code quickly
If we scan the results for a high impact, easy win, we might choose something like:
At 83%, most teams are well on their way and don’t have much further to go on this high-impact dimension of DevOps. This means that, for many, this is a solved problem—and with some quick learning and initiative, any team could be well on its way to consistency on this dimension.
Why it’s valuable to be consistent
We’ve likely all had the experience of releasing some kind of critical bug to production. Breaking production eventually is inevitable, and DevOps prioritizes moving quickly over avoiding breakage at any cost. In this case, “moving quickly” means being able to release a hotfix right away to fix the issue. This is impossible if you have changes on your master branch that you aren’t 100% sure are ready to go.
How to start
It’s hard to beat the GitFlow model for ensuring production can be fixed at any time. GitFlow keeps unreleased work on a separate branch (typically called “develop”) until it is deployed. It is merged to the main branch (typically “master”) only when it is deployed. This leaves master always clean for a hotfix and allows you to follow an abbreviated version of the normal release process during a hotfix.
Getting started is going to mean learning about version control and using it (if you aren’t already—nearly 100% of Drupal respondents report consistently using version control in their work). From there, it’s also important to get as much of your site’s configuration into code, for example, Features for D7 or Configuration Management in D8 so that all work can be pulled around for review by others on the team. Finally, implementing basic continuous integration (CI) for running your test suite, along with continuous delivery (CD) to a preview environment for visual review, will decouple your ongoing feature development from the production pipeline and allow for more strategic deployments of ready branches or hotfixes as needed without jeopardizing the consistency of the master branch.
Fortunately, many services exist for working with Drupal in this way and can be learned and implemented to a basic degree without too much effort.
Another quick-win opportunity: Security
Another quick win in this 2019 Drupal Community DevOps Survey results set could be:
Security can be part of the process while determining how features will be developed. It can also be tested like everything else. And it is clearly a high-impact improvement and should be easy to gain traction, as it starts with a mindset shift that can be learned and practiced.
Why it’s valuable to be consistent
When people think of security issues, they tend to think of a single “hole” an attacker slips through to cause a breach. Modern web applications have hundreds or thousands of these tiny security holes, every one of which could be the source of a security issue that could bankrupt the company. Instead of thinking of security as a boolean (secure/not secure) value, think of it as a scale. To move toward the secure end of the scale, you need to prevent or catch security issues before they make it to production.
To accomplish this, you will need security consciousness at every stage of the process.
How to start
On the business end, consider the types of actions you want users to be able to perform. In architecture, design for defense-in-depth. In development, educate yourself on how to use the security mechanisms provided by the framework. In QA, learn how to exploit an application and actively attempt to do so. Considering security at every stage of the process will reduce (but never eliminate) the volume of security issues the team produces.
In Drupal, it’s important to ask questions about things like user accounts, forms, integrations, secure libraries and frameworks, and security updates. Following are some questions, using user accounts as example responses, to ask as early as possible:
- What are we working on? Users can log in for special access to features.
- What can go wrong? The user’s role is misconfigured to allow too much access, causing either business loss or technical disruption.
- What can we do about it? We can test the roles for misconfiguration on every commit using automated testing and continuous integration.
- Did we do a good job? Do these tests cover everything? What more could be done?
For more ideas, see Threat Modeling Designing for Security.
How to improve
There are ways to automate some types of security checks. They will be specific to your environment, but some ideas that have worked well include:
- Test for XSS vulnerabilities by automatically filling your data structures with “malicious” strings then testing for whether those strings are escaped in your output.
- Monitor your third-party dependencies for insecure libraries and alert when a security update is available. (Bonus: automate a pull request for updating the library.)
- Include SQL injection strings (e.g., drop tables) when filling out forms during automated testing.
In Drupal, automation offers some chances to do some incredibly robust automated tests with very little effort, if you’re clever about it. For example, we can get full-site XSS coverage just by injecting XSS strings at entity load and checking various pages for leakage. There’s also a lot that can be done with permissions, like verifying that nobody but the admin user has “dangerous” permissions.
These kinds of tests are possible because of the Drupal CMS, and we should be working as a community to make them accessible and usable to any team who wants them. Having this level of robust automated testing in Drupal enables us to respond quickly in the event of a zero-day vulnerability, since these tests will catch such issues before deployment that would have been disastrous had the release gone out.
A bigger challenge: Test coverage
The Drupal community’s DevOps dimension that needed the most attention was:
When features are developed, tests can be written to verify whether they are providing business value. How often are the tests written in a way that allows them to be run in a CI environment?
You’ll likely need some pretty good momentum going on your team to tackle this one, but it is well worth it.
How to start
When determining which features need to be tested, consider which ones would cause unplanned work if they broke. For example, if you are creating a recipe database and one of your features is a recipe-entry form, you would want to test that recipes can be entered and the form can be submitted because, without that feature, your application would no longer provide value to your users.
Once your tests are written, they will provide the most value if they are run in a CI system on every commit rather than on a schedule or in an ad-hoc way. This shortens the time it takes for a developer to become aware of a breakage they introduced—the goal is that they become aware of the breakage before they move to the next task so they won’t have to switch contexts to come back to fix it.
How to improve
Even if you have an excellent manual-review process, including functional testing by a human, it will quickly become impossible to manually check every feature of your application. Instead, you should be looking to automated testing to cover (at minimum) your business-critical features. The value of this testing will multiply with the number of developers on the project, the number of commits pushed daily, and the number of business-critical features you have.
There are many options when it comes to testing, but here are two types that people generally use:
- Unit: Exhaustive tests of a single “unit” of code (function, class, etc.). We often do less of this kind of testing for an application but leverage it for particularly complicated or important bits of code. Also, unit testing can force good design practices.
- Functional: Tests of a piece of functionality as seen by a user, such as user log in. This type of testing can cover many different code paths, but typically it doesn’t cover them exhaustively. This is a good choice for applications, and it is typically available using “behavioral” frameworks like Behat or Cucumber.
For Drupal, be sure to learn about Drupal Test Traits, which is a collection of traits “for testing Drupal sites that have user content (versus unpopulated sites)” by Moshe Weitzman, Jibran Ijaz, and Rob Bayliss. DTT was built to make functional testing easy because it operates on your installed site, so you don’t need to worry about mocking up a site install from scratch with every test. It also provides tests that naturally evolve with your site since it uses the configuration you have today, not the configuration you wrote into the test.
Conclusion
Having an understanding of the context and history of DevOps will mean the difference between following and leading in an industry. If there’s still time, please participate in the Drupal Community DevOps Survey so we can include your input in the results.
Rob Bayliss and Kelly Albrecht will present the results of the 2019 Drupal Community DevOps Survey in their presentation DevOps: Why, How, and What and host a follow-up Birds of a Feather discussion at DrupalCon 2019 in Seattle, April 8-12.
