The winds of change have swept the development and testing landscape. The days of waterfall design, planning, and development and then tossing the resulting builds over the wall to a beleaguered QA team have come to an end. Development and QA have started working together closely, and it has become clear that new tools and methodologies are needed to fully realize efficiency gains of a collaborative paradigm. Enter continuous integration. Continuous integration begat continuous delivery, continuous delivery begat DevOps. It's a brave new world!
While testing code is clearly a software quality assurance (SQA) engineering focus, test automation, automated provisioning of test environments, setup and tuning of CI and CD pipelines are vital ingredients in the efficient use of CI/CD and DevOps methodology for testing. Effective use of these methods allows SQA teams to be smaller and more agile, while still maintaining high code quality and release velocity. Indeed, entire organizations benefit from the adoption of CI/CD/DevOps as engineering efforts can be deftly shifted to meet new market opportunities and challenges, with rapid time to market, increased quality and lower overall cost.
Collaborate, Automate or Die
Testing at Puppet Labs is a collaborative effort driven and organized by the SQA team along with contributions from development and operations. With a small SQA team of four individuals, we rely heavily on automated testing, CI, CD and DevOps methodologies and tooling in our testing pipeline. Our development team contributes code to our testing tools. Attempts to build a test harness were only fully successful when the SQA and development teams worked together to build a tool that satisfied the technical needs and unique workflows of each team. Working together on this project resulted in a unified, extensible, robust testing tool used throughout the engineering organization. When planning and scaling infrastructure, we leverage the knowledge and expertise of our operations team. The majority of Puppet Lab's testing infrastructure is puppetized; rather than re-writing all of our own modules and Puppet customizations, we've simply re-used a great deal of our ops team’s modules and processes, resulting in nearly no-cost operational efficiency gains for the SQA team. The Puppet Labs ops team, in turn, benefits from SQA's exercising and extending existing Puppet code.
We benefit from community contributions in the form of bug reports, code patches and in some cases, even contributions to our testing tools. For example, Puppet’s own home-grown testing harness utilizies community member R. Tyler’s Blimpy module for provisioning EC2 nodes, and we used his Jenkins module as a basis for our own module when puppetizing our SQA infrastructure. In kind, we provide bug fixes and additional functionality to these modules and return our code to the community.
The Puppet Test Matrix: Millions of possible combinations
Puppet is a complex software stack, comprised of over 20 components, supported on 40+ OS platforms in a dizzying number of configuration combinations. The Puppet testing eco-system consists of:
- Code from 800 Github repositories
- 60 Operating System platforms
- 50 unique test configurations
- 7 browser versions
- Ruby 1.8.5, 1.8.7 p249 + p358, 1.9.3 p0 + p125
- Puppet 2.7x, 3.x
- Facter 1.6x, 1.7x, 2.0x
- Stdlib 2.1x, 2.2x, 2.3x, 2.4x, 2.5x, 3.0x
- Hiera 1.x
Without the use of automated testing, both CI and CD, testing an entire Puppet Enterprise release would take a team of four QA engineers roughly three months. Clearly then, with such a large testing matrix, it is essential to provide teams with a self-serve CI system allowing ease of test automation, while providing transparency to code health and project status to the entire organization. The continuous delivery aspect of our testing system allows frequent deployment to real world test environments facilitating functional and integration testing. Each build proceeds through a logical progression of increasingly rigorous tests and environmental complexity on the way to being release ready. These frequent integrations, only possible due to our completely automated testing pipeline, are significantly less painful and less time consuming than larger, infrequent efforts.
CI/CD Ecosystem: How to get it done
Our current testing infrastructure supports 60+ test virtual machines, while the next iteration of our testing infrastructure will support 500+ virtual machines.
We have automated, primarily using Puppet, the provisioning and configuration of 'test pods' consisting of a Jenkins CI Server, Jenkins test coordinators, Acceptance/Integration VMs, etc. This allows us to quickly deploy new test environments when we begin new projects and ramp up new development teams.
Puppet Labs Testing Infrastructure, Version 2:
- Puppet-acceptance test harness with EC2, VMWare vSphere, VMWare Fusion integration
- 400 Acceptance/integration testing VMs
- 12 Jenkins Servers
- 20 Jenkins coordinators
- 12 Spec testing systems
- 16 Acceptance test drivers
- 15 Build servers
- 7 Misc appliance machines (LDAP, Active Directory, data base servers)
Implementing a comprehensive CI/CD is a daunting task, but can reap huge rewards for any development team. For anyone trying to build a CI/CD system, I recommend that you avoid the temptation to build a perfect system, and instead, start small, iterate, and build upon successes. A sensible approach might be to automate the provisioning of test appliances and unit level tests, saving more complex integration level and UI testing once the core CI/CD system matures.
Today, the Puppet CI/CD system is mature, stable, and capable, but it took a lot of collaboration with devs, ops, and the community at large to get there. The end result is operational efficiency, a stronger DevOps culture, and better software—delivered faster.