Here, at Skroutz, we’ve been using Puppet for our configuration management needs for several years now. It is a core component of our infrastructure, managing around 450 servers for production, staging and development requirements. Until recently we were still running a Puppet 3 based infrastructure, however the new functionality and performance improvements introduced with Puppet 4, the end-of-life announcement of Puppet 3 support and the upcoming Debian Stretch release with Puppet 4.8.2, made it clear that the time had come to upgrade to Puppet 4, a task that turned out to be challenging and interesting at the same time.
Defining the Roadmap
Puppet 4 comes with a list of several new features, major changes and a lot of deprecations. Our highest priority was to maintain the availability of our (sole) Puppet master during the upgrade, in order not to block regular operations. Dedicating a couple of days to carefully design the upgrade path and identify possible pitfalls, together with a thorough read of the respective Puppet upgrade document helped us a lot towards this direction.
As you might know, we run a Debian-based infrastructure with a strict Debian-packages-preferred policy, so it was desirable to perform the whole upgrade process using packages from Debian proper and not from third-party repositories. At the time we were designing our roadmap, Puppet 4.8 was already in Debian Stretch, so we decided to go with that.
Before the upgrade, our Puppet system consisted of a single 3.7.2 Puppet master running on Ruby Passenger. All of our infrastructure is using Debian Jessie currently, so all Puppet agents had the same version as well. Also, we were using exported resources with the ActiveRecord storeconfigs backend.
The upgrade was primarily centered around the Puppet master, since that’s where most of the heavy changes had to take place. Upgrading the master was essential in order to support Puppet 4 clients (Debian Stretch), while the the Puppet 3 client compatibility patch shipped with the current Debian package (see #832536) guaranteed that our existing Puppet 3 agents would continue working. For us, the most significant changes that needed to be in place before the upgrade, were:
- PuppetDB and ActiveRecord-based storeconfigs
- Parser changes
- Stringified facts
- Directory environment support
- Misc code changes for Puppet 4.x
The most important items for each one of the above phases are highlighted in the rest of this post.
PuppetDB and ActiveRecord-based storeconfigs
We are making use of Puppet’s exported resources and stored configuration facilities and have built a number of tools around them. Under the hood, Puppet was originally using Ruby’s ActiveRecord to implement persistence to an RDBMS, however ActiveRecord-based storeconfigs was deprecated in 3.x and removed completely in Puppet 4.x; it has been replaced by PuppetDB which runs as a separate service.
Consequently, before any further upgrade step, we had to migrate our storeconfigs to PuppetDB. This meant that several components of our infrastructure that were bound to the Puppet 3’s MySQL database needed to be decoupled from it before migrating to PuppetDB. Using primarily the PuppetDB API and a couple of custom Puppet report processors, we manage to preserve all the previous functionality with minimum effort and without diverting significantly from our initial roadmap.
Note that unlike Puppet which has been available in the Debian archive for a long time, PuppetDB is not yet part of Debian. We decided to roll our own package and help move things forward so that PuppetDB would end up in Debian soon-ish.
After a bit of quick research, we decided to target the 4.x PuppetDB release series, as it seemed new but stable enough. However, some compatibility issues had to be taken into account, primarily regarding the terminus, i.e. the glue code that teaches the Puppet master how to use PuppetDB to store and retrieve facts, catalogs and reports:
- The official PuppetDB documentation, states that PuppetDB 2.3.x is the last version compatible with Puppet 3. However,
- having a look at the terminus code, it appears that Puppet 3 support is there in the 3.2.x series as well, albeit undocumented.
Seeing that PuppetDB 2.3.x is using the old v3 query API, while PuppetDB 3.2.x and 4 are using the new v4 API and not wanting to rewrite our tools twice, we decided to target PuppetDB 3.2.4 initially and upgrade to 4.3.2 as soon as we had upgraded our Puppet master to Puppet 4.
After all sub-components were in-place, i.e., scripts were decoupled from the
ActiveRecord storeconfigs and .deb packages had been prepared, we only had to
import our data from ActiveRecord storeconfigs to PuppetDB, in order to ensure
that any resources already collected by our agents will still be collected
after the migration, and the correct configuration will be applied. Thankfully,
using the puppet storeconfigs export and puppetdb import commands, we
achieved to easily migrate all the exported resources to PuppetDB and
subsequently to successfully add PuppetDB to our production stack.
Parser changes
One of the biggest changes coming with Puppet 4, is the new parser. To ease the migration, the current parser for Puppet 4.x is also available on Puppet 3.x versions as the future parser and can be enabled per environment. Note that while both implementations should be compatible, this turned out not to be so and switching the future parser on may become tricky, thus the transition should be handled with care.
To identify and resolve any problems that may arise from enabling the future parser, we decided to setup a testing environment to analyze and compare the output between the two parsers. We started with setting up a second Puppet master Rack instance on our Puppet server, listening on a different port with the future parser enabled, to be able to compile catalogs with the new parser upon request. We created some custom tools around Github’s octocatalog-diff, to identify the differences between catalogs compiled with both the current and the future parser and produce a report with the file line and number the difference was detected at. Alongside, we configured our Buildbot-based CI, to also perform all Puppet builds with the future parser. This made sure that new code posted for review would be future-proof.
During this process, the most remarkable incompatibilities found in our tree were the following:
- Manifests can no longer be imported
- Hyphenated class and module names are no longer valid
- LHS operands in regex expressions must be Strings
- The hiera() function in Puppet 3 does not handle default undef values properly
A massive catalog re-compilation for all our hosts assured us that catalog compilation output was identical with both parsers, and no unforeseen changes would occur after enabling the future parser, as it actually happened.
Stringified facts
Another important milestone of Puppet 4 is typed fact support. In contrast to Puppet 3.x which converts all facts to Strings by default, Puppet 4.x introduces proper data types for facts. This is an agent-side setting, that must be turned off explicitly in Puppet 3, and is off in Puppet 4 (and cannot be turned on), so it is a good idea to turn it off before upgrading to Puppet 4.
Before we enabled typed facts fleet-wide, we first had to ensure that none of our modules, manifests, or custom scripts rely on handling facts as Strings. Comparing to the rest upgrade steps that was probably the most easy task to handle. This change most notably affects boolean comparisons. The canonical way to handle boolean fact comparisons in Puppet 3 would be:
if "${::is_virtual}" == "true" { do something }
Note that
if $::is_virtual { do something }
would always do something, since is_virtual is a string. OTOH, with typed
facts
if $::is_virtual { }
would work as expected, however
if "${::is_virtual}" { }
would not. To ensure a smooth way forward, the recommended method is to
explicitly cast boolean facts to booleans using str2bool() from Puppet’s
stdlib:
if str2bool($::is_virtual) { }
After updating the respective code paths we started to rolling set the
stringify_facts setting to false until all our Puppet deployment is updated
with the new setting.
Directory environment support
The final blocking task before upgrading to Puppet 4.x was enabling directory
environments and migrating to the new code layout. Starting with Puppet 4.x,
directory environments are the only way to organize your code. Since we never
used environments in our deployment, we started by moving everything under the
default production environment. Then, in order to enable directory
environments we simply had to set the environmentpath setting to the new
location and that was it!
Besides the necessity of the current step, enabling directory environments
proved to be a very handy tool in our day-to-day workflow. Taking advantage of
Puppet’s $environment variable and introducing a custom tool to track,
identify, and auto-deploy Git branches under a fixed prefix as new
environments, we are now able to create and remove environments on-the-fly
simplifying the development process of new features drastically.
Misc code changes for Puppet 4.x
As soon as all Puppet 4 prerequisites were met, we could focus on preparing our source code for the final upgrade process. To identify and collect all incompatibilities among the two major releases we followed a similar approach to the one prior to enabling the future parser. We created a new Puppet master VM running Puppet 4.8.2. Using octocatalog-diff again, we could easily request catalogs from both Puppet 3.x and Puppet 4.x masters and then compare their outputs. After the initial screening, we set up our Buildbot to use Puppet 4.x for all builds.
As expected, there were not so many differences between the two releases. That was because we had already enabled the future parser and typed facts and the majority of the deprecated features had been resolved. The most noteworthy issues we’ve encountered during the current phase are outlined below:
- Rename Puppet 4.x reserved words across our tree
- Make v3.x functions compatible with the new v4.x function API
- Handle access to unassigned variables
- The tagmail report processor was removed from Puppet’s core
Again, a massive catalog re-compilation for all our hosts followed, to ensure that everything will work as expected after the upgrade.
Upgrade Puppet master and PuppetDB
Since our Puppet 3 system was considered ready for upgrade, we started with upgrading our Puppet master to Debian Stretch’s 4.8.2 version. To so do, we disabled Puppet fleet-wide for a short time, performed the upgrade of the master server, and then started re-enabling Puppet fleet-wide in small chunks, while monitoring our logging streams for changes. The transition to PuppetDB 4.3.2 was completely transparent and performed smoothly.
Evaluation & future enhancements
Living with our new Puppet setup for about 2 weeks, we are happy to see everything work as intended with no contingencies occurring. Taking into account not only the project’s outcome but also the fact that we performed an in-place upgrade of one of our core components with minimum downtime and no ill side-effects at all, made us feel pleased about the overall exercise. On the project side, defining and trying to keep-up with a quite detailed roadmap during the project’s life-cycle made the overall upgrade process a pleasant exercise.
Additionally, this upgrade paves the way for the upgrade to Stretch, since we are now able to run a mixed Debian Jessie/Stretch infrastructure.
As for our future plans, we’ll be keeping an eye on Puppet Server and Puppet 5 in the longer term.