Open source Puppet documentation

Use this information to understand how the Resource API works: how the resource is defined in the type, how resource management is implemented in the provider, and some of the known limitations of the Resource API.

Resource definition: the type

A type is a definition of a resource that Puppet can manage. The definition contains the resource’s configurable properties and the parameters used to access it.

To make the resource known to the Puppet ecosystem, its definition, or type needs to be registered with Puppet. For example:
Puppet::ResourceApi.register_type(
  name: 'apt_key',
  desc: <<-EOS,
    This type provides Puppet with the capabilities to manage GPG keys needed
    by apt to perform package validation. Apt has it's own GPG keyring that can
    be manipulated through the `apt-key` command.

    apt_key { '6F6B15509CF8E59E6E469F327F438280EF8D349F':
      source => 'http://apt.puppetlabs.com/pubkey.gpg'
    }

    **Autorequires**:
    If Puppet is given the location of a key file which looks like an absolute
    path this type will autorequire that file.
  EOS
  attributes: {
    ensure: {
      type: 'Enum[present, absent]',
      desc: 'Whether this apt key should be present or absent on the target system.'
    },
    id: {
      type:      'Variant[Pattern[/\A(0x)?[0-9a-fA-F]{8}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{16}\Z/], Pattern[/\A(0x)?[0-9a-fA-F]{40}\Z/]]',
      behaviour: :namevar,
      desc:      'The ID of the key you want to manage.',
    },
    source: {
      type: 'String',
      desc: 'Where to retrieve the key from, can be a HTTP(s) URL, or a local file. Files get automatically required.',
    },
    # ...
    created: {
      type:      'String',
      behaviour: :read_only,
      desc:      'Date the key was created, in ISO format.',
    },
  },
  autorequire: {
    file:    '$source', # will evaluate to the value of the `source` attribute
    package: 'apt',
  },
)
The Puppet::ResourceApi.register_type(options) function takes the following keyword arguments:
  • name: the name of the resource type.

  • desc: a doc string that describes the overall working of the resource type, provides examples, and explains prerequisites and known issues.

  • attributes: a hash mapping attribute names to their details. Each attribute is described by a hash containing the Puppet 4 data type, a desc string, a default value, and the behavior of the attribute: namevarread_onlyinit_only, or a parameter.

    • type: the Puppet 4 data type allowed in this attribute.

    • desc: a string describing this attribute. This is used in creating the automated API docs with puppet-strings.

    • default: a default value that the runtime environment uses when you don't specify a value.

    • behavior/behaviour: how the attribute behaves. Currently available values:

      • namevar: marks an attribute as part of the "primary key" or "identity" of the resource. A given set of namevar values needs to distinctively identify an instance.

      • init_only: this attribute can only be set when creating the resource. Its value will be reported going forward, but trying to change it later leads to an error. For example, the base image for a VM or the UID of a user.

      • read_only: values for this attribute will be returned by get(), but set() is not able to change them. Values for this should never be specified in a manifest. For example, the checksum of a file, or the MAC address of a network interface.

      • parameter: these attributes influence how the provider behaves, and cannot be read from the target system. For example, the target file on inifile, or the credentials to access an API.

  • autorequireautobeforeautosubscribe, and autonotify: a hash mapping resource types to titles. The titles must either be constants, or, if the value starts with a dollar sign, a reference to the value of an attribute. If the specified resources exist in the catalog, Puppet will create the relationships that are requested here.

  • features: a list of API feature names, specifying which optional parts of this spec the provider supports. Currently defined features: canonicalizesimple_get_filter, and supports_noop. See provider types for details.

Composite namevars (title_patterns)

Each resource being managed must be identified by a unique title. Usually this is straightforward and a single attribute can be used to act as an identifier. But sometimes you need a composite of two attributes to uniquely identify the resource you want to manage.

If multiple attributes are defined with the namevar behavior, the type will specify title_patterns that tell the Resource API how to get at the attributes from the title. If title_patterns is not specified, a default pattern is applied and matches against the first declared namevar.

Note: The order of the title_patterns is important. You should declare the most specific pattern first and end with the most generic.
Each title pattern contains:
  • pattern, which is a Ruby regular expression containing named captures. The names of the captures must be that of the namevar attributes.

  • desc, a short description of what the pattern matches for.

For example:
Puppet::ResourceApi.register_type(
  name: 'software',
  docs: <<-DOC,
    This type provides Puppet with the capabilities to manage ...
  DOC
  title_patterns: [
    {
      pattern: %r{^(?<package>.*[^-])-(?<manager>.*)$},
      desc: 'Where the package and the manager are provided with a hyphen seperator',
    },
    {
      pattern: %r{^(?<package>.*)$},
      desc: 'Where only the package is provided',
    },
  ],
  attributes: {
    ensure: {
      type:    'Enum[present, absent]',
      desc:    'Whether this resource should be present or absent on the target system.',
      default: 'present',
    },
    package: {
      type:      'String',
      desc:      'The name of the package you want to manage.',
      behaviour: :namevar,
    },
    manager: {
      type:      'String',
      desc:      'The system used to install the package.',
      behaviour: :namevar,
    },
  },
)
These match the first title pattern:
software { php-yum:
  ensure=>'present'
}

software { php-gem:
  ensure=>'absent'
}
This matches the second title pattern:
software { php:
  manager='yum'
  ensure=>'present'
}

Resource implementation: the provider

To make changes, a resource requires an implementation, or provider. It is the code used to retrieve, update and delete the resources of a certain type.

The two fundamental operations to manage resources are reading and writing system state. These operations are implemented as get and set. The implementation itself is a Ruby class in the Puppet::Provider namespace, named after the type using CamelCase.

Note: Due to the way Puppet autoload works, this is in a file called puppet/provider/<type_name>/<type_name>.rb. The class will also have the CamelCased type name twice.
At runtime, the current and intended system states a specific resource. These are represented as Ruby hashes of the resource's attributes and applicable operational parameters:
class Puppet::Provider::AptKey::AptKey
  def get(context)
    [
      {
        name: 'name',
        ensure: 'present',
        created: '2017-01-01',
        # ...
      },
      # ...
    ]
  end

  def set(context, changes)
    changes.each do |name, change|
      is = change.has_key? :is ? change[:is] : get_single(name)
      should = change[:should]
      # ...
    end
  end
end
The get method reports the current state of the managed resources. It returns an enumerable of all existing resources. Each resource is a hash with attribute names as keys, and their respective values as values. It is an error to return values not matching the type specified in the resource type. If a requested resource is not listed in the result, it is considered to not exist on the system. If the get method raises an exception, the provider is marked as unavailable during the current run, and all resources of this type will fail in the current transaction. The exception message will be reported.

The set method updates resources to a new state. The changes parameter gets passed a hash of change requests, keyed by the resource's name. Each value is another hash with the optional :is and :should keys. At least one of the two must be specified. The values will be of the same shape as those returned by get. After the set, all resources should be in the state defined by the :should values.

A missing :should entry indicates that a resource should be removed from the system. Even a type implementing the ensure => [present, absent] attribute pattern must react correctly on a missing :should entry. An :is key may contain the last available system state from a prior get call. If the :is value is nil, the resources were not found by get. If there is no :is key, the runtime did not have a cached state available.

The set method should always return nil. Any progress signaling should be done through the logging utilities described below. If the set method throws an exception, all resources that should change in this call and haven't already been marked with a definite state, will be marked as failed. The runtime will only call the set method if there are changes to be made, especially when resources are marked with noop => true (either locally or through a global flag). The runtime will not pass them to set. See supports_noop for changing this behavior if required.

Both methods take a context parameter which provides utilties from the runtime environment, and is described in more detail there.

Implementing simple providers

In many cases, the resource type follows the conventional patterns of Puppet, and does not gain from the complexities around batch-processing changes. For those cases, the SimpleProvider class supplies a proven foundation that reduces the amount of code necessary to get going.

SimpleProvider requires that your type follows these common conventions:
  • name is the name of your namevar attribute.

  • ensure attribute is present and has the Enum[absent, present] type.

To start using SimpleProvider, inherit from the class like this:
require 'puppet/resource_api/simple_provider'

# Implementation for the wordarray type using the Resource API.
class Puppet::Provider::AptKey::AptKey < Puppet::ResourceApi::SimpleProvider
  # ...
Next, instead of the set method, the provider needs to implement the createupdate or delete methods:
  • create(context, name, should): Called to create a resource.

    • context: provides utilities from the runtime environment.

    • name: the name of the new resource.

    • should: a hash of the attributes for the new instance.

  • update(context, name, should): Called to update a resource.

    • context: provides utilties from the runtime environment.

    • name: the name of the resource to change.

    • should: a hash of the desired state of the attributes.

  • delete(context, name): Called to delete a resource.

    • context: provides utilities from the runtime environment.

    • name: the name of the resource that should be deleted.

The SimpleProvider does basic logging and error handling.

Provider features

There are some use cases where an implementation provides a better experience than the default runtime environment provides. To avoid burdening the simplest providers with that additional complexity, these cases are hidden behind feature flags. To enable the special handling, the resource definition has a feature key to list all features implemented by the provider.

canonicalize

Allows the provider to accept a wide range of formats for values without confusing the user.
Puppet::ResourceApi.register_type(
  name: 'apt_key',
  features: [ 'canonicalize' ],
)

class Puppet::Provider::AptKey::AptKey
  def canonicalize(context, resources)
    resources.each do |r|
      r[:name] = if r[:name].start_with?('0x')
                   r[:name][2..-1].upcase
                 else
                   r[:name].upcase
                 end
    end
  end
The runtime environment needs to compare user input from the manifest (the desired state) with values returned from get (the actual state), to determine whether or not changes need to be affected. In simple cases, a provider only accepts values from the manifest in the same format as get returns. No extra work is required, as a value comparison will suffice. This places a high burden on the user to provide values in an unnaturally constrained format. In the example, the apt_key name is a hexadecimal number that can be written with, and without, the '0x' prefix, and the casing of the digits is irrelevant. A value comparison on the strings causes false positives if the user inputs format that does not match. There is no hexadecimal type in the Puppet language. To address this, the provider can specify the canonicalize feature and implement the canonicalize method.

The canonicalize method transforms its resources argument into the standard format required by the rest of the provider. The resources argument to canonicalize is an enumerable of resource hashes matching the structure returned by get. It returns all passed values in the same structure with the required transformations applied. It is free to reuse or recreate the data structures passed in as arguments. The runtime environment must use canonicalize before comparing user input values with values returned from get. The runtime environment always passes canonicalized values into set. If the runtime environment requires the original values for later processing, it protects itself from modifications to the objects passed into canonicalize, for example by creating a deep copy of the objects.

The context parameter is the same passed to get and set, which provides utilities from the runtime environment, and is described in more detail there.

Note: When the provider implements canonicalization, it aims to always log the canonicalized values. As a result of get and set producing and consuming canonically formatted values, it is not expected to present extra cost.
A side effect of these rules is that the canonicalization of the get method's return value must not change the processed values. Runtime environments may have strict or development modes that check this property.
For example, in the Puppet Discovery runtime environment it is bound to the strict setting, and will follow the established practices:
# puppet resource --strict=error apt_key ensure=present
> runtime exception
# puppet resource --strict=warning apt_key ensure=present
> warning logged but values changed
# puppet resource --strict=off apt_key ensure=present
> values changed

simple_get_filter

Allows for more efficient querying of the system state when only specific parts are required.
Puppet::ResourceApi.register_type(
  name: 'apt_key',
  features: [ 'simple_get_filter' ],
)

class Puppet::Provider::AptKey::AptKey
  def get(context, names = nil)
    [
      {
        name: 'name',
        # ...
      },
    ]
  end
Some resources are very expensive to enumerate. The provider can implement simple_get_filter to signal extended capabilities of the get method to address this. The provider's get method is called with an array of resource names, or nil. The get method must at least return the resources mentioned in the names array, but may return more. If the names parameter is nil, all existing resources should be returned. The names parameter defaults to nil to allow simple runtimes to ignore this feature.

The runtime environment calls get with a minimal set of names, and keeps track of additional instances returned to avoid double querying. To gain the most benefits from batching implementations, the runtime minimizes the number of calls into get.

supports_noop

When a resource is marked with noop => true, either locally or through a global flag, the standard runtime produces the default change report with a noop flag set. In some cases, an implementation provides additional information, for example commands that would get executed, or require additional evaluation before determining the effective changes, such as exec's onlyif attribute. The resource type specifies the supports_noop feature to have set called for all resources, even those flagged with noop. When the noop parameter is set to true, the provider must not change the system state, but only report what it would change. The noop parameter should default to false to allow simple runtimes to ignore this feature.

Puppet::ResourceApi.register_type(
  name: 'apt_key',
  features: [ 'supports_noop' ],
)

class Puppet::Provider::AptKey::AptKey
  def set(context, changes, noop: false)
    changes.each do |name, change|
      is = change.has_key? :is ? change[:is] : get_single(name)
      should = change[:should]
      # ...
      do_something unless noop
    end
  end
end

remote_resource

Declaring this feature restricts the resource from being run locally. It is expected to execute all external interactions through the context.device instance. The way that an instance is set up is runtime specific. In Puppet, it is configured through the  device.conf file, and only available when running under  puppet device. Use Puppet::Util::NetworkDevice::Simple::Device as the base class for all devices. It automatically loads a configuration from the local filesystem of the proxy node where it is running on.

Puppet::ResourceApi.register_type(
  name: 'nx9k_vlan',
  features: [ 'remote_resource' ],
)

require 'puppet/util/network_device/simple/device'
module Puppet::Util::NetworkDevice::Nexus
  class Device < Puppet::Util::NetworkDevice::Simple::Device
    def facts
      # access the device and return facts hash
    end
  end
end

class Puppet::Provider::Nx9k_vlan::Nx9k_vlan
  def set(context, changes, noop: false)
    changes.each do |name, change|
      is = change.has_key? :is ? change[:is] : get_single(name)
      should = change[:should]
      # ...
      context.device.do_something unless noop
    end
  end
end

Runtime environment

The primary runtime environment for the provider is the Puppet agent, a long-running daemon process. The provider can also be used in the puppet apply command, a self contained version of the agent, or the puppet resource command, a short-lived command line interface (CLI) process for listing or managing a single resource type. Other callers that want to access the provider must imitate these environments.

The primary life cycle of resource management in each of these tools is the transaction, a single set of changes, for example a catalog or a CLI invocation. The provider's class will be instantiated once for each transaction. Within that class the provider defines any number of helper methods to support itself. To allow for a transaction to set up the prerequisites for a provider and be used immediately, the provider is instantiated as late as possible. A transaction will usually call get once, and may call set any number of times to make changes.

The object instance that hosts the get and set methods can be used to cache ephemeral state during execution. The provider should not try to cache state outside of its instances. In many cases, such caching won't help as the hosting process will only manage a single transaction. In long-running runtime environments like the agent, the benefit of the caching needs to be balanced with the cost of the cache at rest, and the lifetime of cache entries, which are only useful when they are longer than the regular runinterval.

The runtime environment has the following utilities to provide a uniform experience for its users.

Logging and reporting utilities

The provider needs to signal changes, successes, and failures to the runtime environment. The context is the primary way to do this. It provides a structured logging interface for all provider actions. Using this information, the runtime environments can do automatic processing, emit human readable progress information, and provide status messages for operators.

To provide feedback about the overall operation of the provider, the context has the usual set of loglevel methods that take a string, and pass that up to the runtime environments logging infrastructure. For example:
context.warning("Unexpected state detected, continuing in degraded mode.")
Results in the following message:
Warning: apt_key: Unexpected state detected, continuing in degraded mode.
Other common messages include:
  • debug: Detailed messages to understand everything that is happening at runtime, shown on request.

  • info: Regular progress and status messages, especially useful before long-running operations, or before operations that can fail, to provide context for interactive users.

  • notice: Indicates state changes and other events of notice from the regular operations of the provider.

  • warning: Signals error conditions that do not (yet) prohibit execution of the main part of the provider; for example, deprecation warnings, temporary errors.

  • err: Signals error conditions that have caused normal operations to fail.

  • critical, alert, emerg: should not be used by resource providers.

In simple cases, a provider passes off work to an external tool, logs the details there, and then reports back to Puppet acknowledging these changes. This is called resource status signaling, and looks like this:
@apt_key_cmd.run(context, action, key_id)
context.processed(key_id, is, should)
It reports all changes from is to should, using default messages.

Providers that want to have more control over the logging throughout the processing can use the more specific created(title)updated(title)deleted(title)unchanged(title) methods. To report the change of an attribute, the context provides a attribute_changed(title, attribute, old_value, new_value, message) method.

Most of those messages are expected to be relative to a specific resource instance, and a specific operation on that instance. To enable detailed logging without repeating key arguments, and to provide consistent error logging, the context provides logging context methods to capture the current action and resource instance:
context.updating(title) do
  if apt_key_not_found(title)
    context.warning('Original key not found')
  end

  # Update the key by calling CLI tool
  apt_key(...)

  context.attribute_changed('content', nil, content_hash,
    message: "Replaced with content hash #{content_hash}")
end
This will result in the following messages:
Debug: Apt_key[F1D2D2F9]: Started updating
Warning: Apt_key[F1D2D2F9]: Updating: Original key not found
Debug: Apt_key[F1D2D2F9]: Executing 'apt-key ...'
Debug: Apt_key[F1D2D2F9]: Successfully executed 'apt-key ...'
Notice: Apt_key[F1D2D2F9]: Updating content: Replaced with content hash E242ED3B
Notice: Apt_key[F1D2D2F9]: Successfully updated
In the case of an exception escaping the block, the error is logged appropriately:
Debug: Apt_keyF1D2D2F9]: Started updating
Warning: Apt_key[F1D2D2F9]: Updating: Original key not found
Error: Apt_key[F1D2D2F9]: Updating failed: Something went wrong

Logging contexts process all exceptions. A StandardError is assumed to be regular failures in handling resources, and are consumed after logging. Everything else is assumed to be a fatal application-level issue, and is passed up the stack, ending execution. See the Ruby documentation for details on which exceptions are not a StandardError.

The equivalent long-hand form of manual error handling:
context.updating(title)
begin
  unless title_got_passed_to_set(title)
    raise Puppet::DevError, 'Managing resource outside of requested set: %{title}')
  end

  if apt_key_not_found(title)
    context.warning('Original key not found')
  end

  # Update the key by calling CLI tool
  result = @apt_key_cmd.run(...)

  if result.exitstatus != 0
    context.error(title, "Failed executing apt-key #{...}")
  else
    context.attribute_changed(title, 'content', nil, content_hash,
      message: "Replaced with content hash #{content_hash}")
  end
  context.changed(title)
rescue Exception => e
  context.error(title, e, message: 'Updating failed')
  raise unless e.is_a? StandardError
end
This example is only for demonstration purposes. In the normal course of operations, providers should always use the utility functions.

The following methods are available:

  • Block functions: these functions provide logging and timing around a provider's core actions. If the the passed &block returns, the action is recorded as successful. To signal a failure, the block should raise an exception explaining the problem:

    • creating(titles, message: 'Creating', &block)

    • updating(titles, message: 'Updating', &block)

    • deleting(titles, message: 'Deleting', &block)

    • processing(title, is, should, message: 'Processing', &block): generic processing of a resource, produces default change messages for the difference between is: and should:.

    • failing(titles, message: 'Failing', &block): unlikely to be used often, but provided for completeness. It always records a failure.

  • Action functions:

    • created(titles, message: 'Created')

    • updated(titles, message: 'Updated')

    • deleted(titles, message: 'Deleted')

    • processed(title, is, should): the resource has been processed. It produces default logging for the resource and each attribute

    • failed(titles, message:): the resource has not been updated successfully

  • Attribute Change notifications:

    • attribute_changed(title, attribute, is, should, message: nil): notify the runtime environment that a specific attribute for a specific resource has changed. is and should are the original and the new value of the attribute. Either can be nil.

  • Plain messages:

    • debug(message)

    • debug(titles, message:)

    • info(message)

    • info(titles, message:)

    • notice(message)

    • notice(titles, message:)

    • warning(message)

    • warning(titles, message:)

    • err(message)

    • err(titles, message:)

titles can be a single identifier for a resource or an array of values, if the following block batch processes multiple resources in one pass. If that processing is not atomic, providers should instead use the non-block forms of logging, and provide accurate status reporting on the individual parts of update operations.

A single set() execution may only log messages for instances that have been passed, as part of the changes to process. Logging for instances not requested to be changed will cause an exception - the runtime environment is not prepared for other resources to change.

The provider is free to call different logging methods for different resources in any order it needs to. The only ordering restriction is for all calls specifying the same title. For example, the attribute_changed needs logged before that resource's action logging, and the context needs to be opened before any other logging for this resource.

Type definition

The provider can gain insight into the type definition through these context.type utility methods:
  • attributes: returns a hash containing the type attributes and it's properties.

  • ensurable?: returns true if the type contains the ensure attribute.

  • feature?(feature): returns true if the type supports a given provider feature.

For example:
# example from simple_provider.rb

def set(context, changes)
  changes.each do |name, change|
  is = if context.type.feature?('simple_get_filter')
         change.key?(:is) ? change[:is] : (get(context, [name]) || []).find { |r| r[:name] == name }
       else
         change.key?(:is) ? change[:is] : (get(context) || []).find { |r| r[:name] == name }
       end
  ...

end

Resource API limitations

This Resource API is not a full replacement for the original low-level types and providers method. Here is a list of the current limitations. If they apply to your situation, the low-level types and providers method might be a better solution. The goal of the new Resource API is not to be a replacement of the prior one, but to be a simplified way to get the same results for the majority of use cases.

You can't have multiple providers for the same type

The low-level type and provider method allows multiple providers for the same resource type. This allows the creation of abstract resource types, such as packages, which can span multiple operating systems. Automatic selection of an OS-appropriate provider means less work for the user, as they don't have to address whether the package needs to be managed using apt or yum in their code .

Allowing multiple providers means more complexity and more work for the type or provider developer, including:
  • attribute sprawl

  • disparate feature sets between the different providers for the same abstract type

  • complexity in implementation of both the type and provider pieces stemming from the two issues above

The Resource API does not implement support for multiple providers.

Should support for multiple providers be desirable for a given type, your options are:

  • Use the older, more complex type and provider method, or

  • Implement multiple similar types using the Resource API, and select the platform-appropriate type in Puppet code. For example:
    define package (
      Ensure $ensure,
      Enum[apt, rpm] $provider, # have a hiera 5 dynamic binding to a function choosing a sensible default for the current system
      Optional[String] $source  = undef,
      Optional[String] $version = undef,
      Optional[Hash] $options   = { },
    ) {
      case $provider {
        apt: {
          package_apt { $title:
            ensure          => $ensure,
            source          => $source,
            version         => $version,
            *               => $options,
          }
        }
        rpm: {
          package_rpm { $title:
            ensure => $ensure,
            source => $source,
            *      => $options,
          }
          if defined($version) { fail("RPM doesn't support \$version") }
          # ...
        }
      }
    }

Only built-in Puppet 4 data types are available

Currently, only built-in Puppet 4 data types are usable. This is because the type information is required on the agent, but Puppet has not made it available yet. Even once that is implemented, modules will have to wait until the functionality is widely available before being able to rely on it.

There is no catalog access

There is no way to access the catalog from the provider. Several existing types rely on this to implement advanced functionality. Some of these use cases would be better off being implemented as "external" catalog transformations, instead of munging the catalog from within the compilation process.

No logging for un-managed instances

Previously, the provider could provide log messages for resource instances that were not passed into the set call. In the current implementation, these will cause an error.

Automatic relationships constrained to consts and attribute values

The Puppet 3 type API allows arbitrary code execution for calculating automatic relationship targets. The Resource API is more restrained, but allows understanding the type's needs by inspecting the metadata.

Back to top