Published on 6 October 2012 by
Purpose Fetch and update file data from an S3 bucket
Module branan/s3file
Puppet Version 2.7+
Platforms All, but see ‘Advanced Usage’ for non-Linux

Puppet and Puppet Enterprise come with a basic file server, allowing agents to fetch files from the master. This capability is suitable for small files, but when used with large binaries it can cause performance issues on the master.

S3file provides a simple Puppet type to fetch and update files stored in an Amazon S3 bucket or in your private OpenStack Swift storage environment. This allows you to store large files outside of Puppet, while still keeping the resource model provided by the existing Puppet file types.

S3file is written to be compatible with the old Puppet 2.7 as well as the latest Puppet 3.0, making it easy to integrate with any Puppet deployment.

Installing the module

Complexity Easy
Installation Time <5 minutes

S3file is available from the Puppet Forge. It can be installed with:

 puppet module install branan/s3file.

Resource Overview

The s3file module provides a single defined type, s3file, which downloads a file from cloud storage.

Example Usage

Complexity Easy
Installation Time 5 minutes
include s3file::curl

s3file { ‘/tmp/minecraft_server.jar’:
  source => ‘MinecraftDownload/launcher/minecraft_server.jar’,
  ensure => ‘latest’,
}

This small manifest will:

  • Install cURL
  • Download the minecraft_server.jar file from Amazon S3 and place it in /tmp/
  • Re-download the file whenever the version on S3 differs from the version on the local filesystem.

Advanced Usage

The s3file type has an optional s3_domain parameter. This allows you to point your s3file resource(s) at any S3-like service, including an internal OpenStack Swift server.

If your operating system does not provide cURL, or if you would prefer to manage the cURL package outside of the s3file module, you must ensure that cURL is installed before any s3file instances are evaluated. This can be done with the normal Puppet ordering primitives.

Conclusion

Puppet’s built-in file server is sufficient for most system configuration needs, but there are situations where storing files in S3 is a better option. The s3file module gives you the flexibility to use S3 and similar services without sacrificing the advantages of the Puppet resource model.

Learn More:

Share via:
Posted in:
Tagged:

I have launched a ec2 emr single node cluster and installed your module i.e. s3. But I am getting the below error when I am trying apply a .pp file having your module.

Unknown function validate_re at /etc/puppet/modules/s3file/manifests/init.pp:28 on node ip-xx-xx-xx-xxx.us-west-1.compute.internal
Please help.

Add new comment

The content of this field is kept private and will not be shown publicly.

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.