Integrate Amazon and AWS

Sections

Integrate with AWS EC2

Integrating your Pipelines account with AWS EC2 provides the necessary credentials for you to successfully provision EC2 instances with Pipelines.

For more information on getting started with Amazon and EC2 see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html.

For more information on setting up your Amazon Access Key see docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html.

The AWS IAM user role must have access to ec2. An example IAM policy that would allow ec2 access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "ec2:*",
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

If you need to lock the user down to a specific set of resourses, please see: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-policy-structure.html.

Configure AWS EC2 Credentials

Ensure you are using the correct account before continuing.

  1. In the Pipelines web UI at the top right, click the gear to navigate to the account settings.
  2. Click the Integrations link on the left.
  3. Click the AWS button.
  4. You will find yourself at the AWS integrations credentials page.

  5. Scroll down to AWS EC2 section and click Add Creds.
  6. AWS EC2 Credentials
  7. Enter your AWS EC2 Key Name, Access Key, and Secret Key.
  8. Click Submit when you are ready.

Using multiple EC2 credentials

This feature allows the addition of multiple AWS EC2 credentials from which to launch EC2 instances. This includes launching instances for a Kubernetes cluster in AWS.

Setup Multiple EC2 credentials

To navigate to the EC2 credentials:

  1. Ensure you are on the Pipelines for Applications web UI.
  2. Click the Gear icon at the top right of the Pipelines web UI.
  3. Click the Integrations link on the left.
  4. Click the Amazon Web Services icon.
  5. Select the AWS EC2 tab.

You can add multiple keys by clicking the Add Key link.

ec2 add keys

RBAC Permissions for EC2 Credentials

After adding one or more AWS EC2 credentials, a permission group can be configured to control which users have access to which credentials.

To create group permissions for access to AWS EC2 credentials:

  1. Ensure you are on the Pipelines for Applications web UI.
  2. Click the Gear icon at the top right of the Pipelines web UI.
  3. Click the Groups link on the left.
  4. Either Create New Group or edit an existing group.
  5. Scroll to the bottom of the group permissions to find the Provisioning section.

From here the group permission specifies what EC2 credentials users of this group have and do not have access to.

ec2 key permissions

For more information on users and group permissions see:

Using Multiple EC2 Credentials

Now, when launching EC2 instances (or EC2 clusters) a user will be prompted which credentials to use, based on the available credentials to that user.

ec2 launch

Integrate with Amazon ECR

Integrating your Pipelines account with AWS EC2 Container Registry provides the necessary credentials for you to successfully build and deploy Docker images/containers with Pipelines.

Note: The following credential configuration can be completed when Enabling Docker for new Applications, Enabling Docker for Existing Applications, and in the Docker Settings See the Docker documentation for details.

Configure AWS ECR Credentials

Ensure you are using the correct account before continuing.

  1. In the Pipelines web UI at the top right, click the gear to navigate to the account settings.
  2. Click the Integrations link on the left.
  3. Click the AWS icon.
  4. You will find yourself at the AWS integrations credentials page.

  5. Select the AWS ECR tab.
  6. Click the Add AWS ECR Credentials link.
  7. AWS ECR Credentials
  8. Enter your AWS ECR Key Name, Access Key, and Secret Key.
  9. Click Add when you are ready.

Integrate with S3 Bucket

You can specify an S3 bucket that Pipelines will use to store your bundled releases when you do a distelli push. When you deploy the software the Pipelines agent will retrieve the bundle from your S3 bucket. This document shows you how to create, authorize, and setup that bucket so Pipelines will use it to store your release bundles. For Pipelines to access the S3 bucket you will need to provide:

  • The bucket name
  • An access token
  • A secret key

Overview

When you incorporate an S3 bucket with your Pipelines account Distelli will keep your bundled application releases in your S3 bucket. After adding your S3 credentials to Pipelines on your first application push, Pipelines will create, in your S3 bucket, a folder called distelli-packages.

When you issue a distelli push command and push your application release, the application release will be bundled and uploaded to your S3 bucket using SSL. This bundle is in a gzip tar format. The files bundled depend on your Pipelines Manifest PkgInclude section When deploying an application release, on the destination server the Pipelines agent will retrieve the application bundle from your S3 bucket.

Create an S3 Bucket on AWS

First step will be to create an S3 bucket. If you already have an existing bucket you wish to use, you may skip to the Create an AWS Policy to Grant S3 Bucket Permissions step.

  1. Sign in to the AWS console
  2. In the Amazon Web Services under Storage & Content Delivery select S3 (Scalable Storage in the Cloud).
  3. Click the Create Bucket button.
  4. Enter the Bucket Name (for example, “Pipelines”).
  5. Select a Region.
  6. Click the Create button.

You have created an S3 Bucket. Remember your bucket name.

Create an AWS Policy to Grant S3 Bucket Permissions

These steps will create an AWS policy to grant access to an S3 Bucket.

  1. In the AWS console, click the square logo on the top left to go back to the main console menu.
  2. In the Amazon Web Services under Administration and Security select Identity & Access Management.
  3. In the left Dashboard select Policies.
  4. Click the Create Policy button.
  5. For Create Your Own Policy click the Select button.
  6. Give the policy a Policy Name (for example, “Pipelines_S3_Bucket”).

    Important: For the following steps, replace BUCKETNAME with your AWS S3 bucket name.

  7. Give the policy a Description (for example “Provides Pipelines access to the S3 bucket BUCKETNAME”.
  8. In the policy, enter the following:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::BUCKETNAME/*"
      ]
    }
  ]
}
Did you remember to replace <b>BUCKETNAME</b> with your S3 bucket name?
  1. Click the Validate Policy button.

You should see a This policy is valid. message.

  1. Click Create Policy button.

You have created a policy that can be used to provide a user or group access to the S3 bucket.

Create an AWS User

These steps will create an AWS User to access the S3 Bucket. If you already have an AWS user you would like to use and you know their associated security key and token skip forward to These steps will create a single AWS user account.

  1. Make sure you are in the AWS Console and have selected Identity & Access Management under Administration & Security.
  2. In the left Dashboard select Users.
  3. Click Create New Users button.
  4. Ensure the Generate an access key for each user box is checked.
  5. In field 1 enter a user name (for example “Pipelines”).
  6. Click the Create button.

    Note: The Security Credentials are available for download one time only. These security credentials are required to add your S3 bucket to Pipelines. Keep them safe.

  7. Click the Show User Security Credentials link.
  8. Download the credentials from the Download Credentials button.

    Important: Preserve these credentials in a secure manner that meets your organizations security requirements.

  9. Click Close button.

You have created a new AWS user. You must apply the policy, created above, to the user to allow them access to the S3 bucket.

Apply AWS Policy to AWS User


In this section you will apply the Policy to the AWS User to grant that user access to the S3 bucket.

  1. Make sure you are in the AWS Console and have selected Identity & Access Management under Administration & Security.
  2. In the left Dashboard select Users.
  3. Click the user you wish to apply the policy to.
  4. Find the Permissions section and under Managed Policies, click the Attach Policy button.
  5. Find the policy you created above, Pipelines_S3_Bucket and select it by clicking its checkbox.
  6. Click the Attach Policy button.

With this policy now attached to the user, the user is now configured to access the S3 bucket.

Adding an S3 Bucket to Pipelines

The following steps must be completed in the Pipelines web UI.

  1. Ensure you are signed into Pipelines.
  2. Click the gear icon on the top right.
  3. Click the Integrations link on the left.
  4. Click the AWS button.
  5. Under where it says “AWS S3” click the Add Creds button.
  6. Enter your S3 Bucket Name.
  7. Enter your AWS User S3 Access Token.
  8. Enter your AWS User S3 Secret Key.

Note: If you did not get your AWS user token and key during their one-time availability, or have lost it, see the next section Getting New AWS User Credentials.

Getting New AWS User Credentials

If you have an AWS account where you don’t know the current token and key credentials, you can create new ones. Remember you will be able to view and download the secret key only one time.

  1. Make sure you are in the AWS Console and have selected Identity & Access Management under Administration & Security.
  2. In the left Dashboard select Users.
  3. Click the user you wish to get new credentials for.
  4. Find the Access Keys section under Security Credentials and click the Create Access Key button.

Important: The Security Credentials are available for download one time only. These user credentials are required to add your S3 bucket to Pipelines. Keep them safe.

  1. Click the Show User Security Credentials link.
  2. Download the credentials from the Download Credentials button.

Important: Preserve these credentials in a secure manner that meets your organizations security requirements.

You now have new working known credentials for your AWS user. You may want to remove any extraneous credentials.

Doing a distelli push to AWS S3 Bucket

After you have correctly setup the S3 bucket in Pipelines, when doing a distelli push you can see the bundle being uploaded to S3.

$ distelli push
    Packaged 1 files in 1 directories
    Pushing App: SimpleApp
    Uploading Bundle to S3: [==========] 100%
    Creating Release... DONE
How helpful was this page?
Puppet sites use proprietary and third-party cookies. By using our sites, you agree to our cookie policy.