Pipelines for Applications

Builds are enqueued when an application, in Pipelines, is integrated with a software repository and there is a commit change to that repository.

Builds can also be triggered by Pull Requests and Tags.

Start a Build

To start a build, commit a change to the repository that is integrated with an application in Pipelines. For more information see:

Manual Builds

You can initialize a new build manually from the Application in Pipelines.

To initiate a new build from an application:

  1. Click the Applications link at the top of the Pipelines web UI.
  2. Navigate to the application you wish to build.
  3. In the top right, click the wrench symbol to build.

You are now building your application.

Enable auto build

Auto build is a feature that provides continuous integration and is the first part of continuous delivery and continuous deployment.

Note: To enable auto build your application must be integrated with a repository.

For integrating a repository with Pipelines, see Integrating with a Repository.

Auto build functionality is specific to an application.

The auto build options can be found in an applications overview in the App Pipeline.

Enable Auto Build

If you have not connected your application to a repository see Connecting a Repository to an Application.

  1. In the Pipelines web UI click Applications at the top
  2. Navigate to the application you wish to enable auto build.
  3. Ensure you are on the Overview tab.
  4. If you have not connected a repository. Click Connect Repository and see Connecting a Repository to an Application.
  5. App Pipeline Connect RepoOnce your application is connected to a repository click the Add Pipeline for XXXXXXX branch button. App Pipeline Repo Connected-->
  6. You should find the App Pipeline below. Click the Auto Build settings dropdown.
  7. New Pipeline info part1
  8. Select the auto build options for the selected repository branch.
  9. Auto Build Options
  10. Click the Auto Build settings dropdown to close it.

You have enabled auto build.

For continuous deployments see Enabling Auto Deploy.

Disable Auto Build

  1. In the Pipelines web UI click Applications at the top
  2. Navigate to the application you wish to disable auto build.
  3. Ensure you are on the Overview tab.
  4. Select the branch you wish to disable auto build.
  5. Click the Auto Build settings dropdown.
  6. Uncheck the settings as necessary.
  7. Auto Build Options

You have disabled auto build.

Rebuild a Build

Another way to start a build is to rebuild from an existing build in the build list page.

To enqueue a new build from an existing build:

  1. Click the Builds link at the top of the Pipelines web UI.
  2. Navigate through the list to the build you wish to rebuild.
  3. Click the rebuild symbol on the right in the build panel.

You are now rebuilding your application.

Cancel a build

You can cancel / abort a build from the build list page. To do so:

  1. Click the Builds link at the top of the Pipelines web UI.
  2. Navigate through the list to the build you wish to cancel.
  3. Click the X on the right in the build panel.

Cancel Build

View a build

You can view your builds from the builds list page. To navigate there click the Builds link at the top of the Pipelines web UI.

The builds list page provides a list of builds. This includes builds that are:

  • Queued
  • In progress
  • Successful
  • Failed

The builds list page also provides navigation to many important links.

Links

Each build panel contains links and information.

List Specifics

Build page

Clicking on a build from the builds list page will take you to the build page. The build page has many similar links as the build panel in the build list page. Importantly on the build page is the build log which is helpful in troubleshooting failed builds.

Build log

The build log contains useful information related to your build. It is also invaluable in troubleshooting. The build log can be found on the build page. To navigate to a build page:

  1. Click Builds at the top of the Pipelines web UI.
  2. Click on a build to see the associated log.

To download the build log, click the Download Log button on the build page.

Logging build results

There are several ways to capture information from activities during Pipelines builds.

But first, realize that builds in Pipelines have a complete log of what occurred. If you would like more information, simply output the text during the build, it will be captured in the build log, which you can watch live, or download when the build is complete.

Often customers want to store test results in a database or webpage. Or just provide enhanced information or feedback from their builds to their users. Depending on the scope of information and how you wish to present it, many options are available.

Our engineering team is always on the ready to help you build and customize your solutions in Pipelines. Feel free to reach out to us at [email protected]

This document covers two methods of pushing information from builds to users.

One method that Pipelines supplies right out of the box is the Pipelines manifest ReleaseNotesCmd section.

ReleaseNotesCmd Logging

This section of the Pipelines manifest will take the output of any command and include that in the Pipelines release notes, which can be found on the release details page.

To use this section, include it in your distelli-manifest.yml. Here is an example:

jdoe/example-app:
  ReleaseNotesCmd:
    - DISTELLI_USERNAME="jdoe"
    - BUILD_URL="https://www.distelli.com/$DISTELLI_USERNAME/builds/$DISTELLI_BUILDNUM"
    - LOG_DATE=$(date +"%Y/%m/%d %H:%M:%S")
    - LOG_STATUS="Custom Build ID $AHP_JOB_ID"
    - LOG_STRING="$LOG_DATE $LOG_STATUS Build# $DISTELLI_BUILDNUM $BUILD_URL"
    - echo -e "$LOG_STRING\n"

When navigating to the release notes created from this snippet, they may look like this:

2016/09/13 18:56:03 Custom Build ID "Rev1001.38-beta-x-ha3f" Build# 97155 https://www.distelli.com/jdoe/builds/97155

Realize, above is a simple example. Another example might be including test results:

  ReleaseNotesCmd:
    - cat test_results.output

Or inspect the files (artifacts) built during the build process:

  ReleaseNotesCmd:
    - ls -la target/

Whatever output you have in the ReleaseNotesCmd section of the Pipelines manifest, will be included in your release in Pipelines.

Of note, ReleaseNotesCmd is only run on build success. Failed builds do not create a release, so do not have any release notes.

Logging to AWS S3 Website

This section describes a solution logging to an external log source. This can be accomplished with any external source, not just AWS S3. No matter what external logging you use, the concepts are similar.

This solution includes capturing logging on both AfterBuildSuccess and AfterBuildFailure.

The next few steps will cover how to create a static website on Amazon AWS S3 and dynamically update the content with build information from Pipelines. This will allow you to publish build status, and enhance build information and logging.

Create the S3 Website

The first step is to create the AWS S3 static website. In the AWS Console, navigate to S3.

Next, create a bucket, or navigate to an existing bucket.

In the bucket Properties, under Static Website Hosting; Enable website hosting as exemplified below:

Setup S3 website

Add the Website Content Files

For this exercise, I created a simple website. The website looks like this:

The website

There is an embedded (iframe) of log information that is kept in a different file than index.html. This is simply called stuff.html (your naming may vary).

The code for all the files I used is below.

index.html

<html>
<head>
  <title>Pipelines S3 html build log example</title>
  <link rel="stylesheet" type="text/css" href="styles.css">
  <meta name="Generator" content="Brian McGehee">
  <meta http-equiv="Content-Type" content="text/html">
  <meta name="description" content="Pipelines S3 html build log example">
  <meta name="robots" content="noindex,nofollow">
</head>
<body>

<div class="logo">
<h1>Pipelines S3 html build log example</h1>
<p class="version"><br>
Feedback, suggestions, bugs...<br>
email: <a href="mailto:[email protected]">[email protected]</a></p><br>
</div>

<div>
<iframe src='stuff.html' seamless scrolling='yes' frameborder='1' class="divOptions"></iframe>
</div>

</body>
</html>

styles.css

html {
  margin: 0px;
  padding: 0px;
}
body { 
  background-repeat: repeat
  text-color: #000000;
  margin: 0px; 
  padding: 0px;
}
p { 
  font: 10pt courier; 
  margin-top: 0px; 
  text-align: left;
}
.divOptions {
  position: absolute;
  top: 120px;
  left: 22px;
  width: 1000px;
}
.logo {
  position: absolute;
  top: 22px;
  left: 20px;
  font: 10pt courier;
  margin-top: 0px;
  text-align: left;
}
.version {
  font: 8pt courier; 
  margin-top: 0px; 
  text-align: left;
}
h1 {
  COLOR: #000000;
  FONT-FAMILY: sans;
  FONT-WEIGHT: bold;  
  FONT-SIZE: 18px;
  margin: 0px;
}
a:link { 
  text-decoration: underline; 
  color: #FF0000;
}
a:visited { 
  text-decoration: underline; 
  color: #FF2222;
}
a:hover, a:active { 
  text-decoration: none; 
  color: #444444;
}

favicon.ico

This is your website default icon file. Not necessary.

stuff.html

This file should be initially created with at least 1 line of text to seed the file.

This is the last (bottom) line of the stuff.html logfile.<br>

Now upload all those files to the S3 bucket created earlier.

Setup S3 website

If you try and access the site now, you may see this message:

403

This is a permissions issue.

S3 Website File Permissions

The files must be readable to be served as web content. The simpliest way to do this is grant Everyone Open/Download permission.

Select each file, in succession, in the AWS S3 console, click Properties and expand the Permissions section.

Grant the appropriate permissions.

S3 file permissions

And Save your changes.

With your web browser navigate to the S3 website again.

The Website

Logging to the Website from Pipelines

To complete the following steps, AWS credentials will be required to perform AWS CLI commands. In particular, aws s3 cp. These credentials must have access to provide read/write/grant-permissions to the S3 website bucket file stuff.html.

Adding AWS Credentials to Application in Pipelines

This assumes there exists an application in Pipelines that you wish to log build information to the S3 website. If not, see Creating an Application.

Navigate to the application in Pipelines and click the Branch Build Options link. In the resulting window, enter Pipelines build variables for:

  • AWS_ACCESS_KEY - The AWS Access key.
  • AWS_SECRET_ACCESS_KEY - The AWS Secret key

AWS Build vars

Click Save to save your edits.

Updating the Pipelines Manifest

The Pipelines manifest has your build and deploy instructions. Add the following to your manifest as appropriate.

In the PreBuild section, check to see if AWS CLI is installed, and if not, install it.

  PreBuild:
    # Check to see if aws cli is installed
    - if which aws >/dev/null; then
    -   echo "--AWS CLI exists--"
    - else
    -   echo "--Installing AWS CLI--"
    # Check to see if unzip is installed
    -   if which unzip >/dev/null; then
    -     echo "--Unzip exists--"
    -   else
    -     echo "--Installing unzip--"
    -     sudo apt-get -y install unzip
    -   fi
    -   curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
    -   unzip awscli-bundle.zip
    -   sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
    - fi

In the AfterBuildSuccess and AfterBuildFailure sections, code will be added to provide logging information on the S3 webserver in the file stuff.html.

  AfterBuildSuccess:
    - DISTELLI_USERNAME="jdoe"
    - BUILD_URL="https://www.distelli.com/$DISTELLI_USERNAME/builds/$DISTELLI_BUILDNUM"
    - LOG_DATE="$(date +"%Y/%m/%d %H:%M:%S")"
    - LOG_STATUS="Build Succeeded. Build ID $AHP_JOB_ID"
    - LOG_STRING="$LOG_DATE $LOG_STATUS <a href=\"$BUILD_URL\" target=\"_blank\">Build# $DISTELLI_BUILDNUM</a><br>\n"
    - echo "$LOG_STRING"
    - aws s3 cp s3://jdoe-web/stuff.html stuff.html
    - sed -i "1s|^|$LOG_STRING|" stuff.html
    - aws s3 cp stuff.html s3://jdoe-web/stuff.html --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers
  AfterBuildFailure:
    - DISTELLI_USERNAME="jdoe"
    - BUILD_URL="https://www.distelli.com/$DISTELLI_USERNAME/builds/$DISTELLI_BUILDNUM"
    - LOG_DATE="$(date +"%Y/%m/%d %H:%M:%S")"
    - LOG_STATUS="Build <b><font color=\"red\">Failed</font></b>. Build ID $AHP_JOB_ID"
    - LOG_STRING="$LOG_DATE $LOG_STATUS <a href=\"$BUILD_URL\" target=\"_blank\">Build# $DISTELLI_BUILDNUM</a><br>\n"
    - echo "$LOG_STRING"
    - aws s3 cp s3://jdoe-web/stuff.html stuff.html
    - sed -i "1s|^|$LOG_STRING|" stuff.html
    - aws s3 cp stuff.html s3://jdoe-web/stuff.html --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers  

As you can see, whether the build succeeds or fails, we update the S3 website with the appropriate information. To break down what happens in both AfterBuildSuccess and AfterBuildFailure.

  1. First, we gather all the data and build our message. This includes:
  • DISTELLI_USERNAME - Your Pipelines username.
  • BUILD_URL - The URL back to the Pipelines build log specific to this build.
  • LOG_DATE - The current date and time stamp.
  • LOG_STATUS - The Build status [Success || Failed (in red)].
  • LOG_STRING - A concatenation of the above data.
  • Download the current S3 stuff.html file.
  • Creates the top most recent entry in the stuff.html with the LOG_STRING.
  • Upload the S3 stuff.html file.
  • After running several builds with failure and success, your S3 website page may look similar to this:

    Working website

    Conclusions

    Though this is specifically targeted, it can be expanded to many use cases.

    Scenario: Each build should create its own S3 webpage with a ton of test logging and output.

    Solution: In the AfterBuildSuccess and AfterBuildFailure sections:

    1. As part of the data gathered (noted above) also include:
    • UNIQUE_FILENAME - a unique webpage filename that includes $DISTELLI_BUILDNUM for uniqueness. </ul>
    • Create a new file named UNIQUE_FILENAME.
    • Write the mass amounts of test data to this file in a text readable format.
    • Upload the UNIQUE_FILENAME to S3 with the correct read permissions.
    • Download S3 stuff.html file.
    • Add an entry into stuff.html with a URL link to UNIQUE_FILENAME on S3.
    • Upload stuff.html back to S3.
    • </ol> Here is an example AfterBuildFailure manifest snippet. The only difference in AfterBuildSucces is the LOG_STATUS. ~~~ AfterBuildFailure: - DISTELLI_USERNAME="jdoe" - BUILD_URL="https://www.distelli.com/$DISTELLI_USERNAME/builds/$DISTELLI_BUILDNUM" - S3_BUCKET="s3://jdoe-web" - S3_URL="http://jdoe-web.s3-website-us-west-2.amazonaws.com" - UNIQUE_FILENAME="distelli-build$DISTELLI_BUILDNUM.txt" - UNIQUE_BUCKET="$S3_BUCKET/$UNIQUE_FILENAME" - UNIQUE_URL="$S3_URL/$UNIQUE_FILENAME" - LOG_DATE="$(date +"%Y/%m/%d %H:%M:%S")" - LOG_STATUS="Build <font color=\"red\">Failed</font>." - LOG_STRING="$LOG_DATE $LOG_STATUS <a href=\"$BUILD_URL\" target=\"_blank\">Build# $DISTELLI_BUILDNUM</a> <a href=\"$UNIQUE_URL\" target=\"_blank\">Test Results</a>
      " - echo "$LOG_STRING" > "$UNIQUE_FILENAME" - cat test_results.exa >> "$UNIQUE_FILENAME" - NEW_LINK="http://jdoe-web.s3-website-us-west-2.amazonaws.com/$UNIQUE_FILENAME" - aws s3 cp "$UNIQUE_FILENAME" "$UNIQUE_BUCKET" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers - aws s3 cp "$S3_BUCKET/stuff.html" stuff.html - sed -i "1s|^|$LOG_STRING|" stuff.html - aws s3 cp stuff.html "$S3_BUCKET/stuff.html" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers ~~~ Ultimately this solution gives you a list of links to test results for each build. Working website Publishing results from your builds is easy when your build platform supports anything you can do from the command line.
    Back to top