Pipelines events with Jenkins

Pipelines for Applications — enterprise

Sections

Integrating Pipelines with Jenkins is already feasible using the Pipelines CLI.

The addition of Pipelines event APIs now allows greater integration with Jenkins, including the ability to:

Create Pipelines application push events when Jenkins receives a commit.
Create Pipelines application pull request events when Jenkins receives a pull request.
Create Pipelines application build event when Jenkins starts builds.
Update Pipelines build event when Jenkins completes builds.

This will provide users of Pipelines with links to the repository commits, links to Jenkins builds, and information on when and how long the build took.

This also provides for automated deployments upon succssful Jenkins builds using Pipelines.

Note, this document is geared towards Pipelines enterprise customers who are supporting multiple teams and/or users. Information is supplied on implementing this for a single user account in the Pipelines SaaS also.

Create Pipelines Jenkins user for Enterprise use

This section describes setting up a Pipelines Jenkins user for use in an enterprise team. With this scenario, only one Pipelines user is setup and given access to enterprise team accounts, as necessary. This allows centralized management of Pipelines authentications for Jenkins.

If, instead of an enterprise, this is a single user who would like to integrate with Jenkins build events, then please skip this section and go to Install Jenkins Prerequisites.

Creating a Pipelines Jenkins user for enterprise team use:

Create the Pipelines Jenkins user. Go to https://pipelines.puppet.com/signup and sign up.

sign-up

This user will be used to authenticate API requests to Pipelines. In the next section this user must be added to enterprise team users’ accounts so that the Jenkins user can push other users’ releases.

Add Pipelines Jenkins user to users' accounts.

For every user that wishes to build with Jenkins and deploy with Pipelines, the user must add the Pipelines Jenkins user to their account. This is an iterative process and as new users adopt, they will have to complete these steps.

How to add the Pipelines Jenkins user to an account.

Login to the users account that wishes to do Jenkins builds with Pipelines (not the Pipelines Jenkins user just created).
Click the gear at the top right of the Pipelines web UI.
On the left click Users.
Click the link + Add a New User.
Enter the email that was setup for the Pipelines Jenkins user.

Select the Pipelines Jenkins user and click Add Users.

Now the Pipelines Jenkins user can push application releases from Jenkins to Pipelines.

Install Jenkins pre-requisites

Install Plugins

To perform this integration, Jenkins requires the following plugins are installed:

Credentials plugin - provides a centralized way to define credentials that can be used by the Jenkins instance, plugins, and build jobs.
Credentials Binding plugin - allows the configuration of build jobs to inject credentials as environment variables.
Plain Credentials plugin - a plugin dependency required by the Credentials Binding plugin.
Hudson Post Build Task - provides scripting based on build log output. Can catch failed builds and execute scripts on failure.

Install Utilities

Jenkins also requires that the following utilities are installed on the operating system:

jq 1.5+ - This must be 1.5 or greater!
curl 7.33+

Install Pipelines CLI

Next, it is necessary to install the Pipelines CLI on Jenkins. Please follow these instructions to install the Pipelines CLI:

Login to the Jenkins server.

**Note: You could login as the jenkins user, or sudo to do this as the jenkins user. sudo su jenkins -s /bin/bash
cd to the jenkins user home directory. Typically this is: /var/lib/jenkins.
Create a file in the jenkins user home directory called distelli.yml and enter the following.

~~~ DistelliAccessToken: ‘Distelli Access Token’ DistelliSecretKey: ‘Distelli Secret Key’ ~~~
To retrieve the values required for this file, login to Pipelines as the Pipelines Jenkins user.

**Note: If this an integration to Jenkins with a single user account, then login to Pipelines with the single account.
- Click the gear icon at the top right of the Pipelines web UI.
- Click Agent link on the left menu.
- Click the Show button to see the required Pipelines Access Token and Pipelines Secret Key. Use these values in the distelli.yml file that is being editing on the Jenkins server.
~~~ DistelliAccessToken: ‘12345678901234567890123456’ DistelliSecretKey: ‘1234567890123456789012345678901234567’ ~~~
- Save the file.
Note: If you did the above file creation as any user other then jenkins user, you must ensure the file is owned by the jenkins user. This can typically be done with the following: sudo chown jenkins:jenkins distelli.yml
Install the CLI.

Linux and macOS X

To install on Linux/MAC use either curl or wget with one of the following syntaxes.

wget example

wget -qO- https://pipelines.puppet.com/download/client | sh

curl example

curl -sSL https://pipelines.puppet.com/download/client | sh

Windows

To install on Windows, copy and paste the following powershell command into a command (cmd) window.

powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((new-object net.webclient).DownloadString('https://pipelines.puppet.com/download/client.ps1'))" & SET PATH=%PATH%;%ProgramFiles%/Distelli

The Pipelines CLI is installed. This, and the credentials, can be validated. This must be done as the jenkins user.

distelli login -conf distelli.yml

There should be no response or error.

Add Pipelines API Token to Jenkins

Adding a Pipelines API token to Jenkins requires the following Jenkins plugins to be installed:

Credentials plugin
Credentials Binding plugin
Plain Credentials plugin

See the Install Jenkins Prerequisites section for more information on installing these plugins.

If this is an enterprise user that is using a shared Jenkins account in Pipelines, log in to that account now.

If this is a single user integrating with Jenkins, log in to the single user Pipelines account now.

The following will obtain the Pipelines API token for the account logged into.

Click the gear at the top right in the Pipelines web UI.
Ensure the API selection on the left menu is selected. If there is not a token currently, click the Create API Token button to create one.
Copy the Token.
Login to the Jenkins web inteface.
From the Jenkins dashboard, click the Credentials link.
Click to navigate to the global credentials.
Click Add Credentials.
Set the credentials Kind to Secret text.
Paste the Pipelines API Token into the Secret field.

In the ID field enter DISTELLI_API_TOKEN.
Click OK when done.

The Pipelines API Token has been added to the Jenkins global credentials. Next an application must be created in Pipelines and a distelli-manifest.yml must be created.

Create an Application in Pipelines

To deploy a Jenkins built applcation in Pipelines, there must first be an application in Pipelines.

This application that is created, must not be attached to a repository.

These steps are done from a Pipelines user account that wishes to build on Jenkins, not the Pipelines Jenkins user.

Login to Pipelines with the user account.
Click Applications at the top of the Pipelines web UI.
Click New App button at the top right.
Click No Repository link.

Click I've pushed my Repo button.
Give the application an App Name. This is often the same as the software repository name.

Click Looks good. Create App! button.

A new application has been created in Pipelines. Now the Jenkins build job must be configured to interact with Pipelines.

Create a distelli-manifest.yml

A distelli-manifest.yml file defines the application’s build, test, and deployment instructions. In this scenario, the manifest will not require any build steps as Jenkins is doing the build. But the manifest must indicate what artifacts to keep from the build and the application deployment instructions.

Below is an example manifest. Use this as an initial starting point.

Copy and edit this manifest:

USERNAME/APPLICATION_NAME: 
  # Distelli Manifest example 
  
  Build:
    - echo "Built and tested with Jenkins"
       
  PkgInclude:
    # Include ALL files in the release package
    - '*'

  ###Below is deployment phases   
  
  Env:
    # Set any environment variables in this section.   
    # These are available during all phases of a deploy.
 
  PreInstall:  
    # Use this section to install any deployment pre-requisites.
 
  PostInstall:
    # Use this section to affect any changes before running app.

  Exec:
    # Use this section to run the application.

Change USERNAME to the Pipelines username who owns the application. The Disetelli account in which the application was created.
Change APPLICATION_NAME to the new Pipelines application name that was created.
Save this file.
Check this file into the software repository being built with Jenkins and deployed with Pipelines.

Configure Jenkins Build Job to Update Pipelines Events

There are two scripts to configure in Jenkins.

The “before build” script will create a Pipelines push event. This event represents the commit that triggered the upcoming build. The before build script will also create a Pipelines build event and mark that build event status as Running, and set the build begin date/time.

Following a successful build, an “after build” script will execute a Pipelines CLI distelli push to push the build artifacts up to Pipelines. This push creates a release of your application in Pipelines which can be deployed, automatically, in true continuous delivery/continuous deployment style.

Also in the after build script, the previously created build event will be updated to status “Successful” and the build end date/time will be set.

Jenkins Project Environment Variables

Navigate to the project in Jenkins.
Click Configure link on the left.
Navigate to the Build Environment section.
Check (select) the Use secret text(s) or file(s) selection.

This option is available only if the Jenkins plugin prerequisites are installed.

Under Bindings, click Add.

Select Secret text from the drop down.

In the Variable field, enter DISTELLI_API_TOKEN.
Note: Do not shorten the env variable to DISTELLI_TOKEN. This is a reserved variable.
From the drop down, select The Pipelines API Token that was created above.
Save the changes.

Jenkins Project "Before Build" Script

Navigate to the project in Jenkins.
Click Configure link on the left.
Navigate to the Build section.
Click the Add build step button.

Select Execute shell from the drop down.
In the Shell window, add the following code.


#!/bin/bash
# This script creates a PUSH event in Distelli for the commit
# This script also creates a BUILD event setting the build as "Running"

# Setting global Distelli Env Variables
echo -e "\nSetting global Distelli environment variables\n"
DISTELLI_USERNAME=jdoe
DISTELLI_API_URL="https://api.distelli.com/$DISTELLI_USERNAME"
DISTELLI_APP_NAME="example-jenkins"
DISTELLI_CI_PROVIDER="jenkins"
DISTELLI_BUILD_STATUS="Running"

# Setting repo specific Env Variables
echo -e "Setting repository specific environment variables\n"
if [ ! -z "$GIT_COMMIT" ]; then
  echo -e "Using GIT variables.\n"
  DISTELLI_BUILD_ID=$BUILD_NUMBER
  DISTELLI_BUILD_URL=$BUILD_URL
  DISTELLI_CHANGE_URL=$GIT_URL
  DISTELLI_CHANGE_AUTHOR=$GIT_AUTHOR_NAME
  DISTELLI_CHANGE_AUTHOR_DISPLAY_NAME=$GIT_AUTHOR_NAME
  DISTELLI_CHANGE_TITLE="NoChangeMessage"
  DISTELLI_CHANGE_ID=$GIT_COMMIT
  DISTELLI_BRANCH_NAME=$GIT_BRANCH
  DISTELLI_CHANGE_TARGET=$GIT_URL
elif [ ! -z "$YOUR_OTHER_VARIABLE" ]; then
  # This section is for customizing your own software repo env variables.
  # Change the $YOUR_OTHER_VARIABLE above to one in your software repo env
  # variables that exists, to activate this block.
  echo -e "Using GIT variables.\n"
  DISTELLI_BUILD_ID=$YOUR_BUILD_ID
  DISTELLI_BUILD_URL=$YOUR_BUILD_URL
  DISTELLI_CHANGE_URL=$YOUR_CHANGE_URL
  DISTELLI_CHANGE_AUTHOR=$YOUR_CHANGE_AUTHOR
  DISTELLI_CHANGE_AUTHOR_DISPLAY_NAME=$YOUR_CHANGE_AUTHOR_DISPLAY_NAME
  DISTELLI_CHANGE_TITLE=$YOUR_CHANGE_TITLE
  DISTELLI_CHANGE_ID=$YOUR_CHANGE_ID
  DISTELLI_BRANCH_NAME=$YOUR_BRANCH_NAME
  DISTELLI_CHANGE_TARGET=$YOUR_CHANGE_TARGET
else
  echo -e "No GIT variables available. Using standard Jenkins variables.\n"
  DISTELLI_BUILD_ID=$BUILD_NUMBER
  DISTELLI_BUILD_URL=$BUILD_URL
  DISTELLI_CHANGE_URL="NoChangeURL"
  DISTELLI_CHANGE_AUTHOR="NoChangeAuthor"
  DISTELLI_CHANGE_AUTHOR_DISPLAY_NAME="NoChangeAuthorName"
  DISTELLI_CHANGE_TITLE="NoChangeMessage"
  DISTELLI_CHANGE_ID="NoChangeID"
  DISTELLI_BRANCH_NAME="NoChangeBranch"
  DISTELLI_CHANGE_TARGET="NoChangeTarget"
fi

# Creating Distelli PUSH event
echo -e "\nCreating Distelli PUSH Event\n"

API_JSON=$(printf '{"commit_url":%s, "author_username":%s, "author_name":%s, "commit_msg":%s, "commit_id":%s, "branch":%s, "repo_url":%s}' \
  "$(jq -R . <<<"$DISTELLI_CHANGE_URL")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_AUTHOR")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_AUTHOR_DISPLAY_NAME")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_TITLE")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_ID")" \
  "$(jq -R . <<<"$DISTELLI_BRANCH_NAME")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_TARGET")")
  
 
DISTELLI_RESPONSE=$(curl -s -k -X PUT -H "Content-Type: application/json" "$DISTELLI_API_URL/apps/$DISTELLI_APP_NAME/events/pushEvent?apiToken=$DISTELLI_API_TOKEN" -d "$API_JSON")
DISTELLI_PUSH_EVENT_ID=$(echo $DISTELLI_RESPONSE | jq .event_id | tr -d '"')
echo -e "push_event_id: $DISTELLI_PUSH_EVENT_ID\n"


# Creating Distelli BUILD event
echo -e "\nCreating Distelli BUILD Event\n"

DISTELLI_NOW=$(date -u +%Y-%m-%dT%H:%M:%S.0Z)
API_JSON=$(printf '{"build_status":%s, "build_start":%s, "build_id":%s, "build_provider":%s, "build_url":%s, "repo_url":%s, "commit_url":%s, "author_username":%s, "author_name":%s, "commit_msg":%s, "commit_id":%s, "branch":%s, "parent_event_id":%s}' \
  "$(jq -R . <<<"$DISTELLI_BUILD_STATUS")" \
  "$(jq -R . <<<"$DISTELLI_NOW")" \
  "$(jq -R . <<<"$DISTELLI_BUILD_ID")" \
  "$(jq -R . <<<"$DISTELLI_CI_PROVIDER")" \
  "$(jq -R . <<<"$DISTELLI_BUILD_URL")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_TARGET")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_URL")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_AUTHOR")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_AUTHOR_DISPLAY_NAME")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_TITLE")" \
  "$(jq -R . <<<"$DISTELLI_CHANGE_ID")" \
  "$(jq -R . <<<"$DISTELLI_BRANCH_NAME")" \
  "$(jq -R . <<<"$DISTELLI_PUSH_EVENT_ID")")

  
DISTELLI_RESPONSE=$(curl -s -k -H "Content-Type: application/json" \
  -X PUT "$DISTELLI_API_URL/apps/$DISTELLI_APP_NAME/events/buildEvent?apiToken=$DISTELLI_API_TOKEN" \
  -d "$API_JSON")
  
DISTELLI_BUILD_EVENT_ID=$(echo $DISTELLI_RESPONSE | jq .event_id | tr -d '"')
echo -e "build_event_id: $DISTELLI_BUILD_EVENT_ID\n\n"

# Saving build event id
DISTELLI_TMP_FILENAME="DISTELLI.$JOB_NAME.$BUILD_NUMBER.tmp"
echo "$DISTELLI_BUILD_EVENT_ID" > "$DISTELLI_TMP_FILENAME"

In the above code, you may need to change the following fields:

DISTELLI_USERNAME - the Pipelines username of the user building with Jenkins and deploying with Pipelines.
DISTELLI_API_URL - If using the Pipelines SaaS, this is https://api.distelli.com/$DISTELLI_USERNAME
DISTELLI_APP_NAME - The name of the application in Pipelines. This was configured in the Create an Application in Pipelines step.

Now drag the Execute shell section, just created, above (before) the existing application build commands. This section can be moved by grabbing the grid, to the left of Execute shell, at the top left.

Save your changes.

Jenkins Project "After Build" (Success) Script

Navigate to the project in Jenkins.
Click Configure link on the left.
Navigate to the Build section.
Click the Add build step button.

Select Execute shell from the drop down.
In the Shell window, add the following code.


#!/bin/bash
# This script will update the Build status

# Push the release to Distelli with the Distelli CLI
echo -e "\n\nPushing the Distelli release\n"
DISTELLI_USERNAME=jdoe
DISTELLI_API_URL="https://api.distelli.com/$DISTELLI_USERNAME"
DISTELLI_APP_NAME="example-jenkins"
DISTELLI_TMP_RELEASE="DISTELLIrelease.$JOB_NAME.$BUILD_NUMBER.tmp"
DISTELLI_RESPONSE=$(distelli push -save-release "$DISTELLI_TMP_RELEASE")
DISTELLI_RELEASE_VERSION=$(cat "$DISTELLI_TMP_RELEASE")
DISTELLI_RESPONSE=$(rm "$DISTELLI_TMP_RELEASE")
echo -e "Distelli Release Version: $DISTELLI_RELEASE_VERSION\n"

# Setting global Distelli Env Variables
echo -e "\nSetting global Distelli environment variables\n"
DISTELLI_CI_PROVIDER="jenkins"
DISTELLI_BUILD_STATUS="Success"

# Preparing to update the build event with "Success" and build end time
echo -e "\nPreparing to update build event in Distelli\n"
DISTELLI_NOW=$(date -u +%Y-%m-%dT%H:%M:%S.0Z)
DISTELLI_TMP_FILENAME="DISTELLI.$JOB_NAME.$BUILD_NUMBER.tmp"
DISTELLI_BUILD_EVENT_ID=$(cat "$DISTELLI_TMP_FILENAME")
DISTELLI_RESPONSE=$(rm "$DISTELLI_TMP_FILENAME")

echo -e "Updating build event in Distelli\n"
# Updating build event with "Success" and build end time
API_JSON=$(printf '{"build_status":%s, "build_end":%s, "release_version":%s}' \
  "$(jq -R . <<<"$DISTELLI_BUILD_STATUS")" \
  "$(jq -R . <<<"$DISTELLI_NOW")" \
  "$(jq -R . <<<"$DISTELLI_RELEASE_VERSION")")
  
DISTELLI_RESPONSE=$(curl -s -k -H "Content-Type: application/json" \
  -X POST "$DISTELLI_API_URL/apps/$DISTELLI_APP_NAME/events/$DISTELLI_BUILD_EVENT_ID?apiToken=$DISTELLI_API_TOKEN" \
  -d "$API_JSON")
  
echo -e "Distelli Build Update Response:\n $DISTELLI_RESPONSE\n\n"

In the above code, you may need to change the following fields:

DISTELLI_USERNAME - the Pipelines username of the user building with Jenkins and deploying with Pipelines.
DISTELLI_API_URL - If using the Pipelines SaaS, this is https://api.distelli.com/$DISTELLI_USERNAME
DISTELLI_APP_NAME - The name of the application in Pipelines. This was configured in the create an application step above</a>.

This Execute shell section should be after all build and test scripts have completed. Save your changes.

You are ready to build your application with Jenkins and push releases to Pipelines for automated deployments.

Jenkins Project "After Build Failure" Script

This section decribes how to setup Jenkins to update Pipelines build events in the case of a build failure. This requires the following Jenkins plugin to be installed.

Hudson Post Build Task

Navigate to the project in Jenkins.
Click Configure link on the left.
Navigate to the Post-build Actions section.
Click the Add post-build step button.

Select Post build task from the drop down.

In the Log text field enter marked build as failure.
In the Script section enter the following script:


#!/bin/bash
# This script will update the Build status

DISTELLI_USERNAME=jdoe
DISTELLI_API_URL="https://api.distelli.com/$DISTELLI_USERNAME"
DISTELLI_APP_NAME="example-jenkins"
DISTELLI_TMP_RELEASE="DISTELLIrelease.$JOB_NAME.$BUILD_NUMBER.tmp"

# Setting global Distelli Env Variables
echo -e "\nSetting global Distelli environment variables\n"
DISTELLI_CI_PROVIDER="jenkins"
DISTELLI_BUILD_STATUS="Failed"

# Preparing to update the build event with "Failed" and build end time
echo -e "\nPreparing to update build event in Distelli\n"
DISTELLI_NOW=$(date -u +%Y-%m-%dT%H:%M:%S.0Z)
DISTELLI_TMP_FILENAME="DISTELLI.$JOB_NAME.$BUILD_NUMBER.tmp"
DISTELLI_BUILD_EVENT_ID=$(cat "$DISTELLI_TMP_FILENAME")
DISTELLI_RESPONSE=$(rm "$DISTELLI_TMP_FILENAME")

echo -e "Updating build event in Distelli\n"
# Updating build event with "Success" and build end time
API_JSON=$(printf '{"build_status":%s, "build_end":%s}' \
  "$(jq -R . <<<"$DISTELLI_BUILD_STATUS")" \
  "$(jq -R . <<<"$DISTELLI_NOW")")
  
DISTELLI_RESPONSE=$(curl -s -k -H "Content-Type: application/json" \
  -X POST "$DISTELLI_API_URL/apps/$DISTELLI_APP_NAME/events/$DISTELLI_BUILD_EVENT_ID?apiToken=$DISTELLI_API_TOKEN" \
  -d "$API_JSON")
  
echo -e "Distelli Build Update Response:\n $DISTELLI_RESPONSE\n\n"

In the above code, you may need to change the following fields:

DISTELLI_USERNAME - the Distelli username of the user building with Jenkins and deploying with Distelli.
DISTELLI_API_URL - If using the Distelli SaaS, this is https://api.distelli.com/$DISTELLI_USERNAME
DISTELLI_APP_NAME - The name of the application in Distelli. This was configured in the Create and Application in Distelli step.

Now that you have Jenkins builds being pushed to Pipelines, you may want to look at the following: