Streamlining a Deployment Pipeline with a Custom Jenkins Plug-In

In a previous post (located here) I discussed setting up a deployment pipeline with Jenkins which would retrieve a specified version of one of our web applications from our Nexus repository and deploy it to a specified Glassfish server.  One of the challenges with the Jenkins job was that most of the fields on its deployment page were text fields, allowing users to free form text which was prone to errors.  The original deployment page appears below and allows the user to select the server and application via dropdown box, but leaves all other properties open to free form text.

image1.png (945×640)

The other drawback to the screen above was that each application needed to be listed in a text file which Jenkins would read  in order to populate the Application dropdown box.  This meant that there was a manual step involved with deploying an application that had not been deployed before.

With this in mind I decided to modify the Jenkins Dynamic Parameter Plugin to allow us to auto populate dropdown boxes with valid values based on our environment here at Lynden. I started with this plug-in since it already had functionality to change the contents of dropdown boxes based upon user selections. ie The selection of one dropdown could be used to determine the contents of another dropdown box.  The requirements for this modified plug-in were that it should:

  • Get a list of applications that are in our Nexus release repository
  • Get a list of versions of the selected application from Nexus.
  • Get a list of the application versions that are currently deployed to the Glassfish instance the user has specified for deployment.
  • Enable the user to select a web context to use when deploying the app, or select “Other” if they want to enter a context that does not appear in the list.

The Server dropdown box for this new Deployment plug-in allows the user to select which instance of Glassfish they would like to deploy to.  Each line in the dropdown contains the server name, Glassfish instance name, HTTP port, and admin port of the instance.

JenkinsDeployServerSelect

Once the user has selected a server, they can then select an application from the Application dropdown box.  For this field, the Jenkins plug-in goes out to our Nexus repository and retrieves the names of all of our applications, populating the list with these values.

JenkinsDeployAppSelect

Once the application has been selected, the user can select which version of the application to deploy.  This list of versions is also obtained from Nexus by the plug-in, which displays up to 25 of the most recent versions.

JenkinsDeployAppVersionSelect

Once the user has selected the version to deploy, they can optionally specify a version to undeploy.  The plug-in populates this field by looking at the Glassfish instance that the user selected and querying it to determine if any versions of the specified application are currently deployed to it.  In the example below, HelloWorldWeb 1.1.5 and 1.1.4 are both currently deployed to the Glassfish instance running on “webstg”.  This field is optional, and if a value is selected, the Deploy task will first undeploy the specified version before deploying the new version.

JenkinsDeployAppUndeployVersionSelect

Finally, the user can select a context to use when deploying the application, which is the portion of the URL after the hostname that the app server uses to identify requests for the application.  The two most common context names are included in the drop down, which are <app-name> and <app-name>-<version-number>.  So for example, if HelloWorldWeb-1.1.3 were selected below, the application would have a URL of http://webstg.lynden.com/HelloWorldWeb-1.1.3

JenkinsDeployAppContextSelect

If a different application context is needed, the user can select “Other” from the dropdown box (in the screenshot above), and then enter an app context name in the “Other” text box below the Context drop down box.

JenkinsDeployAppContextOther

At this point the user can select the “Build” button which will invoke an Ant script detailed in the previous post, downloading the binary from Nexus, undeploying the previously deployed version from Glassfish (if specified), deploying the new version of the application to Glassfish, and then saving information about the deployment to a database which can be viewed from a separate web application.


Configuration

Since nearly all the fields are dynamically populated, there isn’t too much work for configuring the job in Jenkins.  Below is the Configuration screen for this deployment job.  There are 2 parameters which must be defined for this job.  The first is the environment this job will deploy to, ie. Test, Staging or Prod. The second is a list of valid Glassfish instances which can be deployed to.  The location of the maven repository and the location of the Glassfish command line interface are defined as system properties in Jenkins.

JenkinsDeployAppConfig

By using a Jenkins plugin to prepopulate the fields which are required for deploying an application we will streamline the deployment process and make it less prone to error.

twitter: @RobTerp

Advertisements

Setting up a Nightly Build Process with Jenkins, SVN and Nexus

We wanted to set up a nightly integration build with our projects so that we could run unit and integration tests on the latest version of our applications and their underlying libraries.  We have a number of libraries that are shared across multiple projects and we wanted this build to run every night and use the latest versions of those libraries even if our applications had a specific release version defined in their Maven pom file.  In this way we would be alerted early if someone added a change to one of the dependency libraries that could potentially break an application when the developer upgraded the dependent library in a future version of the application.

The chart below illustrates our dependencies between our libraries and our applications.

image

Updating Versions Nightly

Both the Crossdock-shared and Messaging-shared libraries depend on the Siesta Framework library.  The Crossdock Web Service and CrossdockMessaging applications both depend on the Crossdock-Shared and Messaging-Shared libraries.  Because of the dependency structure we wanted the SiestaFramework library built first.  The Crossdock-Shared and Messaging-Shared libraries could be built in parallel, but we didn’t want the builds for the Crossdock Web Service and CrossdockMessaging applications to begin until all the libraries had finished building.  We also wanted the nightly build to tag subversion with the build date as well as upload the artifact up to our Nexus “Nightly Build” repository.  The resulting artifact would look something like SiestaFramework-20120720.jar

Also as I had mentioned, even though the CrossdockMessaging app may specify in its pom file it depends on version 5.0.4 of the SiestaFramework library.  For the purposes of the nightly build, we wanted it to use the freshly built SiestaFramework-NIGHTLY-20120720.jar version of the library.

The first problem to tackle was getting the current date into the project’s version number.  For this I started with the Jenkins Zentimestamp plugin.  With this plugin the format of Jenkin’s BUILD_ID timestamp can be changed.  I used this to specify using the format of yyyyMMdd for the timestamp.

image

The next step was to get the timestamp into the version number of the project.  I was able to accomplish this by using the Maven Versions plugin.  One of the things that the Versions plugin can do is to allow you to override the version number in the pom file dynamically at build time.  The code snippet from the SiestaFramework’s pom file is below.

<plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>versions-maven-plugin</artifactId>
   <version>1.3.1</version>
</plugin>

At this point the Jenkin’s job can be configured to invoke the “versions;set” goal, passing in the new version string to use.  The ${BUILD_ID} Jenkins variable will have the newly formatted date string.

image

This will produce an artifact with the name SiestaFramework-NIGHTLY-20120720.jar


Uploading Artifacts to a Nightly Repository

Since this job needed to upload the artifact to a different repository from our Release repository that is defined in our project pom files the “altDeploymentRepository” property was used to pass in the location of the nightly repository.

image

The deployment portion of the SiestaFramework job specifies the location of the nightly repository where ${LYNDEN_NIGHTLY_REPO} is a Jenkin’s variable which contains the nightly repo URL.


Tagging Subversion

Finally, the Jenkins Subversion Tagging Plugin was used to tag SVN if the project was successfully built.  The plugin provides a Post-build Action for the job with the configuration section shown below.

image


Dynamically Updating Dependencies.

So now that the main project is set up, the dependent projects are set up in a similar way, but need to be configured to use the SiestaFramework-NIGHTLY-20120720 of the dependency rather than whatever version they currently have specified in their pom file.  This can be accomplished by changing the pom to use a property for the version number of the dependency.  For example, if the snippet below was the original pom file.

<dependencies>
   <dependency>
      <groupId>com.lynden</groupId>
      <artifactId>SiestaFramework</artifactId>
      <version>5.0.1</version>
   </dependency>
</dependencies>

It could be changed to the following to allow the SiestaFramework version to be set dynamically.

<properties>
   <siesta.version>5.0.1</siesta.version>
</properties>

<dependencies>
   <dependency>
      <groupId>com.lynden</groupId>
      <artifactId>SiestaFramework</artifactId>
      <version>${siesta.version}</version>
   </dependency>
</dependencies>

This version can then be overriden by the Jenkins job. The example below shows the Jenkins configuration for the Crossdock-shared build.

image


Enforcing Build Order

The final step in this process is setting up a structure to enforce the build order of the projects.  The dependencies are set up in such a way that SiestaFramework needs to be built first, and the Crossdock-shared and Messaging-shared libraries can be run concurrently once SiestaFramework finishes. The Crossdock Web Service and CrossdockMessaging application jobs can be run concurrently, but not until after both shared libraries have finished.

Setting up the Crossdock-shared and Messaging-shared jobs to be built after SiestaFramework completes is pretty straightforward.  In the Jenkins job configuration for both the shared libraries, the following build trigger is added.

image

For the requirement of not having the apps build until all libraries have built, I enlisted the help of the Join Plugin.  The Join Plugin can be used to execute a job once all “downstream” jobs have completed.  What does this mean exactly?  Looking at the diagram below, the Crossdock-Shared and the Messaging-Shared jobs are “downstream” from the SiestaFramework job.  Once both of these jobs complete, a Join trigger can be used to start other jobs.

image

In this case rather than having the Join trigger kick off other app jobs directly, I create a dummy Join job.  In this way, as we add more application builds, we don’t need to keep modifying the SiestaFramework job with the new application job we just added.

To illustrate the configuration, SiestaFramework has a new Post-build Action (below)

image

Join-Build is a Jenkins job I configured that does not do anything when executed.  Then our Crossdock Web Service and CrossdockMessaging applications define their builds to trigger as soon as Join-Build has completed.

image

In this way we are able to run builds each night which will update to the latest version of our dependencies as well as tag SVN and archive the binaries to Nexus.  I’d love to hear feedback from anyone who is handling nightly builds via Jenkins, and how they have handled the configuration and build issues.

twitter: @RobTerp