JavaFX is Eye Candy and I Have a Sweet Tooth. (Another Extreme UI Makeover)

In a previous article I showed a redesigned user interface for our Freight Management System (FMS) application using JavaFX as we begin to migrate away from Swing.  We are continuing to design screens for new functionality in FMS, and in this article I will illustrate a makeover I implemented for this new screen using JavaFX, which we had originally had planned to use Swing when the UI was first designed.

The screenshot below shows the original wireframe mock-up of what will be the new “VFC Recovery” screen within our Inbound Planning module of the FMS application.  This module will be used by our warehouse planning personel to display all “Trips” that will be arriving at their terminal.  A trip could be an incoming truck, ship, rail, or flight which has freight destined for the terminal.  On this screen, inbound planners in our warehouses will have the ability to see these incoming trips by arrival date and time.  This is the first layer of nested tables on this tab.  The next layer of tables below the arrival date, is reserved for each trip.  ie in the example below “TOTE*11027” is a ship which is scheduled to arrive on 07-18-12 08:00.  Expanding the trip table, will show the next level of tables, which are the plans for the freight once it arrives at the terminal.  Expanding this table then would show all the shipping containers (called VFCs) that the inbound planners have assigned to their plans, and finally, the VFC rows in this table can be expanded one last time to display all the shipments within the specified container.  As can be seen, the amount of data displayed to the user can get overwhelming pretty quickly.

Wireframe mock-up of VFC Recovery screen

The first step in a makeover of the VFC Recovery screen was to try to separate the data in a way that was intuitive for the users to navigate while also giving them the information they need to make their planning decisions.  Below is a picture of what was designed during a whiteboard session.  The inbound trips to the Lynden Transport terminal (Anchorage in this case), would be displayed in a table in the top portion of the screen.  The table would contain a summary of information related to the incoming trip, such as the mode of travel (ie truck, rail, barge, etc), as well as the origin of the trip, and anticipated date/time of the arrival.  The user could select a trip in this table, and the details of the trip would be displayed in a section below the table.  The details section would contain additional information about the trip, as well as contain an “Accordion” component which would have a separate section for each Plan that the inbound planners would be putting together.  The user could then expand one of these sections to view details on all of the containers that had been assigned to a particular plan.

Results of UI design whiteboard session

Once again I used the JavaFX design tool, Scene Builder to construct the new user interface.  In addition, I wanted to make use of the open source JavaFX components that are part of the JFXtras project, which are located here.  Components that are in the JFXtras project unfortunately can’t be added to the component palette in Scene Builder, but it was still easy enough to manually add them to the FXML code that Scene Builder generates, at which point Scene Builder will display them in the application’s UI as if they had been part of the component palette.  (Exactly how I accomplished this could be a future blog post.)

Below is a screenshot of the new UI for the VFC Recovery tab.  I took a bit of inspiration from what I had seen at Grand Central Station in New York, with my goal to make the trip arrivals portion of the UI look something like what you may see in an airport or train station. Two components in the JFXtras library were perfect for this task.  I used the DotMatrixSegment component to create the arrival board’s column headers as well as display the terminal where the trips were arriving at, in this case Anchorage.  I then used the Splitflap component to display the actual summary data about the trip, including arrival date/time, the trip ID, the origin and also the mode.  I found a nice brushed metal background, and used inner and drop shadows to give the board a sense of depth. In the screen shot below, the trip that the user had selected is highlighted by giving the row a green background.

The details section below the arrival board is formatted in a similar fashion to the table found in the previous article’s JavaFX makeover.  I used CSS to create the gradients, rounded corners, borders, background, etc for the section header, accordion component, and data table.

JavaFX Mock of VFC Recovery Screen

As mentioned in my previous article, this screen will still reside as a TopComponent within a NetBeans RCP application which will manage our other freight management modules that we will be designing with JavaFX as the UI toolkit.

twitter: @RobTerp

Extreme UI Makeover, JavaFX Edition

We have decided to move forward with JavaFX here at Lynden as we design the next version of our Freight Management System (FMS) application.  The initial version of the application was built on the Netbeans Rich Client Platform and will remain so, but we will be moving from the Swing and Jide (Commercial Swing components) component libraries to JavaFX.  The first application module that we are looking at tackling is called “Inbound Planning”.  This module will be used by warehouse managers at our various terminals to plan how and when to pick up and stage freight that is inbound to their location.

Below is a screenshot of the current FMS application with Planning module that is currently in use.  Warehouse managers run a query for their location and a Jide table displays all shipping containers that are inbound to their location.  Containers are grouped by destination, with each destination being represented by a row in the table.

 

Each destination row can then be expanded to reveal a Jide nested table showing all the shipping containers for a particular destination, with each container as a sub row to the destination row.  Finally, each container row can then be expanded to show all the shipments within that specific shipping container, for a total of 3 layers of tables.  In addition, when a user select a shipment, all the details about that shipment are displayed in a separate table in the lower pane of the UI.

 

In the next version of the FMS Planning module we will be adding functionality to the application to allow the users to create plans for the incoming containers.  The initial wire frame diagram below was created for the new planning module. The purpose of the new functionality is to allow the manager to see all incoming containers separated by mode of transit, i.e. ship, rail, truck, etc. and then plan for the container’s arrival accordingly by assigning it to a “Unit Plan”.

The top level table is represented by transit mode, which can then be expanded to reveal all the incoming trucks, ships, and trains to the user’s location. For example, the user could expand the “Steamship” row which would show a sub-table showing all scheduled incoming barges to that location.  The manager would like to ensure that all containers are picked up at the shipyard for an incoming voyage and are properly staged at the warehouse for final delivery to the customer. This could be accomplished by expanding the row for a particular voyage which would display sub rows to group the containers by Full-Load, Consolidation, and Transfer shipments.  The user would then expand the appropriate row to see all the containers that fall within one of those three categories.  In order to assign the container to a Unit Plan, the user would click the “Update” button on each row which would display a dialog asking the user which plan to assign the container to, at which point the container would disappear from this screen indicating that the container’s arrival had been planned.

 

The feedback that was received regarding this wireframe diagram and corresponding workflow is that this is an overwhelming amount of data to display to the user.  There are 5 levels of nested-tables within this framework and when all are fully expanded it is easy to get lost in the resulting sea of nested rows and columns. 

I was tasked with helping to redesign the UI to make the data a bit more manageable for the users as well as provide an efficient means for them to be able to select the containers they are interested in and assign them to the appropriate Unit Plan as well as give them visibility as to which containers still needed to be assigned to a plan.

The result of a whiteboard session appears below.  Transit modes would be displayed within a row of buttons at the top of the screen which would also show  summary data  regarding the number of shipments within each category.  The user could then select one of the categories and the data would be displayed in a table in the main portion of the UI. For example, if the user selected “Steamship”, the table below would show tabs on its left side, one for each inbound voyage.  When the user selects a voyage, the table would populate with all containers on that voyage that have not yet been assigned to a plan. The tabs would have a progress meter to indicate to the user what percentage of containers on the voyage have been assigned to a plan. An “Accordian” widget would be used to partition the data in the table by container type “Full Load”, “Consolidation” or “Transfer”.  Rather than making the user click a button on each row and display a dialog asking which plan the user wanted to assign the container to, the plans are instead shown in a table on the same screen.  The user could then select one or more container rows and then drag them directly over to the plan they wanted to assign those containers to, at which point the containers would disappear from the table, and the progress meter for the voyage would be updated to reflect the changes.

 

 

With the help of the JavaFX design tool, Scene Builder, I was able to design a mock UI of the new Inbound Planning screen. The screenshot below illustrates the “Steamship” mode selected, which changes the button’s color as well as creates a slight glow around the button.The Tote*55433 voyage has been selected and displays all the “Full Load” containers that have not yet been assigned to a plan, but could be dragged over to one of the plans on the right-side table.  The “Sea Land” unit plan is highlighted to illustrated what the table  could look like if the user had dragged any containers over to from the voyage table. (Green and gold are Lynden colors, that which is why the choice of this particular palate).

Scene Builder made it extremely easy to add effects such as gradients to the buttons, tabs and unit plan rows to give the components a glossy look.  Also, adding the reflections to the buttons along the top row was a very straightforward task as well as creating the translucent rows of the Unit Plan table.

As previously mentioned, the Inbound Planning module will still reside as a TopComponent within a NetBeans RCP application which will manage our other freight management modules that we will be designing with JavaFX as the UI toolkit as well.

twitter: @RobTerp

2012 JavaOne Summary

It was quite a busy week last week at JavaOne in San Francisco, but it was one of the best JavaOnes I have attended to date despite the disappointing accommodations at the Powell Hotel (No A/C and bed bugs)  There were so many interesting sessions lined up that it was hard to choose which ones to attend.  I participated on a NetBeans discussion panel as part of NetBeans community day on Sunday before the conference started, and also in a similar panel discussion in a JavaOne general session on Thursday. Listed below are some of the highlights of the sessions that I attended.

---

JavaFX

If there was one thing that I learned from the sessions that I attended related to JavaFX, its that it will be the UI framework of choice moving forward for desktop and rich Internet applications.  Probably more than half the sessions I attended were related to JavaFX in one way or another.  Adding rich content to applications such as gradients, translucency, reflections, blurring, animated transitions, etc. that were possible, but very painful to do in Swing are baked into the JavaFX toolkit.

JavaFX UIs can be constructed using a WYSIWYG designer called Scene Builder which is available as a free download from Oracle’s website.  The tool supports drag and drop of the various JavaFX widgets onto a JavaFX scene.  Scene Builder then generates a .fxml file rather than a Java class, which I feel is the biggest downside to this tool.  The FXML can still be used within a Swing component however as I detailed in a previous post here.  NetBeans IDE can also be integrated with Scene Builder so that Scene Builder will launch when a .fxml file is clicked within the NetBeans IDE.  Below is a screenshot from Scene Builder.

Probably the most interesting session I attended last week was titled “Building JavaFX Interfaces for the Real World”.  The presenter (Simon Ritter) went through an example of developing a JavaFX application using the Microsoft Kinect and the OpenNI (Natural Interaction) library which is a Java API for communicating with the Kinect.  Ritter demoed a black jack card game written with JavaFX for which he used gestures to interact with the game.

Ritter also demoed an interface with NeuroSky, which is a USB EEG sensor for monitoring brain activity.  While it was interesting to see his brain activity plotted with JavaFX chart components and gauge components from Steel Series (located at the JFXtras website), he admitted there wasn’t much practical application for such an interface yet, although it highlighted where things are heading with the future of user interface design.

Here at Lynden we have begun work with designing a new user interface for our Freight Management System (FMS) application using JavaFX on the NetBeans Rich Client Platform, which will give us the best of both worlds.  JavaFX for a rich set of UI components, effects and animations and the NetBeans platform which will provide the underlying framework/plumbing for the application including windowing system, menu, tool bar, property sheets, etc.

Anyone interesting in kicking the tires of JavaFX can tryout the Ensemble application which highlights the various UI components, effects and transitions that are possible with JavaFX.  There are also open source JavaFX components that can be found at the JFXtras website located here, for which there is also a demo “Ensemble” application as well, highlighting the library’s components.

---

NetBeans Rich Client Platform (RCP)

The NetBeans RCP made a strong showing at this year’s JavaOne and is gaining momentum as a solid platform to design rich client applications on.  In fact, two Netbeans RCP applications won a Duke’s Choice Award this year.

The first application from NATO is called “MICE console” which stands for “MASE Integrated Console Environment” and is used to support the execution of air operations in a real-time environment.

The next application called “AgroSense” is an open-source platform for and by the agricultural sector.

Of particular interest this year was integrating JavaFX components within a Netbeans RCP application which I discussed during my demos for the panel discussions that I participated in.   I will be blogging about this further in the near future as we develop our FMS application at Lynden.  I’m also working on an automated trading application built on the NetBeans RCP as well, which I would like to incorporate JavaFX chart components as they seem to be far more polished than the current charts that are available through JFreeChart.

---

Neuroph

I also attended an interesting presentation by Zoran Sevarac regarding a neural network design tool that is written on the NetBeans RCP called Neuroph.  The application allows one to quickly design neural networks for such usages such as facial recognition, music classification, predicting poker hands, etc.  I will be researching this further to see if neural networks can be used to determine how to best handle the loading/unloading of trailers at our warehouses in order to optimize the time spent moving freight from the inbound trucks to the delivery trucks.  I would also like to investigate this in regards to trading stocks/futures/currencies, maybe not as a sole predictor of price movement, but used as aid in an existing trading strategy to help improve performance

---

Javeleon

Allan Gregersen presented on Javeleon which allows you to apply code changes in a running application immediately, without losing any of the application state. The result is that development time is cut down since the application doesn’t need to be shut down and restarted for every change that you would want to see.  We have been using Javeleon for a little over a year now in the development of our NetBeans RCP application and we are able to deploy individual modules to our app running on our development desktop without having to stop this application, thus greatly reducing the amount of time spent waiting to start up the application.

During the demo, Gregersen showed a numer of modifications to a space invaders application that were deployed while the application was running, and also while preserving the application state.

Javeleon supports Swing applications as well as NetBeans RCP applications and has announced support for Java applications servers such as Tomcat, Jetty, and JBoss as well.

---

Hazelcast

Distributed data made easy is what I walked away with from the Hazelcast presentation.  Hazelcast is a lightweight distributed Map (and more) that has a fairly small footprint, a single 1.7MB jar file with no dependencies.  There are also a number of other interesting distributed constructs in addition to maps such as, Queues, Sets, Lists, Locks, Semaphores, Topics, and Executor Services.  We will be investigating distributed caches at Lynden at some point for storing data for quick retrieval by our FMS client application.

---

Raspberry Pi

Finally, I attended a very cool demo on the Raspberry Pi which is a small ARM Linux device designed for the teaching of basic computer science in schools.  The board is the size of a credit card and includes a 700MHz ARM processor and retails for $25.  There a number of articles on getting Java SE Embedded set up and running on the device, which can be found here.

twitter: @RobTerp
twitter: @LimitUpTrading

Add a JavaFX Component Built with Scene Builder and FXML to a NetBeans RCP App

Most of the tutorials and demos I have seen up to this point for integrating JavaFX with NetBeans Rich Client Platform (RCP) applications have created and configured JavaFX components by using the JavaFX Java APIs that are now shipped as part of the JDK as opposed to using the FXML markup language.  This seems logical since if one is already coding a panel or container in Java, it makes sense to instantiate and configure JavaFX components for that container in Java as well.  The disadvantage to this is that you are not able to leverage the benefits of Scene Builder to design your JavaFX components.  Scene Builder is a WYSIWYG design tool for creating components and UIs in JavaFX.  Behind the scenes (pun intended) the application generates an FXML representation of the UI, and does *not* generate Java code.

In the following example I will create a JavaFX component with Scene Builder and add it to a NetBeans RCP TopComponent window alongside a plain vanilla Swing JLabel.

---

Creating the Application

The first step in the process is to create an ordinary NetBeans  RCP application.  We use maven, so I create a new NetBeans application from the Maven Category

In NetBeans IDE there are now 3 projects.

---

Creating a JavaFX Runtime Wrapper Module

The next step is not create a wrapper module for the JavaFX runtime library.  If you are running Java 1.7.0_06 or higher (and you should be if you are developing a JavaFX app), you don’t need to include any of the native libraries in the wrapper module, only the jfxrt.jar itself needs to be included.

I created a new module called jfxrt (JavaFX runtime), and include the jfxrt-2.2.jar in it, which you can find in the JRE’s lib directory.  Below is a screenshot of the jfxrt wrapper module in the IDE.

You will then need to update the pom.xml file for the wrapper to include any JavaFX packages that you want to access within your application.  Below is a code snippet showing the JavaFX packages that my app will need in order to run.

<plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>nbm-maven-plugin</artifactId>
   <version>3.7</version>
   <extensions>true</extensions>
   <configuration>
       <useOSGiDependencies>true</useOSGiDependencies>
       <publicPackages>
          <publicPackage>javafx.scene.layout</publicPackage>
          <publicPackage>javafx.application</publicPackage>
          <publicPackage>javafx.collections</publicPackage>
          <publicPackage>javafx.scene.control</publicPackage>
          <publicPackage>javafx.event</publicPackage>
          <publicPackage>javafx.geometry</publicPackage>
          <publicPackage>javafx.beans.property</publicPackage>
          <publicPackage>javafx.beans.value</publicPackage>
          <publicPackage>javafx.concurrent</publicPackage>
          <publicPackage>javafx.scene.web</publicPackage>
          <publicPackage>javafx.fxml</publicPackage>
       </publicPackages>
   </configuration>
</plugin>

---

Now the fun begins

I create a new NetBeans module called “NumberLabelModule”, which contains a TopComponent to host both the Swing JLabel and JavaFX label.  Unfortunately, it doesn’t appear that the NetBeans IDE WYSIWYG designer, Matisse supports drag-and-drop of JFXPanel, which is the Swing panel that can contain JavaFX components, so the JavaFX portion must be added to the TopComponent by hand. (I attempted to add the JFXPanel to the IDE’s component palette to make use of it in Matisse, but when I attempted to drag the JFXPanel onto the TopComponent the IDE froze and had to be killed with extreme prejudice).

---

Building the JavaFX Label with Scene Builder

For this project the next step is to place a style sheet in the “Other Sources” portion of the project, which the JavaFX component will use for its formatting.

JavaFX Scene Builder is then used to graphically design the new JavaFX label and generate the  FXML markup code, which will be saved right along side the style sheet in the “Other Sources” portion of the project. (NumberLabel.fxml)

---

Adding JavaFX to the TopComponent

Once the component is designed and saved, the .fxml file within the IDE is updated automatically.  The final step is to add the JavaFX component to the TopComponent along with a JLabel.  The TopComponent contains a JPanel that uses a FlowLayout with the JLabel added to it as the first component. (See below).

The JavaFX label will be the 2nd component added to the panel, and this is done in the constructor of the TopComponent as follows.

public final class SwingFXDemoTopComponent extends TopComponent {

    JFXPanel jfxPanel = new JFXPanel();

    public SwingFXDemoTopComponent() {
        initComponents();
        putClientProperty(TopComponent.PROP_DRAGGING_DISABLED, Boolean.TRUE);
        putClientProperty(TopComponent.PROP_UNDOCKING_DISABLED, Boolean.TRUE);
        setName(Bundle.CTL_SwingFXDemoTopComponent());
        setToolTipText(Bundle.HINT_SwingFXDemoTopComponent());

        Platform.setImplicitExit(false);
        // create JavaFX scene
        Platform.runLater(new Runnable() {
            public void run() {
                Parent root;
                try {
                    root = FXMLLoader.load(getClass().getResource("/com/lynden/fxlabel/NumberLabel.fxml"));
                    Scene scene = new Scene(root);
                    jfxPanel.setScene(scene);
                } catch (IOException ex) {
                    Exceptions.printStackTrace(ex);
                }
            }
        });
        jPanel1.add(jfxPanel);
    }

A few more details on the above code:

 Platform.setImplicitExit(false);

This will prevent the JavaFX runtime from exiting when TopComponents are moved around the NetBeans RCP windowing system or when a TopComponent is undocked from the main application frame.  If this is not set, what you will see is that your JavaFX components will disappear when you undock a TopComponent from the main window.

      Platform.runLater(new Runnable() {
            public void run() {
                Parent root;
                try {
                    root = FXMLLoader.load(getClass().getResource("/com/lynden/fxlabel/NumberLabel.fxml"));
                    Scene scene = new Scene(root);
                    jfxPanel.setScene(scene);
                } catch (IOException ex) {
                    Exceptions.printStackTrace(ex);
                }
            }
        });
        jPanel1.add(jfxPanel);

 

The code snippet above will create the JavaFX label within the JavaFX event thread (separate from the Swing event thread).  The first step of the process is to load the FXML representation of the JavaFX label that was built in Scene Builder.  Once the root node is loaded it can then be added to a new Scene.  The Scene can then be added to a JFXPanel, which as mentioned earlier, is a Swing component that can contain JavaFX components.   The final step is to then add the JFXPanel to the JPanel that the JLabel is living on.

---

Final Result

Once this is set up, the application can be run, and displays Swing and JavaFX components (built with FXML) living side by side in perfect harmony.

twitter: @RobTerp

Setting up a Nightly Build Process with Jenkins, SVN and Nexus

We wanted to set up a nightly integration build with our projects so that we could run unit and integration tests on the latest version of our applications and their underlying libraries.  We have a number of libraries that are shared across multiple projects and we wanted this build to run every night and use the latest versions of those libraries even if our applications had a specific release version defined in their Maven pom file.  In this way we would be alerted early if someone added a change to one of the dependency libraries that could potentially break an application when the developer upgraded the dependent library in a future version of the application.

The chart below illustrates our dependencies between our libraries and our applications.

Updating Versions Nightly

Both the Crossdock-shared and Messaging-shared libraries depend on the Siesta Framework library.  The Crossdock Web Service and CrossdockMessaging applications both depend on the Crossdock-Shared and Messaging-Shared libraries.  Because of the dependency structure we wanted the SiestaFramework library built first.  The Crossdock-Shared and Messaging-Shared libraries could be built in parallel, but we didn’t want the builds for the Crossdock Web Service and CrossdockMessaging applications to begin until all the libraries had finished building.  We also wanted the nightly build to tag subversion with the build date as well as upload the artifact up to our Nexus “Nightly Build” repository.  The resulting artifact would look something like SiestaFramework-20120720.jar

Also as I had mentioned, even though the CrossdockMessaging app may specify in its pom file it depends on version 5.0.4 of the SiestaFramework library.  For the purposes of the nightly build, we wanted it to use the freshly built SiestaFramework-NIGHTLY-20120720.jar version of the library.

The first problem to tackle was getting the current date into the project’s version number.  For this I started with the Jenkins Zentimestamp plugin.  With this plugin the format of Jenkin’s BUILD_ID timestamp can be changed.  I used this to specify using the format of yyyyMMdd for the timestamp.

The next step was to get the timestamp into the version number of the project.  I was able to accomplish this by using the Maven Versions plugin.  One of the things that the Versions plugin can do is to allow you to override the version number in the pom file dynamically at build time.  The code snippet from the SiestaFramework’s pom file is below.

<plugin>
   <groupId>org.codehaus.mojo</groupId>
   <artifactId>versions-maven-plugin</artifactId>
   <version>1.3.1</version>
</plugin>

At this point the Jenkin’s job can be configured to invoke the “versions;set” goal, passing in the new version string to use.  The ${BUILD_ID} Jenkins variable will have the newly formatted date string.

This will produce an artifact with the name SiestaFramework-NIGHTLY-20120720.jar

---

Uploading Artifacts to a Nightly Repository

Since this job needed to upload the artifact to a different repository from our Release repository that is defined in our project pom files the “altDeploymentRepository” property was used to pass in the location of the nightly repository.

The deployment portion of the SiestaFramework job specifies the location of the nightly repository where ${LYNDEN_NIGHTLY_REPO} is a Jenkin’s variable which contains the nightly repo URL.

---

Tagging Subversion

Finally, the Jenkins Subversion Tagging Plugin was used to tag SVN if the project was successfully built.  The plugin provides a Post-build Action for the job with the configuration section shown below.

---

Dynamically Updating Dependencies.

So now that the main project is set up, the dependent projects are set up in a similar way, but need to be configured to use the SiestaFramework-NIGHTLY-20120720 of the dependency rather than whatever version they currently have specified in their pom file.  This can be accomplished by changing the pom to use a property for the version number of the dependency.  For example, if the snippet below was the original pom file.

<dependencies>
   <dependency>
      <groupId>com.lynden</groupId>
      <artifactId>SiestaFramework</artifactId>
      <version>5.0.1</version>
   </dependency>
</dependencies>

It could be changed to the following to allow the SiestaFramework version to be set dynamically.

<properties>
   <siesta.version>5.0.1</siesta.version>
</properties>

<dependencies>
   <dependency>
      <groupId>com.lynden</groupId>
      <artifactId>SiestaFramework</artifactId>
      <version>${siesta.version}</version>
   </dependency>
</dependencies>

This version can then be overriden by the Jenkins job. The example below shows the Jenkins configuration for the Crossdock-shared build.

---

Enforcing Build Order

The final step in this process is setting up a structure to enforce the build order of the projects.  The dependencies are set up in such a way that SiestaFramework needs to be built first, and the Crossdock-shared and Messaging-shared libraries can be run concurrently once SiestaFramework finishes. The Crossdock Web Service and CrossdockMessaging application jobs can be run concurrently, but not until after both shared libraries have finished.

Setting up the Crossdock-shared and Messaging-shared jobs to be built after SiestaFramework completes is pretty straightforward.  In the Jenkins job configuration for both the shared libraries, the following build trigger is added.

For the requirement of not having the apps build until all libraries have built, I enlisted the help of the Join Plugin.  The Join Plugin can be used to execute a job once all “downstream” jobs have completed.  What does this mean exactly?  Looking at the diagram below, the Crossdock-Shared and the Messaging-Shared jobs are “downstream” from the SiestaFramework job.  Once both of these jobs complete, a Join trigger can be used to start other jobs.

In this case rather than having the Join trigger kick off other app jobs directly, I create a dummy Join job.  In this way, as we add more application builds, we don’t need to keep modifying the SiestaFramework job with the new application job we just added.

To illustrate the configuration, SiestaFramework has a new Post-build Action (below)

Join-Build is a Jenkins job I configured that does not do anything when executed.  Then our Crossdock Web Service and CrossdockMessaging applications define their builds to trigger as soon as Join-Build has completed.

In this way we are able to run builds each night which will update to the latest version of our dependencies as well as tag SVN and archive the binaries to Nexus.  I’d love to hear feedback from anyone who is handling nightly builds via Jenkins, and how they have handled the configuration and build issues.

twitter: @RobTerp

Designing an Automated Trading Application on the Netbeans Rich Client Platform (Part 1)

Over the past 10 years new opportunities have opened in the stock, futures and currency markets to allow retail traders the ability to produce their own automated trading strategies which was once only the realm of hedge funds and investment banks.  Interactive Brokers was one of the first brokerage firms to offer a Java API to its retail customers. Originally envisioned as way for developers to augment Interactive Brokers Trader Workstation (TWS) desktop application with features such as charting or record keeping, the API has gained popularity as a way to automate trading strategies.

In my first iteration of developing a trading strategy and software to automate the trades I built a Java desktop application using Swing components which would monitor stocks throughout the day and place trades when certain parameters were met, and then exit the trades at the close of the trading day. The software worked well, and it was adequate for the strategy it was designed to trade, however it was not extensible and attempting to implement new trading strategies to automate as well as connect to different brokerage accounts proved difficult and cumbersome.  Also, there are restrictions on how many stocks could be monitored via the broker’s data feed so the software had to be able to accommodate real-time market data feeds from other sources in addition to the broker’s data feed. 

I was introduced to the Netbeans Rich Client Platform (RCP) a couple of years ago and have recently decided to begin porting my application to the platform due to a large number of advantages that it provides.  The Netbeans RCP is built on a modular design principle allowing the developer to define abstract APIs for features and then provide modules which may have different implementations of the API, allowing the application to select at runtime which implementation to use.  Not only does it provide for a cleaner design by separating concerns, but by using the Netbeans Lookup API it also decouples the application and its various components from each other.  There are numerous other features that can be leveraged including a built-in windowing system, text editor, file explorer, toolbar, table and tree table components as well the Action API (just to name a few).

The trading application will make use of the RCP module system to define abstract APIs with the following functionality:

Broker API

• Place and cancel orders for stocks, options, futures, or currencies
• Provide event notification when orders are filled
• Monitor cash balances in the account

Market Data API

• Subscribe to real-time quote data for any ticker symbol
• Subscribe to Level 2 data (market depth/order-book) for any ticker symbol

Historical Data API

• Request historical price data for any ticker symbol

Trading Strategy API

• Define a set of rules for entering and exiting trades
• Ability to use any broker, market data, and historical data API implementations in order to make trading decisions.

The primary implementation for the Broker, Market Data and Historical data API modules will be utilizing Interactive Broker’s Java API,  but other implementations can also be created  as Netbeans modules and then imported into the trading application so that trading strategies can make use of market data from different sources if needed.

New trading strategies can be built as Netbeans modules  implementing the Trading Strategy API, where each strategy can make use of one of the implementations of the various data and broker APIs.  Utilizing the Netbeans Lookup API, strategies can query the platform to get a list of all implementations of the broker and market data APIs providing for loose coupling between the APIs and allowing the user to select which implementation to use at runtime.
Below is a diagram illustrating the organization of the various API components of the application:

In future posts I will go into more detail on how to create an API plug-in for the Netbeans RCP as well as show how to create a concrete implementation of the API.  In the illustration above the abstract broker, market data, and trading strategy APIs are installed into the RCP as plug-ins.  The broker API has a single implementation for Interactive Brokers at this point in time.  The market data API has plug-ins which provide implementations for real-time market data from Yahoo Finance as well as Interactive Brokers real-time market data.  Finally, the trading strategy API has 2 implementations in this example.  The first strategy named “Limit Buyer” will watch the prices of approx 800 stocks and place limit order to buy when certain conditions are met.  The second strategy in the example above, named AUD/NZD Currency Strategy will monitor the exchange rates of the Australian and New Zealand dollars and place orders to buy or sell when certain conditions are met.  

At this point in time the application is functional and is utilizing Interactive Brokers as the main brokerage as well as market data provider.  The AUD/NZD trading strategy is actively being traded through the application, albeit with a rudimentary user interface which is publishing messages to a text area within the strategy’s main tab.  The screenshot below illustrates Interactive Brokers “Trader Workstation” application, the large black application (which is a Java Swing app), as well as the Netbeans RCP automated trading application which is the small white application, with the large text area.  In the screenshot below the application is currently monitoring prices and placing trades for the Australian Dollar, New Zealand dollar, Hong Kong dollar and Japanese yen currencies.

 

This post is just a high level overview on the design of an RCP application to trade in the financial markets.  Future parts to this series will include more information on how to implement abstract APIs and make them available for other portions of the application to use via the Netbeans Lookup API as well as working with some of the Netbeans UI components included with the platform such as tabs, trees and tables, showing how easy it is to render the same data via these different views using the Netbeans Nodes API.  In addition to this I would like to incorporate some JavaFX components into the application such as the charting components that can be found in the core JavaFX library which will provide a graphical representation of some of the data the strategies are monitoring which will be a bit more user friendly than the current large text area.  The integration of JavaFX components within the application will be documented in a future post as well.

You can follow my trading related blog if you would like to see the actual trading results of the application as its being refined at:
http://LimitUpTrading.wordpress.com

twitter: @RobTerp
twitter: @LimitUpTrading

Creating a Deployment Pipeline with Jenkins, Nexus, Ant and Glassfish

In a previous post I discussed how we created a build pipeline using Jenkins to create application binaries and move them into our Nexus repository. (Blog post here)  In this post I will show how we are using Jenkins to pull a versioned binary out of Nexus and deploy to one of our remote test, staging or production Glassfish servers.  By remote I mean that the Glassfish instance does not live on the same box as the Jenkins CI instance, but both machines are on the same network.

In a previous post I also discussed how to set up Glassfish v3 to allow deployments pushed from remote servers (Blog post here), so if you haven’t explicitly configured your Glassfish to allow this feature, you will need to do so before you get started.

On our Jenkins CI box we have an Ant script, which will be executed by a Jenkins job manually kicked off by a user. The script defines tasks to perform the following operations:

• Ensure all needed parameters were entered by the user. (app name, version number, admin username/password, etc).
• Copy the specified version of the application from Nexus to a local temp directory for deployment to a remote Glassfish instance.
• Undeploy a previous version of the app from the target Glassfish instance.  (Optional).
• Deploy the app from the temp directory to the target Glassfish instance.
• Record the deployment info in a deployment tracking database table.  Historical deployment info can then be viewed from a web app.

---

Ant Script

Below are some of the more interesting code snippets from our Ant script that will be doing the heavy lifting for our deployment pipeline.  The first code snippet below defines the Ant tasks needed for deploying and undeploying applications from Glassfish.  These Ant tasks are bundled with Glassfish, but not installed by default.  If you haven’t installed them, you will need to do so from your Glassfish update center admin page.

<taskdef name="sun-appserv-deploy" classname="org.glassfish.ant.tasks.DeployTask">
   <classpath>
      <pathelement location="/nas/apps/cisunwk/glassfish311/glassfish/lib/ant/ant-tasks.jar"/>
   </classpath>
</taskdef>

<taskdef name="sun-appserv-undeploy" classname="org.glassfish.ant.tasks.UndeployTask">
   <classpath>
      <pathelement location="/nas/apps/cisunwk/glassfish311/glassfish/lib/ant/ant-tasks.jar"/>
   </classpath>
</taskdef>

Once we have the tasks defined, we create a new target for pulling the binary from Nexus and copying it to a temporary location from where it will be deployed to Glassfish.

<target name="copy.from.nexus">
   <echo message="copying from nexus"/>    
   <get src="http://cisunwk:8081/nexus/content/repositories/Lynden-Java-Release/com/lynden/${app.name}/${version.number}/${app.name}-${version.number}.${package.type}" dest="/tmp/${app.name}-${version.number}.war"/>
</target>

Next is a target to undeploy a previous version of the application from Glassfish.  This step is optional and only executed if the user specifies a version to undeploy from Jenkins.

<target name="undeploy.from.glassfish" if="env.Undeploy_Version">
   <echo message="Undeploying app: ${app.name}-${undeploy.version}"/>
   <echo file="/tmp/gf.txt" message="AS_ADMIN_PASSWORD=${env.Admin_Password}"/>
   <sun-appserv-undeploy name="${app.name}-${undeploy.version}" host="${server.name}" port="${admin.port}" user="${env.Admin_Username}" passwordfile="/tmp/gf.txt" installDir="/nas/apps/cisunwk/glassfish311"/>
   <delete file="/tmp/gf.txt"/>
</target>

Next, we then define a target to do the deployment of the application to Glassfish.

<target name="deploy.to.glassfish.with.context" if="context.is.set">
    <sun-appserv-deploy file="/tmp/${app.name}-${version.number}.war" name="${app.name}-${version.number}" force="true" host="${server.name}" port="${admin.port}" user="${env.Admin_Username}" passwordfile="/tmp/gf.txt" installDir="/nas/apps/cisunwk/glassfish311" contextroot="${App_Context}"/>
</target>

And then finally, we define a target, which will invoke a server and pass information to it, such as app name, version, who deployed the app, etc.so that it can be recorded in our deployment database.

<target name="tag.uv.deploy.file">
   <tstamp>
      <format property="time" pattern="yyyyMMdd-HHmmss"/>
   </tstamp> 

   <!--Ampersand character for the URL -->
   <property name="A" value="&amp;"/>
   <get src="http://shuyak.lynden.com:8080/DeploymentRecorder/DeploymentRecorderServlet?app=${app.name}${A}date=${time}${A}environment=${deploy.env}${A}serverName=${server.name}${A}serverPort=${server.port}${A}adminPort=${admin.port}${A}serverType=${server.type}${A}version=${version.number}${A}who=${deploy.user}${A}contextName=${App_Context}" dest="/dev/null"/>

   <echo message="tagging deployment info to UV"/>
</target>

---

Jenkins Configuration

Now that we have an Ant script to perform the actions that we need to do a deployment, we set up a Jenkins job to deploy to servers in each one of our environments (test/staging/prod).

---

In order to kick off a deployment to one of our servers, the appropriate environment is selected from the screenshot above, and the “Build Now” link is clicked which presents the user with the screen below. In this case we are deploying to a test Glassfish domain named “bjorn” on unga.lynden.com

The user can select from the drop down list the server they wish to deploy to and the application they wish to deploy.  The version number is a required entry in a text field.  If the script can’t find the specified version in Nexus, the build will fail.  There are also optional parameters for specifying an existing version to undeploy as well as an application context in the event the app name shouldn’t be used as the default context.

---

In the screenshot below we are deploying version 5.0.0 of the Crossdock App to the “bjorn” domain running on unga.lynden.com

---

Once the job completes, if we log into the bjorn Glassfish admin page on unga.lynden.com, we see that Crossdock-5.0.0 has been deployed to the server.

---

The screenshot below is an example of undeploying version 5.0.0 of Crossdock, and deploying version 5.0.1 of Crossdock.  Also, in this example, we are telling the script that we want the web context in Glassfish to be /Crossdock, rather than the default /Crossdock-5.0.1

---

The screenshot of the Glassfish admin page below shows that Crossdock-5.0.0 has been unistalled, and that Crossdock-5.0.1 is now installed with a Context Root of /Crossdock.

---

Deployment History

Finally, as I mentioned previously, the Ant script is also saving the deployment information to a historical deployment table in our database.  We have written a simple web application which will display this historical data.  The screenshot below shows all of the applications that have been deployed to our test environment. (We have similar pages for Staging and Production as well).  Included in this information is the application name, version number, date, and who deployed it, among some other miscellaneous info.

We can then drill into the history of a specific application by clicking the “Crossdock” link on the screen above and get a detailed history about the deployments for that application including version numbers or dates.  We maintain more than 60 different web applications serving various purposes here at Lynden, so this has been a great tool us to see exactly what versions of our applications are currently deployed where, as well as see the history of the deployment of a specific application in the event we need to roll back to a previous version.

As we have learned firsthand, Jenkins is a very useful and versatile tool that is easy to extend for purposes beyond automated builds and continuous integration.

twitter: @RobTerp