Robotium, Jenkins and Ant

by

I think it is time for a new How to/ Tutorial on my blog. Today, I want to give you an overview of how to setup your Robotium test project to build it with the tool Ant. If the project is configured for Ant, I will explain how to integrate it into the continuous integration server Jenkins. With the combination of the build tool Ant and the CI Server Jenkins, you are able to build up your own mobile test automation environment. Written Robotium tests can be executed on schedule or directly after a developer commits new code to the repository.

Three preconditions are needed for this post. First, you need an Robotium test project and an apk file or even better the source of the app you want to test. Second, you need an installed Android SDK on your local machine. How to setup these things can be found here. And last but not least you need an Ant installation on your machine (version >1.8).

Configure your projects

To setup your projects (main and test project) with Ant, you need to create a build file for both projects.

First step is to navigate to your Android main project and execute the following command (path based on operating system or should be part of the system path).

/Applications/android/tools/./android update main-project-name -p .

After executing this command a build.xml file must be created in the project directory. Check with the command ant clean debug if the build.xml is working and if the project was builded successfully.
In the next step you must create a build file also for the test project. To create this file execute the following command in the test project directory. Important: Provide the path to the main project to have a connection to both projects.

/Applications/android/tools/./android update test-project-name -p . -m PATH_TO_MAIN_PROJECT

Besides the build.xml file, the mentioned command will also create an ant.properties and local.properties file where the local settings are stored. Both files need to be part of the test repository.


To check if the created build script is working, start an Android emulator or connect a device to your computer and execute the following command in your test project directory:

ant debug install test

This command will build your Robotium test apk with Ant, install the apk file to the emulator or device and will start the tests. When all these steps passed, your projects are ready to integrate into jenkins.

Install and Setup Jenkins

First step in this section is: Download Jenkins from here and install (install instructions) it on separate server where all your team members have access to. If the installation was successful, Jenkins is available on http://localhost:8080 on the server itself.
Dashboard [Jenkins]

First go to the Manage Jenkins section on the left side to start the configuration of the CI server. Then open the Configure System section. First you have to add the Android SDK path to Jenkins. This is important for starting the emulator and using adb for starting the tests. Add the following section:

Really important is also to set a new Jenkins URL, to have access over the network. Default URL of the server is http://localhost:8080.

Last but not least it is recommended to add an E-Mail Notification to the server. Every time Jenkins tracks an error, an E-Mail will be send to the team.

At the bottom of the settings page click save button.

Plugins

In the next steps you need to install some plugins to Jenkins to get everything up and running. Open the section Manage Jenkins -> Manage Plugins and open the tab ‘Available’. Mandatory is the Android Emulator Plugin, this plugin is needed to create a pre-build step in your Jenkins job to start the Android emulator. Depending on your code versioning you need to install the Git or CVS plugin. Install also the Copy Artifact Plugin, to copy file from one to another Jenkins job.
The next two plugins are recommended. The first one is green balls, because I like a green sign instead of a blue one when the tests are passed :). The second one is the Log Parser Plugin. I will explain later in this post why I recommend this one. Click install and reboot Jenkins.

If you have problems with one of the plugins, please see the plugin documentation.

Create Jenkins Jobs

When you pass these lines of text, you finished successfully the whole setup and environment stuff. The last things you have to do is to create two Jenkins jobs. One for the Android App you want to test, the second job for the Robotium tests.

To create a new Job click New Job link on the top left corner of Jenkins. Choose a name for the project, select Build a free-style software project and click ok.

On the next page you have to define precise steps, Jenkins should execute for you.
To create a job for building an Android app enter the following things:

  1. Enter a name: My Android App
  2. In the section Source Code Management choose e.g. Git and enter the URL of the repository git@YOURDOMAIN.com:my-android-app.git. Make sure to provide also the right branch e.g. master.
  3. Build Triggers: Add a time schedule Poll SCM for building the app after a commit to the git repository was pushed
  4. Invoke Ant: Add build step Ant and enter as target clean debug
  5. Create a Post build action, archive the artifacts apk file under bin/*debug.apk
  6. Choose build additional projects. Enter the Jenkins Job name of the Robotium Tests, this will start the Robotium Tests if the Android app was successfully build
  7. E-Mail Notification: Enter the E-Mail addresses for sending Jenkins Reports
  8. Save the job

Now the first job should be visible on the Jenkins Dashboard.

Press the green play button on the right side to start building the Android App. If the build was successful the dashboard will look like this:

Now you have to do the same again for the Robotium test job. Click New Job, choose a name for the project, select Build a free-style software project and click ok.

  1. Enter a name: My Robotium Tests
  2. In the section Source Code Management choose e.g. Git and enter the URL of the repository git@YOURDOMAIN.com:my-robotium-tests-app.git. Make sure to provide also the right branch e.g. master.
  3. Build Triggers: No build triggers needed because the prev. added job will start the Robotium tests automatically when building the app was successful. Hint: For nightly builds and testing it is recommended to add a build trigger.
  4. Build Environment: If you want to test the Android App on an emulator be sure to select the ‘Run an Android emulator during build’ option.
  5. After selecting this entry you can add a prev. created Android Emulator that should be started within this jenkins job. Be sure the option ‘Use emulator snapshots’ is checked for fast booting the emulator. Note: It is recommended to connect a device (if possible) to the Jenkins server to execute the tests on a real device! The config could look like this:
  6. Invoke Ant: Add build step Ant and enter as target clean debug to build the Robotium tests
  7. Click the button ‘Add Build Step’ and choose ‘Install Android package’. This step is needed to install the Android apk file. Enter the path to the apk file: /Users/jenkins/.jenkins/jobs/My Android App/workspace/bin/MyAndroidApp-debug.apk
  8. Or create a ‘Copy artifacts from another project’ build step to copy the prev. build apk file to your Jenkins test job.
  9. Choose uninstall existing APK first, to be sure that a prev. installed version is uninstalled from emulator or device.
  10. Add another ‘Install Android package’ to install the test apk file. Enter also the path: bin/MyRobotiumTests-debug.apk
  11. Choose uninstall existing APK first, to be sure that a prev. installed version is uninstalled from emulator or device.
  12. Click again ‘Add Build Step’ and choose Execute Shell. This entry is necessary to start the Android Instrumentation Test Runner to start the Robotium tests. Enter /Applications/android/platform-tools/./adb shell am instrument -w com.XYZ.testcase.myRobotiumTests/android.test.InstrumentationTestRunner (recommended to add the Android Path to system environment path to have a relative paths instead of absolute)
  13. E-Mail Notification: Enter the E-Mail addresses for sending Jenkins Reports
  14. Save the job

Again, check the Jenkins dashboard and execute the Robotium tests. If everything is good it should look like:

If you want detailed information about the build process or the test execution you can check the ‘Console Output’. To get the output, open a Jenkins Job, click on an entry of the build history. On the left side of a detailed build there is the entry ‘Console Output’. In this section you find compile information, erros and stacktraces.

Due to the fact that the tests are running inside the emulator or the device, you don’t have access to the JUnit test results. There are two ways of solving that problem. The first one is to root the device to get access to the sdcard and the test reports. With the rooted access you can use the command adb pull PATH_TO_XML_FILE to get the results into Jenkins. In my opinion it is not a good approach to root the test device to get the results because, you should NOT support rooted devices when providing a professional app. The second approach that is working very well, is to use the above mentioned Log Parser Plugin. This plugin is able to parse the console output with your own parsing criteria and mark the build as passed or failed and give you detailed information.

Configure the Log Parser Plugin

Open the Jenkins Management and Configuration section and scroll to the bottom of the page. If the plugin was installed correct, the following section is available:

Provide a description for the parsing rules and add a parsing file from the harddisk. Click Save. A parsing file could look like this:

error /junit.framework.AssertionFailedError/
error /java.lang.NullPointerException/

After creating the file, you have to add the parsing rule to your Robotium Jenkins job. Open the Robotium test job and click on configure. Scroll down to the Post-build actions section and select Console output (build log) parsing. Choose the file from the harddrive and save the job.

If the parsing rule is working, you get the following overview on the detail page of the Jenkins job. The plugin highlights errors, warnings or info in the log and provide a quick overview of the important parts of the current build run and errors.

Please check the plugin documentation for more details with the parsing rules.

That was the tutorial to integrate your Robotium tests into Jenkins using Ant and some plugins. You should now have a complete mobile test automation environment on a centralized place, that will save you a lot of money and time during development and testing!

Update:
Besides using the default “android.test.InstrumentationTestRunner”, you can easily integrate https://github.com/jsankey/android-junit-report into your project which will output  the Robotium test results in JUnit format.

Have fun!