Testing Android apps using Calabash

BDD testing ASP.NET Mvc websites is a well beaten path with the likes of Nunit, Specflow, Selenium WebDriver etc but testing the features of an iPhone or Android app with an automated UI test suite is not so well covered. Xamarin have recently bought out LessPainful (https://www.lesspainful.com/); the guys who make Calabash (http://calaba.sh/). Calabash is an open source project which aims to deliver BDD driven UI tests to the mobile device. LessPainful allows developers to run those tests in the cloud on a huge number of devices. Xamarin are taking LessPainful and adding a C# API (currently Calabash is driven through Ruby). Tests are written using the Cucumber DSL and custom steps are implemented using Ruby. With a C# API, developers will be able to define steps using Nunit which means SpecFlow could be used to add cucumber support and automatic step skeleton creation. Potentially sharing feature files across Android / iPhone as well as HTML 5 mobile app tests.

For now though there is no public access to Xamarin's TestCloud (http://xamarin.com/test-cloud), the new name for LessPainful. The Calabash source is still open and available on GitHub, so anyone can set up and run BDD style tests using Calabash - just no TestCloud access for a while.

At Psonar we wanted to get end to end coverage on UI features and though a lot of the code is covered by unit tests, there's nothing like getting feedback from automated UI tests. Setting up Calabash was not a straight forward "install this, run this; BOOM" type of process. So I've written up my findings for future devs to get their heads around the process, it is a well worthy investment of time. I'm no Ruby/Java dev. The exceptions and error reporting as well as versioning hell that I've just been through hasn't persuaded me to jump ship yet either! Some understanding of the two are very helpful though.

First of all you need to make sure you have Ant (http://ant.apache.org/) installed. I've not had any problems installing this; it's very straightforward "unzip to c:\ant, add ANT_HOME env var". Simples.

Next up you want to download ruby 1.9 + devkit (version 1.9 is important as 2.0 has no compatible ghurkin lex parser - this tripped me up big time):

http://dl.bintray.com/oneclick/rubyinstaller/rubyinstaller-1.9.3-p448.exe?direct https://github.com/downloads/oneclick/rubyinstaller/DevKit-tdm-32-4.5.2-20111229-1559-sfx.exe

Installing Ruby is simple, just double click and run. Installing the DevKit is similar to Ant in that you need to unzip it to c:\DevKit, add that path to your PATH env var. Open a cmd and navigate to c:\DevKit and run the following two commands:


This configures the DevKit in your Ruby installations. Next up you need to use the following command to install the calabash-android gem:


If you are getting errors at this point about not being able to compile native gems, you probably installed the wrong versions of Ruby and DevKit.

All being well you should now be able to create a new folder for your Calabash tests and navigate to it and run:


This will install a default skeleton Calabash directory structure with some example cucumber code. Adding your app to the directory involves (in visual studio):
Select Release build (Calabash only supports release build APK's atm). Ensure you have no SDK linking in the projects properties view. Select Tools/Deploy Android Project. You need to specify the Android debug key located in "AppData\Local\Xamarin\Mono for Android". Specify the following when prompted:

password: android
alias: androiddebugkey
key password: android

Open a new cmd and run:



This fires Android ADB - the debug bridge that effectively outs all of the banter Android has with itself; very handy for debugging problems. Once the app is published and you've followed these steps copy the APK to the Calabash test folder and in another cmd window run:

This will tell calabash to copy the app to the device or emulator and the Calabash server (installed next to your app on the device) will start receiving delegated commands issued from the calabash process running on your computer (which interprets the Cucumber test scripts and turns them into commands sent to the device as JSON). To get you started there are a bunch of pre-canned test steps on the Calabash GitHub Wiki:

https://github.com/calabash/calabash-android/blob/master/ruby-gem/lib/calabash-android/canned_steps.md 

My advice (if you are a Xamarin developer) would be to stick rigidly with the pre-canned steps for the moment. Stepping outside of this safety wall will mean you will more than likely face porting them over to C# at some point in the future. While I am sure Xamarin will maintain the ruby implementation, no one really wants 50/50 Ruby/C# definitions to maintain. The pre-canned steps seem to be fairly comprehensive and allow for a lot of expression so only advanced scenarios would absolutely require custom steps definitions. Although only using pre-canned steps are a lot more verbose (you can't aggregate steps for "Given" style steps for instance) - it's a price worth putting up with for now.

I will try and add a follow up to this post explaining the process of adding iPhone tests - sharing the feature files across both OS's. Very tasty indeed!

w