We’ve been acquired by Xamarin! Click here to learn more about our new Xamarin Test Cloud product
Posted by Karl Krukow - March 28, 2013

Test engineering for Cross-platform

One of the promises of Calabash is cross-platform testing. What we mean by this is the ability to maximize code- and specification reuse when developing the same app (or similar apps) for multiple platforms (e.g. iPhone, iPad, Android Phone, Android Tablet, Mobile Web, Desktop Web,…).

However, it may not immediately apparent how to do this. We do not recommend the use of the Calabash pre-defined steps – these are there only for beginners and to get results quickly. For larger, serious test suites, pre-defined steps should be completely avoided.

We’ve written an article about cross-platform testing with Calabash which comes along with the accompanying sample code based on the open source Wordpress app.

LessPainful also offers a number of services to support the Calabash project. One of the most popular services has been on-site training for companies adopting Calabash. In the training class we teach how you use Calaash efficiently as well as cross-platform and test engineering best practices. Contact us as contact@lesspainful.com for more information.

Cross-platform testing does not mean 100% code-reuse across platforms. This may seem disappointing at first, but when you consider the fact that well-designed UIs are different for different platforms, even if you’re build the same app, then this becomes immediately clear. Instead, what we should hope for is to reuse the parts of the testing code that should be the same: the specifications and step definitions.

For example consider these screens from the wordpress app:

Posted by Jonas Maturana Larsen - March 15, 2013

Testing Multiple Android Apps

The default setup for Calabash-Android is to test a single Android application at a time. calabash-android gen wil generate hooks that will install, launch and uninstall at appropriate times to support the single app use case. However, sometimes it can be useful to interact with multiple apps at the same time. This blog post will dig into details of calabash-android that you might not have touched before.

Notice that this post is written based on the current pre release of Calabash Android (0.4.3.pre3)

Primer: Calabash-Android architecture

Calabash-Android has a client-server architecture where client and server communicate via HTTP. The responsibilites are split like this:

  • Client The client sends commands to the test server using performAction, query a number of other methods. Besides the communication, the client is responsible for installing, starting, stopping the test server.

  • Test Server The test server is an instrumentation app that has special permissions to access your app. When a test server is launched it will start an HTTP server on a specified port, start your app and start waiting for requests.

By the nature of the Android instrumentation framework, each test server can only test a single app, so to achive multi-app-testing we need to do something special.

When you test an app using the generated project structure, the test server will be installed and started on the single device or emulator you have connected to your computer. The test server will listen on port 7102 and localhost:34777 will be setup to forward to device:7102 over USB. All requests from the client is sent to this default test server.

So to test multiple apps, all we need to do is to start multiple test servers. We then configure the client to send requests to each app in turn. Let’s see how we can do that.

Testing Multiple Apps

Let’s assume that we have two apps called app1.apk and app2.apk located in the root of our project folder.

Create the test servers like this:

calabash-android build app1.apk
calabash-android build app2.apk

If you put the following in your features/support/app_installation_hooks.rb and remove features/support/app_life_cycle_hooks.rb you will get two test servers started when executing calabash-android run: one listening on port 7103 and another listening on port 7104.

require 'calabash-android/management/app_installation'

AfterConfiguration do |config|
  path1 = File.expand_path("app1.apk")
  @@app1 = Calabash::Android::Operations::Device.new(
    self, nil, "34801", path1, test_server_path(path1), 7103)

  path2 = File.expand_path("app2.apk")
  @@app2 = Calabash::Android::Operations::Device.new(
    self, nil, "34802", path2, test_server_path(path2), 7104)
end

Before do |scenario|
  extend Calabash::Android::Operations
  @@app2.reinstall_apps
  @@app2.start_test_server_in_background

  @@app1.reinstall_apps
  @@app1.start_test_server_in_background
  set_default_device(@@app1)
end

All requests including those from predifined steps will now be send to app1 running on localhost:34801.

To start communicating with app2 simply call set_default_device(@@app2).

Posted by Karl Krukow - January 15, 2013

Intensive Calabash Training

LessPainful offers a number of services to support the Calabash project. One of the most popular services has been on-site training for companies adopting Calabash. However, until now these courses have been on-site and for single companies only.

We’ll now run our first public training course, which will be held in London on February 11th-12th. To get the most effective training, we’ll accept at most 15 participants so reserve your spot now! To register, please use London training registration form.

Although the first course is in London, we’d be interested in hearing from you if you’d like to attend training, but cannot make it to London. If you prefer a different location or time for the next training let us know by mailing to LessPainful Contact.

Benefits of Training

Learning a new technology on your own can be time-consuming, and it’s always hard to know if you’re doing it right! Calabash is a powerful tool, but different ways of using it lead to different degrees of success. Advanced topics such as structuring for cross-platform testing, using the query language effectively, and testing hybrid apps can be hard to learn on your own.

The Calabash training will be taught by the creators of Calabash, and we have several years experience with mobile testing and development. The training is centered around guided hands-on exercises to ensure you are able to use the tool efficiently in practice. We’ve developed the course based on our experiences running successful on-site private courses. The material has been expanded to cover the latest development, e.g., query support on Android and UIAutomation support on iOS.

The courses is two full days, and we will cover a basic introduction to Ruby and Cucumber, although the main focus on Calabash Android and Calabash iOS.

Please see the London training registration form for a more complete agenda!

Hope to see you there!

Posted by Karl Krukow - December 18, 2012

A Query Language for Android Views

Update: Karl, February 25th, 2013. Updated with newest features in query. This is not a complete specification. For details refer to the wiki.

Back in September we published a roadmap for Calabash in the short to medium term. One of the key things on the roadmap was implementing a query language for Calabash Android. The query language will bring additional expressive power to Android tests. Test scripts can become much more dynamic, which removes the need for many of the extensions in the form of custom actions that users of Calabash Android are currently making.

Query-language support has now been released in Calabash Android. So please update to the latest version (0.4.2 at the time of this writing).

Let’s see some code!

We’ll use the WordPress app (details on how to get up and running here. To get started download wordpress using the following (which requires sub-version, svn, installed)

krukow:~/tmp/query$ mkdir -p android-source
krukow:~/tmp/query$ cd android-source
krukow:~/tmp/query/android-source$ svn co http://android.svn.wordpress.org/tags/2.2.7/
krukow:~/tmp/query/android-source$ cd 2.2.7/
krukow:~/tmp/query/android-source/2.2.7$ ant debug
Buildfile: /Users/krukow/tmp/query/android-source/2.2.7/build.xml
…

Now from the directory containing android-source, connect and Android device (or emulator) and do:

krukow:~/tmp/query$ calabash-android console android-source/2.2.7/bin/Dashboard-debug.apk
No test server found for this combination of app and calabash version. Recreating test server.
Done signing the test server. Moved it to test_servers/63a7fdd310c543c9e27773d4207823d2_0.4.2.apk

Now reinstall and launch:

    irb(main):001:0> reinstall_apps
    …
    => nil
    irb(main):002:0> start_test_server_in_background
    nil

This starts the app on the device:

Simple sample app main screen with two buttons and two edit text fields

Time to actually try query. Let’s query for all buttons in the current view:

irb(main):003:0> buttons = query("button")
[
    [0] {
                    "id" => "button1",
               "enabled" => true,
    "contentDescription" => nil,
                 "class" => "android.widget.Button",
                  "text" => "Accept",
                  "rect" => {
        "center_y" => 434.5,
        "center_x" => 89.0,
          "height" => 55,
               "y" => 407,
           "width" => 146,
               "x" => 16
           },
           "description" => "android.widget.Button@40573488"
    },
    [1] {
                    "id" => "button2",
               "enabled" => true,
    "contentDescription" => nil,
                 "class" => "android.widget.Button",
                  "text" => "Decline",
                  "rect" => {
        "center_y" => 434.5,
        "center_x" => 233.0,
          "height" => 55,
               "y" => 407,
           "width" => 146,
               "x" => 160
           },
           "description" => "android.widget.Button@40558a10"
    }
]

As we can see there are two buttons: one with text "Accept" and id "button1", and one with text "Decline" and id "button2". We can use index to select one of those:

irb(main):004:0> query("button index:0")
[
    [0] {
                    "id" => "button1",
               "enabled" => true,
    "contentDescription" => nil,
                 "class" => "android.widget.Button",
                  "text" => "Accept",
                  "rect" => {
        "center_y" => 434.5,
        "center_x" => 89.0,
          "height" => 55,
               "y" => 407,
           "width" => 146,
               "x" => 16
        },
           "description" => "android.widget.Button@40573488"
    }
]

Filtering based on properties

Using index often leads to brittle tests that may break if the user interface changes. On iOS there is a concept of a “mark” — a general way of identifying views.

The notion of mark also exists in Calabash Android: here it means view selection by one of the properties id, contentDescription or simply text. Here is an example:

irb(main):005:0> query("button marked:'Accept'")
[
    [0] {
            "id" => "button1",
            "text" => "Accept",
            ...
         }
]

In general you can filter based on any “property” of a view. A property is defined as any piece of data you can access by calling a getter on the view. (Actually, any simple data you can extract by calling a no-arg method is a “property”)

For example, let’s filter the buttons by matching against the value of their getText method instead:

irb(main):006:0> query("button text:'Accept'")
#same result

Right now, only integer, string, boolean and null data types are supported. Strings are written in single quotes 'a string'. More exact details on what happens is described in the Wiki page about query syntax.

Filtering based on classes

The first query we saw in this post was query("button") – this is actually querying for all visible views which have a class with “simple name”: "button" (simple name is the last segment of the fully qualified name). The comparison is case insensitive so this will match the class: android.widget.Button.

You can also match on a qualified class name, e.g., query("android.view.View") will match all views of type "android.view.View" (or subtypes thereof).

There is an important distinction between iOS and Android here: on iOS there is a “magic” translation from say label to UILabel (the class in UIView). There is no such translation on Android, only matching based on (sub-) type of a class or simple name for convenience.

Another important distiction is that only fully qualified queries match also sub-types. So if you have a sub-class of android.widget.Button say com.example.AwesomeButton this will not be found with query("button") but it will with query("android.widget.Button"). So, at least right now, the “simple-name” notation is restricted.

Also, there is a short-hand notation for “android.view.View” which is simply “*” (e.g., query("*")).

View hierarchy

On iOS you can navigate the view hierarchy, for example query("tableViewCell index:0 label") finds all labels inside the first table cell.

Here is an Android example:

irb(main):010:0> query("* id:'buttonPanel' button text:'Accept'")
#same result yet again...

This reads as “find a view with id ‘buttonPanel’, then inside of that find a button with text ‘Accept’”.

On iOS you can navigate up and down the hierarchy. This is also supported on Android. Here is a powerful example:

irb(main):011:0> query("* id:'buttonPanel' button text:'Accept' sibling button")
[
    [0] {
                    "id" => "button2",
               "enabled" => true,
    "contentDescription" => nil,
                 "class" => "android.widget.Button",
                  "text" => "Decline",
                  "rect" => {
        "center_y" => 274.5,
        "center_x" => 353.0,
          "height" => 55,
               "y" => 247,
           "width" => 226,
               "x" => 240
        },
           "description" => "android.widget.Button@40558a10"
    }
]

This finds the “sibling” button of “Accept”, namely the “Decline” button. You can also navigate in these directions: descendant, child, parent (descendant being the default).

Property extraction

Just as on iOS, you can extract properties by providing more than one argument to the query function:

irb(main):020:0> query("button",:text)
[
  [0] "Accept",
  [1] "Decline"
]

Here, the android.widget.Button views are found, and text is extracted via a text(), getText() or isText() method.

Several arguments can be supplied, in which case the process continues:

irb(main):021:0> query("button", :text, :length)
[
  [0] 6,
  [1] 7
]

Do you dare try this: query("button", {:setText => "No way!"})?

Touch

If you can find it with query, you can touch it! Touching (and other gestures) can be done in two ways. You can provide a query to the touch function: e.g. touch("button text:'Accept'"). Alternatively you can store the result of a past query in a variable: btn = query("button text:'Accept'") and then later touch(btn) (provided of course the button is still visible at the same location).

Conclusion and What’s next?

As mentioned in the roadmap, we are working at bringing Calabash Android and Calabash iOS closer together. With query-support in the released 0.4.x line, this is already a big step forward in providing a truely cross-platform testing experience for mobile. We’ve also provided an example of cross-platform testing with Calabash which maximizes reuse of test-code across platforms.

Cross-platform example

Getting good community feedback on the query language is important to us. So we encourage everyone to try it out and report experiences and bugs to the Calabash Android Group and to submit improvements as pull requests.

The next blog post will shed some light on how Calabash iOS will improve in the coming months. I think we have some very interesting news here that we’re looking forward to sharing with you… Stay tuned.

Posted by Karl Krukow - December 13, 2012

Call for JVM-support Sponsorships

Background

In June we started a survey of interest in Java/JVM support for writing and executing Calabash tests. In the last 6 months we’ve been speaking at conferences and talking to customers, and JVM support has been a recurring feature request. Our survey for JVM support has now been answered by a number of larger companies all positive around sponsoring JVM support.

So now is the time!

Sponsorships

We now have sufficient interest from several larger companies that need JVM support and have indicated interest in sponsoring the development effort.

We’ve estimated a total cost of $10.000 for us for this development effort. LessPainful itself will sponsor 25% of the effort since it benefits Calabash — this leaves $7500 that we need to find.

Benefits for sponsors

  • Of course, using Java for writing Calabash tests is the most obvious benefit.

  • All sponsors have the option of having their logo added as a sponsor to the Calabash landing page (http://calaba.sh).

  • All sponsors have the option of being mentioned (including the amount sponsored) in an appreciation blog post and tweets by @lesspainful.

  • Any special JVM feature requests are possible and will be seriously considered for sponsors.

Please let us know if you want to contribute to this effort.

Deadline

Deadline for accepting is December 21st 2012 (if you want to sponsor but cannot fully confirm by 21st, let us know, and we will figure something out).

Please contact contact@lesspainful.com to become a sponsor.

After the deadline we will post here whether or not the goal has been reached.

Posted by Karl Krukow - September 2, 2012

Roadmap for Calabash

In this post we’ll talk about the history and future of Calabash. A lot has happened since it was first publicly announced about half a year ago in our introductory blog post on Calabash. We’ll look at the roadmap for Calabash as we see it, but first a bit of history to explain why things are as they are…

Background

Calabash Logo

In January 2012, we decided to change LessPainful’s business model. The most significant change was to open source the software that was used to write automated acceptance and functional tests for our clients' mobile apps: Calabash. This was a lot of work, and continually is. Thankfully, it is also rewarding to see a community thrive.

Infact, it turned out to be the best decision we ever made for LessPainful: it has generated a lot of attention in the mobile space (see links 1-6) below), it has created an active community around the software, and it has given LessPainful a number of very serious larger enterprise customers wanting commercial support, training, Calabash extensions and of course our primary product: Shared and Private Mobile Device Clouds.

Before Calabash was made open source, it existed first as a proprietary library for Android. It was based on Robotium but gave a higher level API and supported Cucumber. The library was used for some time to test our clients' Android apps on real Android devices at LessPainful. It was clear that many of our clients wanted both Android and iOS support. So I implemented a proof-of-concept of the automation infrastructure for iOS which we needed at LessPainful.com (install/uninstall, start/stop, clear device, change language etc).

After doing the POC, we needed a testing library that would let our clients use Cucumber to test also their iOS apps at LessPainful. The iOS library was almost a straight reimplementation of Pete Hodgson’s Frank, changing only what was necessary to support running on our device clouds at LessPainful (it did also change some significant parts like touch synthesis).

What all this means is that Calabash started and has evolved as two separate systems. Although they have the same goals (supporting BDD via Cucumber on Android and iOS), the implementations differ as do the interfaces they expose, apart from Cucumber of course.

Now, part of the value proposition for Calabash is reusing as much as possible of your testing code when building the same app on iOS and Android (i.e., reusing features and test scripts). Improving this situation will be one of the top priorities for Calabash in the near future.

Bringing Android and iOS Closer

Android and iOS logos One of the most important goals for Calabash is to reuse features, step definitions and support code across iOS and Android. We’ve already begun moving in this direction. For example, Calabash Android has moved to embedding an HTTP server instead of the custom TCP based protocol used before. Soon it will also move to a JSON-based protocol similar to Calabash iOS (which is actually more or less the Frankly protocol). This moves the server parts of Calabash iOS and Android closer, meaning that clients can be made closer. Here is a good example: query.

Currently, Calabash iOS has a query language based on UISpec which was inherited from Frank. This query language is immensely useful when writing test scripts – you can query into any object in the view, find its properties, call methods etc. Calabash Android doesn’t support the query language – and it shouldn’t since UISpec is designed around iOS. This is an example where Calabash iOS and Calabash Android will both move away from their current implementation and towards a common query language that makes sense on both platforms. We don’t know exactly what this language will look like yet but UISpec has done well for us so far – so we will design something inspired by that.

This will lead to a set of unified APIs that can be used in your iOS and Android test scripts. This is one of our top priorities for Calabash in the near future.

Extending Reach

Da Vinci Machine Project There are many programmers and QA staff out there that are already familiar with JVM-based languages and tools. Many shops have invested in the JVM both as a platform for development and testing.

While we love Ruby and enjoy working with it, we also understand that not everyone has the time or will to learn a new programming language. So while Ruby is a very nice language to work with and has a great ecosystem and community supporting testing, it does restrict the reach of Calabash.

For this reason and because there is a good interest (more than 20 companies have declared a strong interest in JVM support), we will be developing JVM support for Calabash. BDD will be supported on the JVM via Cucumber JVM, but Calabash JVM itself will be independent of Cucumber.

As a proof of concept, and declaration of intent, we have implemented a minimal and prototype JVM version of Calabash iOS. It is implemented in Clojure, but exposes an API that can be called from Java and other JVM-based languages. The project can be found here, and examples here (Java) and here (Clojure).

While already quite functional, this is only a proof-of-concept. JVM support will not be our first improvement to Calabash. It makes much more sense to first find a good unified API as discussed above. Once we are happy with our unified APIs we can create a JVM implementation much faster.

JVM support is of high priority to us, but we will develop unified APIs first.

Other improvements

Of course we will continually add features and fix bugs on the Calabash versions you already know. Other potential areas of improvement are

  • Device Cloud. Better and easier support for running on the LessPainful Device Cloud. There are actually some very juicy bits here that I can’t reveal yet :) It will actually be released quite soon, and described in a separate blog post.

  • Better hybrid/webview support. While our web view support is actually already quite good, it is also fairly low level. You can query for visible elements and text (this is actually the same API on Android and iOS). You can touch visible elements and enter text. This gets you very far on web views, but it would be nice to have higher level APIs for dealing with common HTML5 components.

  • Image analysis support. We are thinking about looking into better support for comparing images. While we prefer scripts that look for text or UI components in views (since they are more robust), we also recognize that there is value in image-based regression analysis. Nicholas Albion has already done some work on this based on ImageMagick and Zucchini.

  • <Insert Your suggestion here!> Calabash is truely open source. We will listen to community need, and we will acommodate (provided of course requests are inlined with the direction of Calabash).

Stay tuned for more news soon!

Presentation Links

  1. Sauce Labs Mobile Testing Summit, SF 2012
  2. QCon London, London 2012 <– that speaker bio is kinda dated :)
  3. CukeUp, London 2012
  4. Goto Amterdam, Amsterdam 2012
  5. Code Motion, Rome, 2012
  6. StarEast, Orlando, 2012
Posted by Karl Krukow - July 10, 2012

Documentation for Calabash iOS

I’ve done some work to document the previously undocumented API functions needed to write custom steps. I’ve also given the documentation an update in general. It is actually getting fairly good (let me know if you disagree!).

Checkout these highlights:

Documentation Overview

Getting Started Guide

Ruby API

Touch/Event Recording and Playback

Web View/Hybrid Support

And finally:

EVERYTHING.

I’ll be complementing these with upcoming screencasts covering: How to work efficiently with Calabash iOS, Tool support, Ruby Tips and Advanced Touch Synthesis.

Stay tuned :)

Posted by Karl Krukow and Jonas Maturana Larsen - June 13, 2012

Calabash and JVM support

We are trying to determine how broad interest there is in getting Calabash running on the JVM platform, supporting cucumber-jvm.

We would love for people interested in Calabash, to fill out this short form to indicate JVM interest. Comments/discussion here is also ok:

JVM Support interest form

Posted by Karl Krukow - April 1, 2012

Speaking about Calabash at Cukeup! 2012

Things are going well with Calabash – it’s starting to come alive! People are picking it up, they ask questions, post feature requests and get excited. (as are we!)

Thats cool! But we want to spread the word even more. I’ve already given talks at QCon London 2012, Goto Night, Amsterdam and Code motion Rome.

Next up is Cuke!, and I’m very excited about this one. Cucumber powers Calabash, so I’m happy to give something back to the community. I’ll be giving a talk about Calabash at Cukeup! 2012 Wed. April 4 at 14.15 – 14.45. Hope to see you there!

Posted by Karl Krukow - March 7, 2012

An Overview of Calabash iOS

Calabsh iOS is a new open source project for functional testing of iPhone and iPad apps based on Cucumber.

This post describes Calabash iOS in some technical detail. It will give you an architectural overview and compare Calabash iOS with some other functional testing alternatives (hence, it is fairly advanced and assumes that you are familiar with Cucumber and iOS development).

Calabash Architecture

Calabash tests are executed with the Cucumber tool and consist of three parts:

  • Your app, linked with the calabash.framework which embeds an HTTP server in your app (of course, this is only in test builds of your app). The Calabash client library (see next bullet) makes HTTP requests to the server when it wants to do something with the current view.

  • The Calabash client library and step definitions. Calabash supports two types of step definitions: built-in and custom. The built-ins define general and domain-neutral steps that are intended to give you a quick and easy start. They contain steps like: I should see a "..." button and I swipe right. The custom steps are written by you, and they’re specific to your domain or application. Examples could be: I should be on the Login screen, I login as Pete or I add a track to my playlist. To implement a custom step you can use methods defined in the Calabash client library, calabash-cucumber, which is part of Calabash iOS.

  • Your feature files. The feature files describe the app use-cases you want to test. They are simultaneously a specification of the behaviour of your app and an executable test suite.

You can visualize this as:

Calabash-iOS architecture

Calabash iOS supports running on the iOS simulator as well as iPhones, iPads and iPod touches (whether jailbroken or not). The calabash server framework is distributed as a universal framework for your convenience.

The client library, calabash-cucumber, is installed using RubyGems which is as simple as running gem install calabash-cucumber.

Comparison with other functional testing systems

There are several options out there for functional testing on iOS. Apple provides its UIAutomation, there is Zucchini, Frank by Pete Hodgson, KIF by square, FoneMonkey by Gorilla Logic, iCuke by Rob Holland, and probably a few I’ve looked at and forgotten again!

All the projects have their advantages and disadvantages, Calabash included. But we think that Calabash hits a sweet spot that none other the other options cover.

Frank

Frank is the library closest to Calabash; in fact, when we started experimenting with iOS support at LessPainful we started with Frank. Frank is a very cool project by a great person (that’s you Pete), and Calabash is highly inspired by it. But there were a couple of important reasons why we could not use Frank.

First, Frank is licensed using GNU GPLv3 and we wanted to use the less restrictive EPL. Pete actually didn’t want to license Frank as GPL, but since Frank is based on UISpec which is GPL'ed, he had to. We did not want the UISpec project as a dependency.

Second, when we started to look at Frank, it was a bit of a pain to set up (in the mean time, its actually gotten much better). It required modification of you app source, inclusion of static resources and source files. We wanted calabash to be easy to set up, so we made Calabash a universal framework.

Third, Frank is focused on running in the iOS simulator. We wanted something that ran equally well on simulator, iPhone and iPad. There is no fundamental reason that the Frank architecture shouldn’t work on physical devices (indeed Calabash has the same architecture), but many of Frank’s step definitions are simulator only.

Finally, we needed support for some advanced features that Frank just didn’t have. Frank’s touch emulation is derived from UISpec, which has some disadvantages: it only supports tapping, and actually doesn’t seem to work in all cases (for example on a UISwitch). We wanted support for gestures like swipe, pinch-to-zoom, backgrounding and rotation. We wanted support for querying and acting on UIWebViews.

Frank also has the cool Symbiote tool which let’s you explore your app from a browser. We don’t have Symbiote in Calabash, but instead we have a powerful console based tool for interacting and exploring your application (more on this in a later blog post).

KIF, iCuke, FoneMonkey

KIF is another cool project. KIF has an Apache 2.0 license which is ok, but two primary things kept us from going down this route. First, KIF is based on the same touch emulation code as UISpec, and has the same limitations (which actually goes back to a seminal CocoaWithLove post from 2008, Synthesizing touch events on iPhone). Secondly, KIF tests are written in Objective-C, and we specifically wanted Cucumber for the reasons outlined in the Introductory post.

iCuke actually looked very promising two years ago. Unfortunately its author, Rob Holland, has abandoned it, and we had a difficult time proceeding with it.

FoneMonkey, is GPL licensed and doesn’t support Cucumber. FoneMonkey is primarily based on a test recording approach, and we specifically didn’t want recording of test scripts since it often leads to large, unreadable and unmaintainable test script . We do actually support a different form of recording that doesn’t suffer from this problem, but that’s a topic for another blog post.

UIAutomation and Zucchini

Apple’s UIAutomation framework is actually quite advanced in supporting various gestures. It’s also simple to set up since it works without modification to the app (runs via instruments). There are a couple of reasons why we decided not to use UIAutomation.

First, UIAutomation is not open source – this means we have to wait for Apple to make bug fixes, and we can’t extend it. It is also very poorly documented by Apple, and not that much information available on the net.

Second, tests are written in JavaScript using a horrible API, resulting in tests that are too easy to break when making minor changes to the UI (IMHO). The tool for developing and running tests (UIAutomation Instrument) gives an inefficient workflow, for example, by locking the editing file, missing the ability to interactively explore your app and running only parts of a test. Also, screenshotting only works on device.

Third, UIAutomation is hard to get working in a continuous integration setup (in fact, when we first looked at it, CI was completely impossible). Extracting tests results and screenshots in a CI friendly way is hard.

Zucchini, is a new project that’s very cool. It has some great addons for working with app screenshots (we might ‘borrow’ those for Calabash :), and it makes the developer experience much better by using CoffeeScript instead of JavaScript and by coming with some convenience tools. However, by compiling to UIAutomation compatible JavaScript and running via UIAutomation, it has some of the same problems as UIAutomation itself (lack of documentation, Apple control, the fragility of tests, etc.). Also, we didn’t want Calabash users to be required to learn CoffeeScript.

What’s next?

So there you have it! A short overview of the Calabash iOS architecture and an overview of similar frameworks.

In upcoming posts we’ll describe more implementation details of Calabash iOS. For example, we’ll show how to make advanced custom steps using the client library API, we’ll look at support for rotation and gestures like swiping, and even how you can record your own advanced gestures. We’ll also look at an efficient workflow for writing tests in an interactive and exploratory manner.

In the meantime please try out Calabash iOS, read the Project wiki documentation, and tell us what you think: Issues and Discussion.

Posted by Jonas Maturana Larsen - March 7, 2012

An Overview of Calabash Android

This post describes Calabash Android in some technical detail. It will give you an architectural overview of Calabash Android and a resume of the functional testing tools available for Android. It assumes that you are familiar with Cucumber and Android development.

Calabash Android Architecture

When a Calabash Android test is executed both your local computer and a device is involved. The device might be an emulator or an actual physical device. The setup looks like this: Calabash-Android architecture

  • Features. The feature files describe the user-stories you want to test. You can test one or more features in one test run.

  • Step Definitions. Calabash Android comes with a set of predefined step which you can find here. These steps are generic and made to get you up and running fast. When you get more into Calabash you can implement your own custom steps that use the business terms your project uses like I transfer money to my spendings account or I play "Never Gonna Give You Up".

  • Your app. You don’t have to make modifications to your app before testing it.

  • Instrumentation Test Server. This is another app that will be installed and executed the device. This app is based on ActivityInstrumentationTestCase2 from the Android SDK. It is generated by the Calabash Android framework.

Functional Testing tools for Android

The Android SDK comes with a testing framework located in the android.test package. According to Google it is easy to use. However, I’m pretty sure that the person who wrote that either never wrote and maintained a test using ActivityInstrumentationTestCase2 or has totally misunderstood the “easy to use” concept. :–)

When Renas Reda created Robotium things became a lot easier. Robotium is a utility that handles a lot of the small details you really don’t want to be concerned about in a functional test. You are still writing your test using Java and JUnit but a lot of the very annoying things from vanilla Android testing is now taking care of and don’t pollute your tests. Today all the Android projects that I know of that do any kind of testing use Robotium or Calabash. So if you are still fighting android.test and want to clean up your tests you should give Robotium a try and send Renas a happy thought.

Even when using Robotium my experience from several large Android projects is that writing functional testing in Java is just wrong. Over time, the ever growing test suite becomes takes more and more resources to maintain mainly because the readability goes down. But hey! Why wouldn’t it be like that? That is exactly the same experience we had when writing functional test for websites!

Calabash uses Robotium as a library to make the interaction with the tested app as robust as possible while letting the developer or tester write precise and clean test that are easy to maintain.

And that’s why you should give Calabash Android a try.

Testing Hybrid Apps

If you are using Appcelerator, PhoneGap, Sencha Touch or any of the other frameworks for creating hybrid apps. You should really take a look at Calabash. By using the same Cucumber interface as for a native app you can test HTML 5 part of your application. Testing webviews has not been posible with the currently available test frameworks. But with Calabash you can!

Questions

Let me know at jonas@lesspainful.com or @jonasmaturana.

Posted by Karl Krukow - March 7, 2012

Calabash: Functional Testing for Mobile Apps

Jonas and I are proud to release Calabash, a new functional testing tool for Android and iOS mobile apps based on Cucumber. LessPainful has been developing Calabash over the last year working to improve its expressive power, simplicity and stability. Calabash was born out of a need for device automation and test execution at LessPainful, but as of today it is released as open source to the general public: http://github.com/calabash.

The short story

The following section, Introducing Calabash, is intended to describe Calabash for people who don’t already know Cucumber, or who haven’t been looking at functional testing for mobile apps. If this doesn’t apply to you, you can go to the Rationale and Benefits section – otherwise just continue reading :)

Introducing Calabash

Calabash consists of two libraries calabash-android and calabash-ios. As you can imagine, calabash-android is the automation and testing library for Android, and similarly calabash-ios is for iOS.

Calabash-iOS and Calabash-Android are the underlying low-level libraries that empower the Cucumber tool to run automated functional tests on Android and iOS phones and tablets as well as on simulators. These low-level libraries enable QA, business staff and developers to work at a high level by writing tests in a natural language using the terms and concepts of their business domain. For example, here is part of a test written with Calabash.

Feature: Rating a stand
  Scenario: Find and rate a stand from the list
    Given I am on the List
    Then I should see a "rating" button
    And I should not see "Dixie Burger & Gumbo Soup"
    And take picture

    Then I touch the "rating" button
    And I should see "Dixie Burger & Gumbo Soup"
    And take picture

    When I touch "Dixie Burger & Gumbo Soup"
    Then I should see details for "Dixie Burger & Gumbo Soup"

    When I touch the "rate_it" button
    Then I should see the rating panel

    Then I touch "star5"
    And I touch "rate"
    And take picture

If you don’t already know Cucumber, it may surprise you that this description is actually an executable test specification that is part of a test suite for an iPhone app!

The file above is called a feature file in Cucumber and Calabash. A feature file describes the intended behaviour of the app. You can think of a feature file as an executable test specification of a use case of the app. In our example above, we have a feature which describes functionality for rating a food stand at Roskilde Festival. A feature consists of one or more scenarios, corresponding more or less to the possible outcomes of the use cases. In our simple example we have only one scenario the “sunshine” scenario where we find a stand and rate it five stars. There would usually also be scenarios testing other things, e.g., error scenarios. Finally, each scenario consists of a number of steps. The usual form of a feature is:

Feature: name...
  Scenario: ...
    Step
    Step
    ...

  Scenario: ...
    Step
    Step
    ...

In our example, each line after “Scenario” corresponds to a step. In calabash, a step does one of three things: makes a user action (like touch, swipe, scroll, etc.), makes an assertion (like Then I should see details for "..") or takes a screen-shot. The screen-shot step takes a snapshot of how your application looks at the point in the test. Screenshots can then be inspected for graphical errors that are hard to catch using assertions. Also, comparing screenshots across many different Android or iOS devices and operating system can be really useful. This type of visual test report, is part of what the LessPainful Service offers. You can see an example our feature for the Roskilde app above in the LessPainful section.

Overview: How it works

We’ll describe briefly how Calabash works. You can get more details in the blog posts on Calabash Android and Calabash iOS. You can also go straight to the sources: http://github.com/calabash/calabash-android, http://github.com/calabash/calabash-ios and http://github.com/calabash/calabash-ios-server.

Calabash iOS consists of two parts: a client library written in Ruby, and calabash.framework, a server framework written in Objective-C (a Clojure/JVM version of the client is coming too). To use calabash you make a special test target in XCode that links with calabash.framework. The application is otherwise unchanged. The server framework will start an HTTP server inside your app that listens for requests from the client library.

The cucumber tool executes your feature files which contain steps that are defined either in predefined or custom (user defined) steps definitions. Custom steps are written using the API that the client library exposes. The API methods make HTTP requests to the server, which does things like finding components in the UI and taking actions on them. Here is an illustration:

Calabash-iOS architecture

Calabash Android is slightly different even though the idea is much the same. You can read about the architecture of Calabash Android here. The greatest benefit of the different architecture is that you can test your Android app without making any changes to the app.

Rationale and Benefits

Why create a new open source functional test library? There are several reasons.

Uniform interface across platforms
We wanted a single interface for specifying the behaviour of mobile apps across platforms. Many companies are creating the same app (or almost the same app) for both iOS and Android, and pay double costs when it comes to testing. With Calabash, you have the option of maintaining only one feature file per app use-case, even if you are developing for both Android and iOS. The platform or app differences can be factored out into so-called step definitions which are different per platform. (If you're thinking Windows Phone? Then, yes, it is an option, but we haven't gotten round to it yet ;)
Cucumber
We wanted a high-level, domain-centric language for specification. The Cucumber technology gives us this. A Cucumber feature file can be written by QA, domain experts or developers. Developers or technical test staff can then write step definitions to realize the steps on each platform (possibly using one of the many predefined steps or APIs).
Authenticity
We wanted a test automation technology that is as authentic as possible. Test must be able to run equally well on physical, non-jailbroken devices as on simulators. The LessPainful Service takes this to the next level, by providing a test execution environment for running tests concurrently on many different devices (we even do rotation using little robots!).
Expressive power
There are some cool functional testing technologies out there already (like Robotium, Frank, NativeDriver, KIF, Zucchini and UIAutomation). However, in our opinion each of them have advantages and disadvantages, and none of them fulfilled our needs completely (see details in the blog posts Calabash Android and Calabash iOS). We wanted a technology with expressive power: support for complex gestures (swipes, pinch, etc), embedded webviews, rotation, backgrounding ... all while running on non-jailbroken devices.
Extensibility and Community
This is part of the reason why we are open sourcing Calabash. We want you to test your apps and ship better quality. We want Calabash users to be able to extend Calabash itself, and we want to build an open source community that is enthusiastic about the technology.

LessPainful

What is the role of LessPainful in regard to Calabash, and what services do we provide?

LessPainful is the creator of Calabash: we are its core committers and we are dedicated to continuously improving it. Our business model consists of two offerings:

  • We offer commercial support, training and consulting on automated testing of mobile apps. We can also help setup advanced continuous integration build environments on iOS and Android.

  • We provides a test execution service which runs Calabash tests “in the cloud”. The service automates running tests concurrently on several physical, non-jailbroken Android and iOS phones and tablets. We support several operating system versions, as well as device language and regional settings.

The LessPainful service provides visual test reports. This lets you easily compare test results and the visual appearance of your app across different devices. Here is an example test report from our example “food stand” app:

Test report overview

The LessPainful Service is designed to support continuous integration by providing a web service for running tests with every commit or on a daily basis.

Here is part of an Android test running on … different models!

Android test report overview