Early last year I was working for one of our customers to find out an optimum test solution for their upcoming mobile application. The idea was that it should be heavily automated, efficient and cost-effective. The first observations (and the feeling) that I had was that we have very little choice in the way of tool choices. That was unfortunate, but the reality.
Almost a year later, when looking back and reflecting on some of those findings, I can see things have changed immensely. There have been announcements coming from key industry players almost every month. New promises being made! But then again, is it sufficient?
While development of mobile applications has become much simpler, the same is not quite true when it comes to testing them.
I earlier wrote a post on ‘what you should consider’ when you look to adopt a test platform/approach. While it always seems ideal that you go by those suggestions, it is not often possible, partly because you do not have the all-in-one toolset/solution that can tick all the boxes. In the constantly evolving mobile application testing market, there is no gold standard for choosing your test solution either – you have options ranging from feature-rich, flexible open source platforms or more organised and supported licensed platforms, each with their own pros and cons.
I recently discovered a test solution provided by Xamarin, namely Xamarin Test Cloud. While Xamarin is strong in the application development space, it is a relatively new entrant in the testing space. Regardless, I must say I am quite impressed with what they have so far! While it is still in its infancy, and will take them some time to compete with the other key players, in the Xamarin eco-system this might play well.
I will spend some time in this blog on how you can quickly set up yourself to run some tests against the Xamarin Test Cloud. I am consciously not going into the Architecture and other details as you can find many of those references in their website.
What is Xamarin Test Cloud?
Xamarin Test Cloud is the web-based cloud service where you can run your tests against a wide range of physical mobile devices that are managed by Xamarin in their lab. Like some other cloud based service providers, the obvious advantage is you do not have to worry about the device lab and you ony pay for what you use. You can use the Xamarin Test Cloud in two different ways:
- via Xamarin.UITest nuget (Xamarin-provided API)
- via Calabash
How to run tests using Xamarin.UITest?
We will talk about the first option here for the sake of simplicity. Here are my high level steps.
Step 1
The first thing that you need to do is create a Test project and download the Xamarin.UITest and NUnit nuget packages.
As a side note, Xamarin tests use the NUnit framework and do not support MSTest yet. The focus on NUnit is because the tests can be run on OSX / Linux boxes which don’t support MSTest.
Step 2
You can now go ahead and write your tests – in this example, I have used an android app (.apk) from the Selendroid project.
In order to write tests, you need to use the Xamarin.UITest API that provides you with ways to interact with your AUT (mobile application). Xamarin’s website has detailed documentation on the scripting process and how you can write get started pretty quickly.
Here is how you can set up your application to run against the device or emulator. If you want to debug and run your tests against the local emulator, you can install one from Google or other emulators in the market.
A simple unit test looks like the below. It taps on a pop-up button and then dismisses the pop up. You could add assert statements to make it a complete test – something I have not done here!
Running your test against a local emulator
Once you build the solution, you can run the tests using the standard NUnit interface. The only word of advice is to make sure to set the path of your .apk file properly. Ideally the .apk file should be in the same path as in the test DLLs. Note that you do not need to worry about the path in the code when you run against the Test cloud.
The android emulator will load the application and the test framework will run its automation scripts.
Running your tests against Xamarin Test Cloud
Now you can run the same tests against Xamarin Test Cloud – leveraging hundreds of devices! All you need to do is to specify against which devices you want your tests to run.
You need to have a Xamarin subscription to do this (if you don’t have one you can request a trial subscription for 14 days if you want to try it out first).
From the Test Cloud interface once you specify the devices and your test suite, it will provide you with a command that contains the subscription details (an API key), the path to your test DLLs and source path to your application file (.apk). You can run a test using a command line similar to the one below.
[code gutter=”false”]
test-cloud.exe submit aut-test-app-0.12.0.apk apikeyprovidedbyxamarinxxxyyyyyzzzzzzz –devices devicekeygeneratedbytestcloud –series "master" –locale "en_US" –assembly-dir "C:\XamarinTest\XamarinTest\bin\Debug"
[/code]
Note: This is run from the path where the Xamarin.UITest nuget was downloaded. When you follow the path, you will see ‘test-cloud.exe’ residing under ‘packages\Xamarin.UITest.[0.6.8]\tools’. This command line helps to execute the above command and connect to Xamarin Test Cloud.
As an example the same command can be run as below where I have moved to the directory where test-cloud.exe resides and executed the command from there.
The command line will show you the progress of your test run. In this example, I am running my tests against three Android devices in Test Cloud.
Test run results in Test cloud
The Test Cloud infrastructure will track results for you and you can follow them through the Test Cloud web interface as shown below.
Once a test run is finished, the Test Cloud dashboard will indicate the completed status and you can drill down to see the results from each of the devices.
Dashboard and test results
Individual test runs:
You can click on the test runs and find the results for each of the devices. On your left side you should be able to view all your tests and the individual steps:
And when you focus on a single device, you can go something like this:
As you can see, this gives us a fair amount of detail about the test run!
The way I see it, Test Cloud gives us three distinct advantages:
- Diversity of devices: the ability to run our tests against many devices helps us to see the application behaviour against far more devices than we may have had access traditionally. That to me is a huge win – to be able to compare the behaviour.
- Script once: your tests do not need to change for devices – it is the same code!
- Identify the failure points: In mobile application testing, finding the “point of failure” is far more important than to see what is working. In Test Cloud you can instantly get a feedback if certain steps do not work on any specific device.
At the same time, the wishlist of missing features is also long. I think Xamarin will have to continue to add new features to keep the offering relevant and distinct from the competition. An example is easier integration with some of the popular build management systems (it can be done in somewhat roundabout way in its current version), support for Windows Phone and a truly cross-platform environment where you should be able to run your appium (or any other tests). The power and reach will increase ten fold!
Mobile application testing has become a focal point for the industry, especially with the progress of the Appium project and its alignment with Selenium 3 and we can only hope it gets better every day. If you look back at the past two years, this space has already matured many times over and new features are being released almost every week. But so does the complexity and people’s expectation of applications.
The huge year-on-year growth of applications and increasing usage promises an exciting future for mobile applications. This is why it is key for our mobile applications to be well tested because applications become truly ‘exciting’ when they deliver what you want and not when all you get it “Sorry… something went wrong”!