Debugging crashes with Amazon Mobile Device Farm

If there is one thing I love doing as a developer, it is automation. It saves me so much time and energy and helps me make sure my code still works like a charm after all the wonderful changes I’ve just made. And believe me, my changes ARE wonderful :). Today, I’m going to show you how we use solutions like AWS Device Farm at TestFairy and how you can get more from your automation process, by adding the TestFairy SDK to your app.

First, some best practices that help me in my pipeline.

  1. I always make sure that all events that resemble a bug (crashes, failed assertions etc) get be collected across all sources in an easily inspect-able fashion.
  2. Test results are available whenever I commit new functionality to the project codebase.
  3. My team is always able to initiate extensive testing on hundreds of devices to ensure production readiness for a new release.
  4. Our Continuous Testing process depends on the fact that our human testers get automatic app updates in parallel to the automation process so that automatic and manual testing complete and cover each other.

A key factor for a success process is your envoronment:

It starts with your Issue Management system. It must be reliable and help you organize your issues, easily group them to subjects and prioritize. It could be anything between JIRA, Trello, GitHub Issues, MicroFocus or anything that helps you work. If your team isn’t using a good Issue Management System, please stop here.

Next comes your Continuous Integration platform, a.k.a “CI”. It could be the good old Jenkins or great cloud services such as CircleCI. Whichever you use, when new code is pushed on a daily basis, CI helps you automate testing your activities. Most bugs are caught here as developers get notified when a CI run fails. But every once in a while, a bug leaks from a test and the only way to detect it is scaling up the testing environment for a good check up.

As a mobile team, we know that the most challanging bugs are the ones that only happen for some users on some mobile devices and not on others. Those always escape from an emulator run and here, cloud farms such as Amazon Device Farm, Sauce Labs, Perfecto and others solve a real problem. These services offer an easy way to test on the most commonly used Android devices via their cloud. Each device runs your instrumentation separately and reports the result with useful logs and screenshots.

In this post, I’ll show how we test with Amazon Device Farm and how we use TestFairy to debug crashes, and automatically post them to JIRA.

How to Prepare your CI to Run Device Farm and User Acceptance Tests Concurrently

Start by adding the TestFairy SDK to your app for distributing it to your testers.

Create a Device Farm project with the AWS CLI. If you already created a project before, you can skip this step. Make sure your project arn is noted.

If you prefer to use command line tools, here is the command to achieve the same result. This will be only needed once for the entirety of your app development cycle.

aws devicefarm create-project –name androidTest

The call will respond with the following structure.

    "project": {
        "name": "androidTest",
        "arn": "arn:for:the:project",

Take a note of this arn and proceed by creating a test run. Make sure you replace the arn below with the one you noted.

aws devicefarm create-upload –project-arn arn:for:the:project –name myApp.1.2.3.apk –type ANDROID_APP

The response will include a presigned s3 url for you to be able upload our apk/ipa file.

    "upload": {
        "status": "INITIALIZED",
        "name": "myApp.1.2.3.apk",
        "url": "https://very.long.presigned.s3.url",
        "type": "ANDROID_APP",
        "arn": "arn:for:the:app:upload"

Upload the application to the Device Farm.

curl -T myApp.1.2.3.apk "https://very.long.presigned.s3.url"

Create another upload for the instrumentation.

aws devicefarm create-upload –project-arn arn:for:the:project –name –type TEST_TYPE 

Take a look at supported test types here.

    "upload": {
        "status": "INITIALIZED",
        "name": "",
        "url": "https://very.long.presigned.s3.url.for.tests",
        "type": "TEST_TYPE",
        "arn": "arn:for:the:test:upload"

Upload the tests via provided pre-signed url.

curl -T "https://very.long.presigned.s3.url.for.tests"

Create a device pool for the run.

aws devicefarm create-device-pool –project-arn arn:for:the:project –name myDevicePool –rules ‘[{"attribute": "PLATFORM", "operator": "EQUALS", "value": ""ANDROID""}]’

Note the arn returned as well.

    "devicePool": {
        "name": "myDevicePool",
        "arn": "arn:for:the:device:pool"

After completing the steps above, you should have 4 arns noted.

  • Project Arn
  • App Arn
  • Test Arn
  • Device Pool Arn

Provide these into the following command to run your tests on device farm.

aws devicefarm schedule-run –project-arn arn:for:the:project –app-arn arn:for:the:app:upload –device-pool-arn arn:for:the:device:pool –name testRun –test ‘{"type": "TEST_TYPE","testPackageArn":"arn:for:the:test:upload"}’


Now, it is time for the results.

To inspect what’s going on with the tests in Device Farm, simply open up your Amazon Device Farm dashboard.

Click on an individual test that crashes and scroll down to the bottom of the page.

With TestFairy’s help, you can see your crash stacktrace right at the page. Not only that, you can also create a JIRA issue in just one click, that will include the video, logs, events and stacktrace of this session.

Is that awesome or what!

Until Next Time

At TestFairy, we strongly believe that strong integration with complementary product management tools can save developers an uncountable number of hours during app development. Issue trackers, Device Farms and TestFairy combined will streamline this process and boost your productivity.

Next time, I will show how to achieve a similar result when TestFairy SDK is used to instrument not just the app, but the tests themselves! Until next time, have a nice day and enjoy hacking.

If you liked this article, please share it on Twitter, LinkedIn or Facebook, and don’t forget to mention us 🙂