Instrumenting Test Suites on Amazon Device Farm

At TestFairy, we automate our tests whenever we can. We’re using AWS Mobile Device Farm for running our automation and trying to solve as many problems we can, with code.

In a previous post I showed you how to see your stacktrace when having a crash with Amazon Device Farm, and this post is about another problem we solved.

The challenge we had was how to identify the root cause whenever a test fails. To solve this, we run our Android instrumentation tests in a test app hosting a working copy of our SDK in the instrumentation APK. Our approach is open source and you are welcome to modify it for your own needs, no strings attached.

I honestly believe, AWS Device Farm is a remarkable service that allows you to “concurrently” run your test suite on real Android devices connected to the AWS cloud. It is an essential tool in a developer’s toolbox, especially if they have limited access to mobile devices.

  • It is infeasible to buy and maintain a bunch of test devices. especially nowadays since we have to work from home to protect our loved ones.
  • It is often the case that some of the devices I need to test are not available in my country.
  • Android’s liberal approach to allowing OS modifications by OEM manufacturers doesn’t help either.

Device Farm solves all of these by exposing a massive collection of devices to our hands. We embellish our test suite with the TestFairy SDK to be able to inspect what’s going on during a test from our TestFairy dashboard.

Let me accompany you through the steps.

Create APKs

In order to test your app with device farm, you will need one apk for your app and one apk for your instrumentation tests (androidTest).

Run ./gradlew app:assembleRelease and ./gradlew app:assembleConnectedAndroidTest to create your apk files.

Add TestFairy Test Runners

  • Add these dependencies to your app’s build script.
dependencies {
    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
    androidTestImplementation 'testfairy:testfairy-android-sdk:1.+@aar'
}
  • Copy these files into your project.
  • Modify ExampleTestBase.java to set up your TestFairy app token and server endpoint.
  • Prepare your test code like this.
@RunWith(TestFairyAndroidJUnit4ClassRunner.class)
public class ExampleInstrumentedTest extends ExampleTestsBase {

	@Test
	public void exampleTest() {
		// Test goes here
	}
}
  • If running with our class runner is not an option, you can also proceed with manual embellishment like this.
public class ExampleOptionallyInstrumentedTest extends ExampleTestsBase {

	@Rule
	public TestName testName = new TestName();

	@Test
	public void testAppContextPackageNameWithTestFairy() {
		String name = String.format("%s.%s", getClass().getSimpleName(), testName.getMethodName());
		TestFairyInstrumentationUtil.wrapInstrumentation(name, new TestFairyInstrumentationUtil.InstrumentationWrapper() {
			@Override
			public void onRecord() {
				// Test goes here
			}
		});
	}
}

Our open source instrumentation repository showcases a bunch of fine grained approaches as well. Feel free to choose the one that suits your needs.

Upload Tests to Device Farm

  • Build your APKs and proceed to your device farm console.
  • Create a device farm project
  • Create a new run and choose Android/Apple. Upload your app’s apk. If you upload your instrumentation apk by accident, you will receive an error during your tests saying “Unable to find instrumentation target package: <yourpackage>” or something similar.
  • Configure your test as an “Instrumentation” and upload your test apk.
  • Create a device pool or choose an existing one. Then proceed with the environment setup as usual. Make sure you allocate enough time per device so that your tests won’t prematurely stop due to timeouts.

Examining the Results

When AWS allocates an idle device to you, your tests will run and you will be reported the result by the Device Farm dashboard. A run with a bunch of failed tests will look similar to this.

If you included the TestFairy SDK in your instrumentation, you will also be able to take a look at what went wrong with each of your tests. Proceed to your TestFairy dashboard and find the project named after your app. All the sessions you are shown here will match with its corresponding Device Farm test case. Device Farm aggregates similar runs under a uniquely identifiable error group. That is itself useful but if you think a closer look is necessary, TestFairy dashboard will include individual runs as well.

A session with an error will include the entire stacktrace of the failed test case.

From this point, you can make a few changes in your app code, run the tests again, and eliminate issues one by one until you are shown this excellent screen which we all love.

Get the results with less clicks

Did I mention I love automation? Well, navigating through a bunch of dashboards isn’t that automatic. Luckily, TestFairy QuickView Chrome Extension is open source and it was easy to fork and write a bunch of Javascript to do the magic for me (now on Firefox as well!).

If the extension is loaded, you will get the same results without ever leaving the AWS Device Farm dashboard. Take a look at the files section of your tests to see it in action.

Moreover, you will be able to create a ticket in your favorite bug tracker, provided you configured it in your TestFairy dashboard before.

Right below the stack trace, you will see a button named Create Bug.

Click that and in a few seconds, without leaving Device Farm, you will be notifying your team members about the issue you encountered. All session data, trimmed to be able to focus on this specific bug, will automatically be attached to the issue so that your team members have all the necessary info including the video to figure out what went wrong.

Stay Tuned

In our previous post, we discussed how important it is to have Issue Tracking and Continuous Integration to figure out bugs that only happen in a few selected devices in a timely fashion. Today, we showed how that is possible thanks to Device Farm’s remarkably rich device pool and TestFairy QuickView Extension.

We will be back with a bunch of examples for Appium tests in the future as well. Let us know if you think we must integrate with your favorite testing framework.

Until next time – #staysafe