Here is a crazy idea: let’s improve git push to make our CI run Espresso tests on a real device, powered by Sauce Labs Real Device Cloud. That’s right. Every time we have a change in our app code, a real Google Pixel phone will launch our test suite and show us the results, all without leaving the comfort of our Github repo. This is how we test our mobile SDKs, and it is how you should test your app too!
Ingredients
➡️ Github Actions
➡️ Sauce Labs RDC devices
➡️ saucectl
➡️ saucectl-run-action action
Where is the phone?
Sauce Labs data centers host a vast number of real mobile devices, accessible by developers to ensure their apps can confidently run across all Android phones in the wild. You can harness Real Device Cloud (RDC) devices with any testing framework of your choice.
As of 2021, Google Play shows there are at least 15000 different Android device models people use all around the world. We can’t simply buy all of those devices and test TestFairy SDK in each one of them. It is not a race we can ever hope to win without the help of a robust service. If we want to be confident that TestFairy SDK will run without bugs on any device, going cloud is our best shot. Below, you will find the exact code of how we achieve confidence for our SDK with the help of Sauce Labs RDC and Github Actions.
Steps
Let’s start with configuring our repo for saucectl. It is the tool we use under the hood to get access to RDC. saucectl cli takes a look at our current directory (i.e in CI) to decide what kind of tests to run, on which devices with the criteria we define.
Create a new folder named “.sauce” in the project root and put a configuration file named “config.yml” in it.
We will need to define the following items: testing framework, datacenter region, app paths, real phone specification.
apiVersion: v1alpha
kind: espresso
sauce:
region: us-west-1
# Controls how many suites are executed at the same time (sauce test env only).
concurrency: 2
metadata:
name: Testing My Demo App Android with espresso
tags:
- e2e
- release team
- other tag
build: Release $CI_COMMIT_SHORT_SHA
espresso:
app: ./app/build/outputs/apk/debug/app-debug.apk
testApp: ./app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
suites:
- name: "Main suite"
devices:
- name: "Google Pixel 3a"
platformVersion: 11
- id: Google_Pixel_3a_real
# Controls what artifacts to fetch when the suite on Sauce Cloud has finished.
artifacts:
download:
when: always
match:
- junit.xml
directory: ./artifacts/
Once you finish, you can execute saucectl run locally and see your test results on your Sauce Labs dashboard in real time.
This is already really cool but we want to do more. Let’s use Github Actions to run these tests, so that every time a developer makes an update to our repo, the tests are automatically executed.
Our Github repo doesn’t know that we use Sauce Labs yet, so let’s add that.
Navigate to your project secrets: https://github.com/MY_ORGANIZATION/MY_APP/settings/secrets/actions
Add a repository secret named SAUCE_USERNAME and enter your saucelabs.com username in it.
Similarly, add another repository secret named SAUCE_ACCESS_KEY and paste your access key in it.
Now that our repo knows about our Sauce credentials, we can connect to RDC inside all of our Github workflows. For a basic “test on push” scenario, we define the following workflow file (.github/workflows/test-with-saucectl.yml).
name: Test with saucectl
on: [push]
env:
SAUCE_USERNAME: ${{secrets.SAUCE_USERNAME}}
SAUCE_ACCESS_KEY: ${{secrets.SAUCE_ACCESS_KEY}}
jobs:
main:
timeout-minutes: 20
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Setup Java
uses: actions/setup-java@v1
with:
java-version: 11
- name: Setup Android SDK
uses: android-actions/setup-android@v2
- name: Compile
run: |
./gradlew app:assembleDebug
./gradlew app:assembleDebugAndroidTest
shell: bash
- name: Test espresso with saucectl
uses: saucelabs/saucectl-run-action@v1
with:
testing-environment: ""
concurrency: 10
Finally, we commit our changes and push.
Once our changes are merged to main branch for the first time, Github will be able to start scheduling actions whenever a new contribution arrives. ⭐
Try now!
If you’d like to see this in action right now, just fork our open source demo project on Github and update your repo secrets.
Until then, #staysafe
