There is no such thing as a completely perfect test suite, if you are paranoid enough. In TestFairy, we put our tinfoil hats before we run our tests. The SDK is, in our humble opinion, a stateful salad to be precise. Even if the code coverage is 100% and all tests are successful, it is not an indication of a bug free SDK. There is always the possibility of a state, creeping somewhere deep, waiting for false assumptions to crash your app without mercy.
Multi threaded code, by virtue, can be really hard to test. When you want to instrument multiple threads in a single test case, you also need to manage their life-cycle and make sure none of them is leaked. It is inherently a hard problem if your threads execute long running jobs and utilize a fault recovery scheme such as “respawn on exception”.
When a thread is leaked, it is likely that none of your running tests will be affected. especially if you have a big thread pool size.
It is also possible to miss a leak if two consecutive tests do not use the same kind of worker threads for their assertions. It is widely accepted as bad practice to share resources between tests cases after all.
For all the possible problems mentioned above, we decided go with a brute force approach last month. What if we run our tests every X hour with a random order, would it reveal such bugs? We assumed it would.
All of our tests are instrumentation tests run on device. We didn’t care if class files are run in a predictable order, since they never share the SDK. But for each test class, the methods should be run randomly to make sure every once in a while, a false assumption will trigger a stall/crash/assertion failure.
The Code
It is funny that the only piece of additional code we added was like below.
public class ShuffledInstrumentationRunner extends BlockJUnit4ClassRunner {
public ShuffledInstrumentationRunner(Class<?> klass) throws InitializationError {
super(klass);
}
@Override
protected List<FrameworkMethod> computeTestMethods() {
List<FrameworkMethod> methods = new ArrayList<>(super.computeTestMethods());
Collections.shuffle(methods);
return methods;
}
}
Our custom runner overrides computeTestMethods() to meddle with the method list. To finalize, we marked each of our test classes like below to specify our new runner as the main runner.
@RunWith(com.testfairy.ShuffledInstrumentationRunner.class)
public class WorkerThreadTest {
@Before
public void createThreads() {
...
}
@After
public void killThreads() {
...
}
@Test
public void checkThreads() {
...
}
}
How to Check?
In Android Studio, filter your Logcat with “TestRunner: finished:” and look at the lines. If you run your instrumentation multiple times, the order should differ on each run.
Credits
-
Photo by Jeff Frenette on Unsplash
