Open Sourcing Test Butler
Reliable Android Testing, at Your Service
August 4, 2016
Automated testing is a key component to LinkedIn’s 3x3 strategy for releasing mobile applications. As we developed the new LinkedIn Android app, we found that our tests had a major problem: our testing environment was unreliable, so our tests failed intermittently. We needed a solution that would let us rely on our tests to inform us when there was a problem with the app, not the testing environment. For this reason, we created and open sourced Test Butler, a reliable Android testing tool. LinkedIn runs over one million tests each day using Test Butler and we believe that it can provide a benefit to anyone running Android tests.
UI testing on Android can be unstable for a variety of reasons: tests could depend on being run in a certain order, or depend on some shared application state that may not be reset between tests. These types of flakiness are largely up to app developers to address, but the more insidious issues can occur within the Android device itself.
If you’ve ever tried to run a large number of UI tests on Android, you may be familiar with some of the ways the emulator can be unreliable. Animations must be disabled, so that Espresso tests can run reliably. Tests may fail because the emulator CPU randomly goes to sleep, WiFi turns off unexpectedly, or rogue accelerometer data causes the device to change orientations. If a system app crashes on the emulator in the background, the resulting crash or app-not-responding dialog will cause Espresso UI tests to fail. At LinkedIn, we’ve even seen cases where the lock screen on the emulator is randomly triggered, causing tests to fail. This inconsistent behavior was causing our developers to lose trust in our tests and question why we were even writing them in the first place.
LinkedIn is not the first company to encounter testing environment instability, so as we set out to stabilize our tests, we searched for existing solutions that could help. The information we found was pieced together from various StackOverflow links and blog posts, and many of the solutions required adding invasive Android permissions.
For instance, we can prevent the lock screen from appearing by holding a KeyguardLock, but this requires the DISABLE_KEYGUARD permission, which the LinkedIn app definitely does not need. A common recommendation for this problem is to add a debug-specific AndroidManifest.xml file so that these permissions don’t make it into release builds. This method still felt dangerous to us, since developers may not realize these extra permissions are in effect while they are implementing features. And on a more fundamental level, we wanted to avoid leaking testing-specific code into our production codebase.
Another example is disabling animations for Espresso tests. There is an existing process for doing this programmatically, but it requires three distinct steps:
- Add the SET_ANIMATION_SCALE permission to your app
- Grant this permission to your app using Android Debug Bridge (ADB)
- Change the setting in your test code at runtime using reflection
We thought this entire process was too complicated, and we hadn't even solved the crash dialog problem yet!
Introducing Test Butler
Test Butler includes an Android library that your test code can depend on, as well as a companion Android app APK that you can install on your Android emulator before running tests. It provides a number of features that will make your testing safer and more reliable:
- Signature-level permissions: Test Butler is signed with the system keystore for the stock Android emulator, which means it will automatically be granted any signature-level permissions it requests without the need to grant with ADB.
- Disable crash & App Not Responding (ANR) dialogs: Since it’s a system app, Test Butler can install a custom IActivityController on the emulator and suppress system crash & ANR dialogs from appearing and causing test failures.
- Disable animations: Test Butler handles all the steps of disabling animations with a single line of code in your app!
- Lock screen, WiFi, CPU: Test Butler holds locks on the keyguard, WiFi radio, and CPU to ensure they won’t go to sleep and cause test failures.
In addition to system stability enhancements, Test Butler can also help handle other testing tasks that require extra permissions for your app:
- Enable/disable WiFi: Test Butler allows tests to simulate a situation where WiFi is not connected or changes availability at some point.
- Change device orientation: Tests can manually set the orientation of the device during test execution.
- Set location services mode: Test Butler lets your code simulate different location services modes, like battery saver or high accuracy.
- Set application locale: Tests can set a custom Locale object for their application to simulate running the app in another language.
Using Test Butler
Getting started with Test Butler is easy:
- Install the Test Butler app on your emulator before starting your tests.
- Add the Test Butler library as an androidTestCompile dependency in your build.gradle file
- Add the following code to your test runner class:
The TestButler.java class provides other methods you can call from your tests to modify device settings.
Open sourcing Test Butler
We’re happy to announce that Test Butler has been released under the Apache 2.0 license and the code is available on GitHub. Contributions and suggestions are welcome!
Test Butler was created by Drew Hannay and inspired by the Google presentation “Going Green: Cleaning up the Toxic Mobile Environment”. We also had a chance to meet with some of the test infrastructure developers at Google, who provided valuable advice and guidance. Special thanks to Robin Sylvan and Salman Ahmed for help with code reviews.