Open Sourcing Dex Test Parser
February 10, 2017
Part of LinkedIn’s 3x3 pipeline for building and releasing mobile applications involves making the cycle for testing changes to the app as fast and reliable as possible. Achieving this goal requires a complete automation pipeline for every step, from code commit to production release and every test in between. This pipeline needed to be fast enough to fit inside our three-hour window, and reliable enough for us to maintain confidence in our releases.
As we’ve written before, automated tests enable us to make rapid improvements to the mobile LinkedIn application, avoid delays, and eases cross-functional planning with our partners in areas like product and marketing.
Maintaining this confidence at scale can sometimes be challenging. Back in August, we open sourced the Test Butler project, which helps stabilize and control the Android emulators, but we found other problems that needed to be addressed as well.
Out of the box, Android’s default testing framework allows you to run tests using a variety of commands, but they all come down to two categories:
- Run a single test method;
- Run a group of test methods specified by some filter (every test in a certain class, every test with a given annotation, etc.).
The latter category is where we start to see problems. In order to figure out the exact set of test methods to run, Android will start up your application and dynamically scan the Java classpath for any methods matching the specified filter rules. Once it finds all of the tests, it will run them in sequence and report back the results.
There are two main problems here:
- Scanning the classpath dynamically will cause any static state (fields, initializers, etc.) in your app code to be initialized immediately. This doesn’t normally happen in production, where static state is initialized on-demand, and can lead to bugs that only happen in tests, or worse: bugs that only happen in production, but not tests.
- Since tests are found and queued to run from inside your application process, if a test does its job and actually finds a crash, that means that none of the remaining tests will get a chance to run. If a code change introduces many separate test failures, the developer will only see the first failure until it is fixed...and then they will only see the second failure, and so on.
An obvious answer is to always run one test method per launch of your application process. This is what we set out to do, in order to stabilize our tests. However, we needed a way to identify all of the test methods in our app so that our test harness could run them all individually.
Our first approach for this problem was to create an annotation processor that looked for any methods annotated with @Test and saved all of the fully qualified method names to a text file that our test harness could read. This worked, and we used this approach for quite a while. It suffered from a few problems however:
- The annotation processing added an extra step to build time and introduced an extra test artifact (the text file of method names) that needed to be saved if we wanted the ability to run tests later without building again.
- Legacy JUnit 3 tests were unsupported because they don’t use annotations to identify test methods.
We wanted to find a way to address these problems while still allowing our test harness to run each of our tests individually. We realized that all of the info we needed existed in our test apk; we just needed a way to get it out.
Introducing Dex Test Parser
Android apk files store their compiled code in the “Dalvik Executable” format: .dex files. Dex Test Parser is a single-purpose project that reads the bytecode of these dex files and collects the list of all the test methods contained in the apk.
This solution has several benefits:
- Our test runner can determine what tests to run without building the source code. In fact, we don’t even need the source code to be present anymore! This enables features like distributing tests across multiple build machines to be implemented much more easily.
- We can transparently support both JUnit 3 and JUnit 4 tests.
Using Dex Test Parser
Getting started with Dex Test Parser is extremely simple: there’s only one method. You can call this method from Java to get a list of all fully-qualified test names ready to be run by Android instrumentation testing:
List<String> testMethodNames = DexParser.findTestNames(apkPath);
You can also use the Dex Test Parser directly from the command line if you prefer. This will create a file called AllTests.txt in the specified output directory.
java -jar parser.jar path/to/apk path/for/output
Open sourcing Dex Test Parser
We’re happy to announce that Dex Test Parser has been released under the BSD 2-Clause license and the code is available on GitHub. Contributions and suggestions are welcome!
Dex Test Parser was inspired by the Google presentation “Going Green: Cleaning up the Toxic Mobile Environment” and created by Drew Hannay. The test infrastructure team at Google also helped provide us with valuable advice on how to stabilize our Android tests.
This is part of a series of blog posts sharing our learnings and tools with the wider mobile development community. Look for more posts about mobile development on the LinkedIn Engineering blog!