Live Testing with Play Framework and Ember

Here at LinkedIn, Ember.js is one of the many frameworks we use to develop both internal and external-facing web applications. It provides the power to create ambitious web applications while providing extensive testing capabilities to ensure the quality of the product.

Making live testing efficient

One of the ways to ensure quality with our Ember applications is with live (or end-to-end) testing. It is a methodology used to test whether the flow of an application is performing as designed from start to finish. At LinkedIn, we test with real user scenarios, and our focus point is to verify that our ecosystem (app, API, and downstream services) and that the deployment are working as per business rules/functionality defined for identified user flows. But as we know, live tests can be expensive due to the manual intervention involved. Also, they can take a long time to execute and can be flaky due to the many moving parts involved (partner deployments, inconsistency in API latency, UI changes, and more). So, our strategy has been to leverage a small set of live tests to achieve the above goals.

There are multiple solutions that exists today to test web applications end-to-end, most commonly Selenium webdriver or casperJS. These solutions interact with the application similarly to how an end user would in a black box manner. But there are pitfalls with these solutions because their black box approach is prone to explicitly waiting for elements and not knowing when rendering is complete.

This is where Ember.js excels, because Ember’s promise-based actions will execute and continue only once all rendering or processing is complete and the promise is returned. Another reason for using Ember.js for live testing is to have framework parity with what developers are already familiar with. Since our new approach toward development is to have developers write both app and test code, this allows them write tests with syntax they normally use, instead of having to switch between languages.

With these positives points, we explored the possibilities of live testing with Ember.js. The results are described below.

Testing with Ember.js

Before diving into the problems, let's briefly talk about how testing is done in Ember.js. (For more comprehensive details on Ember.js, check out the official docs.)

Ember.js tests are typically executed locally via testem, by spinning up an Express server that serves the Ember application and corresponding assets. A dev build containing the app, test code, and test support and the test page that serves the tests is built. Testem opens browsers pointing to the test page served by the express server. Tests execute, and a test complete hook is called and reporting is displayed.

Live testing with Ember.js

Using Ember’s existing testing functionality to do live testing means that the test needs to live and be served along with the application, similar to how testem serves the app locally.

When we deploy the Ember app, a production build will be packaged up and deployed. A production build only contains the minified version of all application assets, with no unnecessary dev dependencies or tests. So tests and test assets will need to be part of the deployed package.

When testem serves the app and the test index.html, the test index.html is actually served as a static asset. This is because the application does not recognize a /test route, since it is not part of router.js. Hence, we need a way to expose the test route, and have its access restricted only to LinkedIn test execution.

Making live testing work with Ember

There were several hurdles to get over to enable live testing in Ember:

1. Tests must not be served to production members;

2. Tests must be available as part of the build;

3. Tests must be executable against deployed server;

4. Tests must only be executable by LinkedIn test execution runs.

In early versions of ember-cli, tests were built as part of the application.js. So if we wanted to have tests deployed, we could not have tests served to the users. With the help of Trent Willis and the Ember community, changes were made so that ember-cli builds a separate test.js that contains all test code. This created a separation of app and test code to kickstart this experiment.

Secondly, work was done during the build process so that the production build of our application also contains the test-related support files. To do this, we added an in-app add-on to use the preprocessTree hook to return a tree with the necessary test assets to be part of the build. This excludes files that are unnecessary for app deployment, such as unit or acceptance tests, to keep test assets lightweight.

Once the production build with tests was available, we updated the Play server that serves the Ember application so that it recognized a customized test route to serve the test/index.html. We also made sure to restrict access to this route to only LinkedIn test execution.

The last step was to access this test page that is being served by our production servers, and have our test execution pipeline report back on the tests results. To do this, we used Casper.js to bring up the test page in phantomjs and report back the test results once the tests were completed.

Conclusion

With the work discussed above, live tests with Ember.js were enabled. We were able to validate that our application was running fine from the user perspective during deployment as well as conduct periodic health checks. Tests are executed and reported within minutes (1 minute to check out the repo, 2 minutes to execute casper script), with each test executing in seconds.

Although there was additional work to be done for additional validation, better reporting, and executing on additional browsers, this was a good start and a good experiment to use Ember.js for end-to-end testing.