Test Strategy for Samza/Kafka Services

Over a decade ago, test strategies invested heavily in UI-driven tests. Backend and mid-tier services were tested using automated UI-based tests. While UI-based tests have certain merits, such as testing user flows, they are also time-consuming and fragile. The strong coupling of tests with UI caused several problems:

• Tests needed frequent modification due to any major or minor UI changes;

• When a UI test failed, it could be hard to identify the root cause;

• When backend services were tested through UI, they had to be deployed simultaneously.

At LinkedIn, we often had long release cycles due to the unpredictability caused by UI-based tests. That motivated us to take fresh look to address these pain points and improve the overall release velocity. We set out with the goal to transform the release cycle from weeks to hours by moving away from these types of tests.

This post discusses key learnings from de-coupling UI-based tests from backend testing, which we employed for one of the real-time systems at LinkedIn, Content Analytics. These changes to our testing approach, when applied at scale across multiple services, helped us reduce our commit-to-release time from weeks to hours.

Content Analytics

Content Analytics is a platform that provides useful metrics and insights into various social engagement activities (such as shares, likes, and comments) performed by members on a post. For example, it allows us to quickly find answers like “Where is the major base of your audience? Which part of world they are from? What percentage of your connections engage with your content?”

At a high level, various Kafka tracking events are emitted when a member interacts with a post on LinkedIn. The Content Analytics platform processes these streams of events to perform aggregation and decoration before making the data available for clients to consume. More details about the system are covered in the post New Analytics for Sharing on LinkedIn: See Who’s Viewed Your Post.

At LinkedIn, we use stream processing apps like Apache Kafka and Apache Samza to process real-time events.

Change of perspective

Our primary goal when redesigning these tests was to improve release velocity while maintaining a high standard of quality. We made the following changes to our test approach:

• We defined dedicated, non UI-based test-strategies for backend and mid-tier services.

• We began testing and releasing backend and mid-tier services in isolation.

• We adopted a fast-fail approach that identified issues earlier in release cycle.

Our tests need to be consistent and quick to aid fast release cycles. We decided to use the recommended Test Pyramid to ensure the proper distribution of tests:

1. Most: Unit tests, which validate individual units of code and core business logic;

2. Few: Scenario tests, which ensure multiple units of code in a service work as expected;

3. Least: End-to-end tests, which validate core user flows. Tests usually span across multiple services and act as sanity check.

Scenario test framework
Scenario tests are designed to validate the System under Test (SuT) independent of any downstream or upstream dependencies. This is done by mocking or stubbing all external dependencies. We use Mockito to mock downstream calls. The figure below shows the high-level architecture for the test framework.

The scenario tests consist of three primary phases:

• Phase 1 - Setup and configuration: Prior to executing any test, the test framework primes Samza jobs with mock event topics. Events with mock topics are produced, and SuT will process these events like any other real-time Kafka event.

• Phase 2 - Mocking, stubbing, and test logic: We use Mockito’s @Mock annotation to mock downstream calls and Samza tasks. The test generates Kafka events to simulate the desired payload.

• Phase 3 - Validation: The Content Analytics platform produces Kafka events based on the processed payload. The test consumes these events and validates the fields against expected values.

Here is a sample of a complete test.

Success story

Based on the time savings and cost efficiency of this model, we have found great value in moving towards fast feedback and release cycles. We transformed 81 backend and mid-tier services at LinkedIn by reducing the commit-to-release time from weeks to less than two hours, while improving the overall quality. The following are some key success metrics from this new test strategy:

• The entire test suite only takes 2-3 minutes to complete and it is executed as a sanity check against every commit.

• Backend services are tested on stand-alone mode, without any UI tests.

• Tests are reliable and act as a safety net to catch defects early in the development cycle.

• No additional test/deployment setup is required.

Acknowledgements

Developing this testing strategy involved the work of several overlapping teams at LinkedIn. In particular, we’d like to thank Rashmi Jain, Mihir Gandhi, Robin Sylvan, Bharath Kumarasubramanian, and the Content Analytics team.