Introducing and Open Sourcing Flashback, an Internet Mocking Tool

Editor's note: This blog has been updated.

This post was updated on June 15, 2017

Co-authors: Shangshang Feng, Yabin Kang, and Dan Vinegrad

At LinkedIn, we often develop web applications that need to interact with third-party websites. We also employ automatic testing to ensure the quality of our software before it is shipped to production. However, a test is only as useful as it is reliable. With that in mind, it can be highly problematic for a test to have external dependencies, for instance on a third-party website. These external sites may change without notice, suffer from downtime, or otherwise become temporarily inaccessible, as the internet is not 100 percent reliable. If one of our tests relies on being able to communicate with a third-party website, the cause of any failures is hard to pinpoint. A failure could be due to an internal change at LinkedIn, an external change made by the maintainers of the third-party website, or an issue with the network infrastructure. As you can imagine, there are many reasons why interactions with a third-party website may fail. So, you may wonder, how will I deal with this problem? The good news is that there are many internet mocking tools that can help. One such tool is Betamax. It works by intercepting HTTP connections initiated by a web application, and then later replaying them. For a test, Betamax can be used to replace any interaction over HTTP with previously recorded responses, which can be served very reliably.

Initially, we chose to use Betamax in our test automation at LinkedIn. It worked quite well, but we ran into a few problems, such as:

  • For security reasons, our test environment does not have internet access; however, as with most proxies, Betamax requires an internet connection to function properly.

  • We have many use cases that require using authentication protocols, such as OAuth and OpenId. Some of these protocols require complex interactions over HTTP. In order to mock them, we needed a sophisticated model for capturing and replaying the requests.

To address these challenges, we decided to build upon ideas established by Betamax and create our own internet mocking tool, called Flashback. We are also proud to announce that Flashback is now open source.

What is Flashback?

Flashback is designed to mock HTTP and HTTPS resources, like web services and REST APIs, for testing purposes. It records HTTP/HTTPS requests and plays back a previously recorded HTTP transaction—which we call a "scene"—so that no external connection to the internet is required in order to complete testing.

Flashback can also replay scenes based on partial matching of requests. It does so using “match rules.” A match rule associates an incoming request with a previously-recorded request, which is then used to generate a response. For example, the following code snippet implements a basic match rule, where the test method “matches” an incoming request via the URL.

HTTP requests generally contain a URL, method, headers, and body. Flashback allows match rules to be defined for any combination of these components. Flashback also allows users to add passlist or blocklist labels to URL query parameters, headers, and the body.

For instance, in an OAuth authorization flow, the request query parameters may look like the following:

    oauth_consumer_key="jskdjfljsdklfjlsjdfs",
    oauth_nonce="ajskldfjalksjdflkajsdlfjasldfja;lsdkj",
    oauth_signature="asdfjaklsdjflasjdflkajsdklf",
    oauth_signature_method="HMAC-SHA1",
    oauth_timestamp="1318622958",
    oauth_token="asdjfkasjdlfajsdklfjalsdjfalksdjflajsdlfa",
    oauth_version="1.0"

Many of these values will change with every request, since OAuth requires clients to generate a new value for oauth_nonce every time. In our testing, we need to verify values of oauth_consumer_key, oauth_signature_method, and oauth_version while also making sure that oauth_nonce, oauth_signature, oauth_timestamp, and oauth_token exist in the request. Flashback gives us the ability to create our own match rules to achieve this goal. This feature lets us test requests with time-varying data, signatures, tokens, etc. without any changes on the client side.

This flexible matching and the ability to function without connecting to the internet are the attributes that separate Flashback from other mocking solutions. Some other notable features include:

 

  • Flashback is a cross-platform and cross-language solution, with the ability to test both JVM and non-JVM (C++, Python, etc.) apps.
  • Flashback can generate SSL/TLS certificates on the fly to emulate secured channels for HTTPS requests.

 

How to record an HTTP transaction

Recording an HTTP transaction for later playback using Flashback is a relatively straightforward process. Before we dive into the procedure, let us first lay out some terminology:

  • A Scene stores previously-recorded HTTP transactions (in JSON format) that can be replayed later (example).
  • The Root Path is the file path of the directory that contains the Flashback scene data.
  • A Scene Name is the name of a given scene.A Scene Mode is the mode in which the scene is being used—either “record” or “playback.”
  • A Match Rule is a rule that determines if the incoming client request matches the contents of a given scene.
  • Flashback Proxy is an HTTP proxy with two modes of operation, record and playback. 
  • Host and port are the proxy host and port.

In order to record a scene, you must make a real, external request to the destination, and the HTTPS request and response will then be stored in the scene with the match rule that you have specified. When recording, Flashback behaves exactly like a typical “Man in the Middle” (MITM) proxy—it is only in playback mode that the connection flow and data flow are restricted to just between the client and the proxy.

To see Flashback in action, let us create a scene that captures an interaction with example.org by doing the following:

Step 1: Checkout Flashback source code: git clone https://github.com/linkedin/flashback.git

Step 2: Start the Flashback admin server: `./startAdminServer.sh -port 1234`

Step 3: Start Flashback Proxy:

Note the Flashback example above will be started in record mode on localhost, port 5555. The match rule requires an exact match (match HTTP body, headers, and URL). The scene will be stored under /tmp/test1.

Step 4: Flashback is now ready to record, so use it to proxy a request to example.org: `curl http://www.example.org -x localhost:5555 -X GET`

Step 5: Flashback can (optionally) record multiple requests in a single. To finish recording, shutdown Flashback: `curl "http://localhost:1234/admin?action=shutDownFlashback" --data '{}'`

Step 6: To verify what has been recorded, we can view the contents of the scene in the output directory (/tmp/test1). It should look like this.

It’s also easy to use Flashback in your Java code.

To replay a previously-stored scene, use the same basic setup as is used when recording; the only difference is that you set the “Scene Mode” to “playback” in Step 3 above:

One way to verify that the response is from the scene, and not the external source, is to disable your internet connectivity temporarily when you go through Steps 1-6. Another way is to modify your scene file and see if the response is the same as what you have in the file.

Here is an example in Java:

How to record and replay an HTTPS transaction

The process for recording and replaying an HTTPS transaction with Flashback is very similar to that used for HTTP transactions. However, special care needs to be given to the security certificates used for the SSL component of HTTPS. In order for Flashback to act as a MITM proxy, it is necessary to create a Certificate Authority (CA) certificate. This certificate will be used during the creation of the secure channel between the client and Flashback, and will allow Flashback to inspect the data in HTTPS requests it proxies. This certificate should then be stored as a trusted source so that the client will be able to authenticate Flashback when making calls to it. For instructions on how to create a certificate, there are many resources like this one that can be quite helpful. Most companies have their own internal policies for administering and securing certificates—be sure to follow yours!

It is worth noting here that Flashback is intended to be used for testing purpose only. Feel free to integrate Flashback with your service whenever you need it, but note that the record feature of Flashback will need to store everything from the wire, then use it during the replay mode. We recommend that users pay extra attention to ensure that no sensitive member data is being recorded or stored inadvertently. Anything that may violate your company’s data protection or privacy policy is your responsibility.

Once the security certificate is accounted for, the only difference between HTTP and HTTPS in terms of setup for recording is the addition of a few further parameters.

  • rootCertificateInputStream: this can be either a stream or file path that indicates the CA certificate’s filename;

  • rootCertificatePassphrase: this is the passphrase created for the CA certificate;

  • certificateAuthority: these are the CA certificate’s properties.

The code used to record an HTTPS transaction with Flashback, including the above terms, is below.

Replaying an HTTPS transaction with Flashback uses the same process as recording. The only difference is that the scene mode is set to “playback.” This is demonstrated by the code below.

Supporting dynamic changes

In order to allow for flexibility in testing, Flashback lets you dynamically change scenes and match rules. Changing scenes dynamically allows for testing the same requests with different responses (such as success, time_out, rate_limit, etc.). Scene changes only apply to scenarios where we have POSTed data to update the external resource. See the diagram below as an example.

Flashback-Overview

Being able to change the match rule dynamically allows us to test complicated scenarios. For example, we have a use case that requires us to test HTTP calls to both public and private resources of Twitter. For public resources, the HTTP requests are constant, so we can use the “MatchAll” rule. However, for private resources, we need to sign requests with an OAuth consumer secret and an OAuth access token. These requests contain a lot of parameters that have unpredictable values, so the static “MatchAll” rule wouldn’t work.

Use cases

At LinkedIn, Flashback is mainly used for mocking different internet providers in integration tests, as illustrated in the diagrams below. The first diagram shows an internal service inside a LinkedIn production data center interacting with internet providers (such as Google) via a proxy layer. We want to test this internal service in a testing environment.

Flashback-Detail-1

The second and third diagrams show how we can record and playback scenes in different environments. Recording happens in our dev environment, where the user starts Flashback on the same port as the proxy started. All external requests from the internal service to providers will go through Flashback instead of our proxy layer. After the necessary scenes get recorded, we can deploy them to our test environment.

Flashback-Detail-2

In the test environment (which is isolated and has no internet access), Flashback is started on the same port as in the dev environment. All HTTP requests are still coming from the internal service, but the responses will come from Flashback instead of the internet providers.

Flashback-Detail-3

Future directions

We’d like to see if we can support non-HTTP protocols, such as FTP or JDBC, in the future, and maybe even give users the flexibility to inject their own customized protocol using the MITM proxy framework. We will continue improving the Flashback setup API to make it easier to support non-Java languages.

Now available as an open source project

We were fortunate enough to present Flashback at GTAC 2015. At the show, several members of the audience asked if we would be releasing Flashback as an open source project so they could use it for their own testing efforts.

"Mock the Internet" talk at Google Test Automation Conference (GTAC) 2015

We’re happy to announce today that Flashback is now open-sourced and is available under a BSD two-clause license. To get started, visit the GitHub repo at https://github.com/linkedin/flashback.

Acknowledgements

Flashback was created by Shangshang Feng, Yabin Kang, and Dan Vinegrad, and inspired by Betamax. Special thanks to Hwansoo Lee, Eran Leshem, Kunal Kandekar, Keith Dsouza, and Kang Wang for help with code reviews. We would also thank our management—Byron Ma, Yaz Shimizu, Yuliya AverbukhChristopher Hazlett, and Brandon Duncan—for their support in the development and open sourcing of Flashback.