Play Framework: Democratizing Functional Programming for modern Web Programmers

Editor's note: today, we have a guest post from Sadek Drobi, one of the creators of the Play Framework and the CTO of Zenexity. We worked closely with Sadek to bring the Play Framework to LinkedIn and we are excited to have him share some insights with you about the motivation behind Play.

At Zenexity, we build web applications all the time, at different sizes and scales. In recent years, we've seen dramatic growth in Software as a Service (SaaS) and Web Services architectures. This architecture offers many scalability advantages, but it also introduces many new challenges, including data distribution, managing various data formats, flexibility, adaptability, big data, and stream processing.

The Play Framework is an open source framework for web application development created at Zenexity in 2007 and opened to the community in 2009. It targets the Java Virtual Machine (JVM) and focuses on simplicity, developer productivity, and HTTP/Web friendliness. As we got together with Guillaume Bort (founder of the framework) in 2011, we decided to start from scratch and design the Play Framework with a toolset adapted to the new challenges faced by web programmers.

We started working on a brand new architecture and design of Play, with a focus on a programming paradigm that we believe is proven for handling these challenges: functional programming. In this blog post I'd like to share with you the background on what motivated - and continues to motivate - the architecture, design, and evolution of Play.

Motivation

Many classical web frameworks assume a single-box architecture and organize their design around static, unique, change-intolerant data models that try to centralize the entire app's business logic. This approach falls short in a number of areas that motivated the design of Play:

• Data representations: there is a distinction between the raw data and its representation in a single use case. For example, consider a simple piece of data, such as a tweet. If your app happens to use a tweet, then the tweet is rarely your central and single model; it is merely one piece of data among many. Moreover, your app might not be interested in all the data a tweet holds or even need different parts of a tweet at different times: you may just want the text in one use case but the geo location and hashtag in another use case. Therefore, we need tooling that makes it easy to transform and adapt raw data into a variety of representations.
• Data distribution: when breaking your app into different services, you need to exchange data between them. Classical approaches based on runtime reflection and discovery can be heavy-weight, fragile, and have very little programatic power when faced with corner cases. We need a different programming model that is adapted to these kinds of challenges.
• Big data: large data sets present yet another challenge to the classical toolset. Apps will spend more time treating, manipulating, and analyzing data to extract meaningful information. Combined with multi-core processors, it's clear we need new model of programming that makes it easier to deal with large sets of data.
• Real-time data streams: you know what's cooler than big data? Infinite data. When faced with real-time events, you cannot pause the world to launch your batch data treatment. You need to have tools to consume, process, and distribute continuous streams of data in real time.
• Composability: web applications evolved from simple static html pages to include ever more complex and dynamic logic. We need mechanisms to manage this complexity by breaking it down into independent pieces so that we can reason about and focus on one thing at a time. Composability is at the heart of managing complexity and we need our toolset to offer us composable solutions to our challenges.

Introducing functional programming. Gradually.

We believe that functional programming excels at addressing the modern web programming challenges described above. However, even though functional programming has been gaining a lot of popularity lately, we still cannot count on mainstream developers being fluent, or even familiar, with the different concepts and techniques of functional programming. Functional programming is quite different from traditional imperative programming, and it takes time to become comfortable with the new mindset.

This has inspired one more essential motivation of Play design: a progressive learning curve. The complexity of the tools should grow relative to the complexity of the task at hand. The more power you need, the more techniques you need to learn: simple!

Along the same lines, it is important for us to keep the same progressive learning curve between the Play Java and Scala API's. It is perfectly reasonable that a developer can start with Java and progressively learn more about the power of functional programming, slowly introducing Scala where appropriate. In this blog post I will use Scala for illustration, but be aware that the Java API is similar and is moving slowly closer to the Scala API as the Java language evolves.

Play’s structure

The default structure generated for you with a new Play project resembles the well known Model-View-Controller (MVC) structure. A routes file maps http end points to "actions", which are Java or Scala code that produce results for an http request.

The routes file looks something like this:

Each line contains an HTTP method, a URL pattern, and the actual code to execute.

It's worth noting that there is nothing specially baked into Play for this default structure, so you can easily use your own structure and routing. However, the choice of MVC with a routes file by default allows web developers to get started from familiar territory, adhering to the progressive learning curve principle.

Actions are functions

As we can see, routes point to "actions" for handling requests. An "action" looks like this:

Pretty simple: it's just a function that takes a Request and returns a Result. A function is the most important component of functional programming. A function from A to B (written A => B) is a piece of code that, when given a value of type A, will always return a value of type B. A pure function will do nothing but what its type says: no side effects, no I/O, no launching missiles. Therefore, an Action in Play is just a Request => Result function.

This is exactly what makes a function interesting: you can reason about much of the function by just looking at its type. And you can apply it, as many times as you want, without having any fear of destroying planet earth. Being able to reason about a function in this way is crucial to its compositional properties.

Composing actions

Let's look back at our simple Action. What if, for most of actions of my app, I need to extract some user Context from the request? I might have a reusable function Request => Context like this:

How do I get this Context data available in each Action? Through composition! I can compose the reusable Request => Context function with the action-specific Context => Result by defining an ActionWithContext function:

Now I can define my actions as follows:

That is how simple function composition is, look at the types, and compose! The same applies for authentication, authorization, and other reusable aspects of an action. No special tooling, framework hooks, or annotations are needed; since Actions are functions, we are just using pure functional programming to handle more complicated use cases. See the Action Composition docs for more info.

Getting more sophisticated with actions

For more sophisticated use cases, we may need a slightly more sophisticated function than Request => Reponse. An HTTP request actually consists of two parts: a header and a body. The body is itself transmitted as chunks of byte arrays (Array[Byte]), so the type for an Action that needs to process the body will look more like this:

In Play, we can consume the body of a request using something called an Iteratee. An Iteratee[A, B] is a class that can consume objects of type A and eventually produce some output of type B. Using the Iteratee, we can rewrite the action type above as:

That is, an action is a function that takes a RequestHeader, and returns and Iteratee that is capable of consuming the chunks of Array[Byte] that represent the body and eventually returns a Result.

This type better represents an http request, and gives us more power when we want to deal with request body chunks. Of course, if we don’t care about that power we can still stick with the function simple type. Remember our principle: use sophistication only when power is needed.

It's can be useful to compare an Iteratee to an Iterator. With an Iterator, you (the consumer) call the next() method repeatedly to get an item from some collection and process it in some way, storing intermediate state outside of the Iterator itself. For example, if we had a collection that contained a bunch of byte arrays, here is how we would calculate the total length using Java:

With an Iteratee, control is inverted, and someone else (a producer) will feed data into your Iteratee. When you implement an Iteratee, you write code that reacts to each chunk of data by processing it in some way and returning a new Iteratee that has the new intermediate state within it and is ready to consume another chunk of data. Here is an Iteratee that will add up the length of all byte arrays fed to it:

Note that merely defining this Iteratee hasn't done anything; it's waiting for some producer to start feeding it data. In Play, we can use the Enumerator class as a producer (more on Enumerators below):

Make it simple again

Request body chunks as Array[Byte] and Iteratees are very generic and powerful, but they are pretty low level to deal with. We typically want to work with higher level types, such as JSON, which Play represents as the JsValue class.

No fancy framework level hooks will be necessary; we can get back to simpler types using our functional programming toolset and composition. Here's what we'll need to do:

1. Use an Iteratee to consumes chunks of Array[Byte] and produce a JsValue.
2. Convert the JsValue into a Result.

You could do this all from scratch - as well as more complicated variations - but Play has the common cases covered to keep it simple. The first step above - converting Array[Byte] to a friendlier type, such as a JsValue - is a very common use case and Play has a number of pre-built Iteratees, called body parsers, to handle it:

For the second step, we can use the map method implemented on an Iteratee. The map method is from our functional programming toolset (it implements what is called Functor): it lets you pass in a function to transform the output type of an Iteratee from some type S (in this case, JsValue) to some other type T (in this case, Result):

And there you have it: using functional composition, we're back to a simple piece of code that parses the body of a Request as JSON and returns a Result. Even though we're doing something more complicated, the code you actually write is still a simple, reusable one-liner. Composition FTW!

Play has even more sophisticated action and http types to deal with even more complex use cases, such as streaming results and WebSockets. However, the principle is always the same: each time we introduce sophistication in types, we use our functional programming toolset to restore composability and reduce things back down to a very simple form. Check out the Play documentation for more details.

Data manipulation

Many modern web use cases involve going through a list of items, filtering it, grouping its elements, and applying different transformations to its entries. This is another area that is inherently simple and composable in functional programming.

Imagine we have a list of LinkedIn profiles and what we need to do is to get the 10 most popular skills among young folks from that list. To simplify the problem, we can decompose it into simpler tasks, each of them being easier to solve:

1. Keep only profiles of young people
2. Group these profiles by skill
3. For each skill, determine the number of profiles
4. Order results by number of profiles and take the top 10 skills

This kind of problem solving is used a lot in functional programming. What is interesting here is that the solution is pretty isomorphic to this decomposition:

Composability in action! Each step is only concerned about its own output. Intermediate variables here correspond pretty closely to individual steps. Being able to partition the solution into steps is a great property of composability, and a good tool to relax complexity.

Straightforward and composable data manipulation is a very nice advantage of adopting functional programming in Play. Play strives to represent everything as values and functions so that they can be straightforwardly used in a functional style. This includes html templates, which are in fact simple functions, and actions, which use functional programming techniques like Functors and Monads to be composable.

Data parsing

One of the ubiquitous tasks in modern web programming is reading and parsing different formats of data like XML and JSON. We want our tools for reading this data to be flexible (to adapt to data changes), composable (for reuse and relaxing complexity), and programmatic (to handle different special cases with ease).

Let’s take an example of reading some information from a JSON data structure representing a Tweet:

Even a single tweet contains a lot of data. Now imagine that for our app, all we want is to extract image urls included in a tweet. Our Tweet type is much simpler than the raw data:

The challenge is to get a List[Tweet] out of an array of complex JSON objects.

Let’s decompose the problem into parts:

1. Read image urls inside media
2. Take only one url from those
3. Read the tweet Id
4. Read a list of combining id with media url
5. Collect only tweets with media
6. Read tweets
7. Use this JSON parsing code in an Action

We are going to use the Play’s JSON API, a composable library for reading JSON. As you would expect, we deliberately designed this library on top of functional programming techniques and abstractions to guarantee its composition and flexibility. The Play JSON library is just an example of a design that uses functional programming power behind the scene, and it can be applied to XML, BSON, or any other data formats.

Here is some example code that creates a bunch of Reads values that can be used to parse the JSON, step by step, from raw JSON into a Tweet class:

Once again, we are able to break down the data manipulation into discrete steps, many of which could easily be extracted into functions that we can re-use all over the app. We also have the full programmatic power to handle "dirty" data, such as missing urls. This is extremely important while dealing with data formats in the real world.

Composable streams of data

As we already stated above, continuous and infinite data streams are everywhere, and you can’t reason about them without a proper toolset allowing their management and manipulation with ease.

You've already seen one of the main tools we need to deal with streams of data: the Iteratee. We used Iteratees to consume a stream of Array[Byte]and calculate its length or convert it into a JsValue.

Let’s take another example of a data stream use in a web app. Imagine we have a web app that offers a real-time taxi service. All taxis will be publishing their position and status, and clients will be looking for an appropriate taxi car at a given location.

In Play, an Enumerator[A] is a class capable of producing chunks of type A. For example, the stream of all taxi cars events might look like this:

There are different ways to create an Enumerator. For example, if we had some source of taxi events, such as an InputStream from a pub/sub system, we could create the Enumerator as follows:

We know from before that we can consume such a stream reactively using an Iteratee. Here's a simple example that just prints all the taxi events to the console:

With just a few lines of code, we're able to consume a live stream of data: as new taxi events come in, they will be logged to the console.

But what if we need to manipulate the stream on the fly? For example, imagine we're only interested in the statuses of taxis around a particular location. We can filter the stream using an Enumeratee, which is a function that can adapt one type of stream into another type of stream:

What if I also want to filter the results to taxis that can hold at least 4 people? Just compose with one more Enumeratee!

Once again, we use composability for code reuse and for modularity of our program. Thanks to the composable Enumerator abstraction, we can separate logic into different components, just like we did with actions, lists, and JSON readers.

Play and functional programming

These were just a few examples of where we introduced the functional programming toolset in Play to simplify complicated tasks in modern web programming. The same basic techniques of using functions and composition apply all over the codebase, providing a powerful and consistent way of dealing with a variety of problems. Check out the Play documentation for more examples. Of course, Play will continue to evolve, so follow us at http://www.playframework.com/ for more!