Reinventing the Reader monad

Or, designing your own elevated world

This post is the sixth in a series. In the first two posts, I described some of the core functions for dealing with generic data types: map, bind, and so on. In the third post, I discussed "applicative" vs "monadic" style, and how to lift values and functions to be consistent with each other. In the fourth and previous posts, I introduced traverse and sequence as a way of working with lists of elevated values, and we saw this used in a practical example: downloading some URLs.

In this post, we'll finish up by working through another practical example, but this time we'll create our own "elevated world" as a way to deal with awkward code. We'll see that this approach is so common that it has a name -- the "Reader monad".

Series contents

Here's a list of shortcuts to the various functions mentioned in this series:

Part 6: Designing your own elevated world

The scenario we'll be working with in this post is just this:

A customer comes to your site and wants to view information about the products they have purchased.

In this example, we'll assume that you have a API for a key/value store (such as Redis or a NoSql database), and all the information you need is stored there.

So the code we need will look something like this:

How hard can that be?

Well, it turns out to be surprisingly tricky! Luckily, we can find a way to make it easier using the concepts in this series.

Defining the domain and a dummy ApiClient

First let's define the domain types:

  • There will be a CustomerId and ProductId of course.

  • For the product information, we'll just define a simple ProductInfo with a ProductName field.

Here are the types:

For testing our api, let's create an ApiClient class with some Get and Set methods, backed by a static mutable dictionary. This is based on similar APIs such as the Redis client.

Notes:

  • The Get and Set both work with objects, so I've added a casting mechanism.

  • In case of errors such as a failed cast, or a missing key, I'm using the Result type that we've been using throughout this series.

    Therefore, both Get and Set return Results rather than plain objects.

  • To make it more realistic, I've also added dummy methods for Open, Close and Dispose.

  • All methods trace a log to the console.

Let's do some tests:

And the results are:

A first implementation attempt

For our first attempt at implementing the scenario, let's start with the pseudo-code from above:

So far so good, but there is a bit of a problem already.

The getPurchaseInfo function takes a CustId as input, but it can't just output a list of ProductInfos, because there might be a failure. That means that the return type needs to be Result<ProductInfo list>.

Ok, how do we create our productInfosResult?

Well that should be easy. If the productIdsResult is Success, then loop through each id and get the info for each id. If the productIdsResult is Failure, then just return that failure.

Hmmm. It's looking a bit ugly. And I'm having to use a mutable data structure (productInfos) to accumulate each product info and then wrap it in Success.

And there's a worse problem The productInfo that I'm getting from api.Get<ProductInfo> is not a ProductInfo at all, but a Result<ProductInfo>, so productInfos is not the right type at all!

Let's add code to test each ProductInfo result. If it's a success, then add it to the list of product infos, and if it's a failure, then return the failure.

Um, no. That won't work at all. The code above will not compile. We can't do an "early return" in the loop when a failure happens.

So what do we have so far? Some really ugly code that won't even compile.

There has to be a better way.

A second implementation attempt

It would be great if we could hide all this unwrapping and testing of Results. And there is -- computation expressions to the rescue.

If we create a computation expression for Result we can write the code like this:

In let productInfosResult = Result.result { .. } code we create a result computation expression that simplifies all the unwrapping (with let!) and wrapping (with return).

And so this implementation has no explicit xxxResult values anywhere. However, it still has to use a mutable collection class to do the accumulation, because the for productId in productIds do is not actually a real for loop, and we can't replace it with List.map, say.

The result computation expression.

Which brings us onto the implementation of the result computation expression. In the previous posts, ResultBuilder only had two methods, Return and Bind, but in order to get the for..in..do functionality, we have to implement a lot of other methods too, and it ends up being a bit more complicated.

I have a series about the internals of computation expressions, so I don't want to explain all that code here. Instead, for the rest of the post we'll work on refactoring getPurchaseInfo, and by the end of it we'll see that we don't need the result computation expression at all.

Refactoring the function

The problem with the getPurchaseInfo function as it stands is that it mixes concerns: it both creates the ApiClient and does some work with it.

There a number of problems with this approach:

  • If we want to do different work with the API, we have to repeat the open/close part of this code.

    And it's possible that one of the implementations might open the API but forget to close it.

  • It's not testable with a mock API client.

We can solve both of these problems by separating the creation of an ApiClient from its use by parameterizing the action, like this.

The action function that is passed in would look like this, with a parameter for the ApiClient as well as for the CustId:

Note that getPurchaseInfo has two parameters, but executeApiAction expects a function with only one.

No problem! Just use partial application to bake in the first parameter:

That's why the ApiClient is the second parameter in the parameter list -- so that we can do partial application.

More refactoring

We might need to get the product ids for some other purpose, and also the productInfo, so let's refactor those out into separate functions too:

Now, we have these nice core functions getPurchaseIds and getProductInfo, but I'm annoyed that I have to write messy code to glue them together in getPurchaseInfo.

Ideally, what I'd like to do is pipe the output of getPurchaseIds into getProductInfo like this:

Or as a diagram:

But I can't, and there are two reasons why:

  • First, getProductInfo has two parameters. Not just a ProductId but also the ApiClient.

  • Second, even if ApiClient wasn't there, the input of getProductInfo is a simple ProductId but the output of getPurchaseIds is a Result.

Wouldn't it be great if we could solve both of these problems!

Introducing our own elevated world

Let's address the first problem. How can we compose functions when the extra ApiClient parameter keeps getting in the way?

This is what a typical API calling function looks like:

If we look at the type signature we see this, a function with two parameters:

But another way to interpret this function is as a function with one parameter that returns another function. The returned function has an ApiClient parameter and returns the final ouput.

You might think of it like this: I have an input right now, but I won't have an actual ApiClient until later, so let me use the input to create a api-consuming function that can I glue together in various ways right now, without needing a ApiClient at all.

Let's give this api-consuming function a name. Let's call it ApiAction.

In fact, let's do more than that -- let's make it a type!

Unfortunately, as it stands, this is just a type alias for a function, not a separate type. We need to wrap it in a single case union to make it a distinct type.

Rewriting to use ApiAction

Now that we have a real type to use, we can rewrite our core domain functions to use it.

First getPurchaseIds:

The signature is now CustId -> ApiAction<Result<ProductId list>>, which you can interpret as meaning: "give me a CustId and I will give a you a ApiAction that, when given an api, will make a list of ProductIds".

Similarly, getProductInfo can be rewritten to return an ApiAction:

Notice those signatures:

  • CustId -> ApiAction<Result<ProductId list>>

  • ProductId -> ApiAction<Result<ProductInfo>>

This is starting to look awfully familiar. Didn't we see something just like this in the previous post, with Async<Result<_>>?

ApiAction as an elevated world

If we draw diagrams of the various types involved in these two functions, we can clearly see that ApiAction is an elevated world, just like List and Result. And that means that we should be able to use the same techniques as we have used before: map, bind, traverse, etc.

Here's getPurchaseIds as a stack diagram. The input is a CustId and the output is an ApiAction<Result<List<ProductId>>>:

and with getProductInfo the input is a ProductId and the output is an ApiAction<Result<ProductInfo>>:

The combined function that we want, getPurchaseInfo, should look like this:

And now the problem in composing the two functions is very clear: the output of getPurchaseIds can not be used as the input for getProductInfo:

But I think that you can see that we have some hope! There should be some way of manipulating these layers so that they do match up, and then we can compose them easily.

So that's what we will work on next.

Introducting ApiActionResult

In the last post we merged Async and Result into the compound type AsyncResult. We can do the same here, and create the type ApiActionResult.

When we make this change, our two functions become slightly simpler:

Enough diagrams -- let's write some code now.

First, we need to define map, apply, return and bind for ApiAction:

Note that all the functions use a helper function called run which unwraps an ApiAction to get the function inside, and applies this to the api that is also passed in. The result is the value wrapped in the ApiAction.

For example, if we had an ApiAction<int> then run api myAction would result in an int.

And at the bottom, there is a execute function that creates an ApiClient, opens the connection, runs the action, and then closes the connection.

And with the core functions for ApiAction defined, we can go ahead and define the functions for the compound type ApiActionResult, just as we did for AsyncResult in the previous post:

Working out the transforms

Now that we have all the tools in place, we must decide on what transforms to use to change the shape of getProductInfo so that the input matches up.

Should we choose map, or bind, or traverse?

Let's play around with the stacks visually and see what happens for each kind of transform.

Before we get started, let's be explicit about what we are trying to achieve:

  • We have two functions getPurchaseIds and getProductInfo that we want to combine into a single function getPurchaseInfo.

  • We have to manipulate the left side (the input) of getProductInfo so that it matches the output of getPurchaseIds.

  • We have to manipulate the right side (the output) of getProductInfo so that it matches the output of our ideal getPurchaseInfo.

Map

As a reminder, map adds a new stack on both sides. So if we start with a generic world-crossing function like this:

Then, after List.map say, we will have a new List stack on each site.

Here's our getProductInfo before transformation:

And here is what it would look like after using List.map

This might seem promising -- we have a List of ProductId as input now, and if we can stack a ApiActionResult on top we would match the output of getPurchaseId.

But the output is all wrong. We want the ApiActionResult to stay on the top. That is, we don't want a List of ApiActionResult but a ApiActionResult of List.

Bind

Ok, what about bind?

If you recall, bind turns a "diagonal" function into a horizontal function by adding a new stack on the left sides. So for example, whatever the top elevated world is on the right, that will be added to the left.

And here is what our getProductInfo would look like after using ApiActionResult.bind

This is no good to us. We need to have a List of ProductId as input.

Traverse

Finally, let's try traverse.

traverse turns a diagonal function of values into diagonal function with lists wrapping the values. That is, List is added as the top stack on the left hand side, and the second-from-top stack on the right hand side.

if we try that out on getProductInfo we get something very promising.

The input is a list as needed. And the output is perfect. We wanted a ApiAction<Result<List<ProductInfo>>> and we now have it.

So all we need to do now is add an ApiActionResult to the left side.

Well, we just saw this! It's bind. So if we do that as well, we are finished.

And here it is expressed as code:

Or to make it a bit less ugly:

Let's compare that with the earlier version of getPurchaseInfo:

Let's compare the two versions in a table:

Earlier verson

Latest function

Composite function is non-trivial and needs special code to glue the two smaller functions together

Composite function is just piping and composition

Uses the "result" computation expression

No special syntax needed

Has special code to loop through the results

Uses "traverse"

Uses a intermediate (and mutable) List object to accumulate the list of product infos

No intermediate values needed. Just a data pipeline.

Implementing traverse

The code above uses traverse, but we haven't implemented it yet. As I noted earlier, it can be implemented mechanically, following a template.

Here it is:

Testing the implementation

Let's test it!

First we need a helper function to show results:

Next, we need to load the API with some test data:

  • Customer C1 has purchased two products: P1 and P2.

  • Customer C2 has purchased two products: PX and P2.

  • Products P1 and P2 have some info.

  • Product PX does not have any info.

Let's see how this works out for different customer ids.

We'll start with Customer C1. For this customer we expect both product infos to be returned:

And here are the results:

What happens if we use a missing customer, such as CX?

As expected, we get a nice "key not found" failure, and the rest of the operations are skipped as soon as the key is not found.

What about if one of the purchased products has no info? For example, customer C2 purchased PX and P2, but there is no info for PX.

The overall result is a failure. Any bad product causes the whole operation to fail.

But note that the data for product P2 is fetched even though product PX failed. Why? Because we are using the applicative version of traverse, so every element of the list is fetched "in parallel".

If we wanted to only fetch P2 once we knew that PX existed, then we should be using monadic style instead. We already seen how to write a monadic version of traverse, so I leave that as an exercise for you!

Filtering out failures

In the implementation above, the getPurchaseInfo function failed if any product failed to be found. Harsh!

A real application would probably be more forgiving. Probably what should happen is that the failed products are logged, but all the successes are accumulated and returned.

How could we do this?

The answer is simple -- we just need to modify the traverse function to skip failures.

First, we need to create a new helper function for ApiActionResult. It will allow us to pass in two functions, one for the success case and one for the error case:

This helper function helps us match both cases inside a ApiAction without doing complicated unwrapping. We will need this for our traverse that skips failures.

By the way, note that ApiActionResult.bind can be defined in terms of either:

Now we can define our "traverse with logging of failures" function:

The only difference between this and the previous implementation is this bit:

This says that:

  • If the new first element (f head) is a success, lift the inner value (retn h) and cons it with the tail to build a new list.

  • But if the new first element is a failure, then log the inner errors (errs) with the passed in logging function (log)

    and just reuse the current tail.

    In this way, failed elements are not added to the list, but neither do they cause the whole function to fail.

Let's create a new function getPurchasesInfoWithLog and try it with customer C2 and the missing product PX:

The result is a Success now, but only one ProductInfo, for P2, is returned. The log shows that PX was skipped.

The Reader monad

If you look closely at the ApiResult module, you will see that map, bind, and all the other functions do not use any information about the api that is passed around. We could have made it any type and those functions would still have worked.

So in the spirit of "parameterize all the things", why not make it a parameter?

That means that we could have defined ApiAction as follows:

But if it can be anything, why call it ApiAction any more? It could represent any set of things that depend on an object (such as an api) being passed in to them.

We are not the first people to discover this! This type is commonly called the Reader type and is defined like this:

The extra type 'environment plays the same role that ApiClient did in our definition of ApiAction. There is some environment that is passed around as an extra parameter to all your functions, just as a api instance was.

In fact, we can actually define ApiAction in terms of Reader very easily:

The set of functions for Reader are exactly the same as for ApiAction. I have just taken the code and replaced ApiAction with Reader and api with environment!

The type signatures are a bit harder to read now though!

The Reader type, plus bind and return, plus the fact that bind and return implement the monad laws, means that Reader is typically called "the Reader monad" .

I'm not going to delve into the Reader monad here, but I hope that you can see how it is actually a useful thing and not some bizarre ivory tower concept.

The Reader monad vs. an explicit type

Now if you like, you could replace all the ApiAction code above with Reader code, and it would work just the same. But should you?

Personally, I think that while understanding the concept behind the Reader monad is important and useful, I prefer the actual implementation of ApiAction as I defined it originally, an explicit type rather than an alias for Reader<ApiClient,'a>.

Why? Well, F# doesn't have typeclasses, F# doesn't have partial application of type constructors, F# doesn't have "newtype". Basically, F# isn't Haskell! I don't think that idioms that work well in Haskell should be carried over to F# directly when the language does not offer support for it.

If you understand the concepts, you can implement all the necessary transformations in a few lines of code. Yes, it's a little extra work, but the upside is less abstraction and fewer dependencies.

I would make an exception, perhaps, if your team were all Haskell experts, and the Reader monad was familiar to everyone. But for teams of different abilities, I would err on being too concrete rather than too abstract.

Summary

In this post, we worked through another practical example, created our own elevated world which made things much easier, and in the process, accidentally re-invented the reader monad.

If you liked this, you can see a similar practical example, this time for the State monad, in my series on "Dr Frankenfunctor and the Monadster".

The next and final post has a quick summary of the series, and some further reading.

PreviousUsing map, apply, bind and sequence in practiceNextMap and Bind and Apply, a summary

Last updated

Was this helpful?