Choosing between collection functions

A guide for the perplexed

There's more to learning a new language than the language itself. In order to be productive, you need to memorize a big chunk of the standard library and be aware of most of the rest of it. For example, if you know C#, you can pick up Java-the-language quite quickly, but you won't really get up to speed until you are comfortable with the Java Class Library as well.

Similarly, you can't really be effective in F# until you have some familiarity with all the F# functions that work with collections.

In C# there are only a few LINQ methods you need to know1 (Select, Where, and so on). But in F#, there are currently almost 100 functions in the List module (and similar counts in the Seq and Array modules). That's a lot!

1 Yes, there are more, but you can get by with just a few. In F# it's more important to know them all.

If you are coming to F# from C#, then, the large number of list functions can be overwhelming.

So I have written this post to help guide you to the one you want. And for fun, I've done it in a "Choose Your Own Adventure" style!

What collection do I want?

First, a table with information about the different kinds of standard collections. There are five "native" F# ones: list, seq, array, map and set, and ResizeArray and IDictionary are also often used.

Immutable?

Notes

list

Yes

Pros:

  • Pattern matching available.

  • Complex iteration available via recursion.

  • Forward iteration is fast. Prepending is fast.

Cons:

  • Indexed access and other access styles are slow.

seq

Yes

Alias for IEnumerable. Pros:

  • Lazy evaluation

  • Memory efficient (only one element at a time loaded)

  • Can represent an infinite sequence.

  • Interop with .NET libraries that use IEnumerable.

Cons:

  • No pattern matching.

  • Forward only iteration.

  • Indexed access and other access styles are slow.

array

No

Same as BCL Array. Pros:

  • Fast random access

  • Memory efficient and cache locality, especially with structs.

  • Interop with .NET libraries that use Array.

  • Support for 2D, 3D and 4D arrays

Cons:

map

Yes

Immutable dictionary. Requires keys to implement IComparable.

set

Yes

Immutable set. Requires elements to implement IComparable.

ResizeArray

No

Alias for BCL List. Pros and cons similar to array, but resizable.

IDictionary

Yes

For an alternate dictionary that does not requires elements to implement IComparable, you can use the BCL IDictionary. The constructor is dict in F#.

Note that mutation methods such as Add are present, but will cause a runtime error if called.

These are the main collection types that you will encounter in F#, and will be good enough for all common cases.

If you need other kinds of collections though, there are lots of choices:

  • You can use the collection classes in .NET, either the traditional, mutable ones

    or the newer ones such as those in the System.Collections.Immutable namespace.

  • Alternatively, you can use one of the F# collection libraries:

    • FSharpx.Collections, part of the FSharpx series of projects.

    • ExtCore. Some of these are drop-in (almost) replacements for the Map and Set types in FSharp.Core which provide improved performance in specific scenarios (e.g., HashMap). Others provide unique functionality to help tackle specific coding tasks (e.g., LazyList and LruCache).

    • Funq: high performance, immutable data structures for .NET.

    • Persistent: some efficient persistent (immutable) data structures.

About the documentation

All functions are available for list, seq and array in F# v4 unless noted. The Map and Set modules have some of them as well, but I won't be discussing map and set here.

For the function signatures I will use list as the standard collection type. The signatures for the seq and array versions will be similar.

Many of these functions are not yet documented on MSDN so I'm going to link directly to the source code on GitHub, which has the up-to-date comments. Click on the function name for the link.

Note on availability

The availability of these functions may depend on which version of F# you use.

  • In F# version 3 (Visual Studio 2013), there was some degree of inconsistency between Lists, Arrays and Sequences.

  • In F# version 4 (Visual Studio 2015), this inconsistency has been eliminated, and almost all functions are available for all three collection types.

If you want to know what changed between F# v3 and F# v4, please see this chart (from here). The chart shows the new APIs in F# v4 (green), previously-existing APIs (blue), and intentional remaining gaps (white).

Some of the functions documented below are not in this chart -- these are newer still! If you are using an older version of F#, you can simply reimplement them yourself using the code on GitHub.

With that disclaimer out of the way, you can start your adventure!

Table of contents

1. What kind of collection do you have?

What kind of collection do you have?

  • If you don't have a collection, and want to create one, go to section 2.

  • If you already have a collection that you want to work with, go to section 9.

  • If you have two collections that you want to work with, go to section 23.

  • If you have three collections that you want to work with, go to section 24.

  • If you have more than three collections that you want to work with, go to section 25.

  • If you want to combine or uncombine collections, go to section 26.

2. Creating a new collection

So you want to create a new collection. How do you want to create it?

  • If the new collection will be empty or will have one element, go to section 3.

  • If the new collection is a known size, go to section 4.

  • If the new collection is potentially infinite, go to section 7.

  • If you don't know how big the collection will be, go to section 8.

3. Creating a new empty or one-element collection

If you want to create a new empty or one-element collection, use these functions:

If you know the size of the collection in advance, it is generally more efficient to use a different function. See section 4 below.

Usage examples

4. Creating a new collection of known size

  • If all elements of the collection will have the same value, go to section 5.

  • If elements of the collection could be different, go to section 6.

5. Creating a new collection of known size with each element having the same value

If you want to create a new collection of known size with each element having the same value, you want to use replicate:

Array.create is basically the same as replicate (although with a subtly different implementation!) but replicate was only implemented for Array in F# v4.

Usage examples

Note that for zeroCreate, the target type must be known to the compiler.

6. Creating a new collection of known size with each element having a different value

If you want to create a new collection of known size with each element having a potentially different value, you can choose one of three ways:

  • init : length:int -> initializer:(int -> 'T) -> 'T list.

    Creates a collection by calling the given generator on each index.

  • For lists and arrays, you can also use the literal syntax such as [1; 2; 3] (lists) and [|1; 2; 3|] (arrays).

  • For lists and arrays and seqs, you can use the comprehension syntax for .. in .. do .. yield.

Usage examples

Literal syntax allows for an increment as well:

The comprehension syntax is even more flexible because you can yield more than once:

and it can also be used as a quick and dirty inline filter:

Two other tricks:

  • You can use yield! to return a list rather than a single value

  • You can also use recursion

Here is an example of both tricks being used to count up to 10 by twos:

7. Creating a new infinite collection

If you want an infinite list, you have to use a seq rather than a list or array.

  • initInfinite : initializer:(int -> 'T) -> seq<'T>.

    Generates a new sequence which, when iterated, will return successive elements by calling the given function.

  • You can also use a seq comprehension with a recursive loop to generate an infinite sequence.

Usage examples

8. Creating a new collection of indefinite size

Sometimes you don't know how big the collection will be in advance. In this case you need a function that will keep adding elements until it gets a signal to stop. unfold is your friend here, and the "signal to stop" is whether you return a None (stop) or a Some (keep going).

Usage examples

This example reads from the console in a loop until an empty line is entered:

unfold requires that a state be threaded through the generator. You can ignore it (as in the ReadLine example above), or you can use it to keep track of what you have done so far. For example, you can create a Fibonacci series generator using unfold:

9. Working with one list

If you are working with one list and...

  • If you want to get an element at a known position, go to section 10

  • If you want to get one element by searching, go to section 11

  • If you want to get a subset of the collection, go to section 12

  • If you want to partition, chunk, or group a collection into smaller collections, go to section 13

  • If you want to aggregate or summarize the collection into a single value, go to section 14

  • If you want to change the order of the elements, go to section 15

  • If you want to test the elements in the collection, go to section 16

  • If you want to transform each element to something different, go to section 17

  • If you want to iterate over each element, go to section 18

  • If you want to thread state through an iteration, go to section 19

  • If you need to know the index of each element while you are iterating or mapping, go to section 20

  • If you want to transform the whole collection to a different collection type, go to section 21

  • If you want to change the behaviour of the collection as a whole, go to section 22

  • If you want to mutate the collection in place, go to section 27

  • If you want to use a lazy collection with an IDisposable, go to section 28

10. Getting an element at a known position

The following functions get a element in the collection by position:

But what if the collection is empty? Then head and last will fail with an exception (ArgumentException).

And if the index is not found in the collection? Then another exception again (ArgumentException for lists, IndexOutOfRangeException for arrays).

I would therefore recommend that you avoid these functions in general and use the tryXXX equivalents below:

Usage examples

As noted, the item function should be avoided for lists. For example, if you want to process each item in a list, and you come from an imperative background, you might write a loop with something like this:

Don't do that! Use something like map instead. It's both more concise and more efficient:

11. Getting an element by searching

You can search for an element or its index using find and findIndex:

And you can also search backwards:

But what if the item cannot be found? Then these will fail with an exception (KeyNotFoundException).

I would therefore recommend that, as with head and item, you avoid these functions in general and use the tryXXX equivalents below:

If you are doing a map before a find you can often combine the two steps into a single one using pick (or better, tryPick). See below for a usage example.

Usage examples

With pick, rather than returning a bool, you return an option:

Pick vs. Find

That 'pick' function might seem unnecessary, but it is useful when dealing with functions that return options.

For example, say that there is a function tryInt that parses a string and returns Some int if the string is a valid int, otherwise None.

And now say that we want to find the first valid int in a list. The crude way would be:

  • map the list using tryInt

  • find the first one that is a Some using find

  • get the value from inside the option using Option.get

The code might look something like this:

But pick will do all these steps at once! So the code becomes much simpler:

If you want to return many elements in the same way as pick, consider using choose (see section 12).

12. Getting a subset of elements from a collection

The previous section was about getting one element. How can you get more than one element? Well you're in luck! There's lots of functions to choose from.

To extract elements from the front, use one of these:

To extract elements from the rear, use one of these:

To extract other subsets of elements, use one of these:

To reduce the list to distinct elements, use one of these:

Usage examples

Taking elements from the front:

Taking elements from the rear:

To extract other subsets of elements:

To extract a slice:

Note that slicing on lists can be slow, because they are not random access. Slicing on arrays is fast however.

To extract the distinct elements:

Choose vs. Filter

As with pick, the choose function might seem awkward, but it is useful when dealing with functions that return options.

In fact, choose is to filter as pick is to find, Rather than using a boolean filter, the signal is Some vs. None.

As before, say that there is a function tryInt that parses a string and returns Some int if the string is a valid int, otherwise None.

And now say that we want to find all the valid ints in a list. The crude way would be:

  • map the list using tryInt

  • filter to only include the ones that are Some

  • get the value from inside each option using Option.get

The code might look something like this:

But choose will do all these steps at once! So the code becomes much simpler:

If you already have a list of options, you can filter and return the "Some" in one step by passing id into choose:

If you want to return the first element in the same way as choose, consider using pick (see section 11).

If you want to do a similar action as choose but for other wrapper types (such as a Success/Failure result), there is a discussion here.

13. Partitioning, chunking and grouping

There are lots of different ways to split a collection! Have a look at the usage examples to see the differences:

Usage examples

All the functions other than splitAt and pairwise handle edge cases gracefully:

14. Aggregating or summarizing a collection

The most generic way to aggregate the elements in a collection is to use reduce:

and there are specific versions of reduce for frequently used aggregations:

Finally there are some counting functions:

Usage examples

reduce is a variant of fold without an initial state -- see section 19 for more on fold. One way to think of it is just inserting a operator between each element.

is the same as

Here's another example:

Some ways of combining elements depend on the order of combining, and so there are two variants of "reduce":

  • reduce moves forward through the list.

  • reduceBack, not surprisingly, moves backwards through the list.

Here's a demonstration of the difference. First reduce:

Using the same combining function with reduceBack produces a different result! It looks like this:

Again, see section 19 for a more detailed discussion of the related functions fold and foldBack.

The other aggregation functions are much more straightforward.

Most of the aggregation functions do not like empty lists! You might consider using one of the fold functions to be safe -- see section 19.

15. Changing the order of the elements

You can change the order of the elements using reversing, sorting and permuting. All of the following return new collections:

And there are also some array-only functions that sort in place:

Usage examples

16. Testing the elements of a collection

These set of functions all return true or false.

Usage examples

17. Transforming each element to something different

I sometimes like to think of functional programming as "transformation-oriented programming", and map (aka Select in LINQ) is one of the most fundamental ingredients for this approach. In fact, I have devoted a whole series to it here.

Sometimes each element maps to a list, and you want to flatten out all the lists. For this case, use collect (aka SelectMany in LINQ).

Other transformation functions include:

Usage examples

Here are some examples of using map in the conventional way, as a function that accepts a list and a mapping function and returns a new transformed list:

You can also think of map as a function transformer. It turns an element-to-element function into a list-to-list function.

collect works to flatten lists. If you already have a list of lists, you can use collect with id to flatten them.

Seq.cast

Finally, Seq.cast is useful when working with older parts of the BCL that have specialized collection classes rather than generics.

For example, the Regex library has this problem, and so the code below won't compile because MatchCollection is not an IEnumerable<T>

The fix is to cast MatchCollection to a Seq<Match> and then the code will work nicely:

18. Iterating over each element

Normally, when processing a collection, we transform each element to a new value using map. But occasionally we need to process all the elements with a function which doesn't produce a useful value (a "unit function").

Usage examples

The most common examples of unit functions are all about side-effects: printing to the console, updating a database, putting a message on a queue, etc. For the examples below, I will just use printfn as my unit function.

As noted above, the expression inside an iter or for-loop must return unit. In the following examples, we try to add 1 to the element, and get a compiler error:

If you are sure that this is not a logic bug in your code, and you want to get rid of this error, you can pipe the results into ignore:

19. Threading state through an iteration

The fold function is the most basic and powerful function in the collection arsenal. All other functions (other than generators like unfold) can be written in terms of it. See the examples below.

The fold function is often called "fold left" and foldBack is often called "fold right".

The scan function is like fold but returns the intermediate results and thus can be used to trace or monitor the iteration.

Just like the fold twins, scan is often called "scan left" and scanBack is often called "scan right".

Finally, mapFold combines map and fold into one awesome superpower. More complicated than using map and fold separately but also more efficient.

fold examples

One way of thinking about fold is that it is like reduce but with an extra parameter for the initial state:

As with reduce, fold and foldBack can give very different answers.

And here's the foldBack version:

Note that foldBack has a different parameter order to fold: the list is second last, and the initial state is last, which means that piping is not as convenient.

Recursing vs iterating

It's easy to get confused between fold vs. foldBack. I find it helpful to think of fold as being about iteration while foldBack is about recursion.

Let's say we want to calculate the sum of a list. The iterative way would be to use a for-loop. You start with a (mutable) accumulator and thread it through each iteration, updating it as you go.

On the other hand, the recursive approach says that if the list has a head and tail, calculate the sum of the tail (a smaller list) first, and then add the head to it.

Each time the tail gets smaller and smaller until it is empty, at which point you're done.

Which approach is better?

For aggregation, the iterative way is (fold) often easiest to understand. But for things like constructing new lists, the recursive way is (foldBack) is easier to understand.

For example, if we were going to going to create a function from scratch that turned each element into the corresponding string, we might write something like this:

Using foldBack we can transfer that same logic "as is":

  • action for empty list = []

  • action for non-empty list = head.ToString() :: state

Here is the resulting function:

On the other hand, a big advantage of fold is that it is easier to use "inline" because it plays better with piping.

Luckily, you can use fold (for list construction at least) just like foldBack as long as you reverse the list at the end.

Using fold to implement other functions

As I mentioned above, fold is the core function for operating on lists and can emulate most other functions, although perhaps not as efficiently as a custom implementation.

For example, here is map implemented using fold:

And here is filter implemented using fold:

And of course, you can emulate the other functions in a similar way.

scan examples

Earlier, I showed an example of the intermediate steps of fold:

For that example, I had to manually calculate the intermediate states,

Well, if I had used scan, I would have got those intermediate states for free!

scanBack works the same way, but backwards of course:

Just as with foldBack the parameter order for "scan right" is inverted compared with "scan left".

Truncating a string with scan

Here's an example where scan is useful. Say that you have a news site, and you need to make sure headlines fit into 50 chars.

You could just truncate the string at 50, but that would look ugly. Instead you want to have the truncation end at a word boundary.

Here's one way of doing it using scan:

  • Split the headline into words.

  • Use scan to concat the words back together, generating a list of fragments, each with an extra word added.

  • Get the longest fragment under 50 chars.

Note that I'm using Seq.scan rather than Array.scan. This does a lazy scan and avoids having to create fragments that are not needed.

Finally, here is the complete logic as a utility function:

Yes, I know that there is a more efficient implementation than this, but I hope that this little example shows off the power of scan.

mapFold examples

The mapFold function can do a map and a fold in one step, which can be convenient on occasion.

Here's an example of combining an addition and a sum in one step using mapFold:

20. Working with the index of each element

Often, you need the index of the element as you do an iteration. You could use a mutable counter, but why not sit back and let the library do the work for you?

Usage examples

indexed generates a tuple with the index -- a shortcut for a specific use of mapi:

21. Transforming the whole collection to a different collection type

You often need to convert from one kind of collection to another. These functions do this.

The ofXXX functions are used to convert from XXX to the module type. For example, List.ofArray will turn an array into a list.

The toXXX are used to convert from the module type to the type XXX. For example, List.toArray will turn an list into an array.

Usage examples

Using sequences with disposables

One important use of these conversion functions is to convert a lazy enumeration (seq) to a fully evaluated collection such as list. This is particularly important when there is a disposable resource involved, such as file handle or database connection. If the sequence is not converted into a list you may encounter errors accessing the elements. See section 28 for more.

22. Changing the behavior of the collection as a whole

There are some special functions (for Seq only) that change the behavior of the collection as a whole.

  • (Seq only) cache: source:seq<'T> -> seq<'T>.

    Returns a sequence that corresponds to a cached version of the input sequence. This result sequence will have the same elements as the input sequence. The result

    can be enumerated multiple times. The input sequence will be enumerated at most once and only as far as is necessary.

  • (Seq only) readonly : source:seq<'T> -> seq<'T>.

    Builds a new sequence object that delegates to the given sequence object. This ensures the original sequence cannot be rediscovered and mutated by a type cast.

  • (Seq only) delay : generator:(unit -> seq<'T>) -> seq<'T>.

    Returns a sequence that is built from the given delayed specification of a sequence.

cache example

Here's an example of cache in use:

The result of iterating over the sequence twice is as you would expect:

But if we cache the sequence...

... then each item is only printed once:

readonly example

Here's an example of readonly being used to hide the underlying type of the sequence:

delay example

Here's an example of delay.

If we run the code above, we find that just by creating eagerList, we print all the "Evaluating" messages. But creating delayedSeq does not trigger the list iteration.

Only when the sequence is iterated over does the list creation happen:

An alternative to using delay is just to embed the list in a seq like this:

As with delayedSeq, the makeNumbers function will not be called until the sequence is iterated over.

23. Working with two lists

If you have two lists, there are analogues of most of the common functions like map and fold.

Usage examples

These functions are straightforward to use:

Need a function that's not here?

By using fold2 and foldBack2 you can easily create your own functions. For example, some filter2 functions can be defined like this:

See also section 25.

24. Working with three lists

If you have three lists, you only have one built-in function available. But see section 25 for an example of how you can build your own three-list functions.

25. Working with more than three lists

If you are working with more than three lists, there are no built in functions for you.

If this happens infrequently, then you could just collapse the lists into a single tuple using zip2 and/or zip3 in succession, and then process that tuple using map.

Alternatively you can "lift" your function to the world of "zip lists" using applicatives.

If that seems like magic, see this series for a explanation of what this code is doing.

26. Combining and uncombining collections

Finally, there are a number of functions that combine and uncombine collections.

Usage examples

These functions are straightforward to use:

Note that the zip functions require the lengths to be the same.

27. Other array-only functions

Arrays are mutable, and therefore have some functions that are not applicable to lists and sequences.

  • See the "sort in place" functions in section 15

  • Array.blit: source:'T[] -> sourceIndex:int -> target:'T[] -> targetIndex:int -> count:int -> unit.

    Reads a range of elements from the first array and write them into the second.

  • Array.copy: array:'T[] -> 'T[].

    Builds a new array that contains the elements of the given array.

  • Array.fill: target:'T[] -> targetIndex:int -> count:int -> value:'T -> unit.

    Fills a range of elements of the array with the given value.

  • Array.set: array:'T[] -> index:int -> value:'T -> unit.

    Sets an element of an array.

  • In addition to these, all the other BCL array functions are available as well.

I won't give examples. See the MSDN documentation.

28. Using sequences with disposables

One important use of conversion functions like List.ofSeq is to convert a lazy enumeration (seq) to a fully evaluated collection such as list. This is particularly important when there is a disposable resource involved such as file handle or database connection. If the sequence is not converted into a list while the resource is available you may encounter errors accessing the elements later, after the resource has been disposed.

This will be an extended example, so let's start with some helper functions that emulate a database and a UI:

A naive implementation will cause the sequence to be evaluated after the connection is closed:

The output is below. You can see that the connection is closed and only then is the sequence evaluated.

A better implementation will convert the sequence to a list while the connection is open, causing the sequence to be evaluated immediately:

The result is much better. All the records are loaded before the connection is disposed:

A third alternative is to embed the disposable in the sequence itself:

The output shows that now the UI display is also done while the connection is open:

This may be a bad thing (longer time for the connection to stay open) or a good thing (minimal memory use), depending on the context.

29. The end of the adventure

You made it to the end -- well done! Not really much of an adventure, though, was it? No dragons or anything. Nevertheless, I hope it was helpful.

PreviousUnderstanding type inferenceNextThe "Object-oriented programming in F#" Series

Last updated

Was this helpful?