I ended the last post by proposing a syntax for asserting against floating point results:

let ``Comparing floating point numbers``() =
    Given 1.0 |>
    When ``dividing by 3`` |>
    Then it should be (0.333333333 +/- 0.00000001)

How can we implement this? Let’s start with the trivial part: the be function. It is just another “filler” word like it and should:

 let be f = f

That leaves (0.333333333 +/- 0.00000001) to perform the same role as the ``return`` function implemented previously i.e. it is a custom overloaded operator that should have a signature 'a -> TestContext<'a> -> unit. If we restrict 'a to be a float, we could implement it as:

let (+/-) (expected: float) (tolerance: float) (context: TestContext) =
     Assert.AreEqual(expected, context.ToState, tolerance)

Then, assuming we have also defined let ``dividing by 3`` = n / 3.0, the above test passes. However, we will want to comparing other types of floating point numbers, not just floats. Unfortunately we can’t just do this:

let (+/-) expected tolerance (context: TestContext<_>) =
     Assert.AreEqual(expected, context.ToState, tolerance)

The compiler can’t determine which overload of AreEqual to use. Instead, we need to somehow write generic code that is constrained to work with any numeric type. This answer on StackOverflow shows how to use inline functions, explicit member constraints and statically resolved type parameters to (messily) achieve the required result:

let inline (+/-) (expected) (tolerance) (context: TestContext<_>) =
    let zero = LanguagePrimitives.GenericZero
    let actual = context.ToState
    let absDiff = if expected > actual then
      		      expected - actual
		  else
		      actual - expected

    let absTolerance = if tolerance < zero then
		           -tolerance
		       else
			   tolerance

    if absDiff < absTolerance then
        ()
    else
        let msg = sprintf "Expected: %s +/- %s\nBut was:  %s" <| expected.ToString() <| tolerance.ToString() <| actual.ToString()
	raise <| AssertionException(msg)

We basically have to implement Assert.AreEqual(expected, actual, tolerance) ourselves, and I’ve chosen to throw the same exception that NUnit would throw so that it can be a drop-in replacement.

If we then implement ``dividing by 3`` in a similar generic manner:

let inline ``dividing by 3`` (n:^a) : ^a =
    let three:^a = (Seq.init 3 (fun _ -> LanguagePrimitives.GenericOne)) |> Seq.sum
    n / three

Then both of the tests below pass:

[<Test>]
let ``Comparing floats``() =
    Given 1.0 |>
    When ``dividing by 3`` |>
    Then it should be (0.333333333 +/- 0.00000001)

[<Test>]
let ``Comparing decimals``() =
    Given 1.0m |>
    When ``dividing by 3`` |>
    Then it should be (0.333333333m +/- 0.00000001m)

This brings me to the end of this series of posts. Hopefully I’ve illustrated one of the ways in which F# makes it (relatively) easy to write an internal DSL, as well as whetting your appetite for exploring the possibilities of low-ceremony BDD in F#.

Last time we got to the stage where we could write the following test:

let ``Basic example`` () =
    Given (123 |> ``is pushed onto a stack``) |>
    When ``stack is popped`` |>
    Then it should ``return`` 123

I also suggested looking at NaturalSpec (which inspired these posts) to see how it tackles the same problem. If we strip out all extra features that NaturalSpec provides, we are left with:

let Given f = f

let When f = f

let It f = f // NaturalSpec equivalent of Then

let should f x y =
    f x y |> check // Implementation of check not shown

So the real work is all deferred to the should and check functions. The relevance to this post is that an exception that occurs during the action executed by the When is not caught, so we have to use the ExpectedException attribute to assert that an exception has occurred.

I’d like to avoid using the ExpectedException attribute, and instead be able to do something like:

let ``Basic exception Handling`` () =
    Given (Stack<int>()) |>
    When ``stack is popped`` |>
    Then it should throw<InvalidOperationException>

So how do we modify our implementation to support this? The first problem is that we need to catch the exception in the When clause and make it available to the Then clause. In other words, we now need to be able to pass either the system state or an exception to the Then clause. We can model this as:

type TestContext<'a> =
    | State of 'a
    | Exception of exn
	
    member this.ToState =
        match this with
        | State(s) -> s
        | Exception(e) -> failwith "Context does not represent a state."

    member this.ToException = 
        match this with
        | State(s) -> failwith "Context does not represent an exception."
        | Exception(e) -> e

I’ve added ToState and ToException helper properties to avoid repetitive pattern matching in the calling code.

We also need to modify the When function:

let When action precondition = 
    try
        State(action precondition)
    with
    | e -> Exception(e)

The TestContext will be passed to the Then clause, which means that ``return`` will need to be modified to accept a TestContext<'a> instead of an 'a:

let ``return`` (expected: 'a) (context: TestContext<'a>) =
    Assert.AreEqual(expected, context.ToState)

…and yes, the code still compiles and we still have the expected passing and failing tests.

Now we can move on to the throw<'e> function. We can cheat here and look back at the earlier ``result is 123``, which had a signature of 'a -> unit. By analogy throw<'e> should have a signature of TestContext<'a> -> unit:

let throw<'e> (context: TestContext<'a>) =
    Assert.IsInstanceOf<'e>(context.ToException)

However, this fails due to a compiler warning (you do treat warnings as errors, don’t you?):

FS0064: This construct causes code to be less generic than indicated by the type annotations. The type variable ‘a has been constrained to be type ‘int’.

This is because the state we are passing around is an int. In order to make throws properly generic we would need:

let throw<'e, 'a> (context: TestContext<'a>) =
    Assert.IsInstanceOf<'e>(context.ToException)

But that leads to an ugly calling syntax where we have to specify the state type:

let ``Basic exception handling`` () =
    Given (Stack<int>()) |>
    When ``stack is popped`` |>
    Then it should throw<InvalidOperationException, int>

The problem is that throw is dealing with two generic parameters when we only want it to deal with one. Can we extract the exception in one function, then pass it to the next to assert against? Something like this:

let ``Basic exception handling`` () =
    Given (Stack<int>()) |>
    When ``stack is popped`` |>
    Then it should throw any<InvalidOperationException>

The new throw could extract the exception from the test context, then any will be a simplified version of the original throw that gets passed the exception rather than the TextContext.

let throw checkExnType (context: TestContext<'a>) = 
    checkExnType context.ToException

let any<'e> ex =
    Assert.IsInstanceOf<'e>(ex)

More passing tests! However, we want to be sure that the correct InvalidOperationException is being thrown. Something like:

let ``Advanced exception example`` () =
    Given (Stack<int>()) |>
    When ``stack is popped`` |>
    Then it should throw (specific<InvalidOperationException> (fun e -> e.Message = "Stack empty."))

We need to add brackets around the specific function to ensure it gets evaluated first. Then, as long as it returns an exn -> unit (i.e. the same signature as any<'e>) everything should work out:

let specific<'e> (check: exn -> bool) (ex: exn) =
    Assert.IsInstanceOf<'e>(ex)
    Assert.IsTrue(check ex, "Exception condition not met:\n\n" + ex.ToString())

With this, the ``Advanced exception example`` passes. So we now have a DSL that allows us to assert that expected results and expected exceptions have occurred. However, what if the state that we are trying to verify is a floating point number? In that case Then it should return 1.234 is not a good idea.

Wouldn’t it be nice to be able to do the following?

let ``Comparing floating point numbers``() =
        Given 1.0m |>
        When ``dividing by 3`` |>
        Then it should be (0.333333333m +/- 0.00000001m)

More next time…

I’m a fan of the Gherkin DSL for specifying software behaviour. Having a structured but human readable language helps the discussion with business users. However, when it comes to converting the resulting specifications in executable tests, I find the tooling cumbersome. I suspect I would find it more useful if the business users were writing large numbers of tests, but this has never happened on any projects that I’ve been involved with.

I prefer a lighter weight approach where the developers write the tests in plain old code, but hopefully in a format that makes it easy to map them back to the specifications. The problem with this is that most convention-based approaches or BDD frameworks that I’ve encountered leave a big gap between the code syntax and the specification syntax. Then I discovered NaturalSpec, which leverages F#’s DSL-friendly syntax to produce very readable tests, and I was totally blown away by it. Using it you can write tests like the following (taken from solution to the Tennis Kata provided in the NaturalSpec source code):

[<Scenario>]     
let ``When Player1 scores once the score should be Fifteen to Love`` () =   
    Given NewGame
      |> When point_goes_to Player1
      |> It should equal (Fifteen <=> Love)
      |> Verify

It uses the forward pipe operator to thread the test state through the When and Then functions, producing very readable code with no external state. I wanted to understand how it worked under the hood, but my F# skills weren’t up to the task. So, to help myself learn I decided to write my own variant at the same time. I’m going to describe that variant, but if you want a production-grade solution, please use NaturalSpec!

The general form of a Given/When/Then spec is:

Given precondition
When action
Then expectation

We can convert this into (theoretically) valid F# code by piping each clause into the next:

Given precondition |> 
When action |> 
Then expectation

So how do we define the Given/When/Then functions such that they compile? First let’s deal with the Given function:

let Given precondition = 
    precondition

This is just some syntactic sugar to make the test more readable; it does nothing but return the precondition. We’ll need to start looking at function signatures soon, so let’s start with this function, which has a signature of 'a -> 'a. We can use it like:

Given (Stack<int>()) 

or (assuming ``a binary tree of depth 3`` is already defined)

Given ``a binary tree of depth 3``

Notice how F#’s double-backtick notation makes the code more readable. The When function needs to execute an action on the precondition (which is piped in from the result of the Given clause) and pass on the result to the Then:

let When action precondition = 
    action precondition

This has the signature ('a -> 'b) -> 'a -> 'b, and we can then use it in our test like so:

let ``is pushed onto a stack`` n = 
    let s = Stack<int>() 
    s.Push n 
    s 

let ``stack is popped`` (stack: Stack<int>) = 
    let v = stack.Pop() 
    v 

let ``Basic example`` () = 
    Given (123 |> ``is pushed onto a stack``) |> 
    When ``stack is popped`` 

Notice that ``stack is popped`` takes a Stack<int> as its input but returns just the popped value, not the entire stack i.e. we do not need to pass the same kind of state between the clauses.

Now we need to assert against the result. The idea here is to use NUnit’s asserts rather than writing custom ones, but to wrap them in meaningful syntax. So Then needs to test the result (which is piped in from the result of the When clause) against an expectation. Naively, we could assume it takes the form:

let Then expectation result = 
    expectation result 

If we look at the signature, it is ('a -> 'b) -> 'a -> 'b. This looks about right, though we don’t expect Then to return anything meaningful, so would expect 'b to be unit. However, let’s try it “as is” before messing with the signature. As a first attempt how about trying to make the following syntax compile?

let `Basic example` () = 
    Given (123 |> `is pushed onto a stack`) |> 
    When ``stack is popped`` |> 
    Then ``result is 123``

In other words we are trying to make Then ``result is 123`` actual_value have the signature ('a -> 'b) -> 'a -> 'b. But because the whole clause is just going to assert, it will return unit, and then the signature we are gunning for is ('a -> unit) -> 'a -> unit. So, partially applying ``result is 123`` to Then needs to yield a function with signature'a -> unit. So ``result is 123`` must have a signature of'a -> 'unit:

let ``result is 123`` (actual: 'a) = 
    Assert.AreEqual(123, actual)

If we try to compile and run this, we get a passing test…and if we change the Given to push 999 onto the stack, we get the expected failing test. Progress!

However we don’t want to hardcode 123 into the expectation; we would prefer to write:

let ``Basic example`` () = 
    Given (123 |> ``is pushed onto a stack``) |> 
    When ``stack is popped`` |> 
    Then ``result is`` 123 

So now we want to make Then ``result is`` expected_value actual_value have the signature ('a -> unit) -> 'a -> unit. Again, partially applying ``result is`` to Then needs to yield a function with signature 'c -> 'c -> unit (I’ve changed the generic letter for clarity) i.e. 'a must be ('c -> 'c). So, ``result is`` must have the signature ('c -> 'c -> unit):

let ``result is`` (expected: 'a) (actual: 'a) = 
    Assert.AreEqual(expected, actual)

If we try to compile and run this, we still get a passing test…and if we change the expected value to 999, we still get the expected failing test. More progress!

Can we make the syntax more readable and in line with BDD terminology?

let ``Basic example`` () = 
    Given (123 |> ``is pushed onto a stack``) |> 
    When ``stack is popped`` |> 
    Then it should ``return`` 123 

Comparing this to the previous example, we can imagine that ``return`` takes the place of ``result is``, with it and should just being syntactic “filler words”. i.e. we are looking for signature for it and should such that Then return 123 and Then it should return 123 have the same signature. What function does nothing? How about:

let it f = f 

and

let should f = f 

They both have signature 'a -> 'a. So in the above example, with ``return`` having signature ('c -> 'c -> unit), both should return and it should return also have signature ('c -> 'c -> unit). This all looks promising…and yes, the code still compiles and we still have the expected passing and failing tests.

So at this stage the entire implementation (and two test cases) looks like:

namespace ItWorksOnMyMachine

open System.Collections.Generic

open NUnit.Framework

module FSpec =
    let Given precondition =
        precondition

    let When action precondition = 
        action precondition

    let Then expectation actual =
        expectation actual

    let it f = f

    let should f = f

    let ``return`` (expected: 'a) (actual: 'a) =
        Assert.AreEqual(expected, actual)

module FSpecExamples =
    open FSpec

    let ``is pushed onto a stack`` n =
        let s = Stack()
        s.Push n
        s
        
    let ``stack is popped`` (stack: Stack) = 
        let v = stack.Pop()
        v

    [<Test>]
    let ``Basic passing example`` () =
        Given (123 |> ``is pushed onto a stack``) |>
        When ``stack is popped`` |>
        Then it should ``return`` 123
       
    [<Test>]
    [<ExpectedException(typeof<AssertionException>)>]
    let ``Basic failing example`` () =
        Given (999 |> ``is pushed onto a stack``) |>
        When ``stack is popped`` |>
        Then it should ``return`` 123

I would recommend comparing the above implementation (full solution here) to the NaturalSpec code. Even when you strip away all the nice extra features that NaturalSpec provides, you are still left with a very different implementation: one that would never have occurred to me.

Next time I’ll look at an implication of those differences, and try extending my implementation to allow asserting that expected exceptions are thrown, rather than relying on the ExpectedException attribute.