F# Testing DSL Part 2

Last time we got to the stage where we could write the following test:

let ``Basic example`` () =
    Given (123 |> ``is pushed onto a stack``) |>
    When ``stack is popped`` |>
    Then it should ``return`` 123

I also suggested looking at NaturalSpec (which inspired these posts) to see how it tackles the same problem. If we strip out all extra features that NaturalSpec provides, we are left with:

let Given f = f

let When f = f

let It f = f // NaturalSpec equivalent of Then

let should f x y =
    f x y |> check // Implementation of check not shown

So the real work is all deferred to the should and check functions. The relevance to this post is that an exception that occurs during the action executed by the When is not caught, so we have to use the ExpectedException attribute to assert that an exception has occurred.

I’d like to avoid using the ExpectedException attribute, and instead be able to do something like:

let ``Basic exception Handling`` () =
    Given (Stack<int>()) |>
    When ``stack is popped`` |>
    Then it should throw<InvalidOperationException>

So how do we modify our implementation to support this? The first problem is that we need to catch the exception in the When clause and make it available to the Then clause. In other words, we now need to be able to pass either the system state or an exception to the Then clause. We can model this as:

type TestContext<'a> =
    | State of 'a
    | Exception of exn
	
    member this.ToState =
        match this with
        | State(s) -> s
        | Exception(e) -> failwith "Context does not represent a state."

    member this.ToException = 
        match this with
        | State(s) -> failwith "Context does not represent an exception."
        | Exception(e) -> e

I’ve added ToState and ToException helper properties to avoid repetitive pattern matching in the calling code.

We also need to modify the When function:

let When action precondition = 
    try
        State(action precondition)
    with
    | e -> Exception(e)

The TestContext will be passed to the Then clause, which means that ``return`` will need to be modified to accept a TestContext<'a> instead of an 'a:

let ``return`` (expected: 'a) (context: TestContext<'a>) =
    Assert.AreEqual(expected, context.ToState)

…and yes, the code still compiles and we still have the expected passing and failing tests.

Now we can move on to the throw<'e> function. We can cheat here and look back at the earlier ``result is 123``, which had a signature of 'a -> unit. By analogy throw<'e> should have a signature of TestContext<'a> -> unit:

let throw<'e> (context: TestContext<'a>) =
    Assert.IsInstanceOf<'e>(context.ToException)

However, this fails due to a compiler warning (you do treat warnings as errors, don’t you?):

FS0064: This construct causes code to be less generic than indicated by the type annotations. The type variable ‘a has been constrained to be type ‘int’.

This is because the state we are passing around is an int. In order to make throws properly generic we would need:

let throw<'e, 'a> (context: TestContext<'a>) =
    Assert.IsInstanceOf<'e>(context.ToException)

But that leads to an ugly calling syntax where we have to specify the state type:

let ``Basic exception handling`` () =
    Given (Stack<int>()) |>
    When ``stack is popped`` |>
    Then it should throw<InvalidOperationException, int>

The problem is that throw is dealing with two generic parameters when we only want it to deal with one. Can we extract the exception in one function, then pass it to the next to assert against? Something like this:

let ``Basic exception handling`` () =
    Given (Stack<int>()) |>
    When ``stack is popped`` |>
    Then it should throw any<InvalidOperationException>

The new throw could extract the exception from the test context, then any will be a simplified version of the original throw that gets passed the exception rather than the TextContext.

let throw checkExnType (context: TestContext<'a>) = 
    checkExnType context.ToException

let any<'e> ex =
    Assert.IsInstanceOf<'e>(ex)

More passing tests! However, we want to be sure that the correct InvalidOperationException is being thrown. Something like:

let ``Advanced exception example`` () =
    Given (Stack<int>()) |>
    When ``stack is popped`` |>
    Then it should throw (specific<InvalidOperationException> (fun e -> e.Message = "Stack empty."))

We need to add brackets around the specific function to ensure it gets evaluated first. Then, as long as it returns an exn -> unit (i.e. the same signature as any<'e>) everything should work out:

let specific<'e> (check: exn -> bool) (ex: exn) =
    Assert.IsInstanceOf<'e>(ex)
    Assert.IsTrue(check ex, "Exception condition not met:\n\n" + ex.ToString())

With this, the ``Advanced exception example`` passes. So we now have a DSL that allows us to assert that expected results and expected exceptions have occurred. However, what if the state that we are trying to verify is a floating point number? In that case Then it should return 1.234 is not a good idea.

Wouldn’t it be nice to be able to do the following?

let ``Comparing floating point numbers``() =
        Given 1.0m |>
        When ``dividing by 3`` |>
        Then it should be (0.333333333m +/- 0.00000001m)

More next time…

One thought on “F# Testing DSL Part 2

  1. Pingback: F# Testing DSL Part 3 | GET http://localhost…200 OK

Leave a Reply