Property-based TDD

By Matt Baker -
6 minutes - 1160 words

Property-based testing is a type of testing that uses randomly generated inputs to test an attribute or characteristic of the subject under test. You can contrast this with the more traditional example-based testing approach, where you provide specific test cases for your subject under test. Typically, TDD is done using example-based testing. What happens when we use property-based testing in a TDD workflow?

The tools I’m going to use are .Net Core, C#, FsCheck and the FsCheck.XUnit extension.

I’m going to use FizzBuzz as our example problem. If you aren’t familiar, check that out here.

If you’d like to follow along

  1. Create a new project in .Net Core:
dotnet new xunit -o PropertyBasedTesting
cd PropertyBasedTesting/
dotnet add package FsCheck
dotnet add package FsCheck.Xunit
  1. Add a few using statements to your test file:
using System;
using FsCheck;
using FsCheck.Xunit;

Our first test

For our first test, let’s ensure that anything divisible by 3 and not divisible by 5 returns “Fizz”:

[Property]
public Property anything_divisible_by_three_but_not_five_returns_fizz(int input)
{
    Func<bool> property = () => Fizz.Buzz(input) == "Fizz";

    return property.When(input % 3 == 0 && input % 5 != 0);
}

We can pass the first test with:

public class Fizz
{
    public static string Buzz(int input) => "Fizz";
}

Using the fuzzing capabilities that are built into FsCheck, we can generate random inputs for our function. FSCheck also provides conditional properties – this allows us to constrain our input to be only integers that are divisible by three, but not divisible by five.

Our second test

Our second test will ensure that any numbers divisible by 5 and not divisible by 3 will return “Buzz”:

[Property]
public Property anything_divisible_by_five_but_not_three_returns_buzz(int input)
{
    Func<bool> property = () => Fizz.Buzz(input) == "Buzz";

    return property.When(input % 5 == 0 && input % 3 != 0);
}

We can pass this test with:

public class Fizz
{
    public static string Buzz(int input) => input % 5 == 0 ? "Buzz" : "Fizz";
}

Our third test

Next, let’s test to ensure that we return “FizzBuzz” when our input is divisible by both 3 and 5:

[Property]
public Property anything_divisible_by_three_and_five_returns_fizzbuzz(int input)
{
    Func<bool> property = () => Fizz.Buzz(input) == "FizzBuzz";

    return property.When(input % 3 == 0 && input % 5 == 0);
}

I tried to pass this test with the implementation below, but hit an error. The implementation:

public class Fizz
{
    public static string Buzz(int input)
    {
        var output = "";

        if (input % 3 == 0) output += "Fizz";
        if (input % 5 == 0) output += "Buzz";

        return output;
    }
}

FsCheck tried to generate random inputs for me that satisfied the condition When(input % 3 == 0 && input == 5), but it times out before it can come up with enough test cases. After some googling, it turns out that this is expected behavior in FsCheck – it will only try to generate so much test data before it just gives up. We can get around this by writing a custom generator for our test data.

Implementing a custom generator

A custom generator allows us to apply conditionals to the data that is generated for fuzzing. The implementation that I came up with looks like this:

public static class FizzBuzzGenerator
{
    public static Arbitrary<int> Generate()
    {
        return Arb.Default.Int32().Filter(x => x % 3 == 0 && x % 5 == 0);
    }
}

We can instruct our failing test to use the new generator via the Arbitrary attribute:

[Property(Arbitrary = new[] { typeof(FizzBuzzGenerator) })]
public Property anything_divisible_by_three_and_five_returns_fizzbuzz(int input)
{

    var actual = Fizz.Buzz(input);

    return (actual == "FizzBuzz").ToProperty();
}

Our final test

We’re on the home stretch now. The last thing we need to do is write a test to confirm that when the input is not divisible by three or five, we return the input. Our final test:

[Property]
public Property anything_not_divisible_by_three_and_five_should_return_value_as_string(int input)
{
    Func<bool> property = () => Fizz.Buzz(input) == input.ToString();

    return property.When(input % 3 != 0 && input % 5 != 0);
}

This test is currently failing. I have a sense for why it’s failing, but it’s kind of hard to know which input it’s failing on. We can update the test to “label” the failures so that we can see the exact input that is causing the test to fail. In order to add the labeling, we do the following:

[Property]
public Property anything_not_divisible_by_three_and_five_should_return_value_as_string(int input)
{
    Func<bool> property = () => Fizz.Buzz(input) == input.ToString();

    return property.When(input % 3 != 0 && input % 5 != 0).Label($"Failed on input {input}");
}

The Label function will now write a string to the console containing the exact input that we are failing on. With that in hand, we can see the failure and update our FizzBuzz implementation as follows:

public class Fizz
{
    public static string Buzz(int input)
    {
        var output = "";

        if (input % 3 == 0) output += "Fizz";
        if (input % 5 == 0) output += "Buzz";

        return string.IsNullOrEmpty(output) ? input.ToString() : output;
    }
}

A quick retrospective

This exercise was a bit easier than I had anticipated. However, I can see where coming up with properties in a more complex domain would be difficult. There are some great guides for finding properties, but I’ve yet to try this with a “real” application. One of the benefits that I’ve found with TDD is that it helps you discover your design by writing simple tests and allowing your solution to “emerge” over time. With property-based testing, coming up with tests does not feel as “simple”. But – that is my naive assumption based on a lack of experience. I also found myself wondering if you could use example-based tests to discover properties.

For more information on property-based testing, check out the following links:

Our full source code is below.

using System;
using FsCheck;
using FsCheck.Xunit;

namespace property_based_tests
{
    public class FizzBuzzTests
    {
        [Property]
        public Property anything_divisible_by_three_but_not_five_returns_fizz(int input)
        {
            Func<bool> property = () => Fizz.Buzz(input) == "Fizz";

            return property.When(input % 3 == 0 && input % 5 != 0);
        }

        [Property]
        public Property anything_divisible_by_five_but_not_three_returns_buzz(int input)
        {
            Func<bool> property = () => Fizz.Buzz(input) == "Buzz";

            return property.When(input % 5 == 0 && input % 3 != 0);
        }

        [Property(Arbitrary = new[] { typeof(FizzBuzzGenerator) })]
        public Property anything_divisible_by_three_and_five_returns_fizzbuzz(int input)
        {
            var actual = Fizz.Buzz(input);

            return (actual == "FizzBuzz").ToProperty();
        }

        [Property]
        public Property anything_not_divisible_by_three_and_five_should_return_value_as_string(int input)
        {
            Func<bool> property = () => Fizz.Buzz(input) == input.ToString();

            return property.When(input % 3 != 0 && input % 5 != 0).Label($"Failed on input {input}");
        }
    }

    public class Fizz
    {
        public static string Buzz(int input)
        {
            var output = "";

            if (input % 3 == 0) output += "Fizz";
            if (input % 5 == 0) output += "Buzz";

            return string.IsNullOrEmpty(output) ? input.ToString() : output;
        }
    }

    public static class FizzBuzzGenerator
    {
        public static Arbitrary<int> Generate()
        {
            return Arb.Default.Int32().Filter(x => x % 3 == 0 && x % 5 == 0);
        }
    }
}
Categories: technical
Tags: testing, tdd, c-sharp