Technical Article, Testing

Data-driven testing

May 27, 2019
Bill Hodghead

Data driven testing solves the problem: “I want to try a bunch of different test values, but I don’t want to write a different test for each one.”

In data driven testing, you reuse the same test code with different values.

Data-driven testing is so common and valuable that I’ve seen teams set up their unit test templates to always be data driven by default. Team members must change the test if they don’t want it to be data driven.

Data-driven testing is sometimes called “table-driven testing.” However, the inputs don’t have to be rows in a table. They could be any data collection.

To make the code easy, the data for each test includes the expected result.

Here’s a typical data driven test the pseudo code:

For each data row
      Run the test using some of my data in this row as parameters
      Compare the test output to the expected result that’s also in the data
      Log each test result, but continue the test for the remaining rows 
Next row

Password example

Let’s look at an example where we are testing a login function:

bool Login(username, password)

As you might expect, we want to try several valid and invalid tests such as the ones in the table below:

Username	Password	Expected Login Result
<blank>	<blank>	Not logged in
Joe	P@ssword	Logged in
Bob	Wrong	Not logged in
Judy	<blank>	Not logged in

We can do that with the following code:

[Test, Sequential] 
public void LoginWithPositiveAndNegativeValues(
[Values("", "Joe", "Bob", "Judy")] string username, 
	[Values("", "P@ass", "Blah", "")] string password)
	[Values(0, 1, 0, 0)] bool expectedResult)
{
     Assert.That(Login(username, password), Is.EqualTo(expectedResult); 
}

In this case, the values are just coded right into the test, though they could come from an external source like a file or a database. The advantage of putting them in the test is that it’s easy to see everything in one place. Avoid using a database for unit testing. It will slow down your tests too much.

Data driven tests have several advantages. They make it:

Easy to add new test variations
Easy to review multiple tests in one place
Easy to code

The problem with the code above is that it stops on the first failure. If you want it to return all the failures, you need something more like this example in C#:

public struct LoginRow
    {
        public string Username;
        public string Password;
        public bool ExpectedResult;
	public string TestName;
        
        public LoginRow (string username, string password, bool expectedResult, string testname)
        { 
            Username = username;
            Password = password;
            ExpectedResult = expectedResult;
	    TestName = testname;
        }
    }

[TestClass]
public class LoginTests
{
     static readonly List<LoginRow> TestData = new List<LoginRow>();

     [ClassInitialize()]
     public static void BuildTestData(TestContext context)
     {
          TestData.Add(new LoginRow("","",0,"failure username is blank"));
          TestData.Add(new LoginRow("Joe"," P@ass",1,"pass with valid inputs"));
          TestData.Add(new LoginRow("Bob","Blah",0,"failure with bad password"));
          TestData.Add(new LoginRow("Judy","",0,"failure when password is blank"));
    }

    [TestMethod] 
    public void LoginWithPositiveAndNegativeValues(
    {
         bool failed = false;

         foreach (LoginRow test in TestData)
         {
             try
             {
                 Assert.AreEqual(Login(test.Username, test.Password),
                     test.ExpectedResult, test.TestName);
             }
             catch (Exception e)
             {
                 failed = true;
                 Debug.WriteLine(e);
             }
         }
         if (failed) 
             Assert.Fail("One or more tests failed, check debug output.");
    }
}

General guidance

It can be useful to include a description or name of the test in with the data so you know what those values mean.

If you feel like you need more that one input data set, you may want a different technique, such as a test oracle.

If you find that you have more than one loop or other extra complexity, this problem may be better suited for either simple, hard-coded tests (if there are few values), or a test oracle (if there are many values).

Summary

Data-driven testing is the foundation for the rest of the techniques in this series. The next article is on equivalence classes and boundary value analysis.