Mar 4, 2010

MSTest Sucks for Unit Tests

Inflammatory title?  Maybe. But I'm not the only person to complain that MSTest is bad and should only be used when no other alternative is available.  But those who don’t know better will say that when you look at all the unit test frameworks they all do much the same right? So how can MSTest be that bad?
Well, first up, MSTest is slow. To be fair, MSTest when run from the command line runs quick. From the command line you won’t notice much difference at all between MSTest and the other unit testing frameworks, but who runs MSTest from the command line?  Most people run it from within Visual Studio, and when you do, it is really really sow. It does all that assembly copying behind the scenes, it waits for the test results window in the IDE to update and then, finally after all that is done, only then does it run the tests. If you’re writing and running lots of unit tests (you are, aren't you?) then this startup delay is real hindrance to flow and after a while is a major pain in the butt. So much so that many people stop doing tests after they hit a few hundred or so, simply because of the time it takes to run them.
But if speed were the only issue, then maybe the other features would make up for it.  After all it has those cool data driven tests right?
Well, lets consider an example.  Let’s say I’m doing the bowling game kata and I want to use a data driven test to check a number of scenarios.  Here’s what I’d have to do:
1. Create an XML file to act as a data source
<?xml version="1.0" encoding="utf-8" ?>
<Rows>
  <Row>
    <Scores>0123456789X</Scores>
    <Results>0,1,2,3,4,5,6,7,8,9,10</Results>
  </Row>
  <Row>
    <Scores>935345XXXX</Scores>
    <Results>9,3,5,3,4,5,10,10,10,10</Results>
  </Row>
</Rows>
2. Now create a data driven unit test using MSTest
[TestClass()]
public class ScoreEngineTest 
{ 
    private TestContext testContextInstance;

    public TestContext TestContext 
    { 
        get
        {
            return testContextInstance;
        } 
        set
        {
            testContextInstance = value; 
        } 
    } 

[TestMethod] 
[DeploymentItem(@"TestProject1\BowlingScores.xml")] 
[DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "|DataDirectory|\\BowlingScores.xml", "row", DataAccessMethod.Sequential)] 
public void ConvertStringToScoresTest_DataDriven() 
{ 
    ScoreEngine target = new ScoreEngine(); 
    var scores = (String)TestContext.DataRow["Scores"];
    var expectedData = ((string)TestContext.DataRow["Results"]).Split(',');
    var expected = new List<int>();
    foreach (var value in expectedData)
    { 
        expected.Add(int.Parse(value)); 
    } 
    var actual = target.ConvertScores(scores); 
    for (int i = 0; i < expected.Count; i++)
    { 
        if (expected[i] != actual[i]) 
            Assert.Fail("Expected {0}, Actual {1} at element {2}", expected[i], actual[i], i); 
    } 
}

Quite verbose right?  Especially the fact that there is no inbuilt list equality checks nor is there an easy way to use list based data in the data source.  I had to take a string, split it and manually create a list.  Eew!

So let’s look at the XUnit version.

public class Class1
{
    [Theory]
    [InlineData("0123456789X", new int[]{0,1,2,3,4,5,6,7,8,9,10})]
    public void ConvertStringToScoresTest(string scores, int[] results)
    {
        ScoreEngine target = new ScoreEngine();
        var actual = target.ConvertScores(scores);
        Assert.Equal(results,actual.ToArray());
    }
}


Wow! So much shorter! And so much more intuitive. And an Assert statement that actually does equality checks across IEnumerables as you would expect.

So maybe now you see why MSTest gets such a bollocking by so many people.
To be fair, MSTest is improving, and things like Test Impact Analysis in VS2010 are quite nice, but still... the amount of code to writing a simple data driven test is ridiculous and completely unnecessary.

8 comments:

  1. Hi Richard

    I couldn't agree more with you on this!
    Tell MS!!!

    Or
    Can we enhance mstest with that feature?
    I have read that it is quite hard to hook into mstest and extent it...

    My rant
    http://peitor.blogspot.com/2009/05/please-rowtests-in-visual-studio-2010.html

    ReplyDelete
  2. Richard,

    I have a disclaimer to make: I work for Microsoft.

    1) Regarding the slow "assembly copying" point - that behavior could be turned off in the .testrunconfig. Perhaps, it shouldn't be on by default, but (I think) it helps to isolate the test run (e.g. you can build while test is running). Also, if you use a test rig (controller/agent) - deployment becomes a requirement. See:

    http://msdn.microsoft.com/en-us/library/ms182480(VS.80).aspx

    2) Have you looked at CollectionAssert class? It has method that allow to compare collections. For example, see AreEqual, and AreEquivalent. Using CollectionAssert, you can get rid of the for-loop. See:

    http://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.collectionassert_members(v=VS.80).aspx

    Thanks.

    ReplyDelete
  3. @anonymous Why would I want to build code while tests are running? That's crazy talk!

    If I'm deploying to a test rig then I'm not going to be doing unit testing, I'll be doing web tests or integration tests. MSTest is fine for those situations and deployment isn't a problem

    Re CollectionAssert: I hadn't bumped into that before and it's definitely useful, but again proves the point that MSTest isn't good for unit testing - Why don't you have just one Assert class? The segmentation just leads to confusion.

    Thanks for the feedback in any case.

    ReplyDelete
  4. If you want to test hundreds of scenarios with the same test method, then MSTest approach is better. You don't want to contaminate your test class with 1000 [InlineData] attributes, do you?

    Also, an external data source (XML, CSV or database) is much better if your QA team want to be involved.

    ReplyDelete
  5. @Valik

    What you are saying makes sense about using external files for very large datasets and third party involvement.

    In many contexts tests by third parties and very large data sets are really useful tests. They are not what I think of when I think of Unit Tests and TDD, so I think given the scope of the article the criticism is valid.

    ReplyDelete
  6. if youre going to hardcode the InlineData anyway, why not simply put it as a local variable inside the test method or a testclass wide field? the problem i would see is. how would you implement the xml data as the test data in nunit. and please update your post to use CollectioonAssert. itherwise the info in this post is misleading.

    ReplyDelete
  7. For me, MSTest is annoying because I do not know how to run tests from a single test assembly.

    ReplyDelete
    Replies
    1. You can run tests from a single assembly by doing a right click on the test explorer and select "Group by Project". This is not intuitive UI design. "Group by project" should be a top level item in the test explorer.

      Delete