Jan 18 2012

Publish NUnit test results as part of Team Build in Team Foundation Server 2010

Tagged: Team Foundation Server Unit Testing

UPDATE 13.1.1011: I've uploaded a working process template with a sample project to http://www.heikura.info/publishing-xunit-test-results-as-part-of-your-team-build-2010

 

Publishing unit test results to TFS from other sources than MSTest was difficult with TFS 2008. New version makes this easier but still the actual format of the test result causes some problems. I've used http://nunit4teambuild.codeplex.com/ previously to publish NUnit test results to TFS 2008 and it has worked well. Newest version of nunit4teambuild includes support for TFS 2010 but does not include a custom Workflow activity to execute the publishing.

As I use NUnit based unit tests with some of my projects I decided to resolve this issue. Instead of trying to create my own custom Activity I decided to use InvokeProcess-activity and couple of other customisations to the default build process to get the tests running and results published as part of my build to TFS.

Result

Test results from NUnit test runs should be published as part of your build to TFS. Test results should be shown as part of the build log. Following is a sample screenshot taken from build log file with two test runs shown. One of them is MSTest test run and another is NUnit test run. Actual results can be viewed by clicking View Test Results-link. (Not in the picture ofcourse).

Required softwared

You need to install following software into your build machine:

  1. NUnit (I'm using latest recommended version http://www.nunit.org)
  2. nunit4teambuild (Version 2.0 Alpha 1 http://nunit4teambuild.codeplex.com/releases/view/45765)
  3. Team Build 2010

I usually create a c:\tools\ folder and copy executables without installer under named folders under that folder. For example I installed NUnit but copied the nunit4teambuild to folder c:\tools\NUnitPublished on the build server.

Build process changes

Following screenshots show the changes required to the build process. It is a good practice to branch your own build process template and do the changes in it. Open the build process template into the workflow editor by double clicking it at the source control.

Add following Arguments.

You can think the workflow arguments like a method arguments. NUnitBinaryDirectory should point to the folder on the build machine which contains nunit-console.exe. You can also edit those arguments afterwards from the build definition editor. Screenshot below. I probably will be adding few more Arguments later as an update but for now those should be enough.

 

Locate the correct position in workflow . Correct location depends on the settings used in your build definition. I'm using following position as I have specified a TestSettings-file in my solution.

Above is the modified workflow I'm using to run, publish and change the TestResult of my workflow. You should concentrate on Invoke NUnit, Publish NUnit results and If NUNit tests failed - activities. Notice that I'm running NUnit tests before MSTest. This actually was needed because MSTest activity throws errors which are catched by higher level activities.

Add one variable scoped to the selected workflow position. Select the workflow and then add the variable.

This variable will hold the result of the nunit-console.exe. It will return 0 if no tests failed, negative value is something went wrong with while executing the tests and positive number is actually the count of failed tests. 

Run NUnit

Add a invoke process actity to the same position as specified above. Then set the properties of that activity to following.

This will invoke NUnit from the location specified by the NUnitBinaryDirectory - argument. I'm passing the NUnitTests.dll as the assembly containing NUnit tests. This could be changed to find the test assemblies using the pattern specified at build definition. NUnit results will be stored into nunit-tests.xml file for later use. Result of the invoke process will be stored to NUnitResult.

 

Publishing results

Next step is to take the result xml-file of nunit-console.exe and publish results as test run to the TFS.

Drag new InvokeProcess-activity and place it after the previous one. We will invoke nunit4teambuild publishing executable with this activity. Before doing this please check the configuration file of the nunit4teambuild and verify that the TFS 2010 service URI's are pointing at the correct services. Remember to include project collection as part of the URI's or publishing will fail.

Set the properties of the InvokeProcess-activity to following:

 

Here is the whole expression of the Arguments-property:

Following could be enchanced to read the result file name from argument and project name from build variables.

 

You can now save and checkin the new build process. After that use it with a build and verify that results are being published as part of your build. You can debug the InvokeProcess-activities better if you add WriteBuildMessage-activities to stdOutput and errOutput parts of the InvokeProcess. Check the workflow overview in the beginning of this article for a sample. Remember to change the logging to Diagnostics in build definition to see full output.

 

Fail build if tests fail

After the two activies are working you can tweak the process so that the build actually fails when the tests fail.

Add a If-activity after the previous one and set the condition expression to NUnitResult <> 0. Add a Assign-activity to the If-activity. Check the workflow image in the beginning for sample.

Following are the settings for the Assign - activity. It will set the BuildDetail.TestStatus to Failed. This is used later by the build process to fail the build.

 

That is all the changes required. Some of the hardcoded values could be changed to variables to add more flexibility.

 

 

21 Comments

  • pekka said

    Cleaned up the article and fixed the CSS to correctly maintain size of the images.

  • David Gardiner said

    The URL for the web services in my TFS 2010 install are like this:

    http://tfsserver:8080/tfs/_tfs_resources/Build/v3.0/BuildService.asmx

    -dave

  • pekka said

    If you are using ProjectCollections I recommend using the direct address to services under it.

    Example:
    http://tfsserver:8080/tfs/MyCollectionNameHere/Build/v3.0/BuildService.asmx

    You can check that the Uri is pointing to correct location by using you web browser.

  • Jason said

    Great post! Its strange that there isn't a LOT more information out there on this - your post was the only relevant information I was able to find for VS2010! Is there any chance you'd consider posting a sample build script? While your screenshots are quite detailed there's nothing quite like playing with an example!

    Regardless, thanks for the great post!

  • Darrell Sveistrup said

    You have put your logic to run when
    testAssembly.HasTestSettingsFile is true

    What do you have to do in a NUnit test dll to make this condition true.

    Im using the baseline test project from the zip file off http://nunit4teambuild.codeplex.com
    And that condition is always false and just runs the MSTest's not the NUnit logic

  • pekka said

    Sorry for the delayed reply Jason and Darrell. I'll try providing a sample build process as soon I as I get my TFS dev enviroment back up and running. Lost it due to a it being run on very old hardware.

    testAssembly.HasTestSettingsFile is true if you have one testsettings file as part of your solution. If you don't need it you could moving those actions out of that condition.

  • pekka said

    Yes, I know those images aren't scaled correctly. I've just upgraded to latest Orchard release but my custom theme is still not upgraded for it.

  • Richard Banks said

    Hey, thanks! Now I don't have to write this post myself :-) I'll include a link to it on the Nunit4TeamBuild codeplex site if that's OK.

  • pekka said

    Yes, go ahead and add that link. I've been actually planning on doing a revision to this article. Removing those hard coded "config" parameters and moving the actual test execution to its own step.

  • Andy Macdonald said

    Great post! I just wondered if you could clarify a couple of things, what is the content of the "Write NUnit results" activity and the "Convert Server Path to..." activity?

    I've got NUnit executing but it sticks on the "Invoke NUnit" and never moves on despite the test results being put into the xml file.

  • pekka said

    "Write NUnit results" is same as WriteBuildMessage. It will write the stdOutput of nunit-console.exe to your build log. You might need to change your build logging level to something else than the default to actually see those results. "Convert Server Path to.." is the same as in the default build process file. If the process is stuck at "Invoke NUnit" process than the probable cause is that nunit-console is waiting for user input. You might want to check it by trying to execute the same command that the build process is running directly from the build machine. Maybe you have a invalid parameter in the execution or you are trying to execute the command under program files directory. Program files directory requires elevated privileges when running under Windows Server 2008 or Windows 7. Check that working directories match your output directories in the invoke process actions.

  • Thomas said

    Hi, great post. Thanks for that.
    I have the same problem like Andy. It seems this is a NUnit problem. When I start nunit-console.exe it starts 2 processes (nunit-console.exe and nunit-ganent.exe). After test execution the nunit-agent.exe process remains. I prosume the InvoceProcessActivity is waiting untill all started processes are ended before continuing. Therefore the workflow hangs at this activity. But the worst thing is that the nunit-agent.exe process is restarted every time I start nunit-console.exe which means if I start it 10 times from cmd, I have 10 nunit-agent.exe processes.

  • pekka said

    Thomas and Andy it seems that there is a known issue with NUnit 2.5.8 which spawns those nunit-agent processes. You can find a solution from http://stackoverflow.com/questions/4303421/nunit-2-5-8-lots-of-nunit-agent-exe. I haven't tested it yet.

    Supposedly it is fixed in 2.5.9 dev builds of NUnit also.

  • VJ said

    Good and very useful post on NUnit vs Teambuild 2010. I've set up the team build to use NUnit. The test ran successfully creating a xml file. The next step of publishing is also successful without giving any error but i dont see them published in Build summary.

    Result from teambuild 2010:
    C:\util\NUnitPublisher\NUnitTfs.exe -n eCareDomainTests.xml -t eCare -f Debug -b "3.0.2.48" -v 2010
    Loading NUnit Output from eCareDomainTests.xml...
    Transforming NUnit output to .trx file...
    Adding build information to .trx file...
    Publishing .trx file...
    Checking test run does not exist...
    Reading users identity...
    Creating test run...
    Creating test results...
    Updating test run with details...
    Uploading test run details...
    Uploading assemblies...
    Updating test results with details...
    Uploading test result details...

  • Ken said

    great post Pekka, thanks. I'm having the same issue as VJ - the nunit tests run and get published, but I don't get the "1/1 test passed..." message in my build summary. any thoughts?

  • pekka said

    Same thing seems to be happening on my builds lately. I think Microsoft updated something trough Windows Update which is causing something to go wrong with the result publishing. I'm currently on holiday but I will be investigating the problem in more detail next week. If I find the cause I will publish the fix here.

  • Mathias said

    Thank you for that great blog post!

    I've got the same problem as Ken and you, results are not published and the build summary shows 0/0 Test(s) passed. Enjoy your holiday, I'm looking forward to read about a fix for that problem when you're back :-).

  • pekka said

    I just noticed that my builds were failing because someone had added /t:Publish to the command line parameters for MSBuild in my build configuration. After I removed it my tests started working again. So it seems that the problem I had was not caused by any updates released by Microsoft.

    I'll try reproducing your issues later today if I can get my TFS VM to play nice.

  • Thomas said

    Hi,

    I'm trying to set it up but can't get it to work as my template seems different from yours. Could you please post your template ?

    Thanks

  • pekka said

    I've uploaded a working build process template with a sample project as part of a new article which you can find from http://www.heikura.info/blog/publishing-xunit-test-results-as-part-of-your-team-build-2010.

Comments have been disabled for this content.