dateo. Coding Blog

Today let’s explore how you can improve the stability, maintainability, and readability of your CI and CD pipelines using custom test categories in xUnit.

I have worked in a team where xUnit was used as the testing framework in .NET. And with good reason: xUnit is a very versatile and easily extensible platform for writing unit and integration tests in .NET. During this time, developers always added the Trait attribute to each of their test classes, marking them as either unit, integration, or even UI tests. This way, the CI pipeline set up in Azure DevOps was able to group all tests accordingly and execute them in their own separate build step. As of recently, and in a different project and team, the approach is based more on the tests residing in specific namespaces. While that does work, most of the time, it also raises concerns about the maintainability of the CI and CD pipelines, which I want to discuss in this article.

So let’s have a look into those two different approaches.

Approach 1: Namespace-based pipelines

Let us have a glance at the step in the pipeline that performs the actual testing first. Whether in your project you are using Azure, Github, or TeamCity does not matter, often it will probably all boil down to a call of dotnet test.

If your team is keen on relying on the tests having very specific namespaces, the calls for different build steps may look something like this:

# Run all unit tests
dotnet test --filter "(FullyQualifiedName!=Integration.Tests)&(FullyQualifiedName!=System.Tests)"

# Run all integration tests
dotnet test --filter "(FullyQualifiedName=Integration.Tests)&(Category=ReadyForProduction)"

...

Now, without judgment, this approach does of course work, if you manage to have your code organized as follows:

src/
   Core.csproj
   Infrastructure.csproj
   ...
test/
   Core.Tests.csproj
   Integration.Tests.csproj
   ...

In that specific team, the code is distributed over a dozen repositories, and in most of them, the namespaces were correct. Except in one, for which the respective pipeline would not execute the integration tests for weeks, because they were located in a namespace IntegrationTests that would be filtered out downstream of the pipeline. Of course, one might argue, that if developer XY had used the solution template and if developer XY had used the correct namespace, such a thing could have been prevented. Except it wasn’t because people are people. Apologies for being judgemental already.

Let’s consider a different approach.

Approach 2: Category-based pipelines

First, we’ll look again at the respective pipeline command that will execute the tests for us:

# Run all unit tests
dotnet test --filter "Category=Unit"

# Run all integration tests
dotnet test --filter "(Category=Integration)&(Category!=SkipInProduction)"

...

In contrast to approach 1, here we are using a whitelist approach for the basic test categories unit and integration. For sake of completeness, I also added the filtering for tests that might need to be skipped in productive environments.

Now, the solution structure can be organized as follows:

src/
   Core.csproj
   Infrastructure.csproj
   ...
test/
   Application.Tests.csproj
   Core.Tests.csproj
   Integration.Tests.csproj
   MyThirdPartyLibrary.IntegrationTests
   ...

I have deliberately added bad choices for project names here because it does not matter anymore. Using this approach, the pipeline is agnostic of the underlying solution structure.

Let us have a look into the actual implementation in a xUnit project.

Implementation of approach 2 in xUnit

To make this approach as robust as possible, I suggest to implement custom test categories and attributes in xUnit. As a result, your test classes may look something like this:

[Category(TestCategory.Unit)]  
public class UnitTests  
{
	[Fact]  
	public void Unit_Test_1() { ... }
	
	[Fact]  
	public void Unit_Test_2() { ... }

	[Fact]
	public void Unit_Test_3() { ... }
}

[Category(TestCategory.Integration)]  
public class IntegrationTests  
{
	[Fact]  
	public void Integration_Test_1() { ... }
	
	[Fact]  
	public void Integration_Test_2() { ... }

	[Fact]
	[Category(TestCategory.SkipInProduction)]  
	public void Integration_Test_3() { ... }
}

In order to make this work, xUnit requires us to implement both a custom CategoryAttribute and a custom TraitDiscoverer. The latter is the actual class that is used to discover executable unit tests in a test class.

[TraitDiscoverer("Core.Tests.CategoryDiscoverer", "Core.Tests")]
[AttributeUsage(AttributeTargets.Method | AttributeTargets.Class, AllowMultiple = true)]
public class CategoryAttribute : Attribute, ITraitAttribute
{
    public CategoryAttribute(TestCategory category)
    {
    }
}

public class CategoryDiscoverer : ITraitDiscoverer
{
    public const string Key = "Category";

    public IEnumerable<KeyValuePair<string, string>> GetTraits(IAttributeInfo traitAttribute)
    {
        var ctorArgs = traitAttribute.GetConstructorArguments().ToList();
        yield return new KeyValuePair<string, string>(Key, ctorArgs[0].ToString());
    }
}

public enum TestCategory
{
    Unit,
    Integration,
    SkipInProduction
}

These classes can be placed anywhere in a common test project to be shared among your tests. Note that by specifying the attribute usage, you can place your test categories both on methods and classes. Even assembly-wide categories are possible when this code is changed accordingly.

Whenever the tests in this class are executed, xUnit will detect that these tests need to be discovered using the class specified in the CategoryAttribute. It will then use the corresponding implementation of ITraitDiscoverer and turn the constructor arguments into meaningful arguments for the test runner.

xUnit already supports custom test categories using the TraitAttribute on classes and methods like Trait("Category", "Unit"). Since string typing can be prone to errors, I personally prefer the presented approach using an enumeration value, even though that requires the implementation of a custom trait discoverer.

Pros and Cons

You already guessed, I am a fan of approach number 2. You are invited to make up your own mind about this, so let me summarize the pros and cons of using it:

Needless to say, there are things to be argued about in both approaches. Having used approach 2 in the past several years, and having seen approach 1 fail just recently, I do favor test categories over namespace conventions. Let me know what experiences you have made! Maybe there is a thing or two you could use from this article.