In this Article

This article is the 3rd part of a series about Roslyn Source Generators & co. In the 1st article, of this series, we built a Source Generator to generate a new property Items of a smart-enum. In the 2nd article, we added a Roslyn Analyzer and a Code Fix to prevent common mistake(s) and to help out the developers when using this Source Generator. All the code we have written so far was tested manually by executing the code and looking at the outcome. It is time to implement some automated tests to ensure the correct behavior.

Article series

  1. Code sharing of the future
  2. Better experience through Roslyn Analyzers and Code Fixes
  3. Testing Source Generators, Roslyn Analyzers and Code Fixes

More information about the smart enums, the source code, and the documentation of Microsoft.CodeAnalysis.Testing can be found on GitHub:

Different Testing Approaches

When testing Roslyn Source Generators, Analyzers, and Code Fixes then there are two different kinds of tests.

  • One is for testing the behavior of generated code. In our case, it would be a test for the existence of the property Items and whether it returns all defined items. Such tests are for testing the Source Generator only, i.e. the correct behavior of our smart-enums says nothing about the correct behavior of the Analyzer and the Code Fix.
  • The other kind of test verifies the emitted errors and warnings and the generated code, not just the one produced by the Source Generator but also by the Code Fix. With such tests, we are able to tests all pieces we implemented in previous articles.

Preparation

For testing generated code and emitted warnings and errors by the Analyzer, we need a few Nuget packages which are not on nuget.org.
Create a new file NuGet.config in the same folder as the solution file:

<?xml version="1.0" encoding="utf-8"?>

<configuration>
   <packageSources>
      <clear />
      <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
      <add key="dotnet-tools" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json" />
   </packageSources>
</configuration>

Next, create a new library project, DemoTests, which references both the DemoLibrary and the DemoSourceGenerator. Please note, that the reference to DemoSourceGenerator in DemoTests.csproj is missing the attribute ReferenceOutputAssembly="false" this time. For testing, we want both, generation of new source code, i.e. the standard functionality of a Source Generator, and a direct access to the classes DemoSourceGenerator, DemoAnalyzer and DemoCodeFixProvider.

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <TargetFramework>net5.0</TargetFramework>
    </PropertyGroup>

    <ItemGroup>
        <ProjectReference Include="..\DemoLibrary\DemoLibrary.csproj" />
        <ProjectReference Include="..\DemoSourceGenerator\DemoSourceGenerator.csproj"
                          OutputItemType="Analyzer" />
    </ItemGroup>

    <ItemGroup>
        <PackageReference Include="Microsoft.CodeAnalysis.CSharp.Analyzer.Testing.XUnit" 
                          Version="1.0.1-beta1.21208.1" />
        <PackageReference Include="Microsoft.CodeAnalysis.CSharp.CodeFix.Testing.XUnit" 
                          Version="1.0.1-beta1.21208.1" />
        <PackageReference Include="Microsoft.CodeAnalysis.CSharp.CodeRefactoring.Testing.XUnit" 
                          Version="1.0.1-beta1.21208.1" />
        <PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="3.9.0" />

        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.4" />
        <PackageReference Include="FluentAssertions" Version="5.10.3" />
        <PackageReference Include="xunit" Version="2.4.1" />
        <PackageReference Include="xunit.runner.visualstudio" Version="2.4.3" />
    </ItemGroup>

</Project>

In the following tests, we will be using XUnit and Fluent Assertions.

Testing Behavior

In order to test the expected behavior, we need (at least) one enumeration. Copy the ProductCategory from the project DemoConsoleApplication to DemoTests or create a new one.
These kinds of tests are very common, so one positive test should be enough as a starting point. In real scenarios, we should make negative tests and tests for the edge cases as well. An edge case would be an enumeration with no items.

Create a new class ItemsPropertyTests in the project DemoTests with the following content and let the test run.

using FluentAssertions;
using Xunit;

namespace DemoTests
{
   public class ItemsPropertyTests
   {
      [Fact]
      public void Should_return_all_known_items()
      {
         ProductCategory.Items.Should().HaveCount(2)
                        .And.BeEquivalentTo(ProductCategory.Fruits,
                                            ProductCategory.Dairy);
      }
   }
}

The test should be green.

Testing Roslyn Source Generator

Create a new class DemoSourceGeneratorTests in the project DemoTests for testing the output of the DemoSourceGenerator. First, we make a helper method that executes the Source Generator directly and returns the generated output.

using System;
using System.Collections.Generic;
using System.Linq;
using FluentAssertions;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Xunit;

namespace DemoTests
{
   public class DemoSourceGeneratorTests
   {
      private static string GetGeneratedOutput(string sourceCode)
      {
         var syntaxTree = CSharpSyntaxTree.ParseText(sourceCode);
         var references = AppDomain.CurrentDomain.GetAssemblies()
                                   .Where(assembly => !assembly.IsDynamic)
                                   .Select(assembly => MetadataReference.CreateFromFile(assembly.Location))
                                   .Cast<MetadataReference>();

         var compilation = CSharpCompilation.Create("SourceGeneratorTests",
                                                    new[] { syntaxTree },
                                                    references,
                                                    new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary));

         var generator = new DemoSourceGenerator.DemoSourceGenerator();
         CSharpGeneratorDriver.Create(generator)
                              .RunGeneratorsAndUpdateCompilation(compilation,
                                                                 out var outputCompilation,
                                                                 out var diagnostics);

         // optional
         diagnostics.Where(d => d.Severity == DiagnosticSeverity.Error)
                    .Should().BeEmpty();

         return outputCompilation.SyntaxTrees.Skip(1).LastOrDefault()?.ToString();
      }
   }
}

The method GetGeneratedOutput parses and compiles the provided sourceCode and executes the DemoSourceGenerator. The output generated by the Source Generator is the last one in the collection SyntaxTrees. Please note that the generator doesn't produce any output under certain conditions. That's why we Skip the first syntax tree that is the provided sourceCode.

With the helper method GetGeneratedOutput, the actual tests are reduced to the comparison of strings. A positive test looks like as following:

      [Fact]
      public void Should_generate_Items_property_with_2_items()
      {
         var input = @"
using DemoLibrary;

namespace DemoTests
{
   [EnumGeneration]
   public partial class ProductCategory
   {
      public static readonly ProductCategory Fruits = new(""Fruits"");
      public static readonly ProductCategory Dairy = new(""Dairy"");

      public string Name { get; }

      private ProductCategory(string name)
      {
         Name = name;
      }
   }
}";
         GetGeneratedOutput(input)
            .Should().Be(@"// <auto-generated />

using System.Collections.Generic;

namespace DemoTests
{
   partial class ProductCategory
   {
      private static IReadOnlyList<ProductCategory> _items;
      public static IReadOnlyList<ProductCategory> Items => _items ??= GetItems();

      private static IReadOnlyList<ProductCategory> GetItems()
      {
         return new[] { Fruits, Dairy };
      }
   }
}
");
      }

Testing Roslyn Analyzers and Code Fixes

For testing the Analyzers and Code Fixes, it is recommended to create a few helper classes as recommended by Roslyn team. Depending on whether we have Code Fixes or not, we need either an AnalyzerVerifier or an AnalyzerAndCodeFixVerifier. In this article, we will create both helper classes but just use the 2nd one.

Here are the contents of the class AnalyzerVerifier for the sake of completeness.

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using DemoLibrary;
using Microsoft.CodeAnalysis.CSharp.Testing;
using Microsoft.CodeAnalysis.Diagnostics;
using Microsoft.CodeAnalysis.Testing;
using Microsoft.CodeAnalysis.Testing.Verifiers;

namespace DemoTests
{
   public static class AnalyzerVerifier<TAnalyzer>
      where TAnalyzer : DiagnosticAnalyzer, new()
   {
      public static DiagnosticResult Diagnostic(string diagnosticId)
      {
         return CSharpAnalyzerVerifier<TAnalyzer, XUnitVerifier>.Diagnostic(diagnosticId);
      }

      public static async Task VerifyAnalyzerAsync(
         string source,
         params DiagnosticResult[] expected)
      {
         var test = new AnalyzerTest(source, expected);
         await test.RunAsync(CancellationToken.None);
      }

      private class AnalyzerTest : CSharpAnalyzerTest<TAnalyzer, XUnitVerifier>
      {
         public AnalyzerTest(
            string source,
            params DiagnosticResult[] expected)
         {
            TestCode = source;
            ExpectedDiagnostics.AddRange(expected);
            ReferenceAssemblies = ReferenceAssemblies.Net.Net50;

            TestState.AdditionalReferences.Add(typeof(EnumGenerationAttribute).Assembly);
         }
      }
   }
}

What we need for this demo is the AnalyzerAndCodeFixVerifier, so create a new class with the following content:

using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using DemoLibrary;
using Microsoft.CodeAnalysis.CodeFixes;
using Microsoft.CodeAnalysis.CSharp.Testing;
using Microsoft.CodeAnalysis.Diagnostics;
using Microsoft.CodeAnalysis.Testing;
using Microsoft.CodeAnalysis.Testing.Verifiers;

namespace DemoTests
{
   public static class AnalyzerAndCodeFixVerifier<TAnalyzer, TCodeFix>
      where TAnalyzer : DiagnosticAnalyzer, new()
      where TCodeFix : CodeFixProvider, new()
   {
      public static DiagnosticResult Diagnostic(string diagnosticId)
      {
         return CSharpCodeFixVerifier<TAnalyzer, TCodeFix, XUnitVerifier>.Diagnostic(diagnosticId);
      }

      public static async Task VerifyCodeFixAsync(
         string source,
         string fixedSource,
         params DiagnosticResult[] expected)
      {
         var test = new CodeFixTest(source, fixedSource, expected);
         await test.RunAsync(CancellationToken.None);
      }

      private class CodeFixTest : CSharpCodeFixTest<TAnalyzer, TCodeFix, XUnitVerifier>
      {
         public CodeFixTest(
            string source,
            string fixedSource,
            params DiagnosticResult[] expected)
         {
            TestCode = source;
            FixedCode = fixedSource;
            ExpectedDiagnostics.AddRange(expected);
            ReferenceAssemblies = ReferenceAssemblies.Net.Net50;

            TestState.AdditionalReferences.Add(typeof(EnumGenerationAttribute).Assembly);
         }
      }
   }
}

The only thing worth mentioning is the last line where we add the assembly reference DemoLibrary to the testing class CSharpCodeFixTest, otherwise the EnumGenerationAttribute will not be resolved, and the test fails.

Having the helper classes, the actual tests become much easier to write.
Create a new test class DemoAnalyzerAndCodeFixTests to test our DemoAnalyzer and the DemoCodeFixProvider for correct handling of non-partial enumeration classes.

using System.Collections.Generic;
using System.Threading.Tasks;
using DemoSourceGenerator;
using Xunit;
using Verifier = DemoTests.AnalyzerAndCodeFixVerifier<
   DemoSourceGenerator.DemoAnalyzer,
   DemoSourceGenerator.DemoCodeFixProvider>;

namespace DemoTests
{
   public class DemoAnalyzerAndCodeFixTests
   {
      [Fact]
      public async Task Should_trigger_on_non_partial_class()
      {
         var input = @"
using DemoLibrary;

namespace DemoTests
{
   [EnumGeneration]
   public class {|#0:ProductCategory|}
   {
   }
}";

         var expectedOutput = @"
using DemoLibrary;

namespace DemoTests
{
   [EnumGeneration]
   public partial class ProductCategory
   {
   }
}";

         var expectedError = Verifier.Diagnostic(DemoDiagnosticsDescriptors.EnumerationMustBePartial.Id)
                                     .WithLocation(0)
                                     .WithArguments("ProductCategory");
         await Verifier.VerifyCodeFixAsync(input, expectedOutput, expectedError);
      }
   }
}

The magic lies in the using Verifier = ... at the top of the file and Verifier.VerifyCodeFixAsync at the bottom, which get the input source code and expects the expectedOutput along with the expectedError. The rather strange characters {|#0: and |} is markup, so the Verifier knows what location the compilation error will be pointing to.

We were talking about location in the 2nd article. Look for the text classDeclaration.Identifier to get more information on that.

Let's run all three tests, which should be green.

Summary

This article concludes the introductory series about Roslyn Source Generators, Analyzer, and Code Fixes. As a matter of fact, writing tests turned out to be the easiest and fastest task compared to the previous two articles.

If you don't want to miss out on more articles, webinars, videos, and more, subscribe to our monthly dev newsletter.

Related Articles

 | Pawel Gerr

Article series Codesharing of the future Better experience through Roslyn Analyzers and Code Fixes ⬅ Testing Source Generators, Roslyn Analyzers and Code Fixes More background information about the smart-enums and the source code can be found on GitHub: Enum like classes (a.k.a…

Read article
 | Pawel Gerr

Article series Code sharing of the future ⬅ Better experience through Roslyn Analyzers and Code Fixes Testing Source Generators, Roslyn Analyzers and Code Fixes Code Sharing Today The most common approach for sharing code/functionality is providing base/helper classes or…

Read article
 | Pawel Gerr

By switching the to and setting to in the csproj-file we are now benefiting from the introduced with the latest version of C#. By enabling this new feature all type members, input and output parameters are considered to be not-null. If some members or parameters, like can…

Read article