We recently had a strange behaviour in Azure DevOps: When we run our tests there, the test runner found nearly 16’000 tests. If we run the tests for the same solution in Visual Studio, we only had about 2’000 tests. Where did those additional 14’000 tests come from?
We checked every setting in the test pipeline over and over again, but with no success. We even switched to a different build server to make sure that there are no old artifacts sitting around and influencing our tests. Nothing explained the behaviour we had.
We finally found a clue in the 30’000 lines of log messages for the test task. In the section for starting test execution, we get a list of all test projects vstest.console.exe will run:
******* Starting test execution ***********
vstest.console.exe “C:\*\s\tests\Project.Integration\bin\Debug\Project.Integration.dll”
“C:\*\s\tests\Project.Tests\bin\Debug\Project.Tests.dll”
“C:\*\s\tests\Project.ViewModels.Tests\bin\Debug\ Project.ViewModels.Tests.dll”
“C:\*\s\tests\Project.ViewModels.Tests\bin\Debug\Project.Tests.dll”
Well hidden in that long list was the source of our problem: One test project was found in multiple locations and therefore ran multiple times. But why was that specific test project copied into multiple folders?
With the problem narrowed down, it took us not long to find the cause for our mystery: Someone had referenced the test project instead of the project with the helper functions for the tests. That simple mistake led to the copies of the test project in the other project output folders and so was detected multiple times by the test runner.
All we had to do to fix this problem was to remove the project references to the test project. The next push fixed our test numbers on Azure DevOps and everything worked again as expected.