Last week I run into the strangest problem with Azure DevOps since we started using it in 2015. From one day to the next, one of our local build servers took massively more time to do its work. Build steps that took 30 seconds now run for 10 minutes, some even for more than an hour.
Nothing had changed in the application or in our environment, but now we had an unusable situation. I first checked status.dev.azure.com to see if there was a known problem. All services where up and running, no news on Twitter and Google only showed problems like mine from years ago.
Neither updating the build agents nor restarting the server did any good. And the suspected antivirus still had all directories of the build agent excluded. There where no recent patches installed on the server and we did not change anything in the network. While this build server made us such troubles, the other one kept building our software as fast as ever.
One job had a noticeable difference besides the runtime: a few tests for combining PDF files failed. The error message was that the file already existed. Surprisingly, this error persisted with a newly configured agent, that never run anything before. As it turned out, that was the hint that helped me to solve this mystery. The exception happened on the line where we create a temporary file. The folder C:\temp was empty, but as it turns out, the build agents run under the NetworkService account and use the temp folder at this location:
%Windir%\ServiceProfiles\NetworkService\AppData\Local\Temp
Inside this folder where more than 100’000 files and the algorithm behind Path.GetTempFileName() could not return an unused path. I deleted all files in this temp folder, what not only fixed the failing tests but also got rid of the slow build time. Most builds now run a little bit faster than before this incident.
This is one of those unbelievable solutions that take an awful lot of time to figure out. Therefore, if your local build agent starts to slow down, look at the (right) temp folder and clean that up.
If i remember it correctly, the configuration wizard asks you, if you want to run the agent with the NetworkService account or as a different user (which i think its also recommending to do if you plan to run the agent as a service)
Hi Tim,
You are right, the wizard asks you if you want to use the NetworkService account or another. I expect that most people will just hit enter and never bother to check the recommendation in the documentation. If you changed the account, your temp folder will be with this selected account.
Regards,
Johnny