We are currently migrating our larger databases to SQL Server 2019. While the migration went smoothly for the small databases, there was an unexpected problem with the last server: the backup was incredibly slow.
The first larger database we moved to the new server took 10 times longer to back up than on the old server (50 minutes instead of 5). If the backup time would proportionally increase with the database size, our biggest database would use the whole maintenance window just for the backup task. We had to fix this problem immediately.
We tried multiple backup files and storing backup files on different disks, but nothing increased the speed significantly. All we could find in the SQL Server was a very high value for WAIT_XTP_HOST_WAIT. Unfortunately, SQL Server does not reveal what is causing this long wait and we had to keep searching for clues.
By sheer luck I thought about antivirus scanners and lo and behold, on the new machine runs Microsoft Defender. After we stopped Microsoft Defender, the backup run in a similar time to the old server. If we look at the entries in the event log we see how massive this impact is:
Before (with active Microsoft Defender):
BACKUP DATABASE successfully processed 18578717 pages in 3841.345 seconds (37.785 MB/sec).
After (without Microsoft Defender):
BACKUP DATABASE successfully processed 18578717 pages in 439.837 seconds (330.000 MB/sec).
I assume the problem comes from the many documents we manage with the FILESTREAM feature in SQL Server. If every document gets checked for viruses it will take a huge amount of time. However, this part is pure speculation and I have no way to proof it.
If you run into similar problems you should look at Microsoft Defender and how you can change its settings. This could save you days of hunting for a performance bottleneck.