Microsoft Exchange Server: the bug that blocked thousands of emails

New year, new bugs! This time it was Microsoft Exchange Server's turn: the bug blocked the delivery of thousands of emails, which were stuck in the queue . The problem, as illustrated by Microsoft , is related to the lack of controls of the change of date of the new year. An unexpected " millennium bug " that occurred right at the stroke of midnight on January 1, 2022.

The Microsoft Exchange Server bug

The start of the new year began with a bang for thousands of Microsoft Exchange Server users, who found themselves unable to receive emails and messages due to a bug. Exchange admins noticed that servers were no longer able to deliver emails ; looking for the cause of the problem they found a message in the logs with a very specific error:

 Log Name: Application Source: FIPFS Logged: 1/1/2022 1:03:42 AM Event ID: 5300 Level: Error Computer: server1.contoso.com Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.

What is it about? The FIP-FS scanning engine , or the antivirus that scans emails, fails to store the date in an Int32 before converting to long . This type of data can store values ​​from -2.147.483.647 to + 2.147.483.647, but new dates, with one more year, exceed this limit. In the case of January 1st at 00.00 the value to be stored would have been +2.201.010.001, greater than the maximum value of the 32-bit integer.

The scanning engine crashed , sending the entire email and message delivery process into a tailspin , leaving them stuck in the queue. Microsoft's email froze, queues filled up, and users were no longer able to send or receive email.

Thousands of Microsoft Exchange emails got stuck in server queues due to a date bug.
Thousands of Microsoft Exchange emails got stuck in server queues due to a date bug.

How to fix the error?

Microsoft immediately noticed the error on the Exchange Servers and was already working to solve the bug. The fix, however, will not arrive in the short term, and the emails certainly cannot remain blocked, especially now that companies are starting to work again. The Exchange team has published two possible solutions , but both require the intervention of the server admins in order to be put into practice. Administrators can choose whether to follow an automated or manual procedure.

Automated solution

If you want to resort to a ready-made solution, just run a script and follow a few simple steps to temporarily avoid the bug, waiting for the official patch. The first thing to do is to modify the execution policies by setting them to “RemoteSigned” , thus allowing the execution of scripts with a trusted digital signature. At this point you can download the script provided by Microsoft and run it on each company Exchange server.

Manual solution

If you prefer to follow a manual solution, Microsoft has illustrated the steps necessary to solve the problem (always temporarily):

  1. stop the "Microsoft Filtering Management" and "Microsoft Exchange Transport" service. Once the first has been stopped, the system itself will propose a popup to block the second;
  2. make sure the updateservice.exe service is not running;
  3. delete the % ProgramFiles% Microsoft Exchange Server V15 FIP-FS Data Engines amd64 Microsoft folder;
  4. remove all files contained in % ProgramFiles% Microsoft Exchange Server V15 FIP-FS Data Engines metadata ;
  5. restart the two previously stopped services;
  6. open the "Exchange Management Shell", navigate to the scripts folder ( % ProgramFiles% Microsoft Exchange Server V15 Scripts ) and execute the command Update-MalwareFilteringServer.ps1 <server FQDN> ;
  7. always on the Exchange shell execute the command Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell ;
  8. run Get-EngineUpdateInformation and verify that the “UpdateVersion” property is 2112330001.

This value represents a non-existent date, but Microsoft reassures administrators that it is correct and supported by the engine. After executing the script (or after following the manual steps) the server will resume delivering emails and messages correctly, although it may take some time before it returns to full capacity.

The procedure is only necessary for servers that have access to the internet and download updates for the malware. Otherwise the problem should not arise.

The Microsoft Exchange Server article : The bug that blocked thousands of emails comes from Tech CuE | Close-up Engineering .