Microsoft says a portion of servers are returning to service, and a streamlined recovery process is under way to end a global outage that has resulted in nearly 130,000 users worldwide reporting issues accessing its 365 suite of software, including Outlook and Teams.

Outage website Down Detector showed a spike in Microsoft 365 problems, with nearly 130,000 reports of issues affecting access to Word, Excel, PowerPoint, Outlook and Teams as well as enterprise services such as SharePoint and OneDrive. AI assistant Copilot was also affected.

New reports have since fallen from that peak, but many said they couldn’t send and receive emails, or had trouble loading their inboxes altogether.

Posting on X, Microsoft said it was “investigating reports of users experiencing issues accessing Exchange Online mailboxes through multiple connection methods”.

Microsoft later explained that a portion of its infrastructure was “experiencing elevated load” – which it said was “likely contributing to the connectivity errors”. The company determined a number of servers were “unexpectedly unavailable” due to a problem that caused them to fail during automated checks.

At 9pm UTC Thursday, the tech company said it had validated an alternative health check method on a subset of the affected servers, which allowed them to successfully return to service, and that fix was now being applied more broadly.

Related: Jensen Huang built an AI kingdom. Now he’s gone, will Nvidia’s ramparts be breached?

Microsoft was also preparing an automated process to detect and restart unresponsive servers. It said the review of recent changes has identified configuration updates as a potential cause for the issue, but said it was still being investigated.

At 9.41pm UTC, Microsoft said it was working through restarting and validating the affected servers, and looking to expedite the process.

“In parallel, we’re assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.”

But by 10.11pm UTC, Microsoft said it was “continuing to pursue all available pathways to restore service” as the actions taken so far “have not provided the anticipated relief”.

The company said a core process on the affected servers was becoming overwhelmed, causing the servers to stop responding.

At 11.06pm UTC, Microsoft said a “portion of the affected servers” had begun being returned to service and were remaining stable.

“We’ve also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.”

The company said it had streamlined its recovery process to expedite returning the remaining affected servers to service, and Microsoft had implemented protective traffic management to reduce the load to the affected servers.

Customers may experience a gradual recovery as users reconnect to 365 services, Microsoft said.

An earlier service degradation alert on Microsoft’s site said it was “aggressively pursuing all pathways to reduce impact” and said it was “restarting affected servers to return them to service”.

“As a precaution, we’ve paused all deployments and configuration changes while we review recent changes for any correlation to this event.

“We’ve determined that the elevated load isn’t driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure.

“We’re investigating what’s causing these servers to become unavailable while we work to safely return them to service.”

‘A cascading problem’

Mark Gregory, associate professor in the School of Engineering at RMIT University, said the outage appears to be affecting the network layer in Microsoft’s systems, and was potentially similar to the configuration issue which brought down Optus’s network in 2023.

He said when some of the servers have configuration issues, it means customers are forced to share a smaller pool of servers to access the cloud services.

“If you lose part of that pool, then all of the users are trying to access the remaining servers, and that overloads them. And then they go down.

“And so you end up with a cascading problem. Then it takes time to find the underlying problem, in this case, a misconfiguration in the network layer.”

Related: Microsoft posts $92.3bn quarterly revenue after offering buyouts to 60,000 employees

Gregory said it showed companies should invest more in testing environments for configuration changes before deployment in the live network, and said businesses reliant on cloud services such as Microsoft 365 should have backup options.

“From a global perspective, there is someone somewhere that’s that’s been badly affected by this and you end up with lost productivity,” he said.

“They’re being left high and trying now … it then becomes important to have an alternative.”

“Many people would keep a Gmail account, if their Microsoft goes down, they use Gmail. If Gmail goes down, they use Microsoft.”

Both OpenAI and Anthropic reported the issue was affecting the Microsoft 365 connector that links ChatGPT and Claude AI to Microsoft services.

Earlier, the company said it had “successfully restored” problems related to Microsoft Teams in North America, although some users continued to report connectivity issues.

– with Associated Press