The IT Nerd

Last week, Microsoft had a major outage that affected a lot of their services including:

• Teams
• Xbox Live
• Outlook
• Microsoft 365 
• Minecraft
• Azure
• GitHub
• Microsoft Store

At the time, Microsoft said that a networking change caused this. And at the time, I said this:

My question for Microsoft, which I hope they answer is what specifically happened and what will they do to ensure that it doesn’t happen again. Microsoft does give some version of this information out, so I for one will be interested to see what they say.

And now Microsoft has a Preliminary Post Incident Review that goes into more detail that answers the questions that I had:

We determined that a change made to the Microsoft Wide Area Network (WAN) impacted connectivity between clients on the internet to Azure, connectivity across regions, as well as cross-premises connectivity via ExpressRoute. As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them. The command that caused the issue has different behaviors on different network devices, and the command had not been vetted using our full qualification process on the router on which it was executed.

And this is how they responded:

Our monitoring initially detected DNS and WAN related issues from 07:12 UTC. We began investigating by reviewing all recent changes. By 08:10 UTC, the network started to recover automatically. By 08:20 UTC, as the automatic recovery was happening, we identified the problematic command that triggered the issues. Networking telemetry shows that nearly all network devices had recovered by 09:00 UTC, by which point the vast majority of regions and services had recovered. Final networking equipment recovered by 09:35 UTC.

Due to the WAN impact, our automated systems for maintaining the health of the WAN were paused, including the systems for identifying and removing unhealthy devices, and the traffic engineering system for optimizing the flow of data across the network. Due to the pause in these systems, some paths in the network experienced increased packet loss from 09:35 UTC until those systems were manually restarted, restoring the WAN to optimal operating conditions. This recovery was completed at 12:43 UTC.

And this is how they will stop this from happening again:

• We have blocked highly impactful commands from getting executed on the devices (Completed)
• We will require all command execution on the devices to follow safe change guidelines (Estimated completion: February 2023)

This is all good and I really wish that other companies would do the same thing as you’re more likely to trust a company who is open and transparent. Kudos to you Microsoft.

Early this morning, Microsoft had an outage that affected, but were not limited to the following services:

• Teams
• Xbox Live
• Outlook
• Microsoft 365 
• Minecraft
• Azure
• GitHub
• Microsoft Store

The issue started at about 2.30 a.m. EST and ended about 2 hours later. What’s interesting is that Microsoft said this:

So Microsoft made a change that broke a lot of their online services and had to roll it back. That does happen from time to time with the best example that I can think of is Rogers and their July outage. But that creates issues for people who rely on said services. My question for Microsoft, which I hope they answer is what specifically happened and what will they do to ensure that it doesn’t happen again. Microsoft does give some version of this information out, so I for one will be interested to see what they say.

News is filtering out that Microsoft is going to cut 10,000 jobs. Here’s the reason behind this according to a blog post from Microsoft:

We’re living through times of significant change, and as I meet with customers and partners, a few things are clear. First, as we saw customers accelerate their digital spend during the pandemic, we’re now seeing them optimize their digital spend to do more with less. We’re also seeing organizations in every industry and geography exercise caution as some parts of the world are in a recession and other parts are anticipating one. At the same time, the next major wave of computing is being born with advances in AI, as we’re turning the world’s most advanced models into a new computing platform.

As a result of this, this is where the job cuts come in:

First, we will align our cost structure with our revenue and where we see customer demand. Today, we are making changes that will result in the reduction of our overall workforce by 10,000 jobs through the end of FY23 Q3. This represents less than 5 percent of our total employee base, with some notifications happening today. It’s important to note that while we are eliminating roles in some areas, we will continue to hire in key strategic areas. We know this is a challenging time for each person impacted. The senior leadership team and I are committed that as we go through this process, we will do so in the most thoughtful and transparent way possible.

Not all the news is bad though:

Second, we will continue to invest in strategic areas for our future, meaning we are allocating both our capital and talent to areas of secular growth and long-term competitiveness for the company, while divesting in other areas. These are the kinds of hard choices we have made throughout our 47-year history to remain a consequential company in this industry that is unforgiving to anyone who doesn’t adapt to platform shifts. As such, we are taking a $1.2 billion charge in Q2 related to severance costs, changes to our hardware portfolio, and the cost of lease consolidation as we create higher density across our workspaces.

And I suspect, this is an attempt by Microsoft to not be seen as acting like Elon Musk:

And third, we will treat our people with dignity and respect, and act transparently. These decisions are difficult, but necessary. They are especially difficult because they impact people and people’s lives – our colleagues and friends. We are committed to ensuring all those whose roles are eliminated have our full support during these transitions. U.S.-benefit-eligible employees will receive a variety of benefits, including above-market severance pay, continuing healthcare coverage for six months, continued vesting of stock awards for six months, career transition services, and 60 days’ notice prior to termination, regardless of whether such notice is legally required. Benefits for employees outside the U.S. will align with the employment laws in each country.

I fully expect this to be the first of many announcements of this sort that we will hear in the coming days and weeks. As they say on Game Of Thrones, brace yourself.

Happy Friday The 13th. Unless you are running Microsoft Windows because an update to Windows Defender is apparently making the rounds and has some catastrophic effects. Specifically Windows users and system administrators worldwide are complaining that application shortcuts have disappeared from Start menus, desktops, and taskbars. You can read more on places like Reddit for example.

The problem appears to be related to a malfunctioning attack surface reduction (ASR) rule issued with Windows Defender security intelligence update 1.381.2140.0.

For what it’s worth, Microsoft has acknowledged the issue:

The good news is that regular Windows users and consumers aren’t affected by this bug. The bad news is that it will only affect managed machines inside organizations. Which is still hundreds or thousands or even millions of machines inside big businesses that rely on Microsoft’s threat detection security. Thus this is not a trivial issue and it will be interesting to see how Microsoft addresses this.

Today marks the day that Windows 8.1 will reach its end of support. That means that the product will no longer receive security updates, non-security updates, bug fixes, technical support, or online technical content updates. Businesses and individuals around the world will be exposed to a significantly bigger attack surface and increased risk from using an unsupported operating system from Microsoft.

Antonio Sanchez, cybersecurity product marketing principal at cybersecurity software and services provider Fortra says:

“As of January 10^th, any organization that still has Windows 8.1 running in their environment is accepting the additional risk of being breached. This is because Microsoft will no longer be creating security updates for 8.1 for any new vulnerabilities. And if your strategy is to hope there are no new vulnerabilities discovered here is something to keep in mind: Windows 7 had almost 1,000 new vulnerabilities after its end of life.”

My advice would be that if you have not already migrated to Windows 10 or Windows 11, you should do so immediately as there’s very little good reason to be running anything earlier than Windows 10 in 2023.

While I was busy covering the feature dump that Apple did with all its operating systems, I didn’t cover the fact that it was Microsoft’s “Patch Tuesday”. Bleeping Computer has a lot of info on December’s “Patch Tuesday” dump here. And there is truly a lot here for you to read. To help you make sense of it all, I have enlisted the help of Yoav Iellin, Senior Researcher at Silverfort:

Marked as critical, CVE 2022-41076 is one security teams should definitely be aware of as it allows for an attacker to escape the Powershell Constrained Session Configuration to run unapproved commands. Powershell Constrained Session is used across a wide variety of applications so admins need to be aware of where they are exposed and either update, or disable the affected feature. While Microsoft notes this vulnerability is complex to exploit, it can however be triggered by any authenticated user, removing the extra step of escalating privileges.

An interesting, actively exploited vulnerability from an initial access point of view is CVE-2022-44698. This is a flaw in Windows SmartScreen – a component in Microsoft applications designed to reduce the risk of socially engineered malware by checking the reputation of downloaded files prior to installation. Using this vulnerability, an attacker could convince the victim to run a crafted file or access an unsafe link and then bypass protections alerting them to potentially malicious downloads.

Included amongst the usual CVE numbers, Microsoft Security Advisory ADV220005 tells an interesting story. This advisory recounts the detection of malicious drivers submitted and signed by the Microsoft Windows Hardware Developer Program. Components such as this enjoy kernel level access, so would have been able to evade security controls had they not been detected.”

The guidance that Mr. Iellin spoke of can be found here and is very much worth reading. But perhaps that reading should take place after you patch all the things so that the bad guys don’t use today’s “Patch Tuesday” dump to create attacks from.

Bad news if you use Microsoft’s discontinued Boa web server. It’s being targeted by hackers. Microsoft put out a warning about this along with potential remediations, but Security Week has a story about this web server being used in attacks. Which effectively makes this a today problem for anyone who uses Boa.

Sharon Nachshony, Security Researcher, Silverfort had this to say:

     “The Microsoft research highlights a long-standing supply-chain risk to IoT and OT environments from legacy technology. While hard to manage, given the abundance of such technology in critical industries, a rigorous patching regime is essential.

Age-old vulnerabilities such as this provide a jumping-off point for attackers looking to move laterally to more sensitive areas by abusing the identity attack surface. With access to critical areas inside OT environments – their activities can quickly become significantly more impactful.

To stop lateral movement, MFA should be applied to resources such as Command Line interfaces, WMI, Shared Folders and Service Accounts to close down commonly used attack paths.”

If you’re a user of the Boa web server, consider this your invitation to follow Microsoft’s advice so that you don’t get pwned seeing as this is clearly being exploited by threat actors as I type this.

Microsoft yesterday admitted to accidentally exposing sensitive customer data after failing to configure a server security. The involved files were exposed from 2017 to August 2022, including data such as:

• Names
• Email addresses
• Email content
• Company name
• Phone numbers

In addition, Microsoft warned that the exposed data may include “attached files relating to business between a customer and Microsoft or an authorized Microsoft partner.”

SOCRadar claims that the sensitive data of over 65,000 entities in 111 countries on a misconfigured Microsoft server that had been left accessible over the internet.

What could possibly go wrong with that sort of info floating around for anyone to get access to?

John Stevenson, Product Director at Cyren had:

     “Given that Cloud server ‘misconfigurations’ are one of the most common root causes for the loss of personally identifiable information (PII), it is extremely important that organizations stay vigilant for any attempt to target them or their employees, especially through phishing attempts. While there is currently no evidence that the PII accessible from the server has been exploited in the wild, search tools such as the one referenced here are undoubtedly double-edged. At this time, the ‘BlueBleed’ site allows any authenticated user to search the data repository. With the news of this leak, it is essential that organizations look to additional security controls that operate in the inbox to identify targeted, socially engineered email attacks that are routinely missed by Microsoft’s native security controls.”

SOCRadar, which has dubbed the data breach “BlueBleed”, has created a website where concerned companies can search to see if their data has been exposed. You might want to pay a visit to see if your company has been affected.

If you are responsible for an Microsoft Exchange server and it is not Microsoft’s Exchange Online offering, then you should read this story and take action immediately. Microsoft is reporting via a blog post that there’s a zero-day Exchange vulnerability in the wild:

Microsoft is aware of limited targeted attacks using two reported zero-day vulnerabilities affecting Microsoft Exchange Server 2013, Exchange Server 2016, and Exchange Server 2019. The first one, identified as CVE-2022-41040, is a server-side request forgery (SSRF) vulnerability, while the second one, identified as CVE-2022-41082, allows remote code execution (RCE) when Exchange PowerShell is accessible to the attacker. Refer to the Microsoft Security Response Center blog for the mitigation guidance regarding these vulnerabilities.  

CVE-2022-41040 can enable an authenticated attacker to remotely trigger CVE-2022-41082. However, authenticated access to the vulnerable Exchange Server is necessary to successfully exploit either vulnerability, and they can be used separately.

Microsoft Defender Antivirus and Microsoft Defender for Endpoint detect post-exploitation malware and activity associated with these attacks. Microsoft also released a script, available at https://aka.ms/eomtv2, to apply the mitigations for the SSRF vector CVE-2022-41040 to on-premises Exchange servers.

Microsoft will continue to monitor threats that take advantage of these vulnerabilities and take necessary response actions to protect customers.

What makes these exploits so dangerous is this:

While these vulnerabilities require authentication, the authentication needed for exploitation can be that of a standard user. Standard user credentials can be acquired via many different attacks, such as password spray or purchase via the cybercriminal economy.

So, if any user who gets e-mail from an Exchange server has their credentials leak out to threat actors, then the threat actors can use these exploits to pwn the Exchange server. Lovely.

The fact that the attacks at present are targeted implies that a nation state is behind this. There are no signs yet that the exploits have been publicly published. But that’s likely to to change soon. Which is why Exchange admins need to take action now by following this guidance from Microsoft. To reiterate, if you’re responsible for administering an Exchange server that is part of Microsoft’s Exchange Online offering, then you need not worry. If however your Exchange server is on premise, then you have some work to do. And that work is a today problem.

If you’re not familiar with Microsoft BitLocker, it’s the native full disk encryption product for Microsoft Windows. But only the business and enterprise versions. The consumer versions of Windows 10 and 11 don’t have this feature. Enterprises around the world use this as a way to encrypt the data on their hard drive for security reasons. But it appears that threat actors are also using this to launch ransomware attacks according to Microsoft:

Microsoft threat intelligence teams have been tracking multiple ransomware campaigns and have tied these attacks to DEV-0270, also known as Nemesis Kitten, a sub-group of Iranian actor PHOSPHORUS. Microsoft assesses with moderate confidence that DEV-0270 conducts malicious network operations, including widespread vulnerability scanning, on behalf of the government of Iran. However, judging from their geographic and sectoral targeting, which often lacked a strategic value for the regime, we assess with low confidence that some of DEV-0270’s ransomware attacks are a form of moonlighting for personal or company-specific revenue generation. This blog profiles the tactics and techniques behind the DEV-0270/PHOSPHORUS ransomware campaigns. We hope this analysis, which Microsoft is using to protect customers from related attacks, further exposes and disrupts the expansion of DEV-0270’s operations.

DEV-0270 leverages exploits for high-severity vulnerabilities to gain access to devices and is known for the early adoption of newly disclosed vulnerabilities. DEV-0270 also extensively uses living-off-the-land binaries (LOLBINs) throughout the attack chain for discovery and credential access. This extends to its abuse of the built-in BitLocker tool to encrypt files on compromised devices.

In some instances where encryption was successful, the time to ransom (TTR) between initial access and the ransom note was around two days. The group has been observed demanding USD 8,000 for decryption keys. In addition, the actor has been observed pursuing other avenues to generate income through their operations. In one attack, a victim organization refused to pay the ransom, so the actor opted to post the stolen data from the organization for sale packaged in an SQL database dump.

I have to admit that this is novel as the threat actors are using built in tools to pwn their targets. The Microsoft report has mitigation strategies that you should read and implement. Because it seems that we’re going to hear more from this in the weeks and months to come.