The IT Nerd

Numerous websites and services have already reported issues across computers, web browsers and other devices due to the recent expiration of Let’s Encrypt’s root certificate. The best that Let’s Encrypt could do is post this Tweet:

Browsing this forum post indicates that there is no promise of a speedy resolution in getting certificate renewals. 

The fact that major websites and services are affected by this is an #EpicFail. But it gets worse. TechCrunch reported that devices that may face issues include older macOS 2016 and Windows XP (with Service Pack 3) as well as older versions of Playstations and any tools relying on OpenSSL 1.0.2 or earlier. Other experts said PlayStations 4s or earlier devices that have not had their firmware upgraded would not be able to access the Internet. Devices like Android 7.1.1 or earlier will also be affected. 

That’s catastrophic.

Chris Hickman, chief security officer at Keyfactor, a leader in securing digital identities, said this:

“A root CA (certificate authority) does not specifically create a security problem, but rather a disruption to availability to any certificate that chains to that particular root.  This in turn can lead to a number of situations in which users may be forced to click through exception messages, leading to bad user habits or in extreme cases, causing an application to respond as no longer expected, which we’re seeing now with the numerous websites and services that didn’t heed to Let’s Encrypt’s notifications about the upcoming expiration.

When transitioning from an expired root certificate to a new one, in most cases, the greatest issue is a lack of automation to distribute the new root CA certificate to those devices that need to trust it. In many organizations, the root CA certificate stores (otherwise known as roots of trust) are not managed universally. This can lead to situations like only updating parts of the network (say Windows via GPO) but not the entity of all devices that need to trust the new root. In the case of IoT devices the problem is compounded as most IoT devices still rely on firmware and software updates to manage roots of trust. Therefore, these devices are wholly dependent on users taking action. In both cases, a failure to update the roots of trust properly can lead to outages or disruptions in normal use.

Management of the roots of trust for all devices is a part of having a well-defined crypto agility strategy. Any reasonable crypto agility strategy will consider how manage and handle regular lifecycle events including the revocation or expiration of a root CA. The crypto agility strategy will allow executives to quickly identify the scope of the impact on their organization and make an informed decision as to the priorities to their organization as it relates to risk and immediacy of remediation.”

I’ll be watching this issue closely as this personally affects me as my mail server uses Let’s Encrypt certificates and hopefully those mail servers stay up and work.

Well this isn’t good. The free SSL certificate provider Let’s Encrypt is going to revoke 2.6% of the SSL certs issued by them that are currently active, due to a bug in boulder, the Certificate Authority Authorization (CAA) software Let’s Encrypt uses. Here’s the key points from the FAQ that I linked to. Starting with the number of certificates affected:

2.6%. That is 3,048,289 currently-valid certificates are affected, out of ~116 million overall active Let’s Encrypt certificates. Of the affected certificates, about 1 million are duplicates of other affected certificates, in the sense of covering the same set of domain names.

And here’s when the revocation will take place:

In order to complete revocations before the deadline of 2020-03-05 03:00 UTC, we are planning to start revoking affected certificates at 2020-03-04 20:00 UTC (3:00pm US EST). Please continue to renew and replace affected certificates in the meantime. If there are any changes to this start time, updates will be provided in this thread. 

And finally, here’s how to tell if you’re affected:

Here is an online tool that will show you: https://checkhost.unboundtest.com/ 9.4k

Or, on a Linux/BSD-like system, this command will show you example.com‘s current certificate serial number:

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

You can see the list of all affected serial numbers at: https://letsencrypt.org/caaproblem/ 

To be clear, this is not trivial as if you don’t take action, whatever uses an affected certificate will either stop working or not work properly. Thus if you use certificates from this provider, you need to check to see if they are affected. And if they are, you need to take action. Thus if that’s you, I would read the FAQ and go forward from here.