Partner With Vanguard

On February 29, 2020 the popular free SSL/TLS service Let’s Encrypt announced that they found a bug in their code that has caused them to plan to revoke about three million SSL certificates across the globe, meaning these sites are potentially no longer secure.

As you can probably imagine, this has caused quite the uproar in the web industry and has caused significant problems for those affected.

What happened?

Let’s Encrypt allows users to automate the SSL/TLS certificate installation and renewing process. The Let’s Encrypt certificates must be renewed every 90 days, so this automation feature makes Let’s Encrypt highly desirable and convenient.

With automation, comes risk. The bug was found in the automation protocol that handles Certificate Authority Authorization (CAA). The CAA checks the users’ DNS record for an entry that proves that the owner of the website is who they say they are.

The code is supposed to check the CAA records within eight hours of renewing or installing a certificate. Instead of checking each domain one time, it was checking one domain many times, leaving the rest of the websites untouched. While we can assume MOST of these sites are just fine, Let’s Encrypt can’t assume it. That would be against protocol, and could potentially allow a domain name to be stolen,  hacked, and/or vulnerable to a man-in-the-middle attack.

Upon discovery of the error, Let’s Encrypt disclosed the issue to the public and revoked the affected certificates.

Here was a statement issued from an engineer at Let’s Encrypt:

“On 2020-02-29 UTC, Let’s Encrypt found a bug in our CAA code. Our CA software, Boulder, checks for CAA records at the same time it validates a subscriber’s control of a domain name. Most subscribers issue a certificate immediately after domain control validation, but we consider a validation good for 30 days. That means in some cases we need to check CAA records a second time, just before issuance. Specifically, we have to check CAA within 8 hours prior to issuance (per BRs §3.2.2.8), so any domain name that was validated more than 8 hours ago requires rechecking.

The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt.

We confirmed the bug at 2020-02-29 03:08 UTC, and halted issuance at 03:10. We deployed a fix at 05:22 UTC and then re-enabled issuance.

Our preliminary investigation suggests the bug was introduced on 2019-07-25. We will conduct a more detailed investigation and provide a postmortem when it is complete.”

You can find the incident report here.

So what’s next?

The bug is resolved at this point, but it doesn’t mean that the problem is solved. The company has issued a warning to its users of the incident, and is urging those whose certificates were affected to replace them ASAP.  Since that announcement, Let’s Encrypt has been working with their subscribers to replace these certificates. In less than two days, they had already replaced more than half of the certificates that were affected (I can imagine Let’s Encrypt engineers have not slept at all during this timeframe).

In the meantime, they have taken measures to ensure that this will not happen moving forward and are continuing to assist in replacing all of the certificates until this fire has been put out. The certificates that were revoked immediately were flagged as high risk due to their CAA records, however they are refraining from automatically revoking all the certificates by a certain deadline for reasons stated directly from a Let’s Encrypt engineer below:

“Rather than potentially break so many sites and cause concern for their visitors, we have determined that it is in the best interest of the health of the Internet for us to not revoke those certificates by the deadline.”

You are probably asking yourself if you should be worried? If you use Let’s Encrypt and you haven’t received an email from them, then most likely not. But, it is always important to know what is happening in the tech world and be aware of incidents like this.