DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation.
Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner.
Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.
Reads to me like Digicert is hiding behind the TRO to protect themselves from upset customers over following the procedures they should have followed. I don't think they want to update their legal work to prevent customers from legal action over revocation, legal action has been working in their favor so far.
For a company whose entire business is verifying company names and handling the CA process, Digicert seems rather unwilling to actually stick to the process. They can't help having a TRO filed against them by customers with incompetent tech such as Alegeus Technologies LLC, but this isn't the first time they've failed to follow the appropriate processes.
Going through the courts to stop negative discourse seems rather crass for a CA. I don't understand why a company that has been subject of suspicion and doubt lime Digicert would do such a thing unless it's a last-ditch effort to get the heat off their backs.
I'm sure their clients love Digicert for not forcing them to replace their certificates at the appropriate times, but they're in for a surprise when this whole thing blows up and they suddenly need to find another cert vendor when Digicert gets delisted.
Alegeus, a client of DigiCert, filed a court motion to stop DigiCert from revoking their certificate. The courts granted the temporary restraining order.
There is no "update their legal work to prevent customers from legal action" that can avoid a temporary restraining order that is ordered by the courts to provide time to establish facts. Its simply how the legal process works. "they can't help but" seems unfair as the issue was the client Alegeus was unable to replace their certificates in the revocation timeline.
Further "Going through the courts", they didn't go to the courts a client of theirs did.
Digicert is absolutely not blameless. Outside of the TRO, they have failed to meet revocation windows before and yes, are likely using the TRO in this instance as a shield for that.
However the TRO itself is a concerning development that all CA's need to consider depending on their legal jurisdiction, as it could put companies in a bind of being legally ordered to not to complete something they are contractually and legally required to do within the required timeline and interrupt standard security practices in cases like this.
I dunno, seems to me in the TRO case there wasn't anything wrong with the certificates that had been issued. There wasn't any suggestion the the certificates had been issued to the wrong party or anything like that.
When performing DNS verification there's supposed to be an underscore in a record, which protects sites that let their users control arbitrary subdomains - you know, dyndns and things like that. Digicert mistakenly didn't ask the customer to include an underscore. Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
Then Digicert spent a load of the 24 hour disclosure window checking things on their end, then e-mailed the customers after office hours so when they get in the next day they barely had any time to respond.
The only "incompetent tech" here is the assholes who decided on the policy that certificates be urgently revoked, as if they'd been issued to the wrong person, when they hadn't been issued to the wrong customer.
The middle ground here is DigiCert should have been able and continued with the revocation of all the other certificates not impacted by the TRO within the policies and contractual agreements it has.
The policy and its designers aren't incompetent, the intention is to ensure security first when fact-finding can take time to establish a subset of actual abuse. This is common in IT/Security and is clearly communicated in the policy customers agree to as well as something agreed on by all CA's. Security > Uptime is the baseline, and while you may disagree with that and have good arguments for the contrary, the argument is much wider than us and has been extensively hashed out in the working groups and discussions around this and other similar issues and is implemented this way by consensus while weighing valid arguments for different approaches and pros and cons.
>Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
AFAIUI, Alegeus was just 1 customer of Digicert. But Digicert has tons of customers. Just because there wasn't a problem for Alegeus (due to Alegeus not being a dyndns provider), doesn't mean there weren't problems for other Digicert customers.
What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?
DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days.
It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients.
"Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."
I think the eventual correct option is pretty clear.
A TRO like that is based on the company loudly declaring that revoking will cause real damage. That means their use of certificates is incompatible with the web PKI rules and ecosystem. That means they need to be migrated out ASAP, with every certificate authority refusing to take their business.
Make that series of consequences clear, and companies will stop trying that trick.
DigiCert and other CA's need to write in their terms that failure to follow the policies and revocation timelines can/will result in termination of the contract. Alegeus should have been dropped as a customer as soon as the incident was resolved and refused further products/renewals.
Use of a TRO protects you short term, but results in having to migrate to a new CA medium term. You can't stop them from using TRO's but you can make it not worth it.
I think that's pretty unrealistic, considering that X.509 is a de-facto and de-jure standard in a lot of places that also ignore that requirement. It's not always up to that company to make it possible to replace certificates easily, it's an entire chain of vendors (I know at least Salesforce's process would fail here). Unless you want those people to run self-signed/private CA certs.
The other option would be building in a way to revoke other CA's individual certificates if there's some consensus on them being compelled to not revoke them. Not sure if the status quo or this would be more dangerous, but if a TRO can compel a CA to not sign a revocation, can it also compel to sign a certificate?
A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done. Just because Salesforce can't get their stuff together doesn't mean the CA industry needs to bend the knee. Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
If running a private CA or self-signed certificates are even viable, then there are plenty of other workarounds that can be put in place (i.e. not updating the CRL so the software doesn't know about revocations).
If a CA's terms and legal documentation aren't tight enough to prevent a court from compel them to sign a certificate, that CA clearly cannot be trusted. Hopefully that issue can be fixed by writing clearer terms and better agreements, like people have been telling Digicert to do, but perhaps that's not possible at all. In that case, either the company should move to a jurisdiction where that kind of nonsense isn't possible, or it should be removed from global trust stores all together.
> A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done.
You say that and yet Teams has 320 million people using it, and I bet almost none of them enjoy the experience. Sometimes you just have to work with what you're given. Given "incompetence", we might as well throw in all of Azure.
> Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
That might be true for DV, but there's classes of certificates that take at least weeks to obtain, think BIMI VMC or codesigning.
The parent was correct - it's not about the company not using x509 certificates, but not using publicly trusted certificates. There are myriad private/internal PKI solutions available from OpenSSL & bash to millions of dollars of solutions from Big Vendor.
If you can't replace the publicly-trusted certificate quickly, you probably don't need it to be publicly-trusted in the first place.
It’s Saturday and the courts are closed. You park in a car park I own.
I have put up a sign saying I can instantly crush any crossover vehicles I like, as I consider them ugly and lacking in character.
As I load your car into my crusher, you dispute the legality of my sign. But the court is closed until Monday, and the sign says I can crush your car instantly, no waiting.
Should I be allowed to crush your car today? Or should I have to wait until Monday, so the disputed legality can be resolved?
That’s actually a question for you. You won’t truly know if you’re allowed to until a court decides. You can choose to proceed and crush it, and then deal with the consequences of doing so if it turns out you were wrong. Likely you’ll end up owing damages to the owner of the car, perhaps even punitive damages on top.
The TRO actually means you aren't allowed, that's the point. It's an ordered injunction that legally obliges you from not acting until the courts can review the facts.
The better question is, how do you prevent this tactic from working?
For example, suppose there were required to be multiple parties who could issue a revocation, each in a different jurisdiction, and if any of them was ordered not to do it then the others would be required to do it, and would have the technical capacity to do it but not be subject to the jurisdiction of that court.
Well, you can contest the motion for a TRO, for a start. Digicert failed to do so.
You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
Correction, the petition for the TRO was filed ex parte. Digicert did not have any opportunity to respond before it was granted.
They certainly could have filed a response contesting the TRO. Then their customer could have filed another motion, and eventually (7 days later in this case) the judge would have ruled on the substance of it. Their judgement was that it would be preferable to work with the customer to resolve the technical issues with revocation, and submit a joint request to dismiss the TRO. The stated reasoning behind this was that it would be significantly faster than contesting the TRO. This is true: the certs were revoked and the TRO dropped within 3 days.
I think the communication on that point was severely lacking, as they only clarified it three months later and after significant hectoring in two different bug threads: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43
I also think it's reasonable not to take Digicert's statements at face value, given their history. But I think both of the points you made here are wrong:
> You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
Let's be clear about the timeline: Digicert notified their customers that the certs would be revoked. In between the time they notified the customer and the time of revocation (less than 24 hours), the customer got the ex parte restraining order. Are you suggesting that issuers should revoke certificates without notifying their users, so that the users don't have time to get an emergency TRO? I believe that would be in violation of the BRs.
> You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
To add to this, 3 days after the TRO was filed both parties moved to vacate the TRO.
DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED
I'm not sure DigiCert could have done anything about the TRO or the impacted certs, but it should have been able to move forward with the revocation of all other certificates. That IMO is the real issue/failure, alongside the concern/impact of TRO's on security processes in the future.
Just to be clear, the whole incident covered over 80,000 certificates.
The TRO was applicable to only those of one subscriber - just over 70 certificates, yet caused the revocations of all 80k+ to be delayed.
You could just require publishing the revocation immediately with an effective date in the future.
Of course, if that system had been in place, DigiCert would probably be facing hundreds of lawsuits from businesses disrupted through no fault of their own rather than inside baseball PKI drama.
No, Sectigo is the PKI business that was carved out of Comodo, back in 2017. Comodo still exists, doing whatever they do. They have been totally separate entities since then.
Web PKI drama is always astonishing to me because it is one of the only areas of the entire world where a corporation's "fucking around" is seldom ever not followed by a sobering "finding out" period. The various entities that decide what CAs to trust can effectively dismantle any CA business in the world, basically at the drop of a hat. If DigiCert decides to play this game and lose, it would make them the biggest such loser so far. DigiCert is, as far as I know, the largest CA on the Internet. It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting.
Of course, I think that is unlikely, but on the other hand, it's just as cathartic to imagine whatever idiot at DigiCert thought it was a good idea to engage legal here to have the dressing down of a lifetime. I read the thread in question. It doesn't make DigiCert look good, but this action definitely is more damning to them than anything Collan said in my opinion.
Someone should start a real CA business, complete with all the proper audits and everything to get it into the browser trust stores… and call it “Honest Achmed’s Used Cars and Certificates” (have it buy some random used car dealership in the middle of nowhere so the name is not a lie)
Where’s a billionaire troll when you need one? (And a form of billionaire trolling that would upset a lot less people than Musk’s version of it.)
Would be even funnier if the billionaire’s name actually was Achmed
Time and money. Plus right now even if you bought the infra, the staff, paid for and passed the audits, and then waiting while Apple, Mozilla, Google and Oracle (at least) included your roots...Microsoft aren't taking more right now. So you have to wait for some unknown time in the future when/if they start doing that again.
You could purchase a root off an existing CA, subject to the trust stores approving it, and the boatload of cash you'd need to buy it (plus still having the staff and infra to operate it).
Sub-CAs: Not really. Operational risk to the parent CA is huge, you'd be hard pressed to get any current public CA to sign an issuing CA to be operated externally.
Cross-signing still works (though it is the stuff of nightmares in many cases) but again you have to have money and a CA willing to do it!
Anyone can get a cert from lets-encrypt. No users of your website care how trustworthy your CA is. And CA's are too big to fail, so you barely need to worry about your CA being removed by the trust store. So you can only compete on price.
If we want to change this, we need a way for a certificate to be co-signed by multiple CA's. So that the certificate can be presented to the user, and they can figure out if any trusted CA of theirs has signed the certificate.
This way, revoking trust in a CA becomes easier, because people should have multiple signatures on their certificate. That means, all of a sudden, that the quality of a CA actually matters.
Whilst it might seem this is already possible, it is not. Cross-signing is only a thing for intermediate certificates, and does not work. You can also have multiple certificates for the same key, but when starting a TLS session, you must present a single certificate. (This seems to have changed for TLS 1.3, so perhaps this is already possible?)
Let's Encrypt has been around for a decade and is doing pretty well for itself. The manual validation required for EV and OV certificates precludes it from being a passion project, unless managing a call center and a squad of document reviewers is your dream job.
To my understanding, the point of Honest Achmed was to demonstrate that it was possible to write a facially reasonable and compliant CA inclusion request that was also intuitively unacceptable. It successfully demonstrated that the inclusion policies at the time needed to be made more rigorous, which they were.
It's refreshing to see folks who're obviously used to hiding behind clouds of bullshit get skewered by people who both know enough to see through it and have the time and energy to follow every thread to completion.
The most recent digicert thread smells suspiciously similar to those that lead up to the Entrust debacle.
It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores.
How many will agree to that removal? How many will see one more reason to forever turn off automatic updates and decide what to trust themselves, having seen yet another way some faceless entity they never knew about can break things?
It will certainly send a strong message, but likely not the intended one. All it will do is increase the lack of trust in centralised PKI in general.
I'm not sure how to put this nicely, but I'll try my best: The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision. (And corporate users can't do this anyways, since it's not in their hands, and keeping things out of date is not a reasonable option for an organization that has security standards.) Most of them won't know what is going on here, and if they hear about it, probably won't care or know what to do about it. That's the Web PKI infrastructure working as intended, because it would be infeasible for billions of people to properly understand the gravity of all of these decisions. In order to ensure that TLS connections are at least minimally safe, it's pretty much necessary for things to work this way.
I'm not arguing that it's good that a relatively small number of entities (mainly Google, Microsoft and Mozilla) decide which CAs are trustworthy, but that's all the more reason that it's important for all of this Web PKI work to happen completely in the open, so that the few who can spare the time and effort to scrutinize what is going on can help the rest of us have a safer Internet. We don't have a better solution. That's also why DigiCert issuing legal threats because they don't like how one of these issue reports makes them look is a serious problem that can't be tolerated.
> The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.
I'd imagine that they wouldn't do it to "take a stand" so much as "avoid the risk of getting their stuff broken in the short term" in this scenario, regardless of whichever party loses the blame game. See: the recent WordPress drama, which has turned customers away from both parties involved.
As I understand it, the way certificate authorities are removed from the trust store is progressive: they will announce a date after which new certificates from a given CA will no longer be trusted. I think this can be made even more progressive, by limiting the validity period of new certificates that will be trusted. DigiCert will have little recourse other than to let their customers know, and/or start providing certificates issued by another CA that follows the web PKI procedures and remains trusted. (They can still do that, of course, it's just that their own certificates issued by them directly won't be trusted anymore.)
On the flip side, for user impact, it will play out like this: Some bank or other important entity could possibly, for whatever reason, continue using a (presumably expired? unless DigiCert continues issuing anyways; note that most likely, they will not.) DigiCert certificate after the cut off date, which will lead to users receiving errors. Some of them will have HSTS setup, which will lead to an emergency situation where they have to issue a new certificate ASAP, as it will basically halt their business until they do. For places where there is no HSTS, users may be instructed to simply bypass the certificate warning temporarily, and support lines will be absolutely swamped until they actually fix the problem.
The WordPress situation is quite different. You don't have to use WordPress. Users don't even know what the Web PKI is to find an alternative to it, not that there is one or will be one.
Sure some users would keep the CA active out of fear, while everyone paying digicert would also begin to move out of fear.. What portion of the strange sort of sites that couldn't figure out how to get off but could figure out how to renew would bother with paying for a different error message?
The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.
...unless they notice the breakage, and you tell them to do so to stop it.
Ignoring the policy reasons that this won't happen, not all devices can be downgraded, so by then it's too late anyway. Best you can do is bypass a non-HSTS certificate warning.
That all assumes DigiCert continues issuing certificates they know won't be trusted. I assume most likely what will actually happen is they have to start selling certificates from another CA, but failing that they're probably just going to stop issuing certificates before the cutoff.
Is that really the claim, though? If someone chooses to punish DigiCert for compliance with the TRO then DigiCert might have grounds to file for declarative relief from the court, as the court could very well decide to help protect DigiCert from external consequences of their demand.
Perhaps, but only in the case that the delayed revocations were scoped to those certificates covered by the TRO, and not over 1000x more from subscribers who had nothing to do with the company who filed the TRO.
> All it will do is increase the lack of trust in centralised PKI in general.
“In general” is a broad statement, given that the overwhelming majority of people using the Web PKI have no idea that they’re doing so. DigiCert is not a legible part of the value chain for the average users; they won’t notice that the sites they use switch CAs. This strong disfavoritism towards vendors is arguably one of the Web PKI’s greatest strengths.
Many systems do not fetch updates from the Mozilla root store, but from their (possibly Debian-derived) stable distribution of it.
Meaning two highly respected entities, known for being well aware of the wider impact of their careful enforcement of strict policies, need to agree to cause any major breakage. When that happens, I can blindly trust they did the thing needed to keep the unaffected parts of that weird system working as intended.. and then still head to bugzilla and read about the background - to laugh at whoever triggered the mess.
If DigiCert were to lose browser trust (at this point, that's still a big if), it would happen the same way it happened with prior CAs, some of which were pretty big themselves (Symantec): all certificates issued after some date would not be trusted, yes. But all existing certificates would remain valid.
This gives certificate owners ample time to look for a different issuer and no certificate buyer would deliberately purchase a certificate from an issuer when they know some percentage of users will not trust that cert.
So for the end users, everything will keep working: the existing digicert certs stay valid and newly refreshed certs will be signed by a different authority. There is no need to turn off automatic updates over this.
Between Entrust and Symantec, we've already seen this happen to large well-known CAs and everything remained fine (not for the offenders, but, hey, that's the system working as intended)
This is shocking to read. Even attempting to choke the legal speech of web PKI contributors with legalistic bullying is a gross inversion of the purpose and goals of the organization, and IMO warrants revoking everything to do with DigiCert on the spot.
Always two sides to a story but the guy who caused the validation bug at DigiCert already resigned because of it (which is extreme), the Sectigo guy wanted to prevent the bug being closed so he could keep pushing them (in a subjectively prickly fashion) for more answers about their general responsiveness.
A bit of back and forth discourse is fine and expected, but if you keep pushing someone who has their own legal dept they're eventually going to wander over to the coffee machine and have a chat about it with them, then they're going to take a look and it becomes their problem.
So the number one rule would be don't even breath the word "legal" unless you want to invoke them. This particular response is just a letter telling them to back off and it's why you have a legal dept, so they can argue with each other. This one has just found it's way into the open.
There is a understandable perspective that says CAs shouldn't be burdened with legal risk in their discussions, but that's contrary to the fact these guys are commercial entities protecting their interests, so you don't get it both ways unless all your CAs are non-commercial, and even then that would only extend so far.
You've given a detailed rundown like you've been following, so what is your sense about the resignation? Did the guy resign willingly, or is Digicert the kind of management that likely scapegoated him?
He assumed personal responsibility rather than institutional. At the time, everyone in the community expressed alarm that that happened, because 1) and this is obvious, there's no way a single guy would be responsible for the entire fallout and 2) because it sets a bad precedent about what companies can do with their PoC wrt web PKI. It also meant a loss in institutional knowledge about how things worked.
"The actual reason for the underscore is so that services which allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore and be safe from unwanted certificate issuance. It serves the same purpose that /.well-known does for Agreed-Upon Change To Website, and that admin/administrator/webmaster/hostmaster/postmaster do for Constructed Email to Domain Contact. By using DNS records without underscores, DigiCert has violated a security-critical assumption that these services have made.
Therefore, this is truly a security-critical incident, "
That is a pretty brutal f' up of epic proportions... not sure any digicert cert should be trusted after that.
> The public record of Alegeus Technologies LLC v. DigiCert shows no attempt by DigiCert to contest the court’s order prior to the end of its preferred period of nearly 120 hours, even though such a motion could have freed DigiCert to revoke the certificates days earlier.
and
> The other question in comment 28 was for the language establishing DigiCert’s right to revoke Alegeus Technologies certificates. DigiCert has waffled on this point, first implying that this language was to be found on its website but later refusing to confirm that the language on the site applied to Alegeus Technologies at the time.
SPECULATION: Digicert may have offered special terms to Alegeus, and possibly other customers. They may have chosen not to dispute the TRO in court because they did not have grounds to do so under those agreements. They may also have included confidentiality terms in those contracts that prevented them from speaking about it.
OPINION: I am surprised that the forum allowed the issue to be closed without the above quoted questions being satisfied, though it is possible they are addressed elsewhere, I have not done a complete reading of all the linked issues.
> Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied
It's fascinating that we've built a system that has expended perhaps several million dollars of engineering, legal and admin etc time over the issue of a single letter not being capitalized [1], without any demonstrable impact beyond a failure to meet ambiguous specifications.
I do hope that dealing with all of the underlying issues around revocation etc makes the time and effort spent useful, and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues.
Basically the missing '_' was supposed to allow DNS providers who allow programmatic DNS record creation to filter out unauthorised certificate creation. So the certificates created without it could have been unauthorized by the owner of the domain they claim to certify.
Entrust got torpedoed basically for deploying an improvement to the requirements for its certificates slightly before the improvement got officially approved, and the browser people collectively lost their crud over the concept that Entrust didn't immediately revoke all of their... perfectly valid, secure certificates immediately.
Fundamentally, there is no accountability in the web PKI stewards. You want to talk about utter waste and incredible damage to the Internet, you can see it right here, in the people determining who is allowed to issue you sets of magic numbers that browsers have agreed are trustworthy, despite everyone involved behaving like complete children.
And of course, the browser operators all have their own root CAs, so basically have extremely motivated reasons to want to eliminate every commercial provider that isn't one of the monopoly companies.
Meanwhile:
- Compromised certificates are basically a non-issue from a threat model standpoint, every certificate error people hit are just... expired certificates people didn't rotate yet.
- Expired certificates cause issues for the majority of businesses at some point or another, making the internet increasingly fragile and unreliable.
Even taking the (one-sided) depiction of the conversations in DigiCert's letter at face value, maybe Sectigo's guy was being a git at best and intentionally trolling at worst. (I don't think he was, but let's play devil's advocate.) But even then, how did DigiCert think getting legal involved would possibly go well? Sectigo stands to gain publicity and lose nothing by going public with it to the CAB as they did here, and it's not like the CAB is going to play marriage counselor and get the two companies to make up because one of them got their feefees hurt.
Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. I don't know why DigiCert got particularly upset about this one.
> Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion.
As somebody who doesn't spend much time scrolling CAB reports, this was jarring to me.
Digicert's legal action seems nuts, and there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities, but it's hard to see any way that could be productively addressed given the back and forth in the thread. It's like I'm watching a theatrical production staring the most stereotypical corporate drones trading comments with the most stereotypical IRC nerds, both sides doing circles around an interesting topic but too busy trading blows to ever really get to it.
> there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities
As Digicert has repeatedly explained, this is simply how the United States legal system works. Courts have broad and indisputable power to issue temporary restraining orders, and the parties to a case must comply even if doing so violates some promise they made to a third party. (The point of the TRO is to maintain the status quo while the court figures out details like what promises have been made to who.) People in the PKI community who believe that some carefully written policy would enable CAs to reject an invalidation TRO, or convince a court that they cannot issue it, are wrong.
The reason it's never come up before is that no CA had previously attempted to enforce a widespread 24 hour revocation caused by its own error.
I don't think I disagree with you about what courts can do. And (as shown by all the back and forth in the CAB thread, and now here), it is risky: we currently have a situation where CAs can find themselves in a position where taking court-mandated actions (or lack of actions) puts them in jeopardy with the CAB, and violating court orders puts them in jeopardy with the government.
For all the back and forth in the CAB thread, it doesn't seem we're any closer to finding an escape valve for that, which seemingly would have to come from the CAB because there isn't even really a mechanism for courts to decide not to issue TROs about cert revocation, short of somehow taking a case around this to the Supreme Court (which isn't going to happen).
(I think Sectigo's argument is that Digicert did not even attempt to convince the court that it should be allowed to revoke those certificates in the mandated timeframe. If they had attempted and failed, I don't think they would be receiving criticism.)
That's their argument, yes, but it was clearly based on the incorrect belief that there's some emergency button you can press to demand that a court consider your arguments ASAP. As Digicert explained:
> The legal world does not move as fast as the demands of our CA ecosystem. Our legal approach was to work with the complainant’s legal team to get the TRO dismissed in 5 days.
The court would not have dissolved the TRO in anywhere close to 5 days without the complainant's cooperation, even if Digicert had an ironclad argument for doing so. Digicert made
the right choice to get the certificates rotated as fast as possible - and I don't think Sectigo intends to argue it would be better to stand on principle even if that makes the revocation slower.
Should've been more specific. You can ask a court to do all kinds of things, but the individual judge who reads your filing doesn't have to (and in most cases can't) carefully analyze your arguments that day. They need time to think it over, and probably hearings where you and the other party can explain all the arguments for why certain rulings should or shouldn't be made. A contract dispute like this, where one party says they have a right to do something and the other party says they don't, is almost always going to take longer than 1 or 5 or 30 days for a court to figure out.
Temporary restraining orders are the biggest exception. If DigiCert is about to do something crazy like take down all your websites, courts are generally willing to put a temporary stop to it without understanding all the details. "Preserve the status quo" and "prevent irreparable harm" are the buzzwords.
Could they not have had a clause in the contract saying “if you delay a revocation in any manner you owe us $100 million.”?
So a customer could go right ahead and get a TRO but long term it will cost them less than making sure their infrastructure can handle this rare event?
Liquidated damages are possible, but they have to have some relationship to the damages suffered, so $100 million is probably far too high. In any case, I think the complainant was correct that it was DigiCert who caused the entire scenario to happen by issuing certificates which they should have known were invalid.
For further reading, consider these two incidents which resulted in delayed revocation from DigiCert and a bunch of comments about how DigiCert should not be allowing delayed revocation:
Essentially, DigiCert has been delaying the revocation process (twice now) and people are unhappy about that. DigiCert has apparently attempted to silence those unhappy people (Sectigo and their representative Tim Callan) with legal action.
> Contrary to this statement, I received a letter from DigiCert’s lawyers, Wilson Sonsini, regarding posts made by Sectigo’s Chief Compliance Officer in bug 1910322. https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
> We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service.
1. Bugs happen. Critical ones, too. They didn't try to brush this under the carpet, but admitted to it, acted to resolve it and were transparent about it.
2. They worked quickly to make it happen. Would 24h been nice? Sure, but 24h is not much shorter than 120h. In general, 24h is plenty of time for some exploits and 120h doesn't open the window to many more. It would have been very different if it took them months or years to resolve it.
3. They genuinely engaged with the critics on bugzilla, even after Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it.
4. They could have taken legal actions against Sectigo's CCO directly but took the extra step to ask them to stop this nonsense. They didn't demand anything more and even outlined steps Sectigo needed to take to prevent any legal problems down the line, like affirming that their CCO did not make these statements on behalf of Sectigo, an affirmation that they would notify their employees to not make any actions that would violate the laws mentioned in their letter, affirm that their CCO would be instructed not to violate any of the laws outlined in their letter and lastly confirm that, upon consulting with their CCO, they were able to conclude that his statements were not meant to harm DigiCert.
The only ick is the short timeframe they expect a reply within, but that's sadly usual corporate US law practice...
Basically that letter is the result of asking an US law firm for help and telling them to be nice about it and helping their opponent through the process.
In a nutshell:
DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation.
Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner.
Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.
Reads to me like Digicert is hiding behind the TRO to protect themselves from upset customers over following the procedures they should have followed. I don't think they want to update their legal work to prevent customers from legal action over revocation, legal action has been working in their favor so far.
For a company whose entire business is verifying company names and handling the CA process, Digicert seems rather unwilling to actually stick to the process. They can't help having a TRO filed against them by customers with incompetent tech such as Alegeus Technologies LLC, but this isn't the first time they've failed to follow the appropriate processes.
Going through the courts to stop negative discourse seems rather crass for a CA. I don't understand why a company that has been subject of suspicion and doubt lime Digicert would do such a thing unless it's a last-ditch effort to get the heat off their backs.
I'm sure their clients love Digicert for not forcing them to replace their certificates at the appropriate times, but they're in for a surprise when this whole thing blows up and they suddenly need to find another cert vendor when Digicert gets delisted.
This seems like a misunderstanding of the TRO.
Alegeus, a client of DigiCert, filed a court motion to stop DigiCert from revoking their certificate. The courts granted the temporary restraining order.
There is no "update their legal work to prevent customers from legal action" that can avoid a temporary restraining order that is ordered by the courts to provide time to establish facts. Its simply how the legal process works. "they can't help but" seems unfair as the issue was the client Alegeus was unable to replace their certificates in the revocation timeline.
Further "Going through the courts", they didn't go to the courts a client of theirs did.
Digicert is absolutely not blameless. Outside of the TRO, they have failed to meet revocation windows before and yes, are likely using the TRO in this instance as a shield for that.
However the TRO itself is a concerning development that all CA's need to consider depending on their legal jurisdiction, as it could put companies in a bind of being legally ordered to not to complete something they are contractually and legally required to do within the required timeline and interrupt standard security practices in cases like this.
I dunno, seems to me in the TRO case there wasn't anything wrong with the certificates that had been issued. There wasn't any suggestion the the certificates had been issued to the wrong party or anything like that.
When performing DNS verification there's supposed to be an underscore in a record, which protects sites that let their users control arbitrary subdomains - you know, dyndns and things like that. Digicert mistakenly didn't ask the customer to include an underscore. Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
Then Digicert spent a load of the 24 hour disclosure window checking things on their end, then e-mailed the customers after office hours so when they get in the next day they barely had any time to respond.
The only "incompetent tech" here is the assholes who decided on the policy that certificates be urgently revoked, as if they'd been issued to the wrong person, when they hadn't been issued to the wrong customer.
The middle ground here is DigiCert should have been able and continued with the revocation of all the other certificates not impacted by the TRO within the policies and contractual agreements it has.
The policy and its designers aren't incompetent, the intention is to ensure security first when fact-finding can take time to establish a subset of actual abuse. This is common in IT/Security and is clearly communicated in the policy customers agree to as well as something agreed on by all CA's. Security > Uptime is the baseline, and while you may disagree with that and have good arguments for the contrary, the argument is much wider than us and has been extensively hashed out in the working groups and discussions around this and other similar issues and is implemented this way by consensus while weighing valid arguments for different approaches and pros and cons.
>Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
AFAIUI, Alegeus was just 1 customer of Digicert. But Digicert has tons of customers. Just because there wasn't a problem for Alegeus (due to Alegeus not being a dyndns provider), doesn't mean there weren't problems for other Digicert customers.
There was something wrong. They hadn't been issued in accordance with the baseline requirements.
What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?
DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days.
It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients.
Heck, their status page (https://status.digicert.com/incidents/3sccz3v31lc9) even gives instructions for how to request a delayed revocation - even though the initial incident page (https://www.digicert.com/support/certificate-revocation-inci...) says clearly:
"Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."
I think the eventual correct option is pretty clear.
A TRO like that is based on the company loudly declaring that revoking will cause real damage. That means their use of certificates is incompatible with the web PKI rules and ecosystem. That means they need to be migrated out ASAP, with every certificate authority refusing to take their business.
Make that series of consequences clear, and companies will stop trying that trick.
DigiCert and other CA's need to write in their terms that failure to follow the policies and revocation timelines can/will result in termination of the contract. Alegeus should have been dropped as a customer as soon as the incident was resolved and refused further products/renewals.
Use of a TRO protects you short term, but results in having to migrate to a new CA medium term. You can't stop them from using TRO's but you can make it not worth it.
I think that's pretty unrealistic, considering that X.509 is a de-facto and de-jure standard in a lot of places that also ignore that requirement. It's not always up to that company to make it possible to replace certificates easily, it's an entire chain of vendors (I know at least Salesforce's process would fail here). Unless you want those people to run self-signed/private CA certs.
The other option would be building in a way to revoke other CA's individual certificates if there's some consensus on them being compelled to not revoke them. Not sure if the status quo or this would be more dangerous, but if a TRO can compel a CA to not sign a revocation, can it also compel to sign a certificate?
A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done. Just because Salesforce can't get their stuff together doesn't mean the CA industry needs to bend the knee. Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
If running a private CA or self-signed certificates are even viable, then there are plenty of other workarounds that can be put in place (i.e. not updating the CRL so the software doesn't know about revocations).
If a CA's terms and legal documentation aren't tight enough to prevent a court from compel them to sign a certificate, that CA clearly cannot be trusted. Hopefully that issue can be fixed by writing clearer terms and better agreements, like people have been telling Digicert to do, but perhaps that's not possible at all. In that case, either the company should move to a jurisdiction where that kind of nonsense isn't possible, or it should be removed from global trust stores all together.
> A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done.
You say that and yet Teams has 320 million people using it, and I bet almost none of them enjoy the experience. Sometimes you just have to work with what you're given. Given "incompetence", we might as well throw in all of Azure.
> Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
That might be true for DV, but there's classes of certificates that take at least weeks to obtain, think BIMI VMC or codesigning.
The parent was correct - it's not about the company not using x509 certificates, but not using publicly trusted certificates. There are myriad private/internal PKI solutions available from OpenSSL & bash to millions of dollars of solutions from Big Vendor. If you can't replace the publicly-trusted certificate quickly, you probably don't need it to be publicly-trusted in the first place.
How can a court inhibit revocation when every CA declares their right to do so when you purchase a certificate?
It’s Saturday and the courts are closed. You park in a car park I own.
I have put up a sign saying I can instantly crush any crossover vehicles I like, as I consider them ugly and lacking in character.
As I load your car into my crusher, you dispute the legality of my sign. But the court is closed until Monday, and the sign says I can crush your car instantly, no waiting.
Should I be allowed to crush your car today? Or should I have to wait until Monday, so the disputed legality can be resolved?
That’s actually a question for you. You won’t truly know if you’re allowed to until a court decides. You can choose to proceed and crush it, and then deal with the consequences of doing so if it turns out you were wrong. Likely you’ll end up owing damages to the owner of the car, perhaps even punitive damages on top.
I think the same applies here.
The TRO actually means you aren't allowed, that's the point. It's an ordered injunction that legally obliges you from not acting until the courts can review the facts.
The court IS deciding, temporarily.
The better question is, how do you prevent this tactic from working?
For example, suppose there were required to be multiple parties who could issue a revocation, each in a different jurisdiction, and if any of them was ordered not to do it then the others would be required to do it, and would have the technical capacity to do it but not be subject to the jurisdiction of that court.
Well, you can contest the motion for a TRO, for a start. Digicert failed to do so.
You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
Correction, the petition for the TRO was filed ex parte. Digicert did not have any opportunity to respond before it was granted.
They certainly could have filed a response contesting the TRO. Then their customer could have filed another motion, and eventually (7 days later in this case) the judge would have ruled on the substance of it. Their judgement was that it would be preferable to work with the customer to resolve the technical issues with revocation, and submit a joint request to dismiss the TRO. The stated reasoning behind this was that it would be significantly faster than contesting the TRO. This is true: the certs were revoked and the TRO dropped within 3 days.
I think the communication on that point was severely lacking, as they only clarified it three months later and after significant hectoring in two different bug threads: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43
I also think it's reasonable not to take Digicert's statements at face value, given their history. But I think both of the points you made here are wrong:
> You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
Let's be clear about the timeline: Digicert notified their customers that the certs would be revoked. In between the time they notified the customer and the time of revocation (less than 24 hours), the customer got the ex parte restraining order. Are you suggesting that issuers should revoke certificates without notifying their users, so that the users don't have time to get an emergency TRO? I believe that would be in violation of the BRs.
> You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
To add to this, 3 days after the TRO was filed both parties moved to vacate the TRO.
DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED
I'm not sure DigiCert could have done anything about the TRO or the impacted certs, but it should have been able to move forward with the revocation of all other certificates. That IMO is the real issue/failure, alongside the concern/impact of TRO's on security processes in the future.
Just to be clear, the whole incident covered over 80,000 certificates. The TRO was applicable to only those of one subscriber - just over 70 certificates, yet caused the revocations of all 80k+ to be delayed.
You could just require publishing the revocation immediately with an effective date in the future.
Of course, if that system had been in place, DigiCert would probably be facing hundreds of lawsuits from businesses disrupted through no fault of their own rather than inside baseball PKI drama.
I can declare my right to do something as much as I want... that doesn't mean I get to do it, certainly not if a judge decides I can't.
Isn't Sectigo Comodo? Extra rich coming from them.
No, Sectigo is the PKI business that was carved out of Comodo, back in 2017. Comodo still exists, doing whatever they do. They have been totally separate entities since then.
Web PKI drama is always astonishing to me because it is one of the only areas of the entire world where a corporation's "fucking around" is seldom ever not followed by a sobering "finding out" period. The various entities that decide what CAs to trust can effectively dismantle any CA business in the world, basically at the drop of a hat. If DigiCert decides to play this game and lose, it would make them the biggest such loser so far. DigiCert is, as far as I know, the largest CA on the Internet. It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting.
Of course, I think that is unlikely, but on the other hand, it's just as cathartic to imagine whatever idiot at DigiCert thought it was a good idea to engage legal here to have the dressing down of a lifetime. I read the thread in question. It doesn't make DigiCert look good, but this action definitely is more damning to them than anything Collan said in my opinion.
> It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores
Their customers would be forced to give their business to Honest Achmed[1].
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=647959
Someone should start a real CA business, complete with all the proper audits and everything to get it into the browser trust stores… and call it “Honest Achmed’s Used Cars and Certificates” (have it buy some random used car dealership in the middle of nowhere so the name is not a lie)
Where’s a billionaire troll when you need one? (And a form of billionaire trolling that would upset a lot less people than Musk’s version of it.)
Would be even funnier if the billionaire’s name actually was Achmed
What has prevented this (a good legit CA co) from happening before now?
Time and money. Plus right now even if you bought the infra, the staff, paid for and passed the audits, and then waiting while Apple, Mozilla, Google and Oracle (at least) included your roots...Microsoft aren't taking more right now. So you have to wait for some unknown time in the future when/if they start doing that again. You could purchase a root off an existing CA, subject to the trust stores approving it, and the boatload of cash you'd need to buy it (plus still having the staff and infra to operate it).
> Microsoft aren't taking more right now. So you have to wait for some unknown time in the future when/if they start doing that again.
Interesting, I hadn’t heard that, is there anywhere I can read more about it?
> You could purchase a root off an existing CA,
As well as a sub-CA, I remember in theory you can have two independent CAs with cross-signing… but do browsers actually support that?
"Nothing public to point out to" is probably accurate but it is noted publicly here: https://learn.microsoft.com/en-us/security/trusted-root/new-...
Nothing public to point to, sorry.
Sub-CAs: Not really. Operational risk to the parent CA is huge, you'd be hard pressed to get any current public CA to sign an issuing CA to be operated externally. Cross-signing still works (though it is the stuff of nightmares in many cases) but again you have to have money and a CA willing to do it!
Where's the value?
Anyone can get a cert from lets-encrypt. No users of your website care how trustworthy your CA is. And CA's are too big to fail, so you barely need to worry about your CA being removed by the trust store. So you can only compete on price.
If we want to change this, we need a way for a certificate to be co-signed by multiple CA's. So that the certificate can be presented to the user, and they can figure out if any trusted CA of theirs has signed the certificate. This way, revoking trust in a CA becomes easier, because people should have multiple signatures on their certificate. That means, all of a sudden, that the quality of a CA actually matters.
Whilst it might seem this is already possible, it is not. Cross-signing is only a thing for intermediate certificates, and does not work. You can also have multiple certificates for the same key, but when starting a TLS session, you must present a single certificate. (This seems to have changed for TLS 1.3, so perhaps this is already possible?)
Let's Encrypt has been around for a decade and is doing pretty well for itself. The manual validation required for EV and OV certificates precludes it from being a passion project, unless managing a call center and a squad of document reviewers is your dream job.
Let's Encrypt doesn't issue EV and OV. Honest Achmed’s Used Cars and Certificates wouldn't have to either.
I'm actually curious why that wasn't approved
To my understanding, the point of Honest Achmed was to demonstrate that it was possible to write a facially reasonable and compliant CA inclusion request that was also intuitively unacceptable. It successfully demonstrated that the inclusion policies at the time needed to be made more rigorous, which they were.
[flagged]
It's refreshing to see folks who're obviously used to hiding behind clouds of bullshit get skewered by people who both know enough to see through it and have the time and energy to follow every thread to completion.
The most recent digicert thread smells suspiciously similar to those that lead up to the Entrust debacle.
It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores.
How many will agree to that removal? How many will see one more reason to forever turn off automatic updates and decide what to trust themselves, having seen yet another way some faceless entity they never knew about can break things?
It will certainly send a strong message, but likely not the intended one. All it will do is increase the lack of trust in centralised PKI in general.
I'm not sure how to put this nicely, but I'll try my best: The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision. (And corporate users can't do this anyways, since it's not in their hands, and keeping things out of date is not a reasonable option for an organization that has security standards.) Most of them won't know what is going on here, and if they hear about it, probably won't care or know what to do about it. That's the Web PKI infrastructure working as intended, because it would be infeasible for billions of people to properly understand the gravity of all of these decisions. In order to ensure that TLS connections are at least minimally safe, it's pretty much necessary for things to work this way.
I'm not arguing that it's good that a relatively small number of entities (mainly Google, Microsoft and Mozilla) decide which CAs are trustworthy, but that's all the more reason that it's important for all of this Web PKI work to happen completely in the open, so that the few who can spare the time and effort to scrutinize what is going on can help the rest of us have a safer Internet. We don't have a better solution. That's also why DigiCert issuing legal threats because they don't like how one of these issue reports makes them look is a serious problem that can't be tolerated.
> The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.
I'd imagine that they wouldn't do it to "take a stand" so much as "avoid the risk of getting their stuff broken in the short term" in this scenario, regardless of whichever party loses the blame game. See: the recent WordPress drama, which has turned customers away from both parties involved.
As I understand it, the way certificate authorities are removed from the trust store is progressive: they will announce a date after which new certificates from a given CA will no longer be trusted. I think this can be made even more progressive, by limiting the validity period of new certificates that will be trusted. DigiCert will have little recourse other than to let their customers know, and/or start providing certificates issued by another CA that follows the web PKI procedures and remains trusted. (They can still do that, of course, it's just that their own certificates issued by them directly won't be trusted anymore.)
On the flip side, for user impact, it will play out like this: Some bank or other important entity could possibly, for whatever reason, continue using a (presumably expired? unless DigiCert continues issuing anyways; note that most likely, they will not.) DigiCert certificate after the cut off date, which will lead to users receiving errors. Some of them will have HSTS setup, which will lead to an emergency situation where they have to issue a new certificate ASAP, as it will basically halt their business until they do. For places where there is no HSTS, users may be instructed to simply bypass the certificate warning temporarily, and support lines will be absolutely swamped until they actually fix the problem.
The WordPress situation is quite different. You don't have to use WordPress. Users don't even know what the Web PKI is to find an alternative to it, not that there is one or will be one.
Sure some users would keep the CA active out of fear, while everyone paying digicert would also begin to move out of fear.. What portion of the strange sort of sites that couldn't figure out how to get off but could figure out how to renew would bother with paying for a different error message?
The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.
...unless they notice the breakage, and you tell them to do so to stop it.
Ignoring the policy reasons that this won't happen, not all devices can be downgraded, so by then it's too late anyway. Best you can do is bypass a non-HSTS certificate warning.
That all assumes DigiCert continues issuing certificates they know won't be trusted. I assume most likely what will actually happen is they have to start selling certificates from another CA, but failing that they're probably just going to stop issuing certificates before the cutoff.
> mainly Google, Microsoft and Mozilla
They don't have free rein to do whatever they want. If they chose to remove them, they are going to have to be ready to defend themselves in court.
Why exactly do you think certificate authorities have a legal right to appear in trust stores?
Is that really the claim, though? If someone chooses to punish DigiCert for compliance with the TRO then DigiCert might have grounds to file for declarative relief from the court, as the court could very well decide to help protect DigiCert from external consequences of their demand.
Perhaps, but only in the case that the delayed revocations were scoped to those certificates covered by the TRO, and not over 1000x more from subscribers who had nothing to do with the company who filed the TRO.
Both Google and Microsoft tend to act like they have enough money and lawyers to defend whatever behaviour they feel like engaging in.
Those companies are always ready to defend themselves in court, and this is a country with extremely strong free speech rights.
What country?
> All it will do is increase the lack of trust in centralised PKI in general.
“In general” is a broad statement, given that the overwhelming majority of people using the Web PKI have no idea that they’re doing so. DigiCert is not a legible part of the value chain for the average users; they won’t notice that the sites they use switch CAs. This strong disfavoritism towards vendors is arguably one of the Web PKI’s greatest strengths.
Many systems do not fetch updates from the Mozilla root store, but from their (possibly Debian-derived) stable distribution of it. Meaning two highly respected entities, known for being well aware of the wider impact of their careful enforcement of strict policies, need to agree to cause any major breakage. When that happens, I can blindly trust they did the thing needed to keep the unaffected parts of that weird system working as intended.. and then still head to bugzilla and read about the background - to laugh at whoever triggered the mess.
If DigiCert were to lose browser trust (at this point, that's still a big if), it would happen the same way it happened with prior CAs, some of which were pretty big themselves (Symantec): all certificates issued after some date would not be trusted, yes. But all existing certificates would remain valid.
This gives certificate owners ample time to look for a different issuer and no certificate buyer would deliberately purchase a certificate from an issuer when they know some percentage of users will not trust that cert.
So for the end users, everything will keep working: the existing digicert certs stay valid and newly refreshed certs will be signed by a different authority. There is no need to turn off automatic updates over this.
Between Entrust and Symantec, we've already seen this happen to large well-known CAs and everything remained fine (not for the offenders, but, hey, that's the system working as intended)
This is shocking to read. Even attempting to choke the legal speech of web PKI contributors with legalistic bullying is a gross inversion of the purpose and goals of the organization, and IMO warrants revoking everything to do with DigiCert on the spot.
Always two sides to a story but the guy who caused the validation bug at DigiCert already resigned because of it (which is extreme), the Sectigo guy wanted to prevent the bug being closed so he could keep pushing them (in a subjectively prickly fashion) for more answers about their general responsiveness.
A bit of back and forth discourse is fine and expected, but if you keep pushing someone who has their own legal dept they're eventually going to wander over to the coffee machine and have a chat about it with them, then they're going to take a look and it becomes their problem.
So the number one rule would be don't even breath the word "legal" unless you want to invoke them. This particular response is just a letter telling them to back off and it's why you have a legal dept, so they can argue with each other. This one has just found it's way into the open.
There is a understandable perspective that says CAs shouldn't be burdened with legal risk in their discussions, but that's contrary to the fact these guys are commercial entities protecting their interests, so you don't get it both ways unless all your CAs are non-commercial, and even then that would only extend so far.
You've given a detailed rundown like you've been following, so what is your sense about the resignation? Did the guy resign willingly, or is Digicert the kind of management that likely scapegoated him?
He assumed personal responsibility rather than institutional. At the time, everyone in the community expressed alarm that that happened, because 1) and this is obvious, there's no way a single guy would be responsible for the entire fallout and 2) because it sets a bad precedent about what companies can do with their PoC wrt web PKI. It also meant a loss in institutional knowledge about how things worked.
From the bugzilla
"The actual reason for the underscore is so that services which allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore and be safe from unwanted certificate issuance. It serves the same purpose that /.well-known does for Agreed-Upon Change To Website, and that admin/administrator/webmaster/hostmaster/postmaster do for Constructed Email to Domain Contact. By using DNS records without underscores, DigiCert has violated a security-critical assumption that these services have made.
Therefore, this is truly a security-critical incident, "
That is a pretty brutal f' up of epic proportions... not sure any digicert cert should be trusted after that.
Looking at the original report ( https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 ) I can see a couple of questions that DigiCert appears to be avoiding:
> The public record of Alegeus Technologies LLC v. DigiCert shows no attempt by DigiCert to contest the court’s order prior to the end of its preferred period of nearly 120 hours, even though such a motion could have freed DigiCert to revoke the certificates days earlier.
and
> The other question in comment 28 was for the language establishing DigiCert’s right to revoke Alegeus Technologies certificates. DigiCert has waffled on this point, first implying that this language was to be found on its website but later refusing to confirm that the language on the site applied to Alegeus Technologies at the time.
SPECULATION: Digicert may have offered special terms to Alegeus, and possibly other customers. They may have chosen not to dispute the TRO in court because they did not have grounds to do so under those agreements. They may also have included confidentiality terms in those contracts that prevented them from speaking about it.
OPINION: I am surprised that the forum allowed the issue to be closed without the above quoted questions being satisfied, though it is possible they are addressed elsewhere, I have not done a complete reading of all the linked issues.
EDIT to add: DigiCert has a response in a different thread here: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 that would appear to contradict my speculation. Specifically
> Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied
It's fascinating that we've built a system that has expended perhaps several million dollars of engineering, legal and admin etc time over the issue of a single letter not being capitalized [1], without any demonstrable impact beyond a failure to meet ambiguous specifications.
I do hope that dealing with all of the underlying issues around revocation etc makes the time and effort spent useful, and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues.
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1894560
The bug that prompted this was more serious: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c74
Basically the missing '_' was supposed to allow DNS providers who allow programmatic DNS record creation to filter out unauthorised certificate creation. So the certificates created without it could have been unauthorized by the owner of the domain they claim to certify.
I think that the Van Halen Brown M&M anecdote is relevant here.
> and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues.
In your view, are there any meaningful issues going un-addressed currently?
Entrust got torpedoed basically for deploying an improvement to the requirements for its certificates slightly before the improvement got officially approved, and the browser people collectively lost their crud over the concept that Entrust didn't immediately revoke all of their... perfectly valid, secure certificates immediately.
Fundamentally, there is no accountability in the web PKI stewards. You want to talk about utter waste and incredible damage to the Internet, you can see it right here, in the people determining who is allowed to issue you sets of magic numbers that browsers have agreed are trustworthy, despite everyone involved behaving like complete children.
And of course, the browser operators all have their own root CAs, so basically have extremely motivated reasons to want to eliminate every commercial provider that isn't one of the monopoly companies.
Meanwhile:
- Compromised certificates are basically a non-issue from a threat model standpoint, every certificate error people hit are just... expired certificates people didn't rotate yet.
- Expired certificates cause issues for the majority of businesses at some point or another, making the internet increasingly fragile and unreliable.
So what's happened in the ... two months and a bit since the dates mentioned for the letters that this is apparently in reference to?
Even taking the (one-sided) depiction of the conversations in DigiCert's letter at face value, maybe Sectigo's guy was being a git at best and intentionally trolling at worst. (I don't think he was, but let's play devil's advocate.) But even then, how did DigiCert think getting legal involved would possibly go well? Sectigo stands to gain publicity and lose nothing by going public with it to the CAB as they did here, and it's not like the CAB is going to play marriage counselor and get the two companies to make up because one of them got their feefees hurt.
Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. I don't know why DigiCert got particularly upset about this one.
> Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion.
As somebody who doesn't spend much time scrolling CAB reports, this was jarring to me.
Digicert's legal action seems nuts, and there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities, but it's hard to see any way that could be productively addressed given the back and forth in the thread. It's like I'm watching a theatrical production staring the most stereotypical corporate drones trading comments with the most stereotypical IRC nerds, both sides doing circles around an interesting topic but too busy trading blows to ever really get to it.
> there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities
As Digicert has repeatedly explained, this is simply how the United States legal system works. Courts have broad and indisputable power to issue temporary restraining orders, and the parties to a case must comply even if doing so violates some promise they made to a third party. (The point of the TRO is to maintain the status quo while the court figures out details like what promises have been made to who.) People in the PKI community who believe that some carefully written policy would enable CAs to reject an invalidation TRO, or convince a court that they cannot issue it, are wrong.
The reason it's never come up before is that no CA had previously attempted to enforce a widespread 24 hour revocation caused by its own error.
I don't think I disagree with you about what courts can do. And (as shown by all the back and forth in the CAB thread, and now here), it is risky: we currently have a situation where CAs can find themselves in a position where taking court-mandated actions (or lack of actions) puts them in jeopardy with the CAB, and violating court orders puts them in jeopardy with the government.
For all the back and forth in the CAB thread, it doesn't seem we're any closer to finding an escape valve for that, which seemingly would have to come from the CAB because there isn't even really a mechanism for courts to decide not to issue TROs about cert revocation, short of somehow taking a case around this to the Supreme Court (which isn't going to happen).
(I think Sectigo's argument is that Digicert did not even attempt to convince the court that it should be allowed to revoke those certificates in the mandated timeframe. If they had attempted and failed, I don't think they would be receiving criticism.)
That's their argument, yes, but it was clearly based on the incorrect belief that there's some emergency button you can press to demand that a court consider your arguments ASAP. As Digicert explained:
> The legal world does not move as fast as the demands of our CA ecosystem. Our legal approach was to work with the complainant’s legal team to get the TRO dismissed in 5 days.
The court would not have dissolved the TRO in anywhere close to 5 days without the complainant's cooperation, even if Digicert had an ironclad argument for doing so. Digicert made the right choice to get the certificates rotated as fast as possible - and I don't think Sectigo intends to argue it would be better to stand on principle even if that makes the revocation slower.
> the incorrect belief that there's some emergency button you can press to demand that a court consider your arguments ASAP
... isn't that exactly the legal button the complainant pushed to prevent the revocation?
Should've been more specific. You can ask a court to do all kinds of things, but the individual judge who reads your filing doesn't have to (and in most cases can't) carefully analyze your arguments that day. They need time to think it over, and probably hearings where you and the other party can explain all the arguments for why certain rulings should or shouldn't be made. A contract dispute like this, where one party says they have a right to do something and the other party says they don't, is almost always going to take longer than 1 or 5 or 30 days for a court to figure out.
Temporary restraining orders are the biggest exception. If DigiCert is about to do something crazy like take down all your websites, courts are generally willing to put a temporary stop to it without understanding all the details. "Preserve the status quo" and "prevent irreparable harm" are the buzzwords.
Could they not have had a clause in the contract saying “if you delay a revocation in any manner you owe us $100 million.”?
So a customer could go right ahead and get a TRO but long term it will cost them less than making sure their infrastructure can handle this rare event?
Liquidated damages are possible, but they have to have some relationship to the damages suffered, so $100 million is probably far too high. In any case, I think the complainant was correct that it was DigiCert who caused the entire scenario to happen by issuing certificates which they should have known were invalid.
Can someone point to where i can read some context?
DigiCert's legal threat, while obviously biased towards DigiCert, gives some context: https://bug1950144.bmoattachments.org/attachment.cgi?id=9468...
For further reading, consider these two incidents which resulted in delayed revocation from DigiCert and a bunch of comments about how DigiCert should not be allowing delayed revocation:
- Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1894560, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 - incident due to the issuance of some certificates with incorrectly-capitalized phrases in the certificate's Business Category field; baseline requirements require revocation within five days but DigiCert dragged that out much further
- Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1910322, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1910805, DigiCert information page https://www.digicert.com/support/certificate-revocation-inci... - incident due to incorrect CNAME-based domain validation (failure to check that the CNAME started with an underscore); baseline requirements require revocation within 24 hours but DigiCert was stopped by the TRO and revoked after five days.
Essentially, DigiCert has been delaying the revocation process (twice now) and people are unhappy about that. DigiCert has apparently attempted to silence those unhappy people (Sectigo and their representative Tim Callan) with legal action.
I don't believe that that's a legal filing. DigiCert never filed anything with the courts. That's just a letter to Sectigo threatening to sue.
Whoops, sorry, should've said legal threat not filing - fixed.
Can someone link to (or post copies of) the offending questions/statements?
In the second paragraph:
> Contrary to this statement, I received a letter from DigiCert’s lawyers, Wilson Sonsini, regarding posts made by Sectigo’s Chief Compliance Officer in bug 1910322. https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
> The code worked in our original monolithic system but was not implemented properly when we moved to our micro-services systems
:chefs_kiss:
it gets even better:
> We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service.
> multiple random value generation microservices
Wot. They had _multiple_ services generating random numbers?
It might mean multiple instances of the same code (which would explain why there might be commonalities)?
Some might see horror but I see a few genius level engineers 'working' 1hour a month rebuilding the same getrandom syscall.
They hiring? :]
As we all know, software isn't software unless it can immediately scale to having 12 billion users.
Streisand effect 101?
Why do you all think DigiCert handled this badly?
1. Bugs happen. Critical ones, too. They didn't try to brush this under the carpet, but admitted to it, acted to resolve it and were transparent about it.
2. They worked quickly to make it happen. Would 24h been nice? Sure, but 24h is not much shorter than 120h. In general, 24h is plenty of time for some exploits and 120h doesn't open the window to many more. It would have been very different if it took them months or years to resolve it.
3. They genuinely engaged with the critics on bugzilla, even after Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it.
4. They could have taken legal actions against Sectigo's CCO directly but took the extra step to ask them to stop this nonsense. They didn't demand anything more and even outlined steps Sectigo needed to take to prevent any legal problems down the line, like affirming that their CCO did not make these statements on behalf of Sectigo, an affirmation that they would notify their employees to not make any actions that would violate the laws mentioned in their letter, affirm that their CCO would be instructed not to violate any of the laws outlined in their letter and lastly confirm that, upon consulting with their CCO, they were able to conclude that his statements were not meant to harm DigiCert.
The only ick is the short timeframe they expect a reply within, but that's sadly usual corporate US law practice...
Basically that letter is the result of asking an US law firm for help and telling them to be nice about it and helping their opponent through the process.