[Servercert-wg] Ballot SC-75 - Pre-sign linting

[Rob Stradling]

> Do people have strong feelings against RECOMMENDing linting of currently-valid certificates when linting software gets updated, with a threshold of 90 days after the update of the linting software?

ISTM that some people tend to read "SHOULD" or "RECOMMENDED" as "we strongly urge you to do this thing, but don't worry if you don't because it's 100% optional".  IMHO, this interpretation is nearer to "MAY" than to the real definition of "SHOULD".

"SHOULD" needs to be understood as "100% required, except for 'valid reasons in particular circumstances'" (quoting RFC2119), and I think I'm right in saying that in CABForum documents we often use "SHOULD" to define requirements that we anticipate will become "MUST" requirements in the future.

With that in mind, I do not support RECOMMENDing something that is known to be technologically infeasible for high-volume certificate issuers.

> CAs with such large volumes could perform sampling and pick a smaller number of certificates for linting during the linter update

Imposing a minimum percentage sample size (even as low as 1%) might be challenging for some high-volume CAs.  Perhaps we could solve that by letting CAs pick their own sample sizes, but TBH I'm not convinced that sampling is the best or only approach.

ISTM that the goals here are (1) to avoid rolling out buggy linters to production CA systems and (2) to assist linter developers with discovering and fixing bugs in linters.  So why don't we encourage CAs to look closely at the changelog of each new linter version and then do targeted testing of the lints that have been added or changed?  i.e., white box testing instead of black box testing.  Perhaps we could even encourage CAs to review the changes to the linter's source code too.  I think these sorts of approaches are more likely to uncover linter bugs, and more quickly.

But...
Whilst it's great to encourage the CA community to collectively engage in helping to improve the quality of linters, I'm not convinced it's appropriate to define any sort of "SHOULD" requirement in the BRs relating to this.  Similarly, whilst it's great to encourage the CA community to collectively engage in supporting the CT ecosystem, we don't have a "The CA SHOULD operate a CT log" requirement specified anywhere.

________________________________
From: Servercert-wg <servercert-wg-bounces at cabforum.org> on behalf of Dimitris Zacharopoulos (HARICA) via Servercert-wg <servercert-wg at cabforum.org>
Sent: 03 June 2024 06:13
To: Aaron Gable <aaron at letsencrypt.org>; CA/B Forum Server Certificate WG Public Discussion List <servercert-wg at cabforum.org>
Subject: Re: [Servercert-wg] Ballot SC-75 - Pre-sign linting

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.



On 1/6/2024 12:20 π.μ., Aaron Gable wrote:
On Wed, May 29, 2024 at 10:57 PM Dimitris Zacharopoulos (HARICA) via Servercert-wg <servercert-wg at cabforum.org<mailto:servercert-wg at cabforum.org>> wrote:
While we're in this vein, it would also be useful to add a recommendation for CAs to lint all non-expired, non-revoked certificates whenever they install an update of their linting software.

  *   "The CA SHOULD perform Linting on the corpus of its non-expired, non-revoked Subscriber Certificates whenever it updates the Linting software".

What do people think about these proposals?

Just chiming in to say that I don't love this proposal, for a few reasons.

1. Linting software has not always had a great track record of applying new lints (based on new requirements) only to certificates issued after a certain date. Running a new linting tool over old certificates frequently raises warnings or errors which were not actually errors at the time of issuance. Zlint has support for this behavior, but it is not used consistently across all lints in their corpus. A quick glance at pkilint's source does not seem to show any support for this behavior, but I easily could be wrong.

Linters can be expected to throw false positives and it's the CA's responsibility to interpret those results properly. Linting the non-expired, non-revoked certificates is usually a "one-off" task (linting software is usually not updated so frequently) and if the CA decides that a certain lint is a false positive, it could be excluded for that run. In most cases though, updated linters may catch past compliance matters and mis-issuances that are correctly detected and reported. It's always best to risk some false positives (which you can document and move on) than to risk missing real misissuances which will ultimately be revealed by third parties scanning the CT logs.


2. Some CAs have very large certificate corpuses, e.g. Let's Encrypt has 400 million currently-valid certificates. Some linting tools are very slow, e.g. pkilint's `lint_pkix_cert` takes 300ms per run. At that rate, re-linting LE's whole corpus would take four years. I'm sure there are speedups to be had, but they'd have to be several orders of magnitude to make that feasible.

I believe this is a valid case for CAs that have large certificate volumes and linting every single currently-valid certificate is not a viable option, which is why this is a SHOULD requirement. Even so, based on audit methodologies, LE and other CAs with such large volumes could perform sampling and pick a smaller number of certificates for linting during the linter update.


3. Any large systems engineer knows that streaming processing and batch processing infrastructure are very different, with wholly different software and hardware setups to make each efficient. I think it is much more important to incentivize stream-linting (i.e. as issuance happens), and that it would be counterproductive to require CAs to invest in both at the same time.

I'm probably thinking about this a bit differently because linting can be executed in multiple ways. It's the same software that can be executed in the issuance pipeline (pre-sign linting) and after the issuance pipeline (post-sign linting). In the latter case, it can be executed in batch mode that checks multiple certificates. The ballot certainly puts more weight on the stream-linting process but also recommends post-sign linting, at least when the linting software gets updated. I believe this is a good balance that puts all the expectations of using linting tools in the BRs.

Do people have strong feelings against RECOMMENDing linting of currently-valid certificates when linting software gets updated, with a threshold of 90 days after the update of the linting software?


Dimitris.



Thanks,
Aaron

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cabforum.org/pipermail/servercert-wg/attachments/20240603/0fbd14a1/attachment-0001.html>