[Servercert-wg] Ballot SC29: System Configuration Management

[Ryan Sleevi]

On Wed, Apr 1, 2020 at 6:36 AM Dimitris Zacharopoulos (HARICA) via
Servercert-wg <servercert-wg at cabforum.org> wrote:

> The team considers this "Emergency Change Order" to cause unnecessary
> delay in some cases, for some categories of OS patches (e.g. security
> updates). Whenever the vendor has patches that *the vendor* labels as
> security updates, we would like to have the option for these patches to be
> applied as soon as possible,
>

Yes, this is good.


> without requiring additional human review by the CA.
>

No, this is not good.


> We would also like to remind everyone that this ballot affects
> "Certificate Systems, Issuing Systems, Certificate Management Systems,
> Security Support Systems, and Front-End / Internal-Support Systems", which
> covers almost every system under the CA's PKI management.
>

Yes, this is good.


>
>    1. Human review of patches for so many different OS components,
>    sometimes without enough details or the source code available, is too
>    difficult to be done correctly and humans do make mistakes.
>
> Of course - so do OS vendors! But again - what do we mean by "review". I
> don't understand it to be a line-by-line formal code analysis. The issue at
> hand is whether the OS vendor is addressing a security issue which the CA
> is likely to suffer from, and that the patches produced don't introduce any
> extra vulnerability which the CA cannot tolerate.
>
>
> This decision must be left for the CA to make based on its risk assessment.
>

No. The CA's risk assessment is a risk assessment that impacts Relying
Parties. Time and time again, Relying Parties are left holding the
short-end of the stick due to CA risk assessments. The only recourse the
Relying Party has is to simply stop trusting the CA, which is not
unreasonable nor unfathomable, but does cause interoperability issues when
CAs are widely used by Subscribers that fail to uphold the security needs
of Relying Parties.

I want to separate out two aspects of HARICA's reply. It seems like the two
objections are:
1) Whether humans should be deciding whether or not to apply a patch
2) Whether it's OK to automatigically ingest any patches from an OS vendor,
using an OS-vendor provided mechanism, without any form of review or
insight.

I can understand disagreement about 1. However, 2 is simply unacceptable,
because it quickly and rapidly leads to a situation where a CA cannot
describe what the expected state of its systems are. I do not consider "Up
to date" to be a reasonable or acceptable answer here, because it's a
statement without any objective frame of reference (even temporally, it's
suspect based on the delivery mechanisms).

HARICA's suggestion of allowing a third-party to modify their systems
arbitrarily - which is effectively the proposal - is not a risk that should
be fobbed off to Relying Parties.

I wholly understand and appreciate not wanting to undergo lengthy "patch
qualifications". Having worked with both government and Enterprises, I
fully acknowledge a failure to timely patch systems absolutely leaves
systems vulnerable. So I absolutely do not want to introduce undue delays.
However, basic record keeping is not an undue delay, nor is having a
cohesive inventory of the state of the system, at all times, unreasonable.
Among other things, it's absolutely essential for post-mortem analysis to
understand what the system 'should' be and what it 'is', and when, where,
and how attackers caused it to diverge.


> To accept a patch - at least to me - means that you are aware of the issue
> which the patch seeks to address (ie, you've looked at the changelog);
> you've determined that you are likely to suffer from the asserted
> vulnerability; you decide that the exploit of the vulnerability is an
> unacceptable risk; that you believe that the vendor has sufficient QA
> processes in place not to introduce further unacceptable vulnerabilities
> while deploying the updated software; and that you know how to roll back
> the change in the event that it either does not work or introduces
> unacceptable changes in your systems. [There are probably more steps in
> there, but those are the major ones, to my thinking].
>
>
> The belief that the vendor has sufficient QA process has already been
> determined/decided when the CA decided to accept this vendor as.... a
> trusted vendor :-)
>

The belief that the CA has effectively determined whether or not the vendor
has sufficient QA process, and continually assesses that, is... suspect :-)

Can you share the contracts or audits you have with the OS vendor to assess
the CA's risk assessment is well-founded? Or is this just a variant of "Nobody
ever got fired for buying IBM
<https://www.ibm.com/ibm/history/ibm100/us/en/icons/personalcomputer/words/>
?"

I highlight this, because your justification here is that a one-time
evaluation when a CA builds the system is sufficient. The longer the system
is in place, the more incentives the CA has to keep the status quo, even
when the vendor dips below their quality threshhold, because of the
opportunity cost of transitioning. If you don't believe this is the case,
then it's the same fundamental challenge with removing CAs who, after
they're added, dip below a quality threshold: the more in use they are, the
more disruptive it becomes.


> I think our misunderstanding is about the generic use of the term
> "vendor". Our concerns are more focused on OS vendors that the CA trusts
> for doing a good job in addressing security risks at the OS level, doing a
> good job at QA and has a history of delivering security updates/patches
> that do not introduce additional vulnerabilities. CAs should be allowed to
> make this decision once and not daily by approving/screening security
> updates that are critical for the stability and safety of the CA systems.
>

Right, we disagree, because that just externalizes the risk into an
inconsistent mechanism for which Relying Parties have to accept that
disparate CAs with disparate skills and backgrounds will all reach the
intended conclusion. This is no different than arguing that the CA can make
an effective risk calculus for "Any other method" of domain validation, or
the discussions around the determination of Agencies of Registration /
Incorporation.

At its core, the more discretion is afforded, the greater the
inconsistency. It may be that the world is tough and messy and such
discretion is needed. Yet you haven't really demonstrated why or how that
would benefit Relying Parties, nor quite examiend or discussed the risks
that have to be addressed or mitigated. That gives little hope, if the most
ardent supporter can't articulate the tradeoffs involved, while also
highlighting the fundamental risk.

In our understanding, installing security patches without undue delay is
> more secure for the overall security which ultimately benefits the
> end-users. This is the one automation that can be done securely and
> unattended for the benefit of the ecosystem.
>

Yes, we're in strong agreement here. However, as mentioned earlier, you're
conflating "undue delay" with "having a system inventory". Having a human
review of the state of the system is an entirely reasonable and necessary
step, and recording that status is valuable. A CA with an attention to not
having an undue delay can and should ensure such change orders are promptly
and timely reviewed.

If your concern is patch-gapped vulnerabilities, then we can reduce the
timing for applying patches from "6 months" into something much more
sensible. However, that's a separate discussion from having a change
management system.

And I would be horrified to learn if CAs are installing software right now
> _without_ ensuring that software's authenticity and integrity!
>
>
> So would we :-)
>

Can you point to HARICA's analysis of all of its vendor's automatic
software update mechanisms and the relative security strengths? Absent
that, I have to conclude that the situation today is indistinguishable from
installing software without ensuring authenticity and integrity. You might
feel that No True Scotsman/No True Software Vendor Used by CAs would ever
have such issues, but that's not a fair or reasonable assumption. In the
absence of evidence of due diligence, you have to assume hostility or
negligence - that's what being a defender is about.


> We hope to have provided a clear view of our position and why we see real
> security benefit from quick and automated installation of OS security
> patches.
>

These are distinct things, though. Quick installation of OS security
patches is not the same as automated installation of OS security patches.
Again, I'm 100% onboard with quick, but automated is highly suspect here on
its merits, compared to the obvious downside that the CA cannot demonstrate
what state their systems are in or should be in.

The notion of automatically applying patches flies in the face of the
modern best practice, of having a known system image that can be rapidly
replicated to scale and/or replace downed systems. I've worked in
Enterprise admin to know long enough the intersection beyond myriad system
patches absolutely effects the behaviour of the systems in sometimes subtle
and surprising ways - and that's the absolute worst time to find out, when
there is a system disruption. Automated installation, from the OS vendor
directly (as opposed to being mediated by a systems management/deployment
suite), particularly of overlapping patches (i.e. when there is not a
linear and canonical sense of 'version' that defines state), is a danger
zone.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cabforum.org/pipermail/servercert-wg/attachments/20200401/4f6a1537/attachment.html>