[Servercert-wg] Ballot SC29: System Configuration Management

[Neil Dunbar]

On 30/03/2020 11:07, Dimitris Zacharopoulos (HARICA) via Servercert-wg 
wrote:

> We had an internal discussion at HARICA and focused mainly on the OS 
> security patching process. Here are some of the team's observations:
>
>  1. Security patches, especially for 0-day exploits, should be applied
>     as soon as possible. Human review in most cases adds unnecessary
>     delay and increases the risk of the CA being vulnerable to attacks.
>
It's not at all clear to me that this is the case. However, it might 
help to understand what the notion of "review" means in this case. I 
suspect that we're coming at this word from different angles.

Emergency Change Orders are a well known phenomenon in most workflows. 
Now - they _should_ be painful (ie, a senior manager or director should 
be approving the change) because that acts as a discouragement to 
fast-track proper testing via the ECO mechanism.

>  1. Human review doesn't scale as the number of Systems increase.
>     Automation is the only way to handle large number of Systems.
>
Well, no one is asking for a per-system review. If you host a curated 
mirror of your vendor's upstream patches, then the current automation 
systems would work with virtually no alteration - just a repository change.
>
>  1. Human review of patches for so many different OS components,
>     sometimes without enough details or the source code available, is
>     too difficult to be done correctly and humans do make mistakes.
>
Of course - so do OS vendors! But again - what do we mean by "review". I 
don't understand it to be a line-by-line formal code analysis. The issue 
at hand is whether the OS vendor is addressing a security issue which 
the CA is likely to suffer from, and that the patches produced don't 
introduce any extra vulnerability which the CA cannot tolerate.

To accept a patch - at least to me - means that you are aware of the 
issue which the patch seeks to address (ie, you've looked at the 
changelog); you've determined that you are likely to suffer from the 
asserted vulnerability; you decide that the exploit of the vulnerability 
is an unacceptable risk; that you believe that the vendor has sufficient 
QA processes in place not to introduce further unacceptable 
vulnerabilities while deploying the updated software; and that you know 
how to roll back the change in the event that it either does not work or 
introduces unacceptable changes in your systems. [There are probably 
more steps in there, but those are the major ones, to my thinking].

Put another way - if you knew that a patch addressed a vulnerability in 
the Bluetooth stack on a host, but you have no Bluetooth equipment in 
your array, then isn't patching something with no avenue of exploitation 
a source of risk? And if so, is that risk then acceptable? It may, or 
may not be, but I think an auditor would at least want to see that 
someone had made that judgement call.

>  1. The risk of being non-functional is of lesser importance and
>     potential impact, compared to the risk of remaining vulnerable
>     while waiting for a human review.
>
> We believe that OS patches, digitally signed by the software vendor 
> and provided through secure channels, MUST be allowed as a dully 
> approved Change Management Policy practice, to be applied without 
> employing a human review process. Such updates and patches are 
> beforehand thoroughly tested by the software vendors themselves and 
> the CA staff should be discouraged to duplicate such a process with 
> dubious results.
>
Again - I don't think anyone is asking for the CAs to _duplicate_ the QA 
processes of the software vendor - merely to be assured that the patches 
which are going to be installed are proportionate to the risk being 
mitigated.

For the vast majority of (non-emergency) patches which get applied - 
surely we apply them to test systems before we go live on production? 
Isn't that part of a proper review?

And I would be horrified to learn if CAs are installing software right 
now _without_ ensuring that software's authenticity and integrity!

> The recommended change in 1.h:
>
> "Ensure that the CA’s security policies encompass a Change Management 
> Process, following the principles of documentation, approval and 
> testing, and to ensure that all changes to Certificate Systems, 
> Issuing Systems, Certificate Management Systems, Security Support 
> Systems, and Front-End / Internal-Support Systems follow said Change 
> Management Process;"
>
> doesn't seem to clearly allow this practice. We cannot support a 
> ballot that enforces manual human review in applying OS patches which 
> puts systems at risk, even for a short amount of time.

I think that it does allow the practice, even for urgent changes - most 
Change Management Systems encompass the notion of Emergency Change 
Orders. The review can be as little as "we cannot accept the 
vulnerability existing a moment longer; the vendor has produced patch X; 
fire up an ECO and phone the Chief Operating Officer because the risk of 
not deploying the patch is greater than the risk of deploying it".

I guess that my question then boils down to this: other high risk or 
high accountability institutions have rigorous change order management 
processes, which encompass emergency changes too - isn't the onus on CAs 
to explain why their business is so radically different so as not to be 
a proper fit for such widespread processes?

Regards,

Neil

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://cabforum.org/pipermail/servercert-wg/attachments/20200330/b8fe5030/attachment-0001.html>