[cabfpub] [Servercert-wg] CANCEL Notice of Review Period – Ballot SC35

Ryan Sleevi sleevi at google.com
Tue Sep 15 18:34:41 UTC 2020


On Tue, Sep 15, 2020 at 12:25 PM Dimitris Zacharopoulos (HARICA) <
dzacharo at harica.gr> wrote:

> I think you have a point about A + B, so we should have independent review
> periods for each ballot as we did with version 1.6.4 (we had independent
> ballots to be reviewed and then a combined/aggregated new version of the
> BRs).
>
> I also received some personal messages supporting the "aggregated"
> practice as it seems to be much more efficient for a CA's compliance review
> to check against one new BR rather than 2 or 3 separate ones within a very
> short timeframe.
>

As Chair, could I ask you to encourage these to be shared on the list? I
think it's troubling the increased amount of off-list messages used to
justify Forum behaviour. This feels very much like the mode which the Forum
explicitly tried to move past with our move to open lists and minuted
calls, so that we can more accurately understand perspectives and ask
questions. Certainly, I hope it should be uncontroversial that the Forum
helps us understand the many different perspectives and constraints, and
this sort of "appeal to private communication, anonymized, without data"
actively harms the ability of the Forum to function and improve. This is
part of why we have public discussion



> More answers inline.
>
> On 2020-09-15 5:34 μ.μ., Ryan Sleevi wrote:
>
>
>
> On Tue, Sep 15, 2020 at 4:33 AM Dimitris Zacharopoulos (HARICA) <
> dzacharo at harica.gr> wrote:
>
>> On 14/9/2020 8:08 μ.μ., Ryan Sleevi wrote:
>>
>> Yes, I'm aware of the "what", but it's not clear the "why".
>>
>> The act of combining ballots is relatively new, as you can see from
>> https://cabforum.org/baseline-requirements-documents/ . Producing
>> multiple versions of the Guidelines, linear based on when the Ballot
>> concluded, was something our GitHub flow intentionally was to make easy.
>> While that page stopped listing dates of adoption around Ballot 189, you
>> can see previous ballot pairs (e.g. 171+164, 125+118+134+135) that did that.
>>
>> It seems worth figuring out the challenges you're facing, since it was
>> meant to be very easy to create a new version of the document for each
>> ballot, even ballots that conclude closely, and to have IP reviews as such.
>>
>>
>> The administrative overhead of updating public web sites, sending
>> additional emails, and the fact that we would have versions of the
>> Guidelines that would be valid for a few seconds (which seems unreasonable
>> for a public standards document), are some of the reasons behind aggregated
>> final guidelines. Version 1.6.4 aggregated 3 ballots, 1.6.7 and 1.7.1. had
>> 2 and now 1.7.2.
>>
>
>> This process was discussed with Dean and Wayne back in February 2019, and
>> we all considered it compliant with our Bylaws. The results of each IPR
>> review period were sent to the public lists without receiving any
>> objections or concerns.
>>
>> Although we have documented a GitHub workflow that supports the most
>> common case (one ballot, one IPR review, one Final Maintenance Guideline),
>> it should not prohibit aggregated ballots to minimize administrative
>> overhead or the production of Guidelines that have some reasonable validity
>> time.
>>
>> If there are strong objections to this process, we can revise it going
>> forward.
>>
>
> I think we should understand the concerns, so we can make sure we have
> solutions that work.
>
> The concerns for aggregated ballots is that, from an IP perspective, it
> makes multiple ballots share the same fate and can easily delay adoption.
> For example, imagine there are Ballots A and B. A makes a change to one
> section, B makes a change to another, and only through their combined
> implementation does a Member raise an IP concern.
>
> In the serialized approach, A then B, A can be introduced to the IP review
> and not introduce any IP obligations. It can become effective. When B is
> introduced and undergoes IP review, and an exclusion is filed (on the B+A),
> then B is basically sent to the PAG to work through what to do, while A
> remains effective.
>
> In the combined approach, A and B, when A+B trigger the IP review and
> result in the IP obligation. An exclusion is filed, and now A+B are sent to
> a PAG. The act of understanding whether it is A introducing the IP
> obligation or B introducing the IP obligation is left for the PAG to sort,
> with neither A nor B being effective until the PAG reaches recommendations.
>
> This was a concern debated at length throughout our governance process,
> and was indeed one of the core motivations for our IPR policy update
> (which, you may recall, was motivated by the removal of the "any other
> method" introducing obligations on the other enumerated methods). Our IP
> policy was revised to try to make it easier to distinguish whether A or B
> was introducing the obligation, and allow modifications like A (and
> subsequent modifications) to continue, even while B was addressed within
> the PAG.
>
> Beyond that explicit reason, the distinct versioning also helps better
> review changes and tie back to ballots. It treats Ballots as they are -
> logical units of change - rather than aggregations of change. The Forum
> first started with an aggregation-of-change approach (through submitting
> amendments and errata) and quickly confounded the ability to understand the
> reconciled state. The same issue applies to unified ballots. Given that we
> have CAs asserting that it's difficult to find changes in Ballots they
> explicitly voted for, after lengthy reviews, and with policies incorporated
> in at least two root programs, I'm incredibly sensitive to wanting to make
> sure CAs are set up to succeed in following the Requirements, by making
> changes discrete and digestible.
>
> So that's the context about "why" serialized was intended. I'm also
> totally sympathetic to the difficulty involved in producing ballots,
> especially as we haven't quite got automation up to speed, and so I can
> understand why wanting to avoid that work is undesirable. What I take from
> your reply is that there are several tasks that, as a consequence of our
> current tooling, represent undesirable time commitments. It's unclear if
> that's "5 minutes more" or "5 hours more", and so your help in breaking
> down how much time it takes for the following would be greatly useful, as
> it will help prioritize what is most important to resolve. I suspect these
> can become priorities for the Infrastructure WG.
>
> 1) Merge a single Ballot into GitHub
>
>
> This greatly depends on whether the ballot proposer has used the
> documented and recommended (not yet normative/required) process, using
> GitHub to create a redline. Even with GitHub, I've seen different "flavors"
> of redlines, for example with and without pull requests, making it somewhat
> challenging to find the right steps to create a cabforum/documents branch
> that will also create the artifacts (docx document) from which I can create
> the redline.
>

> If we are talking about the Baseline Requirements. this process takes
> between 15 minutes to 1,5 hours in the more weird cases. For the other
> Guidelines, it takes between 30 minutes to 1 hour.
>

Thanks! What I take from this is that one of the steps to improve can be to
front-load some of this (e.g. the creation of PRs earlier, the creation of
GitHub branches that match balloted text). These are things that I think
any of us can do, and sounds like it would help somewhat.


> 2) Produce a Final Guideline and a Redlined Version
>
> Once I have the review period redline ready, it usually takes between 30
> minutes to 1 hour to create the final documents, upload them to the wiki
> (the word versions), produce the final PDF versions and upload them to the
> public web site.
>

Wow! I wouldn't have expected this to be more than 5 minutes, so that's a
really surprising amount of time!  Do you know where the bulk of the time
is, so we can prioritize? That said, it sounds like your response here
aggregated 2-5 - is that roughly right?


> 3) Draft and send the e-mail for the IP review period announcement
>
> This usually takes between 10 and 20 minutes because I am using templates
> on Thunderbird and just replacing the links and attachments.
>
> 4) Upload the Final Guideline and Redlined Version to the website
>
> 5) Update the Website to link to these
>
>
> 10 to 20 minutes for both 4 and 5.
>
> 6) Draft and send the e-mail concluding the IP review period
>
>
> Again 10 to 20 minutes.
>
>
> As a number of these tasks are very automatable, figuring out what we can
> do so that it represents <30 minutes of additional work seems... a
> reasonable target, right? This only comes up when we have multiple ballots
> concluding within a short period of eachother, which although historically
> rare, certainly was more common over the summer as stuff got batched to
> accommodate Covid disruptions.
>
>
> The most challenging and time consuming tasks come with ballots that
> update more than one Guideline, for which we don't have the same tools
> ready to produce docx versions. This requires me to do the GitHub redlines
> for the BRs and docx versions (manually) for EV Guidelines or NSRs.
>
> Even for the BRs, I usually have to remove text after the ToC because it
> repeats information that is supposed to be in the cover page. Some of these
> minor issues need to be fixed, and I would really appreciate the help of
> the Infrastructure Subcommittee.
>

So what I'm taking from this is that, probably in order of priorities for
optimizing:
- Get the EVGs and NCSSRs fully converted and automated (saves 30 - 60
minutes per ballot)
- Create PRs as Ballots are proposed (saves 30 - 60 minutes per ballot)

For the SCWG, we see:
- 20 ballots to the BRs over the past 2 years
- 7 ballots to the EVGs over the past 2 years
- 3 ballots to the NCSSRs over the past 2 years

Meaning that, in the worst case, at 1.5 hours per ballot/doc, that's 15
ballots a year, or 22.5 hours spent working just on ballots and IP reviews
shepherding as individual docs. By aggregating, you've been able to shave
7.5 hours off that time. If we automated, and got it down to 30 minutes per
ballot, that could save another 7.5 hours. So a week or two of work
automating would pay for itself within 5 years (or a day or two of
automating within 1 year). More time than that, however, would be more time
spent vs having the chair do it manually, and might not be worth the time
<https://xkcd.com/1205/> to automate.

Certainly, this breakdown has helped highlight where the most impactful
time savings can be for you, or more generally, for the chair. This is one
of those areas where the added transparency, given all the duties asked of
the Chair, helps us improve the tools and workmode, so that chairing
becomes less of a time commitment and thus something more members can
consider in the future.


> Would your concerns be addressed if we had a separate/distinct IPR review
> for each ballot and once each IPR review is cleared with no essential
> claims filed, produce an aggregate document, or you still prefer separate
> Final documents with distinct versioning to be created? What do others
> think?
>

The IP review necessitates the production of the Final document, so that
sounds like more work, rather than less, and it seems to run in the same
problem of commingling outputs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cabforum.org/pipermail/public/attachments/20200915/a0fabe40/attachment-0003.html>


More information about the Public mailing list