[cabfpub] [Servercert-wg] CANCEL Notice of Review Period – Ballot SC35

Dimitris Zacharopoulos (HARICA) dzacharo at harica.gr
Tue Sep 15 20:18:38 UTC 2020

On 2020-09-15 9:34 μ.μ., Ryan Sleevi wrote:
> On Tue, Sep 15, 2020 at 12:25 PM Dimitris Zacharopoulos (HARICA) 
> <dzacharo at harica.gr <mailto:dzacharo at harica.gr>> wrote:
>     I think you have a point about A + B, so we should have
>     independent review periods for each ballot as we did with version
>     1.6.4 (we had independent ballots to be reviewed and then a
>     combined/aggregated new version of the BRs).
>     I also received some personal messages supporting the "aggregated"
>     practice as it seems to be much more efficient for a CA's
>     compliance review to check against one new BR rather than 2 or 3
>     separate ones within a very short timeframe.
> As Chair, could I ask you to encourage these to be shared on the list? 
> I think it's troubling the increased amount of off-list messages used 
> to justify Forum behaviour. This feels very much like the mode which 
> the Forum explicitly tried to move past with our move to open lists 
> and minuted calls, so that we can more accurately understand 
> perspectives and ask questions. Certainly, I hope it should be 
> uncontroversial that the Forum helps us understand the many different 
> perspectives and constraints, and this sort of "appeal to private 
> communication, anonymized, without data" actively harms the ability of 
> the Forum to function and improve. This is part of why we have public 
> discussion

Sure, I can do that but in any case I forwarded it the argument on the 
list. I also support this argument that aggregating new versions of the 
documents saves time.

If someone wants to send me something in private to check if I agree or 
not, I don't see any harm in doing that. I am sure that direct 
communication among Members, although discouraged, is not prohibited.

>     More answers inline.
>     On 2020-09-15 5:34 μ.μ., Ryan Sleevi wrote:
>>     On Tue, Sep 15, 2020 at 4:33 AM Dimitris Zacharopoulos (HARICA)
>>     <dzacharo at harica.gr <mailto:dzacharo at harica.gr>> wrote:
>>         On 14/9/2020 8:08 μ.μ., Ryan Sleevi wrote:
>>>         Yes, I'm aware of the "what", but it's not clear the "why".
>>>         The act of combining ballots is relatively new, as you can
>>>         see from
>>>         https://cabforum.org/baseline-requirements-documents/ .
>>>         Producing multiple versions of the Guidelines, linear based
>>>         on when the Ballot concluded, was something our GitHub flow
>>>         intentionally was to make easy. While that page stopped
>>>         listing dates of adoption around Ballot 189, you can see
>>>         previous ballot pairs (e.g. 171+164, 125+118+134+135) that
>>>         did that.
>>>         It seems worth figuring out the challenges you're facing,
>>>         since it was meant to be very easy to create a new version
>>>         of the document for each ballot, even ballots that conclude
>>>         closely, and to have IP reviews as such.
>>         The administrative overhead of updating public web sites,
>>         sending additional emails, and the fact that we would have
>>         versions of the Guidelines that would be valid for a few
>>         seconds (which seems unreasonable for a public standards
>>         document), are some of the reasons behind aggregated final
>>         guidelines. Version 1.6.4 aggregated 3 ballots, 1.6.7 and
>>         1.7.1. had 2 and now 1.7.2.
>>         This process was discussed with Dean and Wayne back in
>>         February 2019, and we all considered it compliant with our
>>         Bylaws. The results of each IPR review period were sent to
>>         the public lists without receiving any objections or concerns.
>>         Although we have documented a GitHub workflow that supports
>>         the most common case (one ballot, one IPR review, one Final
>>         Maintenance Guideline), it should not prohibit aggregated
>>         ballots to minimize administrative overhead or the production
>>         of Guidelines that have some reasonable validity time.
>>         If there are strong objections to this process, we can revise
>>         it going forward.
>>     I think we should understand the concerns, so we can make sure we
>>     have solutions that work.
>>     The concerns for aggregated ballots is that, from an IP
>>     perspective, it makes multiple ballots share the same fate and
>>     can easily delay adoption. For example, imagine there are Ballots
>>     A and B. A makes a change to one section, B makes a change to
>>     another, and only through their combined implementation does a
>>     Member raise an IP concern.
>>     In the serialized approach, A then B, A can be introduced to the
>>     IP review and not introduce any IP obligations. It can become
>>     effective. When B is introduced and undergoes IP review, and an
>>     exclusion is filed (on the B+A), then B is basically sent to the
>>     PAG to work through what to do, while A remains effective.
>>     In the combined approach, A and B, when A+B trigger the IP review
>>     and result in the IP obligation. An exclusion is filed, and now
>>     A+B are sent to a PAG. The act of understanding whether it is A
>>     introducing the IP obligation or B introducing the IP obligation
>>     is left for the PAG to sort, with neither A nor B being effective
>>     until the PAG reaches recommendations.
>>     This was a concern debated at length throughout our governance
>>     process, and was indeed one of the core motivations for our IPR
>>     policy update (which, you may recall, was motivated by the
>>     removal of the "any other method" introducing obligations on the
>>     other enumerated methods). Our IP policy was revised to try to
>>     make it easier to distinguish whether A or B was introducing the
>>     obligation, and allow modifications like A (and subsequent
>>     modifications) to continue, even while B was addressed within the
>>     PAG.
>>     Beyond that explicit reason, the distinct versioning also helps
>>     better review changes and tie back to ballots. It treats Ballots
>>     as they are - logical units of change - rather than aggregations
>>     of change. The Forum first started with an aggregation-of-change
>>     approach (through submitting amendments and errata) and quickly
>>     confounded the ability to understand the reconciled state. The
>>     same issue applies to unified ballots. Given that we have CAs
>>     asserting that it's difficult to find changes in Ballots they
>>     explicitly voted for, after lengthy reviews, and with policies
>>     incorporated in at least two root programs, I'm incredibly
>>     sensitive to wanting to make sure CAs are set up to succeed in
>>     following the Requirements, by making changes discrete and
>>     digestible.
>>     So that's the context about "why" serialized was intended. I'm
>>     also totally sympathetic to the difficulty involved in producing
>>     ballots, especially as we haven't quite got automation up to
>>     speed, and so I can understand why wanting to avoid that work is
>>     undesirable. What I take from your reply is that there are
>>     several tasks that, as a consequence of our current tooling,
>>     represent undesirable time commitments. It's unclear if that's "5
>>     minutes more" or "5 hours more", and so your help in breaking
>>     down how much time it takes for the following would be greatly
>>     useful, as it will help prioritize what is most important to
>>     resolve. I suspect these can become priorities for the
>>     Infrastructure WG.
>>     1) Merge a single Ballot into GitHub
>     This greatly depends on whether the ballot proposer has used the
>     documented and recommended (not yet normative/required) process,
>     using GitHub to create a redline. Even with GitHub, I've seen
>     different "flavors" of redlines, for example with and without pull
>     requests, making it somewhat challenging to find the right steps
>     to create a cabforum/documents branch that will also create the
>     artifacts (docx document) from which I can create the redline.
>     If we are talking about the Baseline Requirements. this process
>     takes between 15 minutes to 1,5 hours in the more weird cases. For
>     the other Guidelines, it takes between 30 minutes to 1 hour.
> Thanks! What I take from this is that one of the steps to improve can 
> be to front-load some of this (e.g. the creation of PRs earlier, the 
> creation of GitHub branches that match balloted text). These are 
> things that I think any of us can do, and sounds like it would help 
> somewhat.
>>     2) Produce a Final Guideline and a Redlined Version
>     Once I have the review period redline ready, it usually takes
>     between 30 minutes to 1 hour to create the final documents, upload
>     them to the wiki (the word versions), produce the final PDF
>     versions and upload them to the public web site.
> Wow! I wouldn't have expected this to be more than 5 minutes, so 
> that's a really surprising amount of time!  Do you know where the bulk 
> of the time is, so we can prioritize? That said, it sounds like your 
> response here aggregated 2-5 - is that roughly right?

Yes, because to produce a Final Guideline and a redlined version in a 
non-GitHub version is quite painful. In the GitHub version, things are 
faster but again I need to create the redline manually, compare the two 
.docx versions produced by GitHub (before and after the merge to Main), 
check everything like track changes mode, make sure the ToC is not 
tracked and other minor details. Then, create a PDF without the 
formatting comments (otherwise the PDF contains formatting changes in 
the redline which doesn't look good). Then make the changes to the web 
sites, wiki, etc. I wish I could make that in 5 minutes :-)

This would probably require having our entire public website on GitHub 
and automate the process. That would be really something!

>>     3) Draft and send the e-mail for the IP review period announcement
>     This usually takes between 10 and 20 minutes because I am using
>     templates on Thunderbird and just replacing the links and attachments.
>>     4) Upload the Final Guideline and Redlined Version to the website
>>     5) Update the Website to link to these
>     10 to 20 minutes for both 4 and 5.
>>     6) Draft and send the e-mail concluding the IP review period
>     Again 10 to 20 minutes.
>>     As a number of these tasks are very automatable, figuring out
>>     what we can do so that it represents <30 minutes of additional
>>     work seems... a reasonable target, right? This only comes up when
>>     we have multiple ballots concluding within a short period of
>>     eachother, which although historically rare, certainly was more
>>     common over the summer as stuff got batched to accommodate Covid
>>     disruptions.
>     The most challenging and time consuming tasks come with ballots
>     that update more than one Guideline, for which we don't have the
>     same tools ready to produce docx versions. This requires me to do
>     the GitHub redlines for the BRs and docx versions (manually) for
>     EV Guidelines or NSRs.
>     Even for the BRs, I usually have to remove text after the ToC
>     because it repeats information that is supposed to be in the cover
>     page. Some of these minor issues need to be fixed, and I would
>     really appreciate the help of the Infrastructure Subcommittee.
> So what I'm taking from this is that, probably in order of priorities 
> for optimizing:
> - Get the EVGs and NCSSRs fully converted and automated (saves 30 - 60 
> minutes per ballot)
> - Create PRs as Ballots are proposed (saves 30 - 60 minutes per ballot)
> For the SCWG, we see:
> - 20 ballots to the BRs over the past 2 years
> - 7 ballots to the EVGs over the past 2 years
> - 3 ballots to the NCSSRs over the past 2 years
> Meaning that, in the worst case, at 1.5 hours per ballot/doc, that's 
> 15 ballots a year, or 22.5 hours spent working just on ballots and IP 
> reviews shepherding as individual docs. By aggregating, you've been 
> able to shave 7.5 hours off that time. If we automated, and got it 
> down to 30 minutes per ballot, that could save another 7.5 hours. So a 
> week or two of work automating would pay for itself within 5 years (or 
> a day or two of automating within 1 year). More time than that, 
> however, would be more time spent vs having the chair do it manually, 
> and might not be worth the time <https://xkcd.com/1205/> to automate.
> Certainly, this breakdown has helped highlight where the most 
> impactful time savings can be for you, or more generally, for the 
> chair. This is one of those areas where the added transparency, given 
> all the duties asked of the Chair, helps us improve the tools and 
> workmode, so that chairing becomes less of a time commitment and thus 
> something more members can consider in the future.
>     Would your concerns be addressed if we had a separate/distinct IPR
>     review for each ballot and once each IPR review is cleared with no
>     essential claims filed, produce an aggregate document, or you
>     still prefer separate Final documents with distinct versioning to
>     be created? What do others think?
> The IP review necessitates the production of the Final document, so 
> that sounds like more work, rather than less, and it seems to run in 
> the same problem of commingling outputs.

During my entire time as Chair of the Forum and the SCWG I have tried to 
perform my duties following the Bylaws to the best of my knowledge and 
understanding. Just in case I had misunderstood these requirements, I 
shared this process with the Vice Chairs who are native English 
speakers. We all agreed that aggregating the final Guidelines was 
allowed and not in conflict with our Bylaws.

Can you point me to where you believe this practice is forbidden? I 
honestly don't mind the extra work of creating more Guidelines but it 
seems pointless to create a document that nobody will ever read because 
it will be valid for just a few seconds. Documents with the *same 
effective date* would create a lot of confusion to CAs and Relying Parties.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cabforum.org/pipermail/public/attachments/20200915/1c340f3c/attachment-0003.html>

More information about the Public mailing list