<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Sep 15, 2020 at 12:25 PM Dimitris Zacharopoulos (HARICA) <<a href="mailto:dzacharo@harica.gr">dzacharo@harica.gr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
I think you have a point about A + B, so we should have independent
review periods for each ballot as we did with version 1.6.4 (we had
independent ballots to be reviewed and then a combined/aggregated
new version of the BRs). <br>
<br>
I also received some personal messages supporting the "aggregated"
practice as it seems to be much more efficient for a CA's compliance
review to check against one new BR rather than 2 or 3 separate ones
within a very short timeframe.<br></div></blockquote><div><br></div><div>As Chair, could I ask you to encourage these to be shared on the list? I think it's troubling the increased amount of off-list messages used to justify Forum behaviour. This feels very much like the mode which the Forum explicitly tried to move past with our move to open lists and minuted calls, so that we can more accurately understand perspectives and ask questions. Certainly, I hope it should be uncontroversial that the Forum helps us understand the many different perspectives and constraints, and this sort of "appeal to private communication, anonymized, without data" actively harms the ability of the Forum to function and improve. This is part of why we have public discussion</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<br>
More answers inline.<br>
<br>
<div>On 2020-09-15 5:34 μ.μ., Ryan Sleevi
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 15, 2020 at 4:33
AM Dimitris Zacharopoulos (HARICA) <<a href="mailto:dzacharo@harica.gr" target="_blank">dzacharo@harica.gr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>On 14/9/2020 8:08 μ.μ., Ryan Sleevi wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Yes, I'm aware of the "what", but it's
not clear the "why".
<div><br>
</div>
<div>The act of combining ballots is relatively new,
as you can see from <a href="https://cabforum.org/baseline-requirements-documents/" target="_blank">https://cabforum.org/baseline-requirements-documents/</a>
. Producing multiple versions of the Guidelines,
linear based on when the Ballot concluded, was
something our GitHub flow intentionally was to make
easy. While that page stopped listing dates of
adoption around Ballot 189, you can see previous
ballot pairs (e.g. 171+164, 125+118+134+135) that
did that.</div>
<div><br>
</div>
<div>It seems worth figuring out the challenges you're
facing, since it was meant to be very easy to create
a new version of the document for each ballot, even
ballots that conclude closely, and to have IP
reviews as such.</div>
</div>
</blockquote>
<br>
The administrative overhead of updating public web sites,
sending additional emails, and the fact that we would have
versions of the Guidelines that would be valid for a few
seconds (which seems unreasonable for a public standards
document), are some of the reasons behind aggregated final
guidelines. Version 1.6.4 aggregated 3 ballots, 1.6.7 and
1.7.1. had 2 and now 1.7.2. </div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div> <br>
This process was discussed with Dean and Wayne back in
February 2019, and we all considered it compliant with our
Bylaws. The results of each IPR review period were sent to
the public lists without receiving any objections or
concerns.<br>
<br>
Although we have documented a GitHub workflow that
supports the most common case (one ballot, one IPR review,
one Final Maintenance Guideline), it should not prohibit
aggregated ballots to minimize administrative overhead or
the production of Guidelines that have some reasonable
validity time.<br>
<br>
If there are strong objections to this process, we can
revise it going forward.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I think we should understand the concerns, so we can make
sure we have solutions that work.</div>
<div><br>
</div>
<div>The concerns for aggregated ballots is that, from an IP
perspective, it makes multiple ballots share the same fate
and can easily delay adoption. For example, imagine there
are Ballots A and B. A makes a change to one section, B
makes a change to another, and only through their combined
implementation does a Member raise an IP concern.</div>
<div><br>
</div>
<div>In the serialized approach, A then B, A can be introduced
to the IP review and not introduce any IP obligations. It
can become effective. When B is introduced and undergoes IP
review, and an exclusion is filed (on the B+A), then B is
basically sent to the PAG to work through what to do, while
A remains effective.</div>
<div><br>
</div>
<div>In the combined approach, A and B, when A+B trigger the
IP review and result in the IP obligation. An exclusion is
filed, and now A+B are sent to a PAG. The act of
understanding whether it is A introducing the IP obligation
or B introducing the IP obligation is left for the PAG to
sort, with neither A nor B being effective until the PAG
reaches recommendations.</div>
<div><br>
</div>
<div>This was a concern debated at length throughout our
governance process, and was indeed one of the core
motivations for our IPR policy update (which, you may
recall, was motivated by the removal of the "any other
method" introducing obligations on the other enumerated
methods). Our IP policy was revised to try to make it easier
to distinguish whether A or B was introducing the
obligation, and allow modifications like A (and subsequent
modifications) to continue, even while B was addressed
within the PAG.</div>
<div><br>
</div>
<div>Beyond that explicit reason, the distinct versioning also
helps better review changes and tie back to ballots. It
treats Ballots as they are - logical units of change -
rather than aggregations of change. The Forum first started
with an aggregation-of-change approach (through submitting
amendments and errata) and quickly confounded the ability to
understand the reconciled state. The same issue applies to
unified ballots. Given that we have CAs asserting that it's
difficult to find changes in Ballots they explicitly voted
for, after lengthy reviews, and with policies incorporated
in at least two root programs, I'm incredibly sensitive to
wanting to make sure CAs are set up to succeed in following
the Requirements, by making changes discrete and
digestible. </div>
<div><br>
</div>
<div>So that's the context about "why" serialized was
intended. I'm also totally sympathetic to the difficulty
involved in producing ballots, especially as we haven't
quite got automation up to speed, and so I can understand
why wanting to avoid that work is undesirable. What I take
from your reply is that there are several tasks that, as a
consequence of our current tooling, represent undesirable
time commitments. It's unclear if that's "5 minutes more" or
"5 hours more", and so your help in breaking down how much
time it takes for the following would be greatly useful, as
it will help prioritize what is most important to resolve. I
suspect these can become priorities for the Infrastructure
WG.</div>
<div><br>
</div>
<div>1) Merge a single Ballot into GitHub</div>
</div>
</div>
</blockquote>
<br>
This greatly depends on whether the ballot proposer has used the
documented and recommended (not yet normative/required) process,
using GitHub to create a redline. Even with GitHub, I've seen
different "flavors" of redlines, for example with and without pull
requests, making it somewhat challenging to find the right steps to
create a cabforum/documents branch that will also create the
artifacts (docx document) from which I can create the redline.</div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<br>
If we are talking about the Baseline Requirements. this process
takes between 15 minutes to 1,5 hours in the more weird cases. For
the other Guidelines, it takes between 30 minutes to 1 hour.<br></div></blockquote><div><br></div><div>Thanks! What I take from this is that one of the steps to improve can be to front-load some of this (e.g. the creation of PRs earlier, the creation of GitHub branches that match balloted text). These are things that I think any of us can do, and sounds like it would help somewhat.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>2) Produce a Final Guideline and a Redlined Version</div>
</div>
</div>
</blockquote>
Once I have the review period redline ready, it usually takes
between 30 minutes to 1 hour to create the final documents, upload
them to the wiki (the word versions), produce the final PDF versions
and upload them to the public web site.<br></div></blockquote><div><br></div><div>Wow! I wouldn't have expected this to be more than 5 minutes, so that's a really surprising amount of time! Do you know where the bulk of the time is, so we can prioritize? That said, it sounds like your response here aggregated 2-5 - is that roughly right?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>3) Draft and send the e-mail for the IP review period
announcement</div>
</div>
</div>
</blockquote>
This usually takes between 10 and 20 minutes because I am using
templates on Thunderbird and just replacing the links and
attachments.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>4) Upload the Final Guideline and Redlined Version to the
website</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>5) Update the Website to link to these</div>
</div>
</div>
</blockquote>
<br>
10 to 20 minutes for both 4 and 5.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>6) Draft and send the e-mail concluding the IP review
period</div>
</div>
</div>
</blockquote>
<br>
Again 10 to 20 minutes.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>As a number of these tasks are very automatable, figuring
out what we can do so that it represents <30 minutes of
additional work seems... a reasonable target, right? This
only comes up when we have multiple ballots concluding
within a short period of eachother, which although
historically rare, certainly was more common over the summer
as stuff got batched to accommodate Covid disruptions.</div>
</div>
</div>
</blockquote>
<br>
The most challenging and time consuming tasks come with ballots that
update more than one Guideline, for which we don't have the same
tools ready to produce docx versions. This requires me to do the
GitHub redlines for the BRs and docx versions (manually) for EV
Guidelines or NSRs.<br>
<br>
Even for the BRs, I usually have to remove text after the ToC
because it repeats information that is supposed to be in the cover
page. Some of these minor issues need to be fixed, and I would
really appreciate the help of the Infrastructure Subcommittee.<br></div></blockquote><div><br></div><div>So what I'm taking from this is that, probably in order of priorities for optimizing:</div><div>- Get the EVGs and NCSSRs fully converted and automated (saves 30 - 60 minutes per ballot)</div><div>- Create PRs as Ballots are proposed (saves 30 - 60 minutes per ballot)</div><div><br></div><div>For the SCWG, we see:</div><div>- 20 ballots to the BRs over the past 2 years</div><div>- 7 ballots to the EVGs over the past 2 years</div><div>- 3 ballots to the NCSSRs over the past 2 years</div><div><br></div><div>Meaning that, in the worst case, at 1.5 hours per ballot/doc, that's 15 ballots a year, or 22.5 hours spent working just on ballots and IP reviews shepherding as individual docs. By aggregating, you've been able to shave 7.5 hours off that time. If we automated, and got it down to 30 minutes per ballot, that could save another 7.5 hours. So a week or two of work automating would pay for itself within 5 years (or a day or two of automating within 1 year). More time than that, however, would be more time spent vs having the chair do it manually, and might not be <a href="https://xkcd.com/1205/">worth the time</a> to automate.</div><div><br></div><div>Certainly, this breakdown has helped highlight where the most impactful time savings can be for you, or more generally, for the chair. This is one of those areas where the added transparency, given all the duties asked of the Chair, helps us improve the tools and workmode, so that chairing becomes less of a time commitment and thus something more members can consider in the future.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
Would your concerns be addressed if we had a separate/distinct IPR
review for each ballot and once each IPR review is cleared with no
essential claims filed, produce an aggregate document, or you still
prefer separate Final documents with distinct versioning to be
created? What do others think?<br></div></blockquote><div><br></div><div>The IP review necessitates the production of the Final document, so that sounds like more work, rather than less, and it seems to run in the same problem of commingling outputs.</div></div></div>