<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<br>
<br>
<div class="moz-cite-prefix">On 2020-09-15 9:34 μ.μ., Ryan Sleevi
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CACvaWvb3mnEALdKMk8-H5K-UzjAykUWJcOBK-+X0-etgL_OVAg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 15, 2020 at
12:25 PM Dimitris Zacharopoulos (HARICA) <<a
href="mailto:dzacharo@harica.gr" moz-do-not-send="true">dzacharo@harica.gr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> I think you have a point about A + B, so we should
have independent review periods for each ballot as we did
with version 1.6.4 (we had independent ballots to be
reviewed and then a combined/aggregated new version of the
BRs). <br>
<br>
I also received some personal messages supporting the
"aggregated" practice as it seems to be much more
efficient for a CA's compliance review to check against
one new BR rather than 2 or 3 separate ones within a very
short timeframe.<br>
</div>
</blockquote>
<div><br>
</div>
<div>As Chair, could I ask you to encourage these to be shared
on the list? I think it's troubling the increased amount of
off-list messages used to justify Forum behaviour. This
feels very much like the mode which the Forum explicitly
tried to move past with our move to open lists and minuted
calls, so that we can more accurately understand
perspectives and ask questions. Certainly, I hope it should
be uncontroversial that the Forum helps us understand the
many different perspectives and constraints, and this sort
of "appeal to private communication, anonymized, without
data" actively harms the ability of the Forum to function
and improve. This is part of why we have public discussion</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<br>
Sure, I can do that but in any case I forwarded it the argument on
the list. I also support this argument that aggregating new versions
of the documents saves time.<br>
<br>
If someone wants to send me something in private to check if I agree
or not, I don't see any harm in doing that. I am sure that direct
communication among Members, although discouraged, is not
prohibited.<br>
<br>
<blockquote type="cite"
cite="mid:CACvaWvb3mnEALdKMk8-H5K-UzjAykUWJcOBK-+X0-etgL_OVAg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> <br>
More answers inline.<br>
<br>
<div>On 2020-09-15 5:34 μ.μ., Ryan Sleevi wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, Sep 15,
2020 at 4:33 AM Dimitris Zacharopoulos (HARICA)
<<a href="mailto:dzacharo@harica.gr"
target="_blank" moz-do-not-send="true">dzacharo@harica.gr</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<div>On 14/9/2020 8:08 μ.μ., Ryan Sleevi wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Yes, I'm aware of the "what",
but it's not clear the "why".
<div><br>
</div>
<div>The act of combining ballots is
relatively new, as you can see from <a
href="https://cabforum.org/baseline-requirements-documents/"
target="_blank" moz-do-not-send="true">https://cabforum.org/baseline-requirements-documents/</a>
. Producing multiple versions of the
Guidelines, linear based on when the
Ballot concluded, was something our GitHub
flow intentionally was to make easy. While
that page stopped listing dates of
adoption around Ballot 189, you can see
previous ballot pairs (e.g. 171+164,
125+118+134+135) that did that.</div>
<div><br>
</div>
<div>It seems worth figuring out the
challenges you're facing, since it was
meant to be very easy to create a new
version of the document for each ballot,
even ballots that conclude closely, and to
have IP reviews as such.</div>
</div>
</blockquote>
<br>
The administrative overhead of updating public
web sites, sending additional emails, and the
fact that we would have versions of the
Guidelines that would be valid for a few seconds
(which seems unreasonable for a public standards
document), are some of the reasons behind
aggregated final guidelines. Version 1.6.4
aggregated 3 ballots, 1.6.7 and 1.7.1. had 2 and
now 1.7.2. </div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> <br>
This process was discussed with Dean and Wayne
back in February 2019, and we all considered it
compliant with our Bylaws. The results of each
IPR review period were sent to the public lists
without receiving any objections or concerns.<br>
<br>
Although we have documented a GitHub workflow
that supports the most common case (one ballot,
one IPR review, one Final Maintenance
Guideline), it should not prohibit aggregated
ballots to minimize administrative overhead or
the production of Guidelines that have some
reasonable validity time.<br>
<br>
If there are strong objections to this process,
we can revise it going forward.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I think we should understand the concerns, so
we can make sure we have solutions that work.</div>
<div><br>
</div>
<div>The concerns for aggregated ballots is that,
from an IP perspective, it makes multiple ballots
share the same fate and can easily delay adoption.
For example, imagine there are Ballots A and B. A
makes a change to one section, B makes a change to
another, and only through their combined
implementation does a Member raise an IP concern.</div>
<div><br>
</div>
<div>In the serialized approach, A then B, A can be
introduced to the IP review and not introduce any
IP obligations. It can become effective. When B is
introduced and undergoes IP review, and an
exclusion is filed (on the B+A), then B is
basically sent to the PAG to work through what to
do, while A remains effective.</div>
<div><br>
</div>
<div>In the combined approach, A and B, when A+B
trigger the IP review and result in the IP
obligation. An exclusion is filed, and now A+B are
sent to a PAG. The act of understanding whether it
is A introducing the IP obligation or B
introducing the IP obligation is left for the PAG
to sort, with neither A nor B being effective
until the PAG reaches recommendations.</div>
<div><br>
</div>
<div>This was a concern debated at length throughout
our governance process, and was indeed one of the
core motivations for our IPR policy update (which,
you may recall, was motivated by the removal of
the "any other method" introducing obligations on
the other enumerated methods). Our IP policy was
revised to try to make it easier to distinguish
whether A or B was introducing the obligation, and
allow modifications like A (and subsequent
modifications) to continue, even while B was
addressed within the PAG.</div>
<div><br>
</div>
<div>Beyond that explicit reason, the distinct
versioning also helps better review changes and
tie back to ballots. It treats Ballots as they are
- logical units of change - rather than
aggregations of change. The Forum first started
with an aggregation-of-change approach (through
submitting amendments and errata) and quickly
confounded the ability to understand the
reconciled state. The same issue applies to
unified ballots. Given that we have CAs asserting
that it's difficult to find changes in Ballots
they explicitly voted for, after lengthy reviews,
and with policies incorporated in at least two
root programs, I'm incredibly sensitive to wanting
to make sure CAs are set up to succeed in
following the Requirements, by making changes
discrete and digestible. </div>
<div><br>
</div>
<div>So that's the context about "why" serialized
was intended. I'm also totally sympathetic to the
difficulty involved in producing ballots,
especially as we haven't quite got automation up
to speed, and so I can understand why wanting to
avoid that work is undesirable. What I take from
your reply is that there are several tasks that,
as a consequence of our current tooling, represent
undesirable time commitments. It's unclear if
that's "5 minutes more" or "5 hours more", and so
your help in breaking down how much time it takes
for the following would be greatly useful, as it
will help prioritize what is most important to
resolve. I suspect these can become priorities for
the Infrastructure WG.</div>
<div><br>
</div>
<div>1) Merge a single Ballot into GitHub</div>
</div>
</div>
</blockquote>
<br>
This greatly depends on whether the ballot proposer has
used the documented and recommended (not yet
normative/required) process, using GitHub to create a
redline. Even with GitHub, I've seen different "flavors"
of redlines, for example with and without pull requests,
making it somewhat challenging to find the right steps to
create a cabforum/documents branch that will also create
the artifacts (docx document) from which I can create the
redline.</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> <br>
If we are talking about the Baseline Requirements. this
process takes between 15 minutes to 1,5 hours in the more
weird cases. For the other Guidelines, it takes between 30
minutes to 1 hour.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Thanks! What I take from this is that one of the steps to
improve can be to front-load some of this (e.g. the creation
of PRs earlier, the creation of GitHub branches that match
balloted text). These are things that I think any of us can
do, and sounds like it would help somewhat.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>2) Produce a Final Guideline and a Redlined
Version</div>
</div>
</div>
</blockquote>
Once I have the review period redline ready, it usually
takes between 30 minutes to 1 hour to create the final
documents, upload them to the wiki (the word versions),
produce the final PDF versions and upload them to the
public web site.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Wow! I wouldn't have expected this to be more than 5
minutes, so that's a really surprising amount of time! Do
you know where the bulk of the time is, so we can
prioritize? That said, it sounds like your response here
aggregated 2-5 - is that roughly right?<br>
</div>
</div>
</div>
</blockquote>
<br>
Yes, because to produce a Final Guideline and a redlined version in
a non-GitHub version is quite painful. In the GitHub version, things
are faster but again I need to create the redline manually, compare
the two .docx versions produced by GitHub (before and after the
merge to Main), check everything like track changes mode, make sure
the ToC is not tracked and other minor details. Then, create a PDF
without the formatting comments (otherwise the PDF contains
formatting changes in the redline which doesn't look good). Then
make the changes to the web sites, wiki, etc. I wish I could make
that in 5 minutes :-)<br>
<br>
This would probably require having our entire public website on
GitHub and automate the process. That would be really something!<br>
<br>
<blockquote type="cite"
cite="mid:CACvaWvb3mnEALdKMk8-H5K-UzjAykUWJcOBK-+X0-etgL_OVAg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>3) Draft and send the e-mail for the IP review
period announcement</div>
</div>
</div>
</blockquote>
This usually takes between 10 and 20 minutes because I am
using templates on Thunderbird and just replacing the
links and attachments.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>4) Upload the Final Guideline and Redlined
Version to the website</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>5) Update the Website to link to these</div>
</div>
</div>
</blockquote>
<br>
10 to 20 minutes for both 4 and 5.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div>6) Draft and send the e-mail concluding the IP
review period</div>
</div>
</div>
</blockquote>
<br>
Again 10 to 20 minutes.<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>As a number of these tasks are very
automatable, figuring out what we can do so that
it represents <30 minutes of additional work
seems... a reasonable target, right? This only
comes up when we have multiple ballots concluding
within a short period of eachother, which although
historically rare, certainly was more common over
the summer as stuff got batched to
accommodate Covid disruptions.</div>
</div>
</div>
</blockquote>
<br>
The most challenging and time consuming tasks come with
ballots that update more than one Guideline, for which we
don't have the same tools ready to produce docx versions.
This requires me to do the GitHub redlines for the BRs and
docx versions (manually) for EV Guidelines or NSRs.<br>
<br>
Even for the BRs, I usually have to remove text after the
ToC because it repeats information that is supposed to be
in the cover page. Some of these minor issues need to be
fixed, and I would really appreciate the help of the
Infrastructure Subcommittee.<br>
</div>
</blockquote>
<div><br>
</div>
<div>So what I'm taking from this is that, probably in order
of priorities for optimizing:</div>
<div>- Get the EVGs and NCSSRs fully converted and automated
(saves 30 - 60 minutes per ballot)</div>
<div>- Create PRs as Ballots are proposed (saves 30 - 60
minutes per ballot)</div>
<div><br>
</div>
<div>For the SCWG, we see:</div>
<div>- 20 ballots to the BRs over the past 2 years</div>
<div>- 7 ballots to the EVGs over the past 2 years</div>
<div>- 3 ballots to the NCSSRs over the past 2 years</div>
<div><br>
</div>
<div>Meaning that, in the worst case, at 1.5 hours per
ballot/doc, that's 15 ballots a year, or 22.5 hours spent
working just on ballots and IP reviews shepherding as
individual docs. By aggregating, you've been able to shave
7.5 hours off that time. If we automated, and got it down to
30 minutes per ballot, that could save another 7.5 hours. So
a week or two of work automating would pay for itself within
5 years (or a day or two of automating within 1 year). More
time than that, however, would be more time spent vs having
the chair do it manually, and might not be <a
href="https://xkcd.com/1205/" moz-do-not-send="true">worth
the time</a> to automate.</div>
<div><br>
</div>
<div>Certainly, this breakdown has helped highlight where the
most impactful time savings can be for you, or more
generally, for the chair. This is one of those areas where
the added transparency, given all the duties asked of the
Chair, helps us improve the tools and workmode, so that
chairing becomes less of a time commitment and thus
something more members can consider in the future.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div> Would your concerns be addressed if we had a
separate/distinct IPR review for each ballot and once each
IPR review is cleared with no essential claims filed,
produce an aggregate document, or you still prefer
separate Final documents with distinct versioning to be
created? What do others think?<br>
</div>
</blockquote>
<div><br>
</div>
<div>The IP review necessitates the production of the Final
document, so that sounds like more work, rather than less,
and it seems to run in the same problem of
commingling outputs.</div>
</div>
</div>
</blockquote>
<br>
During my entire time as Chair of the Forum and the SCWG I have
tried to perform my duties following the Bylaws to the best of my
knowledge and understanding. Just in case I had misunderstood these
requirements, I shared this process with the Vice Chairs who are
native English speakers. We all agreed that aggregating the final
Guidelines was allowed and not in conflict with our Bylaws.<br>
<br>
Can you point me to where you believe this practice is forbidden? I
honestly don't mind the extra work of creating more Guidelines but
it seems pointless to create a document that nobody will ever read
because it will be valid for just a few seconds. Documents with the
<b>same effective date</b> would create a lot of confusion to CAs
and Relying Parties.<br>
<br>
<br>
<br>
</body>
</html>