[cabfpub] CAA concerns (and potential solutions)

[Peter Bowen]

> On Oct 28, 2016, at 1:19 PM, Jacob Hoffman-Andrews <jsha at letsencrypt.org> wrote:
> 
> On Fri, Oct 28, 2016 at 9:52 AM, Peter Bowen via Public <public at cabforum.org <mailto:public at cabforum.org>> wrote:
> OK.  I used a machine with a locally running caching resolver.  I then did the following as a test:
> 
> Fetch top-1m.csv.zip
> Unzip it
> head -n 200 top-1m.csv | cut -d, -f2 | sed -r -e ’s/^/www./‘ > top200.txt
> echo 'nameserver 127.0.0.1’ > resolv.conf
> time (ruby -e 'IO.foreach("top200.txt"){|l| l.strip!; loop {|x|puts l + "."; l=l.split(".")[1..-1].join("."); break if l.empty?}}' | while read N; do drill -c resolv.conf "$N" IN TXT; done)
> 
> real	1m5.424s
> user	0m0.476s
> sys	0m0.296s
> 
> Thanks for putting together this test. There are a few ways in which it's inefficient. Queries are made serially, while a production system would make them in parallel. Process creation overhead is large, as you mentioned. Also, you are fetching across many different domains, which may have widely varying performance characteristics, but the main case we are discussing is issuance to a large number of subdomains.
> 
> I have a somewhat more realistic test setup I've used for DNS scanning. I used a commodity VPS with 8 cores running Unbound, plus this Go tool: https://github.com/jsha/go/blob/master/caa-lookups/lookups.go <https://github.com/jsha/go/blob/master/caa-lookups/lookups.go>.
> 
> time (seq --format '%g.hoffman-andrews.com <http://g.hoffman-andrews.com/>' 1 100000 | ./caa-lookups -server localhost:1053 -parallel 1500 -checkCAA) | grep -v ': ok'
> 
> real    1m34.374s
> user    0m45.656s
> sys     0m43.392s
> 
> That works out to a rate of about 1000 qps right off the bat, and I'm sure it can be optimized further. Or one could add a few more cores, or a slightly faster network card. I'm confident that this setup could support the issuance rate of 6M per hour (1670 qps) that you cite.

I redid the test moving to 10000 domains, but still one FQDN per Alexa known domain.  I also switched to a threaded option in Ruby (but note that threads in MRI are somewhat slow).  The result was an average of .0078 seconds to do a full CAA check starting with the full FQDN, as specified in the CAA RFC.  This would have been lower, but a percentage of these had DNS server failures which resulted in the thread having to wait until a multi-second timeout to continue.

So I agree, this seems very doable, as long as there is good internet access.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cabforum.org/pipermail/public/attachments/20161028/afa0df24/attachment-0003.html>