[HTTPS-Everywhere] Clarifying the process for testing rulesets

[Peter Eckersley]

On Thu, Oct 11, 2012 at 03:42:34AM -0400, mezzanine at Safe-mail.net wrote:
> The page at https://www.eff.org/https-everywhere/rulesets talks about
> how to create rulesets and also touches upon the step of testing a
> ruleset. Even so, some aspects of how to test a ruleset are unclear.
> For example, it may be the case where a specific domain or subdomain
> appears to support HTTPS on more than one page, but with the question
> as to whether each and every page under the domain supports HTTPS.
> Should the tester try to visit and examine almost every page under the
> site before submitting the ruleset? 

This is clearly not possible in many cases.  Having said that, the more
complicated a site becomes, the more you would ideally want to have tested it
extensively.  A good heuristic might be that if a site supports accounts, the
ruleset author should make and use one, and try a bunch of the site's features
to ensure that things aren't exploding.

> For that matter, how much testing and evaluation is performed on a ruleset
> after it is submitted? Among other things, is it the case that a new ruleset
> may be added to a development release of HTTPS Everywhere before being added
> to a release version? 

This is almost always the case, yes, although occasionally things slip through
in git.  With the 3.0 release, it seems as though we caught and fixed a lot of
bugs in the new rulesets with the 3.0development releases, but another big
wave of reports (more per unit time, fewer total, thus far) came in after the
3.0 release went stable.

The other testing that occurs at present is the syntax sanity checks that are
used if you build the extension.  There are some cert-validity test scripts in
existence, but I'm not sure if any of them know about transvalidity (missing
intermediate CA certificates which browsers will cache, causing a cert to
switch from invalid to valid) yet, and none are run automatically.

We have plans to try and implement/deploy an automated ruleset testing framework,
possibly based on https://github.com/hiviah/https-everywhere-checker,
possibly using Selenium or Marionette, but none of that is close to ready.

> It is one's presumption that it is sufficient to test
> a ruleset prior to submission by using it with the HTTPS Everywhere
> extension and the Firefox browser, and that it is not necessary to use
> the Google Chrome browser, even though the HTTPS Everywhere extension
> is available in alpha form for the Chrome browser. 

Interesting point.  Currenly, we have one major difference between the Chrome
and Firefox platforms, which is that rulesets creating mixed content are okay
on Firefox, but sometimes quite problematic on Chrome:

https://trac.torproject.org/projects/tor/ticket/6975

There are some rumours that Firefox is going to copy Chrome's mixed-content
blocking strategy at some point, in which case we may have a large and sudden set
of changes to make to the ruleset library.  It's therefore a good idea to
start watching for and labelling rulesets as "mixedcontent" where possible.

> Although it seems sensible to subject a submitted ruleset to a certain
> amount of post- submission testing (among other things, screening for
> potential security issues), it would also seem advantageous for
> pre-submission and post-submission testing of a ruleset to complement each
> other, and to avoid (if possible) the situation where the two overlap
> excessively.

Yup :). 
> 
> Ticket #2160 (https://trac.torproject.org/projects/tor/ticket/2160) on
> the Tor Bug Tracker & Wiki talks about documenting the ruleset review
> process for HTTPS Everywhere. The documentation that this ticket would
> involve would be meant both for ruleset submitters and also
> administrators/reviewers, according to the ticket description. The
> rulesets for HTTPS Everywhere include coverage of sites that would
> seem to be very large, such as sites for certain universities. Has
> anyone had any experience or any comments regarding testing rulesets
> prior to submitting them, particularly rulesets for large sites?

It feels as though the ruleset review process is still fluid enough that
documenting it might be premature :)

-- 
Peter Eckersley                            pde at eff.org
Technology Projects Director      Tel  +1 415 436 9333 x131
Electronic Frontier Foundation    Fax  +1 415 436 9993