[HTTPS-Everywhere] Clarifying the process for testing rulesets

[Peter Eckersley]

I think this just increases the urgency of working on this:

https://trac.torproject.org/projects/tor/ticket/6975

I will write a patch so that the Firefox version knows to disable mixed
content rules on version 18+.

On Sat, Oct 20, 2012 at 05:32:08AM -0700, Fruitless Creek wrote:
> Related?
> 
> "Disable insecure content loading on HTTPS pages"
> 
> https://www.mozilla.org/en-US/firefox/18.0a2/auroranotes/
> 
> 
> 
> 
> ________________________________
>  From: Peter Eckersley <pde at eff.org>
> To: mezzanine at Safe-mail.net 
> Cc: https-everywhere at eff.org 
> Sent: Wednesday, October 17, 2012 11:53 PM
> Subject: Re: [HTTPS-Everywhere] Clarifying the process for testing rulesets
>  
> On Thu, Oct 11, 2012 at 03:42:34AM -0400, mezzanine at Safe-mail.net wrote:
> > The page at https://www.eff.org/https-everywhere/rulesets talks about
> > how to create rulesets and also touches upon the step of testing a
> > ruleset. Even so, some aspects of how to test a ruleset are unclear.
> > For example, it may be the case where a specific domain or subdomain
> > appears to support HTTPS on more than one page, but with the question
> > as to whether each and every page under the domain supports HTTPS.
> > Should the tester try to visit and examine almost every page under the
> > site before submitting the ruleset? 
> 
> This is clearly not possible in many cases.  Having said that, the more
> complicated a site becomes, the more you would ideally want to have tested it
> extensively.  A good heuristic might be that if a site supports accounts, the
> ruleset author should make and use one, and try a bunch of the site's features
> to ensure that things aren't exploding.
> 
> > For that matter, how much testing and evaluation is performed on a ruleset
> > after it is submitted? Among other things, is it the case that a new ruleset
> > may be added to a development release of HTTPS Everywhere before being added
> > to a release version? 
> 
> This is almost always the case, yes, although occasionally things slip through
> in git.  With the 3.0 release, it seems as though we caught and fixed a lot of
> bugs in the new rulesets with the 3.0development releases, but another big
> wave of reports (more per unit time, fewer total, thus far) came in after the
> 3.0 release went stable.
> 
> The other testing that occurs at present is the syntax sanity checks that are
> used if you build the extension.  There are some cert-validity test scripts in
> existence, but I'm not sure if any of them know about transvalidity (missing
> intermediate CA certificates which browsers will cache, causing a cert to
> switch from invalid to valid) yet, and none are run automatically.
> 
> We have plans to try and implement/deploy an automated ruleset testing framework,
> possibly based on https://github.com/hiviah/https-everywhere-checker,
> possibly using Selenium or Marionette, but none of that is close to ready.
> 
> > It is one's presumption that it is sufficient to test
> > a ruleset prior to submission by using it with the HTTPS Everywhere
> > extension and the Firefox browser, and that it is not necessary to use
> > the Google Chrome browser, even though the HTTPS Everywhere extension
> > is available in alpha form for the Chrome browser. 
> 
> Interesting point.  Currenly, we have one major difference between the Chrome
> and Firefox platforms, which is that rulesets creating mixed content are okay
> on Firefox, but sometimes quite problematic on Chrome:
> 
> https://trac.torproject.org/projects/tor/ticket/6975
> 
> There are some rumours that Firefox is going to copy Chrome's mixed-content
> blocking strategy at some point, in which case we may have a large and sudden set
> of changes to make to the ruleset library.  It's therefore a good idea to
> start watching for and labelling rulesets as "mixedcontent" where possible.
> 
> > Although it seems sensible to subject a submitted ruleset to a certain
> > amount of post- submission testing (among other things, screening for
> > potential security issues), it would also seem advantageous for
> > pre-submission and post-submission testing of a ruleset to complement each
> > other, and to avoid (if possible) the situation where the two overlap
> > excessively.
> 
> Yup :). 
> > 
> > Ticket #2160 (https://trac.torproject.org/projects/tor/ticket/2160) on
> > the Tor Bug Tracker & Wiki talks about documenting the ruleset review
> > process for HTTPS Everywhere. The documentation that this ticket would
> > involve would be meant both for ruleset submitters and also
> > administrators/reviewers, according to the ticket description. The
> > rulesets for HTTPS Everywhere include coverage of sites that would
> > seem to be very large, such as sites for certain universities. Has
> > anyone had any experience or any comments regarding testing rulesets
> > prior to submitting them, particularly rulesets for large sites?
> 
> It feels as though the ruleset review process is still fluid enough that
> documenting it might be premature :)
> 
> -- 
> Peter Eckersley                            pde at eff.org
> Technology Projects Director      Tel  +1 415 436 9333 x131
> Electronic Frontier Foundation    Fax  +1 415 436 9993
> 
> _______________________________________________
> HTTPS-everywhere mailing list
> HTTPS-everywhere at mail1.eff.org
> https://mail1.eff.org/mailman/listinfo/https-everywhere

-- 
Peter Eckersley                            pde at eff.org
Technology Projects Director      Tel  +1 415 436 9333 x131
Electronic Frontier Foundation    Fax  +1 415 436 9993