[HTTPS-Everywhere] Automatic testing of rules to discover rules that broke (e.g. by site redesign)

[Peter Eckersley]

By the way, Seth managed to use this script to author a bugfix commit like this:

https://gitweb.torproject.org/https-everywhere.git/commitdiff/31e73569f7ef0670490b7111cf069a8fae758d5d

But there are still hundreds of other rulesets that your scripts generate
warnings about that need to be examined.  It will be quite a lot of work to do
this automatically (for instance, your code will generate warnings about
"transvalid" certs that are usually valid in real-world Firefoxes because of
cached intermediate CAs, but the Python requests library doesn't know about
this caching.  We could hook things up to the SSL Observatory to try to
deal with this, but it's quite a bit of work:
https://git.eff.org/?p=observatory.git;a=blob;f=transvalid.py;h=b0a5ce6ab89e35c30ea74384acbe8c45781b8695;hb=HEAD )

Another option would be to start running these kinds of scripts on a nightly
basis, and checking the output into git or otherwise publishing it, so that
ruleset authors can see when there's possible breakage to be investigated.

On Wed, May 09, 2012 at 09:13:56PM -0400, Colonel Graff wrote:
> Hey Ondrej,
> 
> Peter and Scott have both been made aware of this, but these are the
> first steps towards what you suggested. Already with them I've found a
> few broken websites that I've fixed. Don't run them on your machine
> though. They're highly imperfect. Feel free to add/improve them
> though.
> https://gitorious.org/https-everywhere-fork/https-everywhere-fork/blobs/graff/single_rule_response.py
> https://gitorious.org/https-everywhere-fork/https-everywhere-fork/blobs/graff/trivial-response.py
> 
> For now these will just test the destination of the redirects that
> don't have backreferences. No breadth first search yet. That wouldn't
> be trivial. I might go ahead and write that later, but not right now.
> As a non-CS person (I'm far more of a hobbyist) I haven't had the joy
> of dealing with databases yet. If you'd like to add that to this or
> send me a good tutorial for the future, I'd appreciate it.
> 
> On Tue, May 1, 2012 at 10:03 AM, Colonel Graff
> <graffatcolmingov at gmail.com> wrote:
> > On Sun, Apr 29, 2012 at 12:15 AM, Peter Eckersley <pde at eff.org> wrote:
> >>
> >> On Sat, Apr 28, 2012 at 06:50:01PM +0200, Ondrej Mikle wrote:
> >> > Hi,
> >> >
> >> > seeing how many times "broken rule" topics appear in
> >> > https-everywhere-rules,
> >> > have you considered some automatic tests? (Re-reading the text of this
> >> > mail
> >> > again, I might be attempting to solve a problem you may not really
> >> > have.)
> >>
> >> We would /love/ to have someone implement a proper test suite of this
> >> sort!
> >> We even offered it as a Google Summer of Code project this year, though we
> >> didn't get any takers :(
> >>
> >> --
> >> Peter Eckersley                            pde at eff.org
> >> Technology Projects Director      Tel  +1 415 436 9333 x131
> >> Electronic Frontier Foundation    Fax  +1 415 436 9993
> >>
> >> _______________________________________________
> >> HTTPS-everywhere mailing list
> >> HTTPS-everywhere at mail1.eff.org
> >> https://mail1.eff.org/mailman/listinfo/https-everywhere
> >
> >
> > I'm not entirely certain of the best way to do this, but if anyone's looking
> > at doing it in Python (since we already use python for rule validation)
> > check out Kenneth Reitz's excellent requests library. It handles https (if I
> > remember correctly) and makes URL handling a lot easier.
> 
> _______________________________________________
> HTTPS-everywhere mailing list
> HTTPS-everywhere at mail1.eff.org
> https://mail1.eff.org/mailman/listinfo/https-everywhere

-- 
Peter Eckersley                            pde at eff.org
Technology Projects Director      Tel  +1 415 436 9333 x131
Electronic Frontier Foundation    Fax  +1 415 436 9993