[HTTPS-Everywhere] Automatic testing of rules to discover rules that broke (e.g. by site redesign)

[Ondrej Mikle]

On 05/18/2012 12:26 AM, Colonel Graff wrote:
> On Thu, May 17, 2012 at 5:34 PM, Peter Eckersley <pde at eff.org> wrote:
>> But there are still hundreds of other rulesets that your scripts generate
>> warnings about that need to be examined.  It will be quite a lot of work to do
>> this automatically (for instance, your code will generate warnings about
>> "transvalid" certs that are usually valid in real-world Firefoxes because of
>> cached intermediate CAs, but the Python requests library doesn't know about
>> this caching.  We could hook things up to the SSL Observatory to try to
>> deal with this, but it's quite a bit of work:
>> https://git.eff.org/?p=observatory.git;a=blob;f=transvalid.py;h=b0a5ce6ab89e35c30ea74384acbe8c45781b8695;hb=HEAD )
>>
> Well I finished my last final last night so I have a lot more time on
> my hands. I'll look more into this and see if there's a good way of
> checking it without having to hook into the SSL Observatory.

I've found a relatively simple way to deal with transvalid certs that
should be consistent how Firefox with cached intermediate certs sees
them. Requires PyCURL though (with openssl support enabled, which is
default in most distros; but reportedly doesn't work on Windows).

1. Export all the certs from a Firefox profile with this extension to
some directory, e.g. $HOME/local/Firefox_certs:
https://addons.mozilla.org/en-US/firefox/addon/export-all-certificates/

Note that the above method will also export the dozen or so "blacklisted
certs" for login.yahoo.com etc., that need to be deleted by hand (those
can be spotted quite easily, under "Servers" tab of FF's Certificate
Manager, the certs have "*" under "Server" field).

2. Convert them from DER to PEM:

for I in *.der; do openssl x509 -inform der -outform pem -in "$I" \
  -out "$(basename $I .der)".pem; done

3. Use openssl's "c_rehash" to create symlinks (if c_rehash is missing
in your openssl, grab it from openssl source tarball, it's a perl script
in "tools" directory):

c_rehash $HOME/local/Firefox_certs

Now you can "simulate" Firefox cert validation by giving the path to
PyCURL, example:

#---

#!/usr/bin/python
import pycurl
import cStringIO

buf = cStringIO.StringIO()

c = pycurl.Curl()
c.setopt(c.URL, 'https://sonicwall.21productions.com')
c.setopt(c.WRITEFUNCTION, buf.write)
c.setopt(c.CONNECTTIMEOUT, 5)
c.setopt(c.TIMEOUT, 8)
#following line does the trick of setting CApath to what FF sees
c.setopt(c.CAPATH, "/home/username/local/Firefox_certs")
c.perform()

print buf.getvalue()
buf.close()

#---

I've picked the above URL from SSL Observatory as an example of a
transvalid cert (GoDaddy intermediate missing in chain). Here is the
dump of certificates from FF with that intermediate cached (already
c_rehash'd, but still with the blacklisted certs):

https://constructibleuniverse.net/FF_certs/FF_certs_intermediate.tgz

With the above multiple profiles can be created, e.g. one for "squeaky
clean FF profile", one for CAcert, one for "typical FF profile after
years". Thus various rules can be tested depending on what the
"platform" attribute says.

> 
>> Another option would be to start running these kinds of scripts on a nightly
>> basis, and checking the output into git or otherwise publishing it, so that
>> ruleset authors can see when there's possible breakage to be investigated.
>>
> 
> Well, my concern would be that these websites might start to realize
> that every night they're getting the same request from the same IP
> addresses and get suspicious. We obviously can send user-agent
> information along with this to identify the script for people, but
> even then, it might result in suspicious system administrators
> IP-blocking the script.

I don't think they would mind. I had just single complaint in ~9 months
for scanning certs daily (or add an explanation URL to the User Agent
string).

Ondrej