[HTTPS-Everywhere] HTTPS Everywhere ruleset checker alpha version

[Ondrej Mikle]

After couple of days working around pycurl's quirks I have to say it's quite a
collection of special cases (see comments inline).

On 08/05/2012 11:43 PM, Peter Eckersley wrote:
> Firefox seems happy to talk to https://www.zotero.org, though
> 
> 2) gnutls cert validation snafu on www.wyndhamrentals.com (that chain looks
> valid and isn't even transvalid, according to openssl)

That's gnutls being strict about the TLS RFC again - the only thing wrong with
the chain is that it's in wrong order. And it trips up gnutls which is not as
forgiving as openssl (also tested it on my own server, gnutls didn't like
incorrect chain order either).

Even though I kind of like gnutls's strict adhering to protocol, for practical
purposes we'll have to stick with openssl or nss (I'd favor nss if it wasn't for
the occasional weird HTTP 400s).

> 3) We probably need to augment this script to recognise rules for sites with
> transvalid certs, which are still about as "correct" as they can be until the
> browsers fix the underlying problem of transvalidity.

Originally I planned a feature that would retrieve the intermediate certs
on-demand from a observatory, but creating a minimal set in a "transvalid"
directory would probably work as well (and wouldn't be such an overkill).

I could also switch the per-rule-selection-of-CA-dir to make it per-run (might
be more useful for the transvalid case).


Pycurl, threads and subprocesses:

Python threads don't work well in two cases: CPU-intensive computation due to
GIL and crashing native libraries (pycurl case turns out to be the second one).

I've tried also worker subprocesses (with modules like multiprocess) to avoid
pycurl/openssl/gnutls/nss error taking the main process with it, but it has
quite huge CPU/mem overhead (more than 2x) compared to threads.

The best approach so far seems to be to use threads, but spawn a short-lived
subprocesses to fetch one page and pass all data serialized through cPickle (via
pipes or stdin/stdout, multiprocessing.Queue is for some reason a hog on CPU too).

Side observation about pycurl: setting option pycurl.NOSIGNAL avoids the
segfaults, but underlying TLS libs still can bring the process down with SIGPIPE
when for instance server's port 443 is open but doesn't send any data and
immediately closes.

Ondrej

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.eff.org/pipermail/https-everywhere/attachments/20120810/76e7e4ab/attachment.sig>