[HTTPS-Everywhere] [GSoC] HTTPS Everywhere's Secure Ruleset Updater - Final Report

[Jacob S Hoffman-Andrews]

Thanks very much for your contributions, Red! It's been great to 
have you on the team this summer, and the work you've done is really 
valuable.

On 08/18/2014 08:11 PM, Red wrote:
> Hello everyone!
>
> Today is the official "pencils down" date for Google Summer of Code, so
> I am going to outline the results of my work this Summer as a way of
> concluding my project.
>
> The first thing I had done at the start of my project was learn about
> Mozilla's XPCOM components and how they are used to interface with Gecko
> and other Firefox internals from extension code.  I also learned about
> special files included with any extension such as install.rdf which are
> used to describe an extension's version information and
> signature-verifying public key, among other things. This lead me to
> understanding how extensions are updated and discover the kinds of
> limitations I would find in trying to update HTTPS Everywhere's rulesets
> library without changing the rest of the extension.
>
> The next thing I did was draft the specification for the format of the
> JSON manifest used to describe ruleset updates and explain how the
> process of updating the rulesets library would work.  This work has
> since been built on and improved, and the existing specification
> document can be found in my fork of the extension, and is linked below[0].
> This document describes everything one needs to know about the ruleset
> updater. It details:
>
> * How an object signing certificate can be generated using certutil from
> the NSS toolset.
> * How to create an update.json manifest file using the
> ruleset_update_manifest.py utility script I created.[1]
> * How to sign the update.json file using pk1sign and an object signing
> certificate.
> * How update information is fetched and how download failures are handled.
> * How the contents of update.json and the rulesets library database file
> are verified to be authentic.
> * What the significance of each field in update.json is.
> * Approximately how the ruleset updating mechanism functions in pseudocode.
>
> After the details of how updating would work were outlined in
> updateJSONSpec.md, I began implementing the ruleset updater module code
> for the extension.  I also started working on a way to write tests for
> HTTPS Everywhere that I would initially use to test my code. I
> discovered that Firefox extensions could import XPCOM components
> exported by other extensions added to a user's browser, and so began
> work on a separate extension[2] that could be installed and used solely
> for testing HTTPS Everywhere.  This work was soon improved upon by my
> mentor, Yan, who had the idea of creating a skeleton of an extension
> inside of HTTPS Everywhere's own codebase[3].  This extension used the
> more modern Addon SDK, which includes a unit testing suite, enabling
> HTTPS Everywhere developers to start writing proper unit tests that
> could be included in the HTTPS-Everywhere repository and run from the
> command line.
>
> While the unit testing environment I now had available was very
> effective, I had run into some surprising difficulty with Mozilla's
> nsIDataSignatureVerified XPCOM component, used to verify the signatures
> we were generating.  By this point, Yan and I had agreed to try to
> compute the signature over the SHA256 hash of update.json's contents, so
> that this data could be more easily transcribed from a regular work
> computer to the airgapped machine EFF uses to house an offline signing
> key.  It was at this point that we found that the key and signatures
> OpenSSL were generating were, somehow, incompatible with
> nsIDataSignatureVerifier, and Yan managed to get the verification unit
> test to pass by using certutil and pk1sign.  The same problem of
> verifying signatures was encountered again when I got around to testing
> the actual ruleset updater code again, and had to generate new data,
> despite using certutil and pk1sign.  Quite unfortunately, this problem
> drained almost four whole weeks of my time, as I had been looking into
> solutions involving porting a Firefox extension signing tool called
> Uhura (written in Perl), which meant also trying to understand what the
> tool was doing with ASN.1 specifications, among other things.  Finally,
> I found that the nsIDataSignatureVerifier tool successfully verified the
> signature I had created for testing when it was not the hash of
> update.json's contents that were signed and supplied to the verifier,
> but the contents of the file themselves.  The component is poorly
> documented and, after reaching out to developers in Mozilla's ecosystem,
> I found that it isn't very well understood in general.  As such, even
> now I do not really understand why the verifier did not behave as
> expected in this scenario.  Regardless, while it is somewhat less
> convenient here to sign update.json, the signature verification process
> works.
>
> Finally, I was able to move on to solve the problems of downloading and
> verifying the authenticity of the rulesets database file, and of
> reinitializing the extension's rulesets with those of the new database
> file. These problems were technically nuanced, but have been solved.
> The ruleset updater has been integrated into my fork of the extension's
> repository on github.  My tests of its performance have passed, and I am
> pleased to say that it appears to be working.  I have since submitted a
> pull request[4] to have my work reviewed and merged, and have even made
> changes requested by Yan, who has reviewed my work already.
>
> I'd like to conclude by giving my thanks to everyone in both of these
> mailing lists.  I have received some invaluable support and advice that
> has had a profoundly positive impact on my experience.  I learned a lot
> from everyone that lent a hand, and I feel the project has benefited
> greatly from each of the contributions I received to my efforts.  My
> mentor, Yan, has also been a pleasure to work with.  Our weekly meetings
> added very significant structure to my workflow, helped to keep me
> organized, and were also a great opportunity for me to pique her brain
> for ideas and possible solutions to some of the tougher problems I've
> encountered.  I sincerely regard this as having been a positive
> experience, and I hope that everyone who has helped me until now is
> satisfied with my efforts and the actual results of my work.
>
> Thanks, everyone!
>
> [0] -
> https://github.com/redwire/https-everywhere/blob/master/doc/updateJSONSpec.md
> [1] -
> https://github.com/redwire/https-everywhere/blob/master/utils/ruleset_update_manifest.py
> [2] - https://github.com/redwire/https-everywhere-testing
> [3] -
> https://github.com/EFForg/https-everywhere/tree/master/https-everywhere-tests
> [4] - https://github.com/EFForg/https-everywhere/pull/437
>
>
>
> _______________________________________________
> HTTPS-Everywhere mailing list
> HTTPS-Everywhere at lists.eff.org
> https://lists.eff.org/mailman/listinfo/https-everywhere

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.eff.org/pipermail/https-everywhere/attachments/20140819/29c44ca9/attachment.html>