[HTTPS-Everywhere] [GSoC] HTTPS Everywhere's Secure Ruleset Updater - Final Report

[yan]

Agreed! It was great working with you!

-Yan

On 08/19/2014 03:32 PM, Jacob S Hoffman-Andrews wrote:
> Thanks very much for your contributions, Red! It's been great to have
> you on the team this summer, and the work you've done is really valuable.
> 
> On 08/18/2014 08:11 PM, Red wrote:
>> Hello everyone!
>>
>> Today is the official "pencils down" date for Google Summer of Code, so
>> I am going to outline the results of my work this Summer as a way of
>> concluding my project.
>>
>> The first thing I had done at the start of my project was learn about
>> Mozilla's XPCOM components and how they are used to interface with Gecko
>> and other Firefox internals from extension code.  I also learned about
>> special files included with any extension such as install.rdf which are
>> used to describe an extension's version information and
>> signature-verifying public key, among other things. This lead me to
>> understanding how extensions are updated and discover the kinds of
>> limitations I would find in trying to update HTTPS Everywhere's rulesets
>> library without changing the rest of the extension.
>>
>> The next thing I did was draft the specification for the format of the
>> JSON manifest used to describe ruleset updates and explain how the
>> process of updating the rulesets library would work.  This work has
>> since been built on and improved, and the existing specification
>> document can be found in my fork of the extension, and is linked below[0].
>> This document describes everything one needs to know about the ruleset
>> updater. It details:
>>
>> * How an object signing certificate can be generated using certutil from
>> the NSS toolset.
>> * How to create an update.json manifest file using the
>> ruleset_update_manifest.py utility script I created.[1]
>> * How to sign the update.json file using pk1sign and an object signing
>> certificate.
>> * How update information is fetched and how download failures are handled.
>> * How the contents of update.json and the rulesets library database file
>> are verified to be authentic.
>> * What the significance of each field in update.json is.
>> * Approximately how the ruleset updating mechanism functions in pseudocode.
>>
>> After the details of how updating would work were outlined in
>> updateJSONSpec.md, I began implementing the ruleset updater module code
>> for the extension.  I also started working on a way to write tests for
>> HTTPS Everywhere that I would initially use to test my code. I
>> discovered that Firefox extensions could import XPCOM components
>> exported by other extensions added to a user's browser, and so began
>> work on a separate extension[2] that could be installed and used solely
>> for testing HTTPS Everywhere.  This work was soon improved upon by my
>> mentor, Yan, who had the idea of creating a skeleton of an extension
>> inside of HTTPS Everywhere's own codebase[3].  This extension used the
>> more modern Addon SDK, which includes a unit testing suite, enabling
>> HTTPS Everywhere developers to start writing proper unit tests that
>> could be included in the HTTPS-Everywhere repository and run from the
>> command line.
>>
>> While the unit testing environment I now had available was very
>> effective, I had run into some surprising difficulty with Mozilla's
>> nsIDataSignatureVerified XPCOM component, used to verify the signatures
>> we were generating.  By this point, Yan and I had agreed to try to
>> compute the signature over the SHA256 hash of update.json's contents, so
>> that this data could be more easily transcribed from a regular work
>> computer to the airgapped machine EFF uses to house an offline signing
>> key.  It was at this point that we found that the key and signatures
>> OpenSSL were generating were, somehow, incompatible with
>> nsIDataSignatureVerifier, and Yan managed to get the verification unit
>> test to pass by using certutil and pk1sign.  The same problem of
>> verifying signatures was encountered again when I got around to testing
>> the actual ruleset updater code again, and had to generate new data,
>> despite using certutil and pk1sign.  Quite unfortunately, this problem
>> drained almost four whole weeks of my time, as I had been looking into
>> solutions involving porting a Firefox extension signing tool called
>> Uhura (written in Perl), which meant also trying to understand what the
>> tool was doing with ASN.1 specifications, among other things.  Finally,
>> I found that the nsIDataSignatureVerifier tool successfully verified the
>> signature I had created for testing when it was not the hash of
>> update.json's contents that were signed and supplied to the verifier,
>> but the contents of the file themselves.  The component is poorly
>> documented and, after reaching out to developers in Mozilla's ecosystem,
>> I found that it isn't very well understood in general.  As such, even
>> now I do not really understand why the verifier did not behave as
>> expected in this scenario.  Regardless, while it is somewhat less
>> convenient here to sign update.json, the signature verification process
>> works.
>>
>> Finally, I was able to move on to solve the problems of downloading and
>> verifying the authenticity of the rulesets database file, and of
>> reinitializing the extension's rulesets with those of the new database
>> file. These problems were technically nuanced, but have been solved. 
>> The ruleset updater has been integrated into my fork of the extension's
>> repository on github.  My tests of its performance have passed, and I am
>> pleased to say that it appears to be working.  I have since submitted a
>> pull request[4] to have my work reviewed and merged, and have even made
>> changes requested by Yan, who has reviewed my work already.
>>
>> I'd like to conclude by giving my thanks to everyone in both of these
>> mailing lists.  I have received some invaluable support and advice that
>> has had a profoundly positive impact on my experience.  I learned a lot
>> from everyone that lent a hand, and I feel the project has benefited
>> greatly from each of the contributions I received to my efforts.  My
>> mentor, Yan, has also been a pleasure to work with.  Our weekly meetings
>> added very significant structure to my workflow, helped to keep me
>> organized, and were also a great opportunity for me to pique her brain
>> for ideas and possible solutions to some of the tougher problems I've
>> encountered.  I sincerely regard this as having been a positive
>> experience, and I hope that everyone who has helped me until now is
>> satisfied with my efforts and the actual results of my work.
>>
>> Thanks, everyone!
>>
>> [0] -
>> https://github.com/redwire/https-everywhere/blob/master/doc/updateJSONSpec.md
>> [1] -
>> https://github.com/redwire/https-everywhere/blob/master/utils/ruleset_update_manifest.py
>> [2] - https://github.com/redwire/https-everywhere-testing
>> [3] -
>> https://github.com/EFForg/https-everywhere/tree/master/https-everywhere-tests
>> [4] - https://github.com/EFForg/https-everywhere/pull/437
>>
>>
>>
>> _______________________________________________
>> HTTPS-Everywhere mailing list
>> HTTPS-Everywhere at lists.eff.org
>> https://lists.eff.org/mailman/listinfo/https-everywhere
>