[HTTPS-Everywhere] Draft specification for file used to check for ruleset updates

[Red]

On 2014-06-18, 11:43 AM, Yan Zhu wrote:
> Nice work! I have some comments on the spec below (why does gist not
> allow you to post inline comments??)
This is another good reason to probably stick the document into my fork
of https-everywhere somewhere, huh? I'll do that, after all.
https://github.com/redwire/https-everywhere/blob/makeJSONManifest/doc/updateJSONSpec.md

>> However, fetching files using standard XMLHTTPRequests from within the
>> extension is trivial to accomplish and standards-compliant.
>> What does standards-compliant mean here? Maybe omit.
Here, "standards-compliant" meant that it doesn't require trying to
force any parts of the browser to do weird things with the updater (make
premature requests to look for updates, apply changes to only a part of
the extension, ...).
I'll change the wording a bit to make that more clear.
>> And finally, the public key to hardcode into the HTTPS-Everywhere
>> extension to enable it to verify such signatures can be output to
>> pubkey.pem via the command [...]
>> This command should be modified to generate a base64-encoded signature,
>> which is what nsIDataSignatureVerifier expects.
Right you are! I added a line detailing use of `openssl base64` to
accomplish this last night.
>> Note that the header/footer of pubkey.pem need to be stripped before
>> giving it as input to nsIDataSignatureVerifier.
I'll include instructions regarding that.
>> If an attempt to fetch or verify an update fails, the extension should
>> request update.json again every 5 + R minutes, where R is a random
>> number between 0 and 5.
>>
>> The padding should be a random integer in milliseconds, so it's probably
>> better to specify this time interval in milliseconds.
Alright. I've made all the conversions for that section and also
specified the initial three-hour
value in terms of milliseconds for consistency.
>> Every time the extension finds that the data provided by update.json
>> to be inauthentic, either as a result of the hash of the database file
>> not matching or the signature not verifying, the extension must send a
>> POST request to a hardcoded url containing the data in the update.json
>> file that it tried and failed to verify.
>>
>> Replace "hardcoded URL" with "hardcoded failure-reporting URL" for
>> clarity. Add that the format for failure reports is yet to be determined.
Good point. That should be there.
>> This will, of course, only occur after the extension has determined
>> that an update to the ruleset database is available, and verified the
>> content of the update.json file to be authentic.
>>
>> Replace with "Replacing the local rulesets library will only occur after
>> validating update.json (pseudocode below)."
Alright.
>> The format for the date is " , ". For example, "08 June, 2014".
>> Why not 08-06-2014 for easier date comparison in JS if we ever need to
>> do it?
Arg, I keep forgetting angle-brackets are special in Markdown.  I
definitely see your point and agree that we might
as well use that kind of date format.  I chose the former because I
figured it was more human readable, which counted for a lot if we were
only going to display it to the user.
>>  hash is a SHA1 (for now) hash of the database file's raw content.
> What encoding should we use for the hash? The default from
> https://developer.mozilla.org/en-US/docs/Mozilla/Tech/XPCOM/Reference/Interface/nsICryptoHash#finish%28%29
> is base64, so I would lean towards that.
I also looked into that and modified the script I have to build
update.json to do this a few days ago.
https://github.com/redwire/https-everywhere/blob/makeJSONManifest/utils/ruleset_update_manifest.py#L89
> Also, it's better to specify SHA1 somewhere in the update.json file in
> case anyone is reading it independently. This could either be an
> additional field, or we could use the format
> "sha1/5R0zeLx7EWRxqw6HRlgCRxNLHDo=" (<name of hash function>/<base
> 64-encoded string).
The fact that SHA1 is used is specified in the first paragraph of
"Verification and Version Checking".
Specifically: "SHA1 is currently being used as the hashing algorithm."
>> Pseudocode of update procedure [...]
> line 9: "update.version" should be "updateData.version"
>
> "inauthentic" should be initialized to true instead of false so that if
> isValidSignature throws an error or gets skipped, nothing is updated
> (generally safer).
>
> updateData.source should be validated to be a valid eff.org URL before
> fetching it.
>
> if any of the XHR  fails (due to network disconnectivity, for instance),
> we should try again in 5 minutes plus padding.
This was a bit trickier to get into the pseudocode cleanly. You'll
probably want to review it again to
make sure everything looks good. I improved on everything you mentioned
there, hopefully without
being too specific- it is just pseudocode after all.
> On 06/17/2014 06:18 PM, Red wrote:
>> I have recently been working on updating the ruleset updater spec to
>> reflect the changes that Yan and I discussed during our meeting last
>> Friday (notes at
>> https://gist.github.com/redwire/b62f03905a826e79947a#week-5), so I
>> wanted to inform everyone of the changes.  If anyone would like to have
>> a look over the document, I'd be happy to receive any suggestions for
>> improvements.
>> https://gist.github.com/redwire/2e1d8377ea58e43edb40
>>
>> Another thing I want to report is that I have finished updating the
>> utility script I have also been working on to automate most of the
>> process of building the update.json file.  One thing I have added into
>> it is a simple mechanism for creating and applying sanity checks to
>> input values.  One such test is the "valid_eff_url" function I wrote
>> that attempts to match the provided source (directing the extension to
>> the location of the ruleset database file) to a valid eff.org URL
>> (ending with .sqlite).  I wrote the regular expression here specifically
>> to be exactly the same as what would be used to repeat this sanity check
>> in the update mechanism itself.  I have tested the regex in both Python
>> and Javascript and found it behaves the same in Javascript and Python,
>> but as Yan suggested, it seems obvious that several others should review
>> this regex to be sure.  Here is a direct link to the beginning of the
>> function.
>> https://github.com/redwire/https-everywhere/blob/makeJSONManifest/utils/ruleset_update_manifest.py#L58


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 341 bytes
Desc: OpenPGP digital signature
URL: <https://lists.eff.org/pipermail/https-everywhere/attachments/20140619/81a86f31/attachment.sig>