[HTTPS-Everywhere] persistent user-generated rules

Tue Jan 7 12:01:24 PST 2014

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 01/05/2014 09:30 AM, Vijay P wrote:
> Hi Yan,
> 
> Thanks for the quick reply!
> 

Hi Vijay!

No worries, though my replies will probably be slower in the future. :)

> As far as the XML change is concerned, why is speed of processing
> a concern? Is it because it slows development? Or because making a 
> deployment binary takes too long?

That's not the main problem. It's not as obvious in Chrome, but we've
noticed that HTTPS Everywhere takes a lot of CPU at startup in
Firefox, presumably due to loading the ruleset library. On my
Thinkpad, the current stable release raises CPU usage by 20-30% for a
few seconds whenever it's enabled/disabled.

Actually, simply enabling/disabling a rule on a site that has only one
rule (trac.torproject.org, for instance) seems to raise CPU by that
amount for about a second!

I'd be curious to see benchmarks on JSON parsing vs. XML parsing in
chrome vs. Firefox over the entire ruleset library. It looks like we
use https://developer.mozilla.org/en-US/docs/nsIDOMParser in FF. In
Chrome, we pretend that we're loading the rulesets from XHR and use
responseXML. My impression is that Chrome is more efficient.

There are a massive number of tickets related to this (actually, more
related to the feature you're implementing than the XML problem per
se). For instance:

* https://trac.torproject.org/projects/tor/ticket/4804
* https://trac.torproject.org/projects/tor/ticket/6118
* https://trac.torproject.org/projects/tor/ticket/9868
* https://trac.torproject.org/projects/tor/ticket/10174

> There are solutions for both of these possibilities that don't
> require a wholesale change to XML.  I'd argue that a shift to JSON
> would make sense for simplifying the process of adding rules, and
> unifying the structure of user-generated and static rules, not so
> much as a performance fix.

The usability win is in itself worth it IMO, even without any
performance benefits. :)

> 
> I personally think there would be great benefit to allow people to 
> "suggest" / upload their custom rules. It is non-trivial to send a 
> special request to add a site, and if the goal is to maximize the
> reach of this tool, making it as easy as possible to add random
> sites to the extension, we should make that at easy as possible.
> We could then have some process to integrate suggested rules into
> static rule-set at some point in the future.
> 
> (In fact, I'd even argue that long-term, the rulesets should be
> totally decoupled from the extension, the extension should be able
> to download rulesets from any source, with the EFF source being the
> default source)
> 

Yep, lots of discussion about this in the links above. Regarding
making it easier for people to add rules, what do you think about
putting a mailto:https-everywhere-rules at lists.eff.org button in
popup.html that automatically generates a git patch-formatted email
containing the new rule?

If we were to decouple rulesets from the extension (which is a great
idea, because then we could continuously update users' ruleset
subscriptions without going through the time-consuming process of
publishing an extension update), I think we'd have to research the
following:

* Should we allow people to subscribe to external ruleset feeds? This
is what AdblockPlus does. Probably yes, as you've suggested.

* Do Firefox/Chrome both offer synchronization API's that perform
signature verification? Probably not, so we'd have to write some code
to do this. I would say that being able to sign ruleset updates and
have clients verify them is a hard requirement for EFF.

* Would having a highly-customized ruleset subscription make users
fingerprintable? Say that an advertising company registers
[0-9a-z].ads.com and then sets itself up as a source for HTTPS
Everywhere rules. Unknown to the users, each person who downloads an
HTTPS Everywhere ruleset subscription from this source gets a
different rule for *.ads.com. The ad company could then track users
via essentially this method: http://www.unrest.ca/hsts-privacy-vs-security

But ignoring the external rulesets feature request for now, I think
your pull request is super helpful. Thanks a ton! I'll leave some
comments on the Github page.

- -Yan

> Vijay
> 
> 
> 
> On Sun, Jan 5, 2014 at 12:09 AM, Yan Zhu <yan at eff.org 
> <mailto:yan at eff.org>> wrote:
> 
> 
> 
> On 01/04/2014 08:40 PM, Vijay P wrote:
> 
> 
> 
>> On Sat, Jan 4, 2014 at 11:39 PM, Vijay P <vijayp at qwsrt.com
> <mailto:vijayp at qwsrt.com>
>> <mailto:vijayp at qwsrt.com <mailto:vijayp at qwsrt.com>>> wrote:
> 
>> Hi everyone,
> 
>> I'm a huge fan of the EFF and  HTTPS Everywhere extension!
> 
>> The one feature I felt it really needed was the ability for
>> users to add arbitrary rules, and to have those rules persistent
>> across restarts, and the ability for the users to submit rules to
>> the EFF so that the extension can be updated more frequently.
> 
>> I've put together a change that is a partial implementation of 
>> this feature, you can build it by checking out my branch below.
>> I'd love comments and/or suggestions before I go too far down the
>> path of implementation. Stuff in "what's missing" still has to
>> be implemented, but it should be functional as is.
> 
> 
> Thanks! This is a great idea that some people have requested. I'll 
> check it out next week along with Jacob's feature.
> 
>> What it does: - enables the httpseverywhere action icon thing on 
>> _all_ https pages - when on an https page, allows you to create
>> a rule for that host. It pre-fills the matching regex and
>> redirect-to fields with sane defaults.
> 
> 
>> What's missing: - cookie rules, and exceptions are unimplemented
>> - you can't edit or delete rules (but disabling them works). -
>> there is no way to upload these rules (not sure how you guys want
>> to handle that; I'd probably use a GAE endpoint, but you guys
>> might have thoughts about this)
> 
> 
> I wouldn't worry about this for now, since people who really want
> to submit rules can do so by email or pull requests. :)
> 
>> Future work? - my changelist uses java objects
> 
>> Uh, I (obviously) meant javascript objects
> 
> 
> 
> Phew, my heart stopped for a second there!
> 
> 
>> to send and persist rules on the extension. Perhaps we should 
>> change the XML structure to using JSON and match the object
>> format used by user rules at some point? Jacob Hoffman-Andrews
>> (we used to work together at Google a long time ago) mentioned
>> that there was discussion on this list about this topic before I
>> joined, and I'd be happy to help with that if people would find
>> it useful.
> 
> This is indeed something that we've been talking about more and
> more as ruleset loading/processing gets slower. I will very happily
> review any pull requests to change the XML ruleset structure to
> JSON! Perhaps convert all the existing rules to JSON first and
> check whether there's a noticeable performance improvement from
> doing so?
> 
> Another possibility is to convert default.rulesets into a sqlite 
> database, which we can work with using the web SQL API's in Chrome
> or Mozilla's storage API's in Firefox 
> (https://developer.mozilla.org/en-US/docs/Storage). Feel free to
> play with that idea.
> 
> -Yan
> 
> 
> 
> https://github.com/vijayp/https-everywhere/compare/vijayp.add-persistent-user-rules
>
> 
> 
> 
> 

- -- 
Yan Zhu                           yan at eff.org
Technologist                      Tel  +1 415 436 9333 x134
Electronic Frontier Foundation    Fax  +1 415 436 9993
-----BEGIN PGP SIGNATURE-----

iQEcBAEBCgAGBQJSzF0RAAoJENC7YDZD/dnsX+IH/021aEkiL/DVBDW6IYejo7hB
8pIDvWcENZuulz59UCC+mndnXEIHB6NrfxXL/T01/U1EPR6jbPk4orxdWjysUnQy
4O1ICIVKN328Pwrp73an7OIOpBcg1qdGHDFPXrDs1wlNb7fIQ/l6VsOKEgoZI6u7
5/IW24aOPY5oPdyLj+YKzox/MnUKqbGmewUWmvda8Mhy90fviweoyRuZ9vQQri3Q
NTnjALwQtZaa+tw1tQGnsrOL0l8TG/cmiohsYKUwk5mTc2FgTkNyRBMdPNfrSdwd
M3sc58xfNnNB+xVOr9O2l/1Jr8V/26DA0ckocXdSzJWrwJZR6pjhO/EREQUE+x0=
=iyli
-----END PGP SIGNATURE-----