<div dir="ltr"><div><div>Hey Yan,<br><br>I had a bit of free time this evening, so I decided it was a good time to learn Python :P<br><br></div>This script works with python3.3. I need somebody to check the code (my eyes are sore at the moment), but basically: (the script works ONLY in a directory that contains the src/chrome/content/rules folders - dirtree below)<br>
</div><div><br></div><div>1) you unzip the Alexa Top1M in the same directory as the script<br></div><div>2) you generate a git diff with this command (it's also in the script - probably you know a better alternative)<br>
<i>git diff --name-status master..remotes/origin/stable src/chrome/content/rules >> newRules.diff</i><br></div><div> and put the<i> newRules.diff </i>file in the same directory as merger.py and top-1m.csv<br>
</div><div>3) You launch merger.py<br><br></div><div>What it does now is just:<br></div><div>- printing "FOUND: src/chrome/content/rules/RULENAME.xml" if one of the targets in the rulefile is found in the Alexa list<br>
</div><div>- printing "File not found: <i>filename" </i>for those file with a weird encoding that were messing up my cool script (just two of them, can't remember their names)<br><br></div><div>It should be easy to tweak to - for example - copy a "FOUND" rule in a specific directory somewhere else for review/merge with stable<br>
<br></div><div>During the run, it splits the git diff in two segments: the "action" letter (A,D,M) and the rule path. I'm considering <b>only</b> rules with an "A", meaning new rules added.<br><br>
</div><div>Directory tree:<br>\-<br></div><div> --\src<br></div><div> --\chrome<br></div><div> --\content<br></div><div> --\rules<br></div><div> --\ [*.xml]<br></div><div> -- merger.py<br> -- newRules.diff<br>
</div><div> -- top-1m.csv<br></div><div><br><br></div><div>(hope it's clear, please tell me if it's not. I'll try to do something better :) )<br><br></div><div>Please, somebody, review the script and tell me (a) if it's actually working and (b) what can I do to improve it!<br>
</div><div><br>Cheers,<br><br>Claudio<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Jan 14, 2014 at 6:18 AM, Yan Zhu <span dir="ltr"><<a href="mailto:yan@eff.org" target="_blank">yan@eff.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA512<br>
<br>
<br>
<br>
</div><div class="im">On 01/13/2014 10:03 PM, Drake, Brian wrote:<br>
<br>
> I’m still concerned about the other part of my message. Right now,<br>
> it seems that, to review a ruleset properly, there are at least<br>
> four places that I need to check:<br>
><br>
> 1. Mailing list archives 2. <a href="http://trac.torproject.org" target="_blank">trac.torproject.org</a><br>
</div>> <<a href="http://trac.torproject.org" target="_blank">http://trac.torproject.org</a>> bug tracker 3. Github bug tracker 4.<br>
<div class="im">> Git (to find out the history of the ruleset, especially if I’m<br>
> using a stable release but want to account for ruleset changes in<br>
> the development branch)<br>
><br>
<br>
</div>Definitely open to suggestions about how to consolidate these, though<br>
I find that mailing list + 2 bug trackers is manageable as long as I'm<br>
not looking too far back in time (i.e., pre-December 2013).<br>
<br>
But usually, I consider a new ruleset to be properly reviewed if<br>
someone has built a test FF/Chrome extension with it included and<br>
tested it out.<br>
<br>
- -Yan<br>
<div class="im"><br>
<br>
><br>
> -- Brian Drake<br>
><br>
> All content created by me: Copyright<br>
> <<a href="http://www.wipo.int/treaties/en/ip/berne/trtdocs_wo001.html" target="_blank">http://www.wipo.int/treaties/en/ip/berne/trtdocs_wo001.html</a>> ©<br>
> 2014 Brian Drake. All rights reserved.<br>
><br>
</div><div class="im">> On Tue, Jan 14, 2014 at 0536 (UTC), Yan Zhu <<a href="mailto:yan@eff.org">yan@eff.org</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:yan@eff.org">yan@eff.org</a>>> wrote:<br>
><br>
><br>
><br>
> On 01/13/2014 09:18 PM, Drake, Brian wrote:<br>
>> Maybe people could opt-in to … is this where we would say<br>
>> “telemetry”? We could collect information about how much the<br>
>> rules actually get used, as well as things like redirect loops,<br>
>> to try to determine if a rule has been tested enough with no<br>
>> problems being found.<br>
><br>
> This is theoretically a good idea, except in practice there are<br>
> some obstacles:<br>
><br>
> 1. Stuff like automatically detecting when a page appears "broken"<br>
> or even just Javascript redirects is really, really hard. People<br>
> have tried using metrics like the Levenshtein distance between the<br>
> DOM tree of the HTTP and HTTPS sites, but nothing so far really<br>
> works.<br>
><br>
> 2. Given that automatically detecting breakage is tricky, it seems<br>
> that one of our best ways to figure out when something breaks is<br>
> to see how often users disable certain rules. This is hopefully<br>
> going to get merged soon (see other thread).<br>
><br>
> 3. Info like "how often a rule gets used" is hard to collect<br>
> safely, in the sense that collecting enough of it tends to<br>
> inadvertently create the risk of deanonymizing users. EFF tries as<br>
> hard as it can not to collect and store fingerprintable data on its<br>
> servers. :)<br>
><br>
><br>
>> What we desperately need as well is an easy way to find any<br>
>> issues already reported with a ruleset.<br>
><br>
>> For example, I when I was working on <a href="http://boohoo.com" target="_blank">boohoo.com</a><br>
</div></div>>> <<a href="http://boohoo.com" target="_blank">http://boohoo.com</a>> <<a href="http://boohoo.com" target="_blank">http://boohoo.com</a>>, I found many rulesets in<br>
<div class="im">>> the development branch (but not yet in stable) that were<br>
>> relevant, carefully checked the rules in them, and found many<br>
>> issues [1]. But since I am not familiar with any of those<br>
>> domains, I might have missed something. Or I might have reported<br>
>> issues that were already known. I have no idea.<br>
><br>
>> [1]<br>
><br>
> <a href="https://lists.eff.org/pipermail/https-everywhere-rules/2014-January/001792.html" target="_blank">https://lists.eff.org/pipermail/https-everywhere-rules/2014-January/001792.html</a><br>
><br>
><br>
>> -- Brian Drake<br>
><br>
>> All content created by me: Copyright<br>
>> <<a href="http://www.wipo.int/treaties/en/ip/berne/trtdocs_wo001.html" target="_blank">http://www.wipo.int/treaties/en/ip/berne/trtdocs_wo001.html</a>> ©<br>
>> 2014 Brian Drake. All rights reserved.<br>
><br>
>> On Tue, Jan 14, 2014 at 0435 (UTC), Yan Zhu <<a href="mailto:yan@eff.org">yan@eff.org</a><br>
> <mailto:<a href="mailto:yan@eff.org">yan@eff.org</a>><br>
</div><div class="im">>> <mailto:<a href="mailto:yan@eff.org">yan@eff.org</a> <mailto:<a href="mailto:yan@eff.org">yan@eff.org</a>>>> wrote:<br>
><br>
><br>
><br>
>> On 01/13/2014 06:29 PM, Drake, Brian wrote:<br>
>>> What is the process for moving a ruleset from the development<br>
>>> branch to the stable branch?<br>
><br>
>> Thank you thank you thank you for asking that question. I opened<br>
>> a ticket for this exact problem a few weeks ago:<br>
>> <a href="https://trac.torproject.org/projects/tor/ticket/10310" target="_blank">https://trac.torproject.org/projects/tor/ticket/10310</a><br>
><br>
>> Right now, the answer is "when yan or peter thinks it's<br>
>> important and probably been tested enough." I'll also merge<br>
>> something from dev to stable if someone pokes me about it<br>
>> specifically (ex: in the case of the stackexchange rule, since<br>
>> that was a blocker for Tor launching their own stackexchange<br>
>> site).<br>
><br>
>> Anyway, whoever works on that ticket linked above gets my<br>
>> undying love.<br>
><br>
>> -Yan<br>
><br>
><br>
><br>
><br>
><br>
<br>
- --<br>
Yan Zhu <a href="mailto:yan@eff.org">yan@eff.org</a><br>
Technologist Tel <a href="tel:%2B1%20415%20436%209333%20x134" value="+14154369333">+1 415 436 9333 x134</a><br>
Electronic Frontier Foundation Fax <a href="tel:%2B1%20415%20436%209993" value="+14154369993">+1 415 436 9993</a><br>
-----BEGIN PGP SIGNATURE-----<br>
<br>
</div>iQEcBAEBCgAGBQJS1NbGAAoJENC7YDZD/dnsOpgIAIqPUvXXyi3pGHfIhZrlvDOi<br>
1gszqGnBmipCwPepve5AHUgZw2u4rapqOb908KcPPF8L0AOE93tPgWG12RsmXwHh<br>
heNvgWDY+K1y+sCzd1vEm+0pm5gW/e3trrvat47tK3OZTTVC32n4i8ywLbGDheQ0<br>
pWxcFGsm/72+3Gz4h1H5VwdTHsSjd1VJgEqwlGXPn5a3eAqcpXWdEqUzgbnB8b4y<br>
+T+149FkXl8G4tUHjtAeEFqoTI04hS3b1S7/n6bRjyUnyohbQS/k59tchCobQHQm<br>
gY0m96U/wVATjTVZOqlx5o1h5tPUNdCukxGPHieNcZEXyHWTDqTsEz7qAnkL6lc=<br>
=rdhI<br>
-----END PGP SIGNATURE-----<br>
</blockquote></div><br></div>