[HTTPS-Everywhere] Stupid Perl Tricks: ssl_check2.pl
Whizz Mo
https at whizzmo.com
Wed Nov 10 18:15:35 PST 2010
Quite possibly! [ At least you still have your cat/dog/truck/wife right?
]
Digging deeper, it seems that some aX.twimg.com hosts support SSL while
others do not. This makes troubleshooting... interesting. I don't have a
good solution for this one just yet. One of the assumptions that I made
when writing this script was that a host would either support or not support
SSL connections reliably. Silly me!
I suppose I'll have to add some support for multiple A (or AAAA?) records at
some point. Feature creep, and all that, you know.
I did make a tweak to the script to allow it to examine HTTPS responses that
were "close" but not quite identical, and determine the number of
initially-similar bytes. (See attached) For me, the output of this feature
looks like this:
29 sad. HTTPS hash does NOT match HTTP hash for https://twitter.com.
... but the first 1742 / 45360 bytes are the same! Manual check
required.
Here are 80 bytes from both strings, starting at offset 1732:
http: ref="
http://a1.twimg.com/a/1289433550/images/twitter_57.png" rel="apple-touch-ic
https: ref="
https://s3.amazonaws.com/twitter_production/a/1289433550/images/twitter_57.
Note that the starting byte of each listed string is 10 chars before the
variance occurs. For sites with rotating ad banners, this may be an
issue. I did mention that this was a QnD hack, right? I suppose I could
add a CLI parameter to "fetch html content x times" and hope that, with a
sufficiently large x, a pair of HTTP/HTTPS responses would match. This
solution has O(n^2) complexity, which I would like to avoid if possible.
Any suggestions?
On Wed, Nov 10, 2010 at 5:48 PM, Peter Eckersley <pde at eff.org> wrote:
> Whizz, this script is great but I'm wondering if it's still somewhat
> buggy...
>
> perl ./ssl_check2.pl http://twitter.com
>
> Getting http://twitter.com ...Done.
> Got 44511 bytes in 1 secs (44511 bytes / sec)
> Found 29 reference(s) to check.
>
> Checking reference URLs...
> 1 zzz. HTTPS request timeout. Added a0.twimg.com to badhosts list.g
> 2 Skipping url
> http://a0.twimg.com/a/1289339734/images/whatsnew/video-sample-ss.png(known-bad host
> a0.twimg.com).
> 3 Skipping url
> http://a0.twimg.com/a/1289339734/javascripts/widgets/widget.js?1289366423(known-bad host
> a0.twimg.com).
> 4 Skipping url
> http://a0.twimg.com/profile_images/118608576/twitter_sc_logo_normal.jpg(known-bad host
> a0.twimg.com).
> 5 Skipping url
> http://a0.twimg.com/profile_images/120242004/finaltwitter_normal.jpg(known-bad host
> a0.twimg.com).
> 6 Skipping url
> http://a0.twimg.com/profile_images/49918572/half-face-ice_normal.jpg(known-bad host
> a0.twimg.com).
> 7 zzz. HTTPS request timeout. Added a1.twimg.com to badhosts list.
> 8 Skipping url http://a1.twimg.com/a/1289339734/images/twitter_57.png(known-bad host
> a1.twimg.com).
> 9 Skipping url
> http://a1.twimg.com/a/1289339734/stylesheets/fronts.css?1289366423(known-bad host
> a1.twimg.com).
> 10 Skipping url
> http://a1.twimg.com/profile_images/220756397/afwd-twitter-logo_normal.gif(known-bad host
> a1.twimg.com).
> 11 Skipping url
> http://a1.twimg.com/profile_images/263029233/slide1_normal.jpg (known-bad
> host a1.twimg.com).
> 12 Skipping url
> http://a1.twimg.com/profile_images/381297805/mobile_normal.png (known-bad
> host a1.twimg.com).
> 13 Skipping url
> http://a1.twimg.com/profile_images/52564417/twitter_normal.jpg (known-bad
> host a1.twimg.com).
> 14 Skipping url
> http://a1.twimg.com/profile_images/555579649/steve_case_wsj_normal.jpg(known-bad host
> a1.twimg.com).
> 15 Skipping url
> http://a1.twimg.com/profile_images/601329413/twitter_logo_normal.jpg(known-bad host
> a1.twimg.com).
> 16 Skipping url
> http://a1.twimg.com/profile_images/670252813/136489main_pia04413-feature-browse_normal.jpg(known-bad host
> a1.twimg.com).
> 17 Skipping url
> http://a1.twimg.com/profile_images/727884617/rainbow_normal.jpg (known-bad
> host a1.twimg.com).
> 18 Skipping url
> http://a1.twimg.com/profile_images/866556637/teatime__normal.jpg(known-bad host
> a1.twimg.com).
> 19 zzz. HTTPS request timeout. Added a2.twimg.com to badhosts list.
> 20 Skipping url
> http://a2.twimg.com/profile_images/1114845454/daily-parent-tip_normal.png(known-bad host
> a2.twimg.com).
> 21 Skipping url
> http://a2.twimg.com/profile_images/264983646/2008_author_shot_copy_normal.jpg(known-bad host
> a2.twimg.com).
> 22 Skipping url
> http://a2.twimg.com/profile_images/458966890/twitterprofilephoto_normal.jpg(known-bad host
> a2.twimg.com).
> 23 Skipping url
> http://a2.twimg.com/profile_images/544732942/logorgb2_justh_normal.png(known-bad host
> a2.twimg.com).
> 24 Skipping url
> http://a2.twimg.com/profile_images/91810842/ai_250x250_twit_normal.jpg(known-bad host
> a2.twimg.com).
> 25 zzz. HTTPS request timeout. Added a3.twimg.com to badhosts list.g
> 26 Skipping url
> http://a3.twimg.com/profile_images/291571823/unknown-6_normal.jpeg(known-bad host
> a3.twimg.com).
> 27 Skipping url
> http://a3.twimg.com/profile_images/748445671/shopaneer-002-36x36_normal.jpg(known-bad host
> a3.twimg.com).
> 28 zzz. HTTPS request timeout. Added ajax.googleapis.com to badhosts
> list.
> 29 zzz. HTTPS request timeout. Added twitter.com to badhosts list.
>
>
>
>
> Results:
> Total links: 29
> Working links: 0
> Non-Working links: 29 (100%)
> HTTP request fail: 0
> HTTPS request fail: 6 (20.68%)
> Hash mismatch: 0
> Links with a known-bad host: 23 (79.31%)
>
> Bad hosts:
> a0.twimg.com
> a1.twimg.com
> a2.twimg.com
> a3.twimg.com
> ajax.googleapis.com
> twitter.com
>
> HTTPS fail urls:
>
>
> https://a0.twimg.com/a/1289339734/images/fronts/logo_withbird_home.png
> https://a1.twimg.com/a/1289339734/images/favicon.ico
> https://a2.twimg.com/a/1289339734/javascripts/fronts.js
>
> https://a3.twimg.com/profile_images/1148176527/1110-twitter_normal.jpg
> https://ajax.googleapis.com/ajax/libs/jquery/1.3.0/jquery.min.js
> https://twitter.com
>
>
> Verdict: Verdict: This page IS NOT a candidate for *simple* domain-wide
> forced encryption, but may be a candidate for URL-rewriting or path-based
> forcing. More research is required.
>
>
> On Thu, Nov 04, 2010 at 07:06:51PM -0700, Whizz Mo wrote:
> > In case no one has bothered to write this already, attached is a simple
> perl
> > script to check an http url for https compatibility.
> > This script:
> >
> > 1. fetches the http url
> > 2. parses it for fetchable links (images, scripts, frames, other
> hrefs)
> > 3. fetches the fetchable links in http and https
> > 4. compares the http and https responses.
> > 5. prints report. (See attached text file for a sample)
> >
> > Usage:
> > perl ssl_check2.pl http://somesite.com/
> >
> > Output is currently command-line only. (Do not run this script from the
> > Windows Run Command box.)
> >
> > Caveats:
> >
> > - This is very quick and dirty code, and should be considered
> > "experimental". May format your hard drive, kick your dog, steal your
> > truck, and run off with your wife.
> > - This script will parse a frame url, but will not (recursively) parse
> > the content of the frame. [To-do list]
> >
> >
> >
> >
> > Thanks,
> > Whizz
>
> > Getting http://slashdot.org/ ...Done.
> > Got 117515 bytes in 1 secs (117515 bytes / sec)
> > Found 4 reference(s) to check.
> >
> > Checking reference URLs...
> > 1 YAY! HTTPS appears ok for
> https://b.scorecardresearch.com/p?c1=2&c2=6035546&c3=&c4=&c5=&c6=&c15=&cj=1.
> > 2 sad. HTTPS hash does NOT match HTTP hash for
> https://jlinks.industrybrains.com/jsct?sid=941&ct=slashdot_ros&num=3&layt=300x250imgad&fmt=simp
> .
> > ... but the first 1058 / 8670 bytes are the same! Manual check
> required.
> > Here are 80 bytes from both strings, starting at offset 1048:
> > http:
> mp;lid=682045&cid=151113&pr=2&tstamp=20101104214514&iip=260.309.
> > https:
> mp;lid=685533&cid=151895&pr=2&tstamp=20101104214515&iip=260.309.
> > 3 zzz. HTTPS request timeout. Added rss.slashdot.org to badhosts
> list.
> > 4 YAY! HTTPS appears ok for https://slashdot.org/ .
> >
> >
> >
> >
> > Results:
> > Total links: 4
> > Working links: 2 (50%)
> > Semi-working links: 1 (25%) [See "HTTPS possible urls" below]
> > Non-Working links: 1 (25%)
> > HTTP request fail: 0
> > HTTPS request fail: 1 (25%)
> > Hash mismatch: 1 (25%)
> > Links with a known-bad host: 0
> >
> > Bad hosts:
> > rss.slashdot.org
> >
> > HTTPS OK urls:
> >
> https://b.scorecardresearch.com/p?c1=2&c2=6035546&c3=&c4=&c5=&c6=&c15=&cj=1
> > https://slashdot.org/
> >
> > HTTPS possible urls:
> >
> https://jlinks.industrybrains.com/jsct?sid=941&ct=slashdot_ros&num=3&layt=300x250imgad&fmt=simp
> >
> > HTTPS fail urls:
> > https://rss.slashdot.org/slashdot/slashdot
> >
> >
> > Verdict: This page IS NOT a candidate for *simple* domain-wide forced
> encryption, but may be a candidate for URL-rewriting or path-based forcing.
> More research is required.
>
>
> > _______________________________________________
> > HTTPS-everywhere mailing list
> > HTTPS-everywhere at mail1.eff.org
> > https://mail1.eff.org/mailman/listinfo/https-everywhere
>
>
> --
> Peter Eckersley pde at eff.org
> Senior Staff Technologist Tel +1 415 436 9333 x131
> Electronic Frontier Foundation Fax +1 415 436 9993
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.eff.org/pipermail/https-everywhere/attachments/20101110/e826899b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ssl_check2.pl
Type: application/octet-stream
Size: 11155 bytes
Desc: not available
URL: <http://lists.eff.org/pipermail/https-everywhere/attachments/20101110/e826899b/attachment.obj>
More information about the HTTPS-everywhere
mailing list