[HTTPS-Everywhere] Stupid Perl Tricks: ssl_check2.pl

Wed Nov 10 17:48:59 PST 2010

Whizz, this script is great but I'm wondering if it's still somewhat buggy... 

perl ./ssl_check2.pl http://twitter.com

Getting http://twitter.com ...Done.
  Got 44511 bytes in 1 secs (44511 bytes / sec)
  Found 29 reference(s) to check.

Checking reference URLs...
  1    zzz.  HTTPS request timeout.  Added a0.twimg.com to badhosts list.g
  2    Skipping url http://a0.twimg.com/a/1289339734/images/whatsnew/video-sample-ss.png (known-bad host a0.twimg.com).
  3    Skipping url http://a0.twimg.com/a/1289339734/javascripts/widgets/widget.js?1289366423 (known-bad host a0.twimg.com).
  4    Skipping url http://a0.twimg.com/profile_images/118608576/twitter_sc_logo_normal.jpg (known-bad host a0.twimg.com).
  5    Skipping url http://a0.twimg.com/profile_images/120242004/finaltwitter_normal.jpg (known-bad host a0.twimg.com).
  6    Skipping url http://a0.twimg.com/profile_images/49918572/half-face-ice_normal.jpg (known-bad host a0.twimg.com).
  7    zzz.  HTTPS request timeout.  Added a1.twimg.com to badhosts list.
  8    Skipping url http://a1.twimg.com/a/1289339734/images/twitter_57.png (known-bad host a1.twimg.com).
  9    Skipping url http://a1.twimg.com/a/1289339734/stylesheets/fronts.css?1289366423 (known-bad host a1.twimg.com).
  10    Skipping url http://a1.twimg.com/profile_images/220756397/afwd-twitter-logo_normal.gif (known-bad host a1.twimg.com).
  11    Skipping url http://a1.twimg.com/profile_images/263029233/slide1_normal.jpg (known-bad host a1.twimg.com).
  12    Skipping url http://a1.twimg.com/profile_images/381297805/mobile_normal.png (known-bad host a1.twimg.com).
  13    Skipping url http://a1.twimg.com/profile_images/52564417/twitter_normal.jpg (known-bad host a1.twimg.com).
  14    Skipping url http://a1.twimg.com/profile_images/555579649/steve_case_wsj_normal.jpg (known-bad host a1.twimg.com).
  15    Skipping url http://a1.twimg.com/profile_images/601329413/twitter_logo_normal.jpg (known-bad host a1.twimg.com).
  16    Skipping url http://a1.twimg.com/profile_images/670252813/136489main_pia04413-feature-browse_normal.jpg (known-bad host a1.twimg.com).
  17    Skipping url http://a1.twimg.com/profile_images/727884617/rainbow_normal.jpg (known-bad host a1.twimg.com).
  18    Skipping url http://a1.twimg.com/profile_images/866556637/teatime__normal.jpg (known-bad host a1.twimg.com).
  19    zzz.  HTTPS request timeout.  Added a2.twimg.com to badhosts list.
  20    Skipping url http://a2.twimg.com/profile_images/1114845454/daily-parent-tip_normal.png (known-bad host a2.twimg.com).
  21    Skipping url http://a2.twimg.com/profile_images/264983646/2008_author_shot_copy_normal.jpg (known-bad host a2.twimg.com).
  22    Skipping url http://a2.twimg.com/profile_images/458966890/twitterprofilephoto_normal.jpg (known-bad host a2.twimg.com).
  23    Skipping url http://a2.twimg.com/profile_images/544732942/logorgb2_justh_normal.png (known-bad host a2.twimg.com).
  24    Skipping url http://a2.twimg.com/profile_images/91810842/ai_250x250_twit_normal.jpg (known-bad host a2.twimg.com).
  25    zzz.  HTTPS request timeout.  Added a3.twimg.com to badhosts list.g
  26    Skipping url http://a3.twimg.com/profile_images/291571823/unknown-6_normal.jpeg (known-bad host a3.twimg.com).
  27    Skipping url http://a3.twimg.com/profile_images/748445671/shopaneer-002-36x36_normal.jpg (known-bad host a3.twimg.com).
  28    zzz.  HTTPS request timeout.  Added ajax.googleapis.com to badhosts list.
  29    zzz.  HTTPS request timeout.  Added twitter.com to badhosts list.

Results:
	Total links: 29
	Working links: 0 
	Non-Working links: 29 (100%)
	  HTTP request fail: 0
	  HTTPS request fail: 6 (20.68%)
	  Hash mismatch: 0
	  Links with a known-bad host: 23 (79.31%)

	Bad hosts:
	  a0.twimg.com
	  a1.twimg.com
	  a2.twimg.com
	  a3.twimg.com
	  ajax.googleapis.com
	  twitter.com

	HTTPS fail urls:

	  https://a0.twimg.com/a/1289339734/images/fronts/logo_withbird_home.png
	  https://a1.twimg.com/a/1289339734/images/favicon.ico
	  https://a2.twimg.com/a/1289339734/javascripts/fronts.js
	  https://a3.twimg.com/profile_images/1148176527/1110-twitter_normal.jpg
	  https://ajax.googleapis.com/ajax/libs/jquery/1.3.0/jquery.min.js
	  https://twitter.com

Verdict: Verdict: This page IS NOT a candidate for *simple* domain-wide forced encryption, but may be a candidate for URL-rewriting or path-based forcing.  More research is required.

On Thu, Nov 04, 2010 at 07:06:51PM -0700, Whizz Mo wrote:
> In case no one has bothered to write this already, attached is a simple perl
> script to check an http url for https compatibility.
> This script:
> 
>    1. fetches the http url
>    2. parses it for fetchable links (images, scripts, frames, other hrefs)
>    3. fetches the fetchable links in http and https
>    4. compares the http and https responses.
>    5. prints report.      (See attached text file for a sample)
> 
> Usage:
>        perl ssl_check2.pl http://somesite.com/
> 
> Output is currently command-line only.   (Do not run this script from the
> Windows Run Command box.)
> 
> Caveats:
> 
>    - This is very quick and dirty code, and should be considered
>    "experimental".  May format your hard drive, kick your dog, steal your
>    truck, and run off with your wife.
>    - This script will parse a frame url, but will not (recursively) parse
>    the content of the frame.  [To-do list]
> 
> 
> 
> 
> Thanks,
> Whizz

> Getting http://slashdot.org/ ...Done.
>   Got 117515 bytes in 1 secs (117515 bytes / sec)
>   Found 4 reference(s) to check.
> 
> Checking reference URLs...
>   1    YAY!  HTTPS appears ok for https://b.scorecardresearch.com/p?c1=2&c2=6035546&c3=&c4=&c5=&c6=&c15=&cj=1 .
>   2    sad.  HTTPS hash does NOT match HTTP hash for https://jlinks.industrybrains.com/jsct?sid=941&ct=slashdot_ros&num=3&layt=300x250imgad&fmt=simp.
>          ... but the first 1058 / 8670 bytes are the same!  Manual check required.
>         Here are 80 bytes from both strings, starting at offset 1048:
>                 http:  mp;lid=682045&cid=151113&pr=2&tstamp=20101104214514&iip=260.309.
>                 https:  mp;lid=685533&cid=151895&pr=2&tstamp=20101104214515&iip=260.309.
>   3    zzz.  HTTPS request timeout.  Added rss.slashdot.org to badhosts list.
>   4    YAY!  HTTPS appears ok for https://slashdot.org/ .
> 
> 
> 
> 
> Results:
>         Total links: 4
>         Working links: 2 (50%)
>         Semi-working links: 1 (25%) [See "HTTPS possible urls" below]
>         Non-Working links: 1 (25%)
>           HTTP request fail: 0
>           HTTPS request fail: 1 (25%)
>           Hash mismatch: 1 (25%)
>           Links with a known-bad host: 0
> 
>         Bad hosts:
>           rss.slashdot.org
> 
>         HTTPS OK urls:
>           https://b.scorecardresearch.com/p?c1=2&c2=6035546&c3=&c4=&c5=&c6=&c15=&cj=1
>           https://slashdot.org/
> 
>         HTTPS possible urls:
>           https://jlinks.industrybrains.com/jsct?sid=941&ct=slashdot_ros&num=3&layt=300x250imgad&fmt=simp
> 
>         HTTPS fail urls:
>           https://rss.slashdot.org/slashdot/slashdot
> 
> 
> Verdict: This page IS NOT a candidate for *simple* domain-wide forced encryption, but may be a candidate for URL-rewriting or path-based forcing.  More research is required.

> _______________________________________________
> HTTPS-everywhere mailing list
> HTTPS-everywhere at mail1.eff.org
> https://mail1.eff.org/mailman/listinfo/https-everywhere

-- 
Peter Eckersley                            pde at eff.org
Senior Staff Technologist         Tel  +1 415 436 9333 x131
Electronic Frontier Foundation    Fax  +1 415 436 9993