Quite possibly! [ At least you still have your cat/dog/truck/wife right? ] <br><br>Digging deeper, it seems that some <a href="http://aX.twimg.com">aX.twimg.com</a> hosts support SSL while others do not. This makes troubleshooting... interesting. I don't have a good solution for this one just yet. One of the assumptions that I made when writing this script was that a host would either support or not support SSL connections reliably. Silly me! <br>
I suppose I'll have to add some support for multiple A (or AAAA?) records at some point. Feature creep, and all that, you know.<br><br>I did make a tweak to the script to allow it to examine HTTPS responses that were "close" but not quite identical, and determine the number of initially-similar bytes. (See attached) For me, the output of this feature looks like this:<br>
<div style="margin-left: 40px;"><span style="color: rgb(51, 51, 255);"> 29 sad. HTTPS hash does NOT match HTTP hash for <a href="https://twitter.com">https://twitter.com</a>.</span><br style="color: rgb(51, 51, 255);">
<span style="color: rgb(51, 51, 255);"> ... but the first 1742 / 45360 bytes are the same! Manual check required.</span><br style="color: rgb(51, 51, 255);"><span style="color: rgb(51, 51, 255);"> Here are 80 bytes from both strings, starting at offset 1732:</span><br style="color: rgb(51, 51, 255);">
<span style="color: rgb(51, 51, 255);"> http: ref="<a href="http://a1.twimg.com/a/1289433550/images/twitter_57.png">http://a1.twimg.com/a/1289433550/images/twitter_57.png</a>" rel="apple-touch-ic</span><br style="color: rgb(51, 51, 255);">
<span style="color: rgb(51, 51, 255);"> https: ref="<a href="https://s3.amazonaws.com/twitter_production/a/1289433550/images/twitter_57">https://s3.amazonaws.com/twitter_production/a/1289433550/images/twitter_57</a>.</span><br>
</div><br>Note that the starting byte of each listed string is 10 chars before the variance occurs. For sites with rotating ad banners, this may be an issue. I did mention that this was a QnD hack, right? I suppose I could add a CLI parameter to "fetch html content x times" and hope that, with a sufficiently large x, a pair of HTTP/HTTPS responses would match. This solution has O(n^2) complexity, which I would like to avoid if possible. Any suggestions?<br>
<br><br><br><div class="gmail_quote">On Wed, Nov 10, 2010 at 5:48 PM, Peter Eckersley <span dir="ltr"><<a href="mailto:pde@eff.org">pde@eff.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Whizz, this script is great but I'm wondering if it's still somewhat buggy...<br>
<br>
perl ./<a href="http://ssl_check2.pl" target="_blank">ssl_check2.pl</a> <a href="http://twitter.com" target="_blank">http://twitter.com</a><br>
<br>
Getting <a href="http://twitter.com" target="_blank">http://twitter.com</a> ...Done.<br>
Got 44511 bytes in 1 secs (44511 bytes / sec)<br>
Found 29 reference(s) to check.<br>
<br>
Checking reference URLs...<br>
1 zzz. HTTPS request timeout. Added <a href="http://a0.twimg.com" target="_blank">a0.twimg.com</a> to badhosts list.g<br>
2 Skipping url <a href="http://a0.twimg.com/a/1289339734/images/whatsnew/video-sample-ss.png" target="_blank">http://a0.twimg.com/a/1289339734/images/whatsnew/video-sample-ss.png</a> (known-bad host <a href="http://a0.twimg.com" target="_blank">a0.twimg.com</a>).<br>
3 Skipping url <a href="http://a0.twimg.com/a/1289339734/javascripts/widgets/widget.js?1289366423" target="_blank">http://a0.twimg.com/a/1289339734/javascripts/widgets/widget.js?1289366423</a> (known-bad host <a href="http://a0.twimg.com" target="_blank">a0.twimg.com</a>).<br>
4 Skipping url <a href="http://a0.twimg.com/profile_images/118608576/twitter_sc_logo_normal.jpg" target="_blank">http://a0.twimg.com/profile_images/118608576/twitter_sc_logo_normal.jpg</a> (known-bad host <a href="http://a0.twimg.com" target="_blank">a0.twimg.com</a>).<br>
5 Skipping url <a href="http://a0.twimg.com/profile_images/120242004/finaltwitter_normal.jpg" target="_blank">http://a0.twimg.com/profile_images/120242004/finaltwitter_normal.jpg</a> (known-bad host <a href="http://a0.twimg.com" target="_blank">a0.twimg.com</a>).<br>
6 Skipping url <a href="http://a0.twimg.com/profile_images/49918572/half-face-ice_normal.jpg" target="_blank">http://a0.twimg.com/profile_images/49918572/half-face-ice_normal.jpg</a> (known-bad host <a href="http://a0.twimg.com" target="_blank">a0.twimg.com</a>).<br>
7 zzz. HTTPS request timeout. Added <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a> to badhosts list.<br>
8 Skipping url <a href="http://a1.twimg.com/a/1289339734/images/twitter_57.png" target="_blank">http://a1.twimg.com/a/1289339734/images/twitter_57.png</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
9 Skipping url <a href="http://a1.twimg.com/a/1289339734/stylesheets/fronts.css?1289366423" target="_blank">http://a1.twimg.com/a/1289339734/stylesheets/fronts.css?1289366423</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
10 Skipping url <a href="http://a1.twimg.com/profile_images/220756397/afwd-twitter-logo_normal.gif" target="_blank">http://a1.twimg.com/profile_images/220756397/afwd-twitter-logo_normal.gif</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
11 Skipping url <a href="http://a1.twimg.com/profile_images/263029233/slide1_normal.jpg" target="_blank">http://a1.twimg.com/profile_images/263029233/slide1_normal.jpg</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
12 Skipping url <a href="http://a1.twimg.com/profile_images/381297805/mobile_normal.png" target="_blank">http://a1.twimg.com/profile_images/381297805/mobile_normal.png</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
13 Skipping url <a href="http://a1.twimg.com/profile_images/52564417/twitter_normal.jpg" target="_blank">http://a1.twimg.com/profile_images/52564417/twitter_normal.jpg</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
14 Skipping url <a href="http://a1.twimg.com/profile_images/555579649/steve_case_wsj_normal.jpg" target="_blank">http://a1.twimg.com/profile_images/555579649/steve_case_wsj_normal.jpg</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
15 Skipping url <a href="http://a1.twimg.com/profile_images/601329413/twitter_logo_normal.jpg" target="_blank">http://a1.twimg.com/profile_images/601329413/twitter_logo_normal.jpg</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
16 Skipping url <a href="http://a1.twimg.com/profile_images/670252813/136489main_pia04413-feature-browse_normal.jpg" target="_blank">http://a1.twimg.com/profile_images/670252813/136489main_pia04413-feature-browse_normal.jpg</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
17 Skipping url <a href="http://a1.twimg.com/profile_images/727884617/rainbow_normal.jpg" target="_blank">http://a1.twimg.com/profile_images/727884617/rainbow_normal.jpg</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
18 Skipping url <a href="http://a1.twimg.com/profile_images/866556637/teatime__normal.jpg" target="_blank">http://a1.twimg.com/profile_images/866556637/teatime__normal.jpg</a> (known-bad host <a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a>).<br>
19 zzz. HTTPS request timeout. Added <a href="http://a2.twimg.com" target="_blank">a2.twimg.com</a> to badhosts list.<br>
20 Skipping url <a href="http://a2.twimg.com/profile_images/1114845454/daily-parent-tip_normal.png" target="_blank">http://a2.twimg.com/profile_images/1114845454/daily-parent-tip_normal.png</a> (known-bad host <a href="http://a2.twimg.com" target="_blank">a2.twimg.com</a>).<br>
21 Skipping url <a href="http://a2.twimg.com/profile_images/264983646/2008_author_shot_copy_normal.jpg" target="_blank">http://a2.twimg.com/profile_images/264983646/2008_author_shot_copy_normal.jpg</a> (known-bad host <a href="http://a2.twimg.com" target="_blank">a2.twimg.com</a>).<br>
22 Skipping url <a href="http://a2.twimg.com/profile_images/458966890/twitterprofilephoto_normal.jpg" target="_blank">http://a2.twimg.com/profile_images/458966890/twitterprofilephoto_normal.jpg</a> (known-bad host <a href="http://a2.twimg.com" target="_blank">a2.twimg.com</a>).<br>
23 Skipping url <a href="http://a2.twimg.com/profile_images/544732942/logorgb2_justh_normal.png" target="_blank">http://a2.twimg.com/profile_images/544732942/logorgb2_justh_normal.png</a> (known-bad host <a href="http://a2.twimg.com" target="_blank">a2.twimg.com</a>).<br>
24 Skipping url <a href="http://a2.twimg.com/profile_images/91810842/ai_250x250_twit_normal.jpg" target="_blank">http://a2.twimg.com/profile_images/91810842/ai_250x250_twit_normal.jpg</a> (known-bad host <a href="http://a2.twimg.com" target="_blank">a2.twimg.com</a>).<br>
25 zzz. HTTPS request timeout. Added <a href="http://a3.twimg.com" target="_blank">a3.twimg.com</a> to badhosts list.g<br>
26 Skipping url <a href="http://a3.twimg.com/profile_images/291571823/unknown-6_normal.jpeg" target="_blank">http://a3.twimg.com/profile_images/291571823/unknown-6_normal.jpeg</a> (known-bad host <a href="http://a3.twimg.com" target="_blank">a3.twimg.com</a>).<br>
27 Skipping url <a href="http://a3.twimg.com/profile_images/748445671/shopaneer-002-36x36_normal.jpg" target="_blank">http://a3.twimg.com/profile_images/748445671/shopaneer-002-36x36_normal.jpg</a> (known-bad host <a href="http://a3.twimg.com" target="_blank">a3.twimg.com</a>).<br>
28 zzz. HTTPS request timeout. Added <a href="http://ajax.googleapis.com" target="_blank">ajax.googleapis.com</a> to badhosts list.<br>
29 zzz. HTTPS request timeout. Added <a href="http://twitter.com" target="_blank">twitter.com</a> to badhosts list.<br>
<br>
<br>
<br>
<br>
Results:<br>
Total links: 29<br>
Working links: 0<br>
Non-Working links: 29 (100%)<br>
HTTP request fail: 0<br>
HTTPS request fail: 6 (20.68%)<br>
Hash mismatch: 0<br>
Links with a known-bad host: 23 (79.31%)<br>
<br>
Bad hosts:<br>
<a href="http://a0.twimg.com" target="_blank">a0.twimg.com</a><br>
<a href="http://a1.twimg.com" target="_blank">a1.twimg.com</a><br>
<a href="http://a2.twimg.com" target="_blank">a2.twimg.com</a><br>
<a href="http://a3.twimg.com" target="_blank">a3.twimg.com</a><br>
<a href="http://ajax.googleapis.com" target="_blank">ajax.googleapis.com</a><br>
<a href="http://twitter.com" target="_blank">twitter.com</a><br>
<br>
HTTPS fail urls:<br>
<br>
<a href="https://a0.twimg.com/a/1289339734/images/fronts/logo_withbird_home.png" target="_blank">https://a0.twimg.com/a/1289339734/images/fronts/logo_withbird_home.png</a><br>
<a href="https://a1.twimg.com/a/1289339734/images/favicon.ico" target="_blank">https://a1.twimg.com/a/1289339734/images/favicon.ico</a><br>
<a href="https://a2.twimg.com/a/1289339734/javascripts/fronts.js" target="_blank">https://a2.twimg.com/a/1289339734/javascripts/fronts.js</a><br>
<a href="https://a3.twimg.com/profile_images/1148176527/1110-twitter_normal.jpg" target="_blank">https://a3.twimg.com/profile_images/1148176527/1110-twitter_normal.jpg</a><br>
<a href="https://ajax.googleapis.com/ajax/libs/jquery/1.3.0/jquery.min.js" target="_blank">https://ajax.googleapis.com/ajax/libs/jquery/1.3.0/jquery.min.js</a><br>
<a href="https://twitter.com" target="_blank">https://twitter.com</a><br>
<br>
<br>
Verdict: Verdict: This page IS NOT a candidate for *simple* domain-wide forced encryption, but may be a candidate for URL-rewriting or path-based forcing. More research is required.<br>
<div class="im"><br>
<br>
On Thu, Nov 04, 2010 at 07:06:51PM -0700, Whizz Mo wrote:<br>
> In case no one has bothered to write this already, attached is a simple perl<br>
> script to check an http url for https compatibility.<br>
> This script:<br>
><br>
</div><div class="im">> 1. fetches the http url<br>
</div>> 2. parses it for fetchable links (images, scripts, frames, other hrefs)<br>
> 3. fetches the fetchable links in http and https<br>
> 4. compares the http and https responses.<br>
> 5. prints report. (See attached text file for a sample)<br>
<div class="im">><br>
> Usage:<br>
> perl <a href="http://ssl_check2.pl" target="_blank">ssl_check2.pl</a> <a href="http://somesite.com/" target="_blank">http://somesite.com/</a><br>
><br>
> Output is currently command-line only. (Do not run this script from the<br>
> Windows Run Command box.)<br>
><br>
> Caveats:<br>
><br>
> - This is very quick and dirty code, and should be considered<br>
> "experimental". May format your hard drive, kick your dog, steal your<br>
> truck, and run off with your wife.<br>
> - This script will parse a frame url, but will not (recursively) parse<br>
> the content of the frame. [To-do list]<br>
><br>
><br>
><br>
><br>
> Thanks,<br>
> Whizz<br>
<br>
</div>> Getting <a href="http://slashdot.org/" target="_blank">http://slashdot.org/</a> ...Done.<br>
> Got 117515 bytes in 1 secs (117515 bytes / sec)<br>
> Found 4 reference(s) to check.<br>
><br>
> Checking reference URLs...<br>
> 1 YAY! HTTPS appears ok for <a href="https://b.scorecardresearch.com/p?c1=2&c2=6035546&c3=&c4=&c5=&c6=&c15=&cj=1" target="_blank">https://b.scorecardresearch.com/p?c1=2&c2=6035546&c3=&c4=&c5=&c6=&c15=&cj=1</a> .<br>
> 2 sad. HTTPS hash does NOT match HTTP hash for <a href="https://jlinks.industrybrains.com/jsct?sid=941&ct=slashdot_ros&num=3&layt=300x250imgad&fmt=simp" target="_blank">https://jlinks.industrybrains.com/jsct?sid=941&ct=slashdot_ros&num=3&layt=300x250imgad&fmt=simp</a>.<br>
> ... but the first 1058 / 8670 bytes are the same! Manual check required.<br>
> Here are 80 bytes from both strings, starting at offset 1048:<br>
> http: mp;lid=682045&cid=151113&pr=2&tstamp=20101104214514&iip=260.309.<br>
> https: mp;lid=685533&cid=151895&pr=2&tstamp=20101104214515&iip=260.309.<br>
> 3 zzz. HTTPS request timeout. Added <a href="http://rss.slashdot.org" target="_blank">rss.slashdot.org</a> to badhosts list.<br>
> 4 YAY! HTTPS appears ok for <a href="https://slashdot.org/" target="_blank">https://slashdot.org/</a> .<br>
><br>
><br>
><br>
><br>
> Results:<br>
> Total links: 4<br>
> Working links: 2 (50%)<br>
> Semi-working links: 1 (25%) [See "HTTPS possible urls" below]<br>
> Non-Working links: 1 (25%)<br>
> HTTP request fail: 0<br>
> HTTPS request fail: 1 (25%)<br>
> Hash mismatch: 1 (25%)<br>
> Links with a known-bad host: 0<br>
><br>
> Bad hosts:<br>
> <a href="http://rss.slashdot.org" target="_blank">rss.slashdot.org</a><br>
><br>
> HTTPS OK urls:<br>
> <a href="https://b.scorecardresearch.com/p?c1=2&c2=6035546&c3=&c4=&c5=&c6=&c15=&cj=1" target="_blank">https://b.scorecardresearch.com/p?c1=2&c2=6035546&c3=&c4=&c5=&c6=&c15=&cj=1</a><br>
> <a href="https://slashdot.org/" target="_blank">https://slashdot.org/</a><br>
><br>
> HTTPS possible urls:<br>
> <a href="https://jlinks.industrybrains.com/jsct?sid=941&ct=slashdot_ros&num=3&layt=300x250imgad&fmt=simp" target="_blank">https://jlinks.industrybrains.com/jsct?sid=941&ct=slashdot_ros&num=3&layt=300x250imgad&fmt=simp</a><br>
><br>
> HTTPS fail urls:<br>
> <a href="https://rss.slashdot.org/slashdot/slashdot" target="_blank">https://rss.slashdot.org/slashdot/slashdot</a><br>
><br>
><br>
> Verdict: This page IS NOT a candidate for *simple* domain-wide forced encryption, but may be a candidate for URL-rewriting or path-based forcing. More research is required.<br>
<div><div></div><div class="h5"><br>
<br>
> _______________________________________________<br>
> HTTPS-everywhere mailing list<br>
> <a href="mailto:HTTPS-everywhere@mail1.eff.org">HTTPS-everywhere@mail1.eff.org</a><br>
> <a href="https://mail1.eff.org/mailman/listinfo/https-everywhere" target="_blank">https://mail1.eff.org/mailman/listinfo/https-everywhere</a><br>
<br>
<br>
--<br>
Peter Eckersley <a href="mailto:pde@eff.org">pde@eff.org</a><br>
Senior Staff Technologist Tel +1 415 436 9333 x131<br>
Electronic Frontier Foundation Fax +1 415 436 9993<br>
</div></div></blockquote></div><br>