[HTTPS-Everywhere] Finding HSTS preloadable sites using HTTPS-Everywhere

[Søren Fuglede Jørgensen]

Here's an application of the HTTPS-Everywhere ruleset database that people here might find useful/curious/cute/worth knowing about/completely pointless:

A lot of users submit rulesets for sites that specify HSTS policies. While of course not completely useless, they do help somewhat less than non-HSTS rulesets do, and since some performance issues exist with HTTPS-Everywhere, I was curious about how big a part of the ruleset database is made up by such rulesets.

While trying to figure out how many rulesets we were talking about, I decided that a more interesting question was how many of these rulesets were actually already covered by the HSTS preload lists. Here are some rough numbers:

Total hosts checked:                          19613
Failed HTTPS connections:                     10910
Good connections with no HSTS:                 7028
HSTS supported:                                1675
- Preload header specified:                     279
- Can be included in Google's preload list*:    152
Google preload list size:                      3558
Overlap between our preloadables and Google's:  123
Hosts that can be added to preload lists:        29

* Google requires 'includeSubDomains' and a large HSTS expiration time.

Methodology and some notes: Since I was primarily curious in preloadability, only the top part of domains were checked (thus the large number of failed connections). Moreover, the redirect logic of the code was a bit too crude and so some part of the 29 newly preloadable hosts that were found are false positives (but it took like 10 hours to run the test so I'm not likely to redo it).

- Søren