I have written before about how overzealous design of cryptographic protocols often results in their non-use. Protocol engineers are trained to be thorough and complete. They rankle at leaving in vulnerabilities, even against the most extreme threats. But the perfect is often the enemy of the good. None of the various protocols to encrypt E-mail have ever reached even a modicum of success in the public space. It's a very rare VoIP call (other than Skype) that is encrypted.

The two most successful encryption protocols in the public space are SSL/TLS (which provide the HTTPS system among other things) and Skype. At a level below that are some of the VPN applications and SSH.

TLS (the successor to SSL) is very widely deployed but still very rarely used. Only the most tiny fraction of web sessions are encrypted. Many sites don't support it at all. Some will accept HTTPS but immediately push you back to HTTP. In most cases, sites will have you log in via HTTPS so your password is secure, and then send you back to unencrypted HTTP, where anybody on the wireless network can watch all your traffic. It's a rare site that lets you conduct your entire series of web interactions entirely encrypted. This site fails in that regard. More common is the use of TLS for POP3 and IMAP sessions, both because it's easy, there is only one TCP session, and the set of users who access the server is a small and controlled set. The same is true with VPNs -- one session, and typically the users are all required by their employer to use the VPN, so it gets deployed. IPSec code exists in many systems, but is rarely used in stranger-to-stranger communications (or even friend-to-friend) due to the nightmares of key management.

TLS's complexity makes sense for "sessions" but has problems when you use it for transactions, such as web hits. Transactions want to be short. They consist of a request, and a response, and perhaps an ACK. Adding extra back and forths to negotiate encryption can double or triple the network cost of the transactions.

Skype became a huge success at encrypting because it is done with ZUI -- the user is not even aware of the crypto. It just happens. SSH takes an approach that is deliberately vulnerable to man-in-the-middle attacks on the first session in order to reduce the UI, and it has almost completely replaced unencrypted telnet among the command line crowd.

I write about this because now Google is finally doing an experiment to let people have their whole gmail session be encrypted with HTTPS. This is great news. But hidden in the great news is the fact that Google is evaluating the "cost" of doing this. There also may be some backlash if Google does this on web search, as it means that ordinary sites will stop getting to see the search query in the "Referer" field until they too switch to HTTPS and Google sends traffic to them over HTTPS. (That's because, for security reasons, the HTTPS design says that if I made a query encrypted, I don't want that query to be repeated in the clear when I follow a link to a non-encrypted site.) Many sites do a lot of log analysis to see what search terms are bringing in traffic, and may object when that goes away. Several things have stood in the way of deploying the encrypted web:

Cost of certificates

First of all, to use TLS without annoyance, you had to buy, and keep buying, a certificate from a seller on the trusted list of all major browsers. In the old days these cost hundreds of dollars per year, which large sites had no problem with but small sites balked at. Even today the hassle of getting a certificate and maintaining it scares away so many sites. In the ZUI approach, we need a way for sites to get, free and with very little UI, a low-security certificate good only for securing basic traffic.

One simple approach would be a certificate server which allowed any site to request a certificate and verify it owned the domain in question but putting a response to a challenge in a URL on that domain on a web server on a random port below 1024. The web server, when it is being installed, could conduct this exchange without the installing user having any involvement, and gain a low-security certificate proving that at least back then, the requester had control of the domain or IP. (There are some DNS cache poisoning gotchas to worry about here.) With such a certificate, the yellow lock would be shown as barely locked. We might call this a "Light Certificate." The startssl system offers free SSL Certs but not automatically and they will get an error in Internet Explorer.

One could even imagine a certificate which only certifies a given domain on a given IP. You would need to (automatically) get another certificate if you ever changed IPs. If you want to do complex stuff, such as round-robin DNS or Dynamic DNS, get a more general certificate. However, this would not solve the problem of local IPs behind NAT that can't be certified by remote CAs. It does mitigate the issue that somebody who takes over your machine temporarily could then get a cert to pretend to be it on the general internet, which is a problem with all low-authentication CA approaches.

On top of this, cheaper identity verified certificates would have been worthwhile for those sites willing to spend a little cash to get a stronger lock icon.

UI Cost of certificates

Another downside of running secured traffic is all the warnings that come in many browsers when you transition from a secured page to an unsecured one, or when a page contains secured text but includes unsecured images or other embedded items. These warnings have become the "alarm that constantly goes off" and is thus routinely ignored. We might want to not have these warnings when moving from a light certificate to unencrypted. Or make them less intrusive, such as a little animation as the security level changes, a change of colour in a toolbar background or something. Again, if the alarm is going to go off all the time it is not doing you any good.

Client authentication is another can of worms. For privacy reasons, clients must not identify themselves to any server who asks. But it is useful to be able to do this, and more secure and invisible to use public key encryption and digital signature when you need it than to use userids and passwords as we do today.

In addition, users have shown it takes a lot of prodding for them to get and install client side certificates. The more they can do with user generated certificates, or free certificates that can be requested by programs with minimal UI for the user, the better. Earlier, I wrote about the idea of authenticated actions as an alternative to logins and single sign-on.

Physical cost of the encryption

In the early days -- SSL was developed 15 years ago -- people thought seriously about the CPU cost of doing the ordinary symmetric encryption on the actual data. To do it properly, it was felt back then, required buying a card for your server which would do the encrypt, a real serious cost. Thanks to 15 years of Moore's law, you would think that this would no longer be an issue, but you will still hear some talk about it.

There is also resistance to the cost of the public key encryption (typically RSA) done in the initial handshake for TLS. RSA is much more expensive than ordinary encryption. But it is done only once, at the start of a session, and should not be standing in anybody's way. RSA uses large keys and large certificates, however, and people with bandwidth concerns (mostly for their users) have reason to object to it. To take a tiny transaction, such as the fetching of the lightweight Google home page (3kb in size) and make it involve tens of kilobytes is something one can still express some concern about, even today.

There is an answer to that, in elliptic curve cryptography, which is able to use much smaller keys and certificates. However, for a variety of reasons, some of them silly, it is less popular for these applications.

Cost of the handshakes

The biggest cost, one that Moore's law doesn't help at all, is the handshake to start a TLS based web fetch.

With regular HTTP, you send your unencrypted web URL request out, and the server answers back with the web page you asked for. Aside from the TCP SYN/ACK setup roundtrip there is just the basic request, response (and close).

With TLS, in the full version there is also a double round-trip handshake. After that the data can flow securely for another round trip. While Moore's law can make the calculations at either end be faster, and more bandwidth can reduce the cost of the large certificates and keys being bandied about, little can reduce the round trip times of handshakes to servers that are far away. Everything is going to start a bit slower.

No good fallbacks

Another curse was the use of two protocols HTTP and HTTPS. You can write a URL that points to one or the other, but there is no way to say, "Use HTTPS, but if that doesn't work fall back to HTTP." Right now the most common alternative is that you go to a site with HTTP, and then the site notices your browser supports HTTPS (they almost all do these days) and it redirects you to the https site for login. This means the initial request is sent unencrypted, and is even slower. Even though there are many sites that have reliably supported https for over a decade, it is very rare to see an external link use an HTTPS URL. It was drummed into us in the early days that the browser might not support HTTPS, and so such a URL might break things. Even today, when a web tool that can't handle HTTPS is the thing that should be considered broken, you will not find many external site links using it.

We could define extra attributes to put on links that say, "It's OK to fall back" but of course the older tools would not understand these attributes. We could have an attribute that says, "Really try https on this link if you are able" and that might help a bit better. Another useful attribute might imply, "There is no need to warn the user that this link goes to an unsecured web site, we know what we're doing." Leave the warnings for times when something has actually gone wrong, rather than for something planned. Attempting to enforce policy with warnings seems to work fairly rarely.

There has been some talk of using DNS to handle both fallbacks and fall "ups" to encourage encryption. For example, one could put an extra field in the DNS record for a domain (for example using a SRV field) to say that, "If you're getting a request to go to port 80 for HTTP here, instead use HTTPS." A browser that was aware of this would know to go secure and there is no need to change the target links. This also allows the introduction of newer, more efficient protocols without having to change any of the web's HTML (a huge plus) which can then be adopted as browsers and servers start supporting them. Security designers tend to disdain the "fall-up" concept where at any moment you can't be sure if your traffic will be encrypted or in-the-clear. Fear of this has resulted in most traffic being in-the-clear, rather than most traffic being encrypted.

With proper deployment of more security in DNS, it's also possible to put public keys and lists of supported protocols into DNS records, so that browsers can make their initial request encrypted using that key, and cut back on the handshakes. This way even the first request can be free of the extra handshakes.

Alternatives

It is possible to design a transaction protocol (ie. for web fetches) which is not nearly as costly in roundtrips. Particularly, if a client is able to remember the public key of the web sites it visits, there can be a non-handshake protocol, where the request consists of "Here, encrypted with our public key X, is my request, my generated symmetric key and optionally my own key and certificate" and the response is "Here, encrypted with the symmetric key you offered is the response" or "No, we don't accept that key or method any more, here's what we want to use" which requires another round trip and offers the full potential of the advanced protocols -- but only once per user.

It has a problem with compromised keys, however. But so do all the other systems. But for ordinary site encryption, which is 99% of your web hits, it would be fine. Sites could elect to opt out, and say, "don't use this, we want a full TLS handshake for any transaction to domains matching this pattern" to assure that the most complex systems are used for financial transactions or logins associated with them.

Of course, switching to a system like this would not be easy. SSL and TLS took time to get adopted, and they had a big adoption advantage -- there was no other way to do a secure web session, and big commercial sites were demanding something. Now that they have something that they can use for bank logins, there is much less pressure to do it better. And for ordinary web sites, just not much pressure at all to make their traffic secure.

There is an effort afoot to put an encryption handshake into the TCP handshake which died on the vine because most IETF people felt it was a layering violation.

There would in fact be a tremendous efficiency gain if web transactions could be made vastly simpler, bypassing even the usual rules of TCP. An ideal web transaction protocol might be done with a mixture of UDP and a kludged TCP. In such a system, the initial request would go out as UDP (encrypted using keys learned from DNS) and effectively being the SYN. The response could come back, and if it all fit in one packet, that would be it and an ACK/CLOSE would be sent in response. If it did not fit, the response would be the SYN/ACK and the rest would follow like regular TCP. This would commonly be the case for requests that will be known to fetch more components such as images keep-alive style. However, the short transactions would be just a few packets.

To get even more efficient, one could imagine a system where you don't yet know the DNS records of the host your going to talk to. In this case you should send the web request to the DNS server, and ask it to forward it on (via UDP) to the web server, which would then respond directly to you (telling you its authenticated DNS information for future reference.) Now that would be efficient, though a problem requiring proxies in the modern NAT world. Also you need a way to stop the DNS server from snooping on the initial request, which is difficult as you don't yet know anything about the target site or its keys. Identity based certificates could provide a cool solution to this, though they need trusted 3rd parties.