Submitted by brad on Wed, 2010-03-24 18:02.
Today an interesting paper (written with the assistance of the EFF) was released. The authors have found evidence that governments are compromising trusted “certificate authorities” by issuing warrants to them, compelling them to create a false certificate for a site whose encrypted traffic they want to snoop on.
That’s just one of the many ways in which web traffic is highly insecure. The biggest reason, though, is that the vast majority of all web traffic takes place “in the clear” with no encryption at all. This happens because SSL/TLS, the “https” system is hard to set up, hard to use, considered expensive and subject to many false-alarm warnings. The tendency of security professionals to deprecate anything but perfect security often leaves us with no security at all. My philosophy is different. To paraphrase Einstein:
Ordinary traffic should be made as secure as can be made easy to use, but no more secure
In this vein, I have prepared a new article on how to make the web much more secure, and it makes sense to release it today in light of the newly published threat. My approach, which calls for new browser behaviour and some optional new practices for sites, calls for the following:
- Make TLS more lightweight so that nobody is bothered by the cost of it
- Automatic provisioning (Zero UI) for self-signed certificates for domains and IPs.
- A different meaning for the lock icon: Strong (Locked), Ordinary (no icon) and in-the-clear (unlocked).
- A new philosophy of browser warnings with a focus on real threats and on changes in security, rather than static states deemed insecure.
- A means so sites can provide a file with advisories for browsers about what warnings make sense at this site.
There is one goal in mind here: The web must become encrypted by default, with no effort on the part of site operators and users, and false positive warnings that go off too frequently and make security poor and hard to use must be eliminated.
If you have interest in browser design and security policy I welcome your comments on A new way to secure the web.
Submitted by brad on Tue, 2010-01-26 05:28.
I’m at DLD in Munich, and going to Davos tomorrow. While at DLD I made a brief mention during a panel on identity and tracking of my concept of the privacy dangers of the AIs of the future, which are able to extract things from recorded data (like faces) that we can’t do today.
I mentioned a new idea, however, which is a search engine which focuses on the negative, because though advanced algorithms it can tell the difference between positive and negative content.
We’re quite interested in dirt. Every eBay user who looks at a seller’s feedback would like to see only the negative comments, as the positive ones tell almost no information. eBay doesn’t want to show this, they want people to see eBay sellers as positive and to bid.
But a lot of the time if we are investigating a company we might do business with or even a person, we want to focus on the negative. A company with few complaints is of interest to us. AI software will exist to find such complaints, and possibly even to do things like understand photos and know which ones might be a source of embarrassment, or read on postings on message boards and tell which ones are damning. This is hard to do well today, but will change over time.
This will have deep consequences to concepts of reputation. Those with a big online presence certainly have bad stuff written by or about them out there. Normally, however, it is buried in the large volume of stuff, and doesn’t get high search engine rankings. However, our human thirst for gossip and dirt will result in some search engines will push it to the top. In addition, there will be those wanting to game this with deliberate libel of their enemies and competitors. Today they can do this but their libels will be hidden in the large volume of information.
Some have proposed that in the future it will be necessary to pay a service to libel you, and spread lots of false material that buries and discredits any libel left by enemies (as well as true negative comments.) The AIs may be able to spot the difference, but that’s an arms race which can’t easily be predicted.
It is likely that all the bad in our lives will haunt us even more than we already fear. Efforts by some countries to pass laws which let people delete alleged libels will not work, and may bring even more attention to the materials. While you might be able to remove your tag from a photo on facebook, once that photo makes it into a system that can do face recognition the tag will come back and do so in ways beyond your control.
Submitted by brad on Thu, 2010-01-21 16:28.
These days it is getting very common to make videos of presentations, and even to do live streams of them. And most of these presentations have slides in Powerpoint or Keynote or whatever. But this always sucks, because the camera operator — if there is one — never moves between the speaker and the slide the way I want. You can’t please everybody of course.
In the proprietary “web meeting” space there are several tools that will let people do a video presentation and sync it with slides, ideally by pre-transmitting the slide deck so it is rendered in full resolution at the other end, along with the video. In this industry there are also some video players where you can seek along in the video and it automatically seeks within the slides. This can be a bit complex if the slides try to do funny animations but it can be done.
Obviously it would be nice to see a flash player that understands it is playing a video and also playing slides (even video of slides, though it would be better to do it in higher quality since it isn’t usually full motion video.) Sites like youtube could support it. However, getting the synchronization requires that you have a program on the presenting computer, which you may not readily control.
One simple idea would be a button the camera operator could push to say “Copy this frame to the slide window.” Then the camera would, when there is a new slide, move or switch over there, and the button would be pushed, and the camera could go immediately back to the speaker. Usually though the camera crew has access to the projector feed and would not need to actually point a camera, in fact some systems “switch” to the slides by just changing the video feed. A program which sends the projector feed with huge compresion (in other words, an I frame for any slide change and nothing after) would also work well. No need to send all the fancy transitions.
But it would be good to send the slides not as mpeg, but as PNG style, to be sharper if you can’t get access to the slides themselves. I want a free tool, so I can’t ask for the world, yet, but even something as basic as this would make my watching of remote presentations and talks much better. And it would make people watching my talks have a better time too — a dozen or so of them are out on the web.
I’m in O’Hare waiting to fly to Munich for DLD. More details to come.
Submitted by brad on Fri, 2010-01-15 17:50.
There’s a phenomenon we’re seeing more and more often. A company screws over a customer, but this customer now has a means to reach a large audience through the internet, and as a result it becomes a PR disaster for the company. The most famous case recently was United Breaks Guitars where Nova Scotia musician David Carroll had his luggage mistreated and didn’t get good service, so he wrote a funny song and music video about it. 7 million views later, a lot of damage was done to United Airlines’ reputation.
I’ve done this myself to companies who refuse to fix things. I will write a page about the incident sometimes, and due to my high google pagerank, the page will show up high. Do a Google search for Qwest Long Distance and you’ll see the first hit is Qwest, and the 2nd is my boring but frustrating story of bad service. I’m not the only one to have done this. Over 200 people per month visit that page — which has been up for almost a decade — and you have to assume they have lost more business than it would have cost to make things right.
Now I think all good companies should make things right whenever they can to show that the errors are rare enough that they can afford to go the extra mile and fix them. If you won’t fix them, it means you must have a lot of them.
However, companies are soon going to realize that there are a whole raft of “minor celebrities” like David Carroll and even myself who can do far more damage than they can tolerate. Companies have always given top notch service to A-list celebrities, and even to B list. Not just gift bags at the Oscars. When I was kid, my father was A-list for a time in Canada, and that meant that when he got on a plane with a coach ticket, the flight attendant escorted him to first class. That was in the days before first class was always full due to upgrades, of course.
But there are tens of thousands, maybe hundreds of thousands of people who can be a risk for a company if they piss us off. All bloggers with a decent audience (and even some who have an audience that includes the A-list bloggers.) People with high search engine rank. People who can simply write well to get their story out there — in particular people who are good at making a story funny and entertaining. And of course, musicians and people who are good at video editing and producing viral videos. Perhaps them most of all.
So I predict that before long services will spring up to enumerate these D-list and E-list celebrities and potential celebs. Everybody will get graded. And a flag will show up in the customer service computer for the top few percentiles saying, “this one is an influencer.” It will say, “you are authorized, though you are just a script monkey customer rep, to do more for this customer.” Or you might just be direct right to a more powerful rep. This “long tail elite” may just start getting better service and even better deals, so long as they identify themselves first.
Companies have done this for some time based on how good a customer you are, ie. how much you spend. If you are a big spending customer, you get the magic 800 number or just get routed to the better service due to your frequent flyer number or even caller-ID. But I’m talking about doing this not just for those who spend a lot, but for those who influence a lot of spending — or could influence it in a negative way.
And of course they are working hard to make us identify ourselves in every transaction, just not yet for this. People who review products for a living will need to be sure they are anonymous when they buy and ask for service. But oddly, negative reviews from people who review stuff for a living are becoming less important than the horror story from the negative guy. Since most product reviewers at magazines are unwilling to go through the horrors of real customer service, they call the PR flacks and get top-rated service, and then explain in the review that they did this (if they are honest.)
If you’re not in the long-tail elite, this is all a bad sign. You’ll never get much satisfaction, and the number of horror stories on the net will go down below what the true level should be. Of course you will be able to join the long tail elite if you want to, since I am sure those who track it will note the names of people who regularly show up on consumer complaint message boards that have high readership or rank. But that’s a lot of work.
It doesn’t really do a lot of good for the rest of the world if perks are given to the long tail elite. Better just for companies to get good enough that they make mistakes rarely, and thus can afford to go the extra distance to fix them when it happens.
Submitted by brad on Tue, 2009-12-15 16:46.
I think URL shorteners are are a curse, but thanks to Twitter they are growing vastly in use. If you don’t know, URL shorteners are sites that will generate a compact encoded URL for you to turn a very long link into a short one that’s easier to cut and paste, and in particular these days, one that fits in the 140 character constraint on Twitter.
I understand the attraction, and not just on twitter. Some sites generate hugely long URLs which fold over many lines if put in text files or entered for display in comments and other locations. The result, though, is that you can no longer determine where the link will take you from the URL. This hurts the UI of the web, and makes it possible to fool people into going to attack sites or Rick Astley videos. Because of this, some better twitter clients re-expand the shortened URLs when displaying on a larger screen.
Anyway, here’s an idea for the Twitter clients and URL shorteners, if they must be used. In a tweet, figure out how much room there is to put the compacted URL, and work with a shortener that will let you generate a URL of exactly that length. And if that length has some room, try to put in some elements from the original URL so I can see them. For example, you can probably fit the domain name, especially if you strip off the “www.” from it (in the visible part, not in the real URL.) Try to leave as many things that look like real words, and strip things that look like character encoded binary codes and numbers. Of course, in the end you’ll need something to make the short URL unique, but not that much. Of course, if there already is a URL created for the target, re-use that.
Google just did its own URL shortener. I’m not quite sure what the motives of URL shortener sites are. While sometimes I see redirects that pause at the intermediate site, nobody wants that and so few ever use such sites. The search engines must have started ignoring URL redirect sites when it comes to pagerank long ago. They take donations and run ads on the pages where people create the tiny URLs, but when it comes to ones used on Twitter, these are almost all automatically generated, so the user never sees the site.
Submitted by brad on Tue, 2009-11-17 16:18.
(Update: I had a formatting error in the original posting, this has been fixed.)
A few weeks ago when I wrote about the non deployment of SSL I touched on an old idea I had to make web transactions vastly more efficient. I recently read about Google’s proposed SPDY protocol which goes in a completely opposite direction, attempting to solve the problem of large numbers of parallel requests to a web server by multiplexing them all in a single streaming protocol that works inside a TCP session.
While calling attention to that, let me outline what I think would be the fastest way to do very simple web transactions. It may be that such simple transactions are no longer common, but it’s worth considering.
Today the way this works is pretty complex:
- You do a DNS request for www.example.com via a UDP request to your DNS server. In the pure case this also means first asking where “.com” is but your DNS server almost surely knows that. Instead, a UDP request is sent to the “.com” master server.
- The “.com” master server returns with the address of the server for example.com.
- You send a DNS request to the example.com server, asking where “www.example.com is.”
- The example.com DNS server sends a UDP response back with the IP address of www.example.com
- You open a TCP session to that address. First, you send a “SYN” packet.
- The site responds with a SYN/ACK packet.
- You respond to the SYN/ACK with an ACK packet. You also send the packet with your HTTP “GET” reqequest for “/page.html.” This is a distinct packet but there is no roundtrip so this can be viewed as one step. You may also close off your sending with a FIN packet.
- The site sends back data with the contents of the page. If the page is short it may come in one packet. If it is long, there may be several packets.
- There will also be acknowledgement packets as the multiple data packets arrive in each direction. You will send at least one ACK.
The other server will ACK your FIN.
- The remote server will close the session with a FIN packet.
- You will ACK the FIN packet.
You may not be familiar with all this, but the main thing to understand is that there are a lot of roundtrips going on. If the servers are far away and the time to transmit is long, it can take a long time for all these round trips.
It gets worse when you want to set up a secure, encrypted connection using TLS/SSL. On top of all the TCP, there are additional handshakes for the encryption. For full security, you must encrypt before you send the GET because the contents of the URL name should be kept encrypted.
A simple alternative
Consider a protocol for simple transactions where the DNS server plays a role, and short transactions use UDP. I am going to call this the “Web Transaction Protocol” or WTP. (There is a WAP variant called that but WAP is fading.)
- You send, via a UDP packet, not just a DNS request but your full GET request to the DNS server you know about, either for .com or for example.com. You also include an IP and port to which responses to the request can be sent.
- The DNS server, which knows where the target machine is (or next level DNS server) forwards the full GET request for you to that server. It also sends back the normal DNS answer to you via UDP, including a flag to say it forwarded the request for you (or that it refused to, which is the default for servers that don’t even know about this.) It is important to note that quite commonly, the DNS server for example.com and the www.example.com web server will be on the same LAN, or even be the same machine, so there is no hop time involved.
- The web server, receiving your request, considers the size and complexity of the response. If the response is short and simple, it sends it in one UDP packet, though possibly more than one, to your specified address. If no ACK is received in reasonable time, send it again a few times until you get one.
- When you receive the response, you send an ACK back via UDP. You’re done.
The above transaction would take place incredibly fast compared to the standard approach. If you know the DNS server for example.com, it will usually mean a single packet to that server, and a single packet coming back — one round trip — to get your answer. If you only know the server for .com, it would mean a single packet to the .com server which is forwarded to the example.com server for you. Since the master servers tend to be in the “center” of the network and are multiplied out so there is one near you, this is not much more than a single round trip. read more »
Submitted by brad on Wed, 2009-10-28 22:03.
I just returned from Jeff Pulver’s “140 Characters” conference in L.A. which was about Twitter. I asked many people if they get Twitter — not if they understand how it’s useful, but why it is such a hot item, and whether it deserves to be, with billion dollar valuations and many talking about it as the most important platform.
Some suggested Twitter is not as big as it appears, with a larger churn than expected and some plateau appearing in new users. Others think it is still shooting for the moon.
The first value in twitter I found was as a broadcast SMS. While I would not text all my friends when I go to a restaurant or a club, having a way so that they will easily know that (and might join me) is valuable. Other services have tried to do things like this but Twitter is the one that succeeded in spite of not being aimed at any specific application like this.
This explains the secret of Twitter. By being simple (and forcing brevity) it was able to be universal. By being more universal it could more easily attain critical mass within groups of friends. While an app dedicated to some social or location based application might do it better, it needs to get a critical mass of friends using it to work. Once Twitter got that mass, it had a leg up at being that platform.
At first, people wondered if Twitter’s simplicity (and requirement for brevity) was a bug or a feature. It definitely seems to have worked as a feature. By keeping things short, Twitter makes is less scary to follow people. It’s hard for me to get new subscribers to this blog, because subscribing to the blog means you will see my moderately long posts every day or two, and that’s an investment in reading. To subscribe to somebody’s Twitter feed is no big commitment. Thus people can get a million followers there, when no blog has that. In addition, the brevity makes it a good match for the mobile phone, which is the primary way people use Twitter. (Though usually the smart phone, not the old SMS way.)
And yet it is hard not to be frustrated at Twitter for being so simple. There are so many things people do with Twitter that could be done better by some more specialized or complex tool. Yet it does not happen.
Twitter has made me revise slightly my two axes of social media — serial vs. browsed and reader-friendly vs. writer friendly. Twitter is generally serial, and I would say it is writer-friendly (it is easy to tweet) but not so reader friendly (the volume gets too high.)
However, Twitter, in its latest mode, is something different. It is “sampled.” In normal serial media, you usually consume all of it. You come in to read and the tool shows you all the new items in the stream. Your goal is to read them all, and the publishers tend to expect it. Most Twitter users now follow far too many people to read it all, so the best they can do is sample — they come it at various times of day and find out what their stalkees are up to right then. Of course, other media have also been sampled, including newspapers and message boards, just because people don’t have time, or because they go away for too long to catch up. On Twitter, however, going away for even a couple of hours will give you too many tweets to catch up on.
This makes Twitter an odd choice as a publishing tool. If I publish on this blog, I expect most of my RSS subscribers will see it, even if they check a week later. If I tweet something, only a small fraction of the followers will see it — only if they happen to read shortly after I write it, and sometimes not even then. Perhaps some who follow only a few will see it later, or those who specifically check on my postings. (You can’t. Mine are protected, which turns out to be a mistake on Twitter but there are nasty privacy results from not being protected.)
TV has an unusual history in this regard. In the early days, there were so few stations that many people watched, at one time or another, all the major shows. As TV grew to many channels, it became a sampled medium. You would channel surf, and stop at things that were interesting, and know that most of the stream was going by. When the Tivo arose, TV became a subscription medium, where you identify the programs you like, and you see only those, with perhaps some suggestions thrown in to sample from.
Online media, however, and social media in particular were not intended to be sampled. Sure, everybody would just skip over the high volume of their mailing lists and news feeds when coming back from a vacation, but this was the exception and not the rule.
The question is, will Twitter’s nature as a sampled medium be a bug or a feature? It seems like a bug but so did the simplicity. It makes it easy to get followers, which the narcissists and the PR flacks love, but many of the tweets get missed (unless they get picked up as a meme and re-tweeted) and nobody loves that.
On Protection: It is typical to tweet not just blog-like items but the personal story of your day. Where you went and when. This is fine as a thing to tell friends in the moment, but with a public twitter feed, it’s being recorded forever by many different players. The ephemeral aspects of your life become permanent. But if you do protect your feed, you can’t do a lot of things on twitter. What you write won’t be seen by others who search for hashtags. You can’t reply to people who don’t follow you. You’re an outsider. The only way to solve this would be to make Twitter really proprietary, blocking all the services that are republishing it, analysing it and indexing it. In this case, dedicated applications make more sense. For example, while location based apps need my location, they don’t need to record it for more than a short period. They can safely erase it, and still provide me a good app. They can only do this if they are proprietary, because if they give my location to other tools it is hard to stop them from recording it, and making it all public. There’s no good answer here.
Submitted by brad on Wed, 2009-09-23 17:50.
It seems that with more and more of the online transactions I engage in — and sometimes even when I don’t buy anything — I will get a request to participate in a customer satisfaction survey. Not just some of the time in some cases, but with every purchase. I’m also seeing it on web sites — sometimes just for visiting a web site I will get a request to do a survey, either while reading, or upon clicking on a link away from the site.
On the surface this may seem like the company is showing they care. But in reality it is just the marketing group’s thirst for numbers both to actually improve things and to give them something to do. But there’s a problem with doing it all the time, or most of the time.
First, it doesn’t scale. I do a lot of transactions, and in the future I will do even more. I can’t possibly fill out a survey on each, and I certainly don’t want to. As such I find the requests an annoyance, almost spam. And I bet a lot of other people do.
And that actually means that if you ask too much, you now will get a self-selected subset of people who either have lots of free time, or who have something pointed to say (ie. they got a bad experience, or perhaps rarely a very good one.) So your survey becomes valueless as data collection the more people you ask to do it, or rather the more refusals you get. Oddly, you will get more useful results asking fewer people.
Sort of. Because if other people keep asking everybody, it creates the same burn-out and even a survey that is only requested from 1 user out of 1000 will still see high rejection and self-selection. There is no answer but for everybody to truly only survey a tiny random subset of the transactions, and offer a real reward (not some bogus coupon) to get participation.
I also get phone surveys today from companies I have actually done business with. I ask them, “Do you have this survey on the web?” So far, they always say no, so I say, “I won’t do it on the phone, sorry. If you had it on the web I might have.” I’m lying a bit, in that the probability is still low I would do it, but it’s a lot higher. I can do a web survey in 1/10th the time it takes to get quizzed on the phone, and my time is valuable. Telling me I need to do it on the phone instead of the web says the company doesn’t care about my time, and so I won’t do it and the company loses points.
Sadly, I don’t see companies learning these lessons, unless they hire better stats people to manage their surveys.
Also, I don’t want a reminder from everybody I buy from on eBay to leave feedback. In fact, remind me twice and I’ll leave negative feedback if I’m in a bad mood. I prefer to leave feedback in bulk, that way every transaction isn’t really multiple transactions. Much better if ebay sends me a reminder once a month to leave feedback for those I didn’t report on, and takes me right to the bulk feedback page.
Submitted by brad on Sun, 2009-06-07 16:29.
Twenty years ago (Monday) on June 8th, 1989, I did the public launch of ClariNet.com, my electronic newspaper business, which would
be delivered using USENET protocols (there was no HTTP yet) over the internet.
ClariNet was the first company created to use the internet as its platform for business, and as such this event has a claim at being the birth of the “dot-com” concept which so affected the world in the two intervening decades. There are other definitions and other contenders which I discuss in the article below.
In those days, the internet consisted of regional networks, who were mostly non-profit cooperatives, and the government funded “NSFNet” backbone which linked them up. That backbone had a no-commercial-use policy, but I found a way around it. In addition, a nascent commercial internet was arising with companies like UUNet and PSINet, and the seeds of internet-based business were growing. There was no web, of course. The internet’s community lived in e-Mail and USENET. Those, and FTP file transfer were the means of publishing. When Tim Berners-Lee would coin the term “the web” a few years later, he would call all these the web, and HTML/HTTP a new addition and glue connecting them.
I decided I should write a history of those early days, where the seeds of the company came from and what it was like before most of the world had even heard of the internet. It is a story of the origins and early perils and successes, and not so much of the boom times that came in the mid-90s. It also contains a few standalone anecdotes, such as the story of how I accidentally implemented a system so reliable, even those authorized to do so failed to shut it down (which I call “M5 reliability” after the Star Trek computer), stories of too-early eBook publishing and more.
There’s also a little bit about some of the other early internet and e-publishing businesses such as BBN, UUNet, Stargate, public access unix, Netcom, Comtex and the first Internet World trade show.
Extra, extra, read all about it: The history of ClariNet.com and the dawn of the dot-coms.
Submitted by brad on Sat, 2009-03-14 16:43.
As you may know, I allow anonymous comments on this blog. Generally, when a blog is small, you don’t want to do too much to discourage participation. Making people sign up for an account (particularly with email verification) is too much of a barrier when your comment volume is small. You can’t allow raw posting these days because of spammers — you need some sort of captcha or other proof-of-humanity — but in most cases moderate readership sites can allow fairly easy participation.
Once a site gets very popular, it probably wants to move to authenticated user posting only. In this case, once the comment forums are getting noisy, you want to raise the bar and discourage participation by people who are not serious. My sub blog on Battlestar Galactica has gotten quite popular of late, and is attracting 100 or more comments per post, even though it has only 1/10th the subscribers of the main blog. Almost all post using the anonymous mechanism which lets them fill in a name, but does nothing to verify it. Many still post under the default name of “Anonymous.”
Some sites let you login using external IDs, such as OpenID, or accounts at Google or Yahoo. On this site, you can log in using any ID from the drupal network, in theory.
However, drupal (which is the software running this site) and most other comment/board systems are not very good at providing an intermediate state, which I will call “casual comments.” Here’s what I would like to see:
- Unauthenticated posters may fill in parameters as they can now (like name, email, URL) and check a box to be remembered. They would get a long-term cookie set. The first post would indicate the user was new.
- Any future posts from that browser would use that remembered ID. In fact, they would need to delete the cookie or ask the site to do so in order to change the parameters.
- If they use the cookie, they could do things like edit their postings and several of the things that registered users can do.
- If they don’t pick a name, a random pseudonym would be assigned. The pseudonym would never be re-used.
- Even people who don’t ask to be remembered would get a random pseudonym. Again, such pseudonyms would not be re-used by other posters or registered users. They might get a new one every time they post. Possibly it could be tied to their IP, though not necessarily traceable back to it, but of course IPs change at many ISPs.
- If they lose the cookie (or move to another computer) they can’t post under that name, and must create a new one. If they want to post under the same name from many machines, create an account.
- The casual commenters don’t need to do more special things like create new threads, and can be quite limited in other ways.
In essence, a mini-account with no authorization or verification. These pseudonyms would be marked as unverified in postings. A posting count might be displayed. A mechanism should also exist to convert the pseudonym to a real account you can login from. Indeed, for many sites the day will come when they want to turn off casual commenting if it is getting abused, and thus many casual commenters will want to convert their cookies into accounts.
The main goal would be to remove confusion over who is posting in anonymous postings, and to stop impersonation, or accusations of impersonation, among casual posters.
I don’t think it should be too hard to make a module for drupal to modify the comment system like this if I knew drupal better.
Submitted by brad on Mon, 2008-04-07 14:58.
Ok, admit it, who likes blogging in to a vacuum. You want to know how many people are actually reading your blog.
I have created a simple Perl script that scans your blog’s log file and attempts to calculate how many people read the blog and the RSS feeds.
You can download the feed reader script. I release it under GPL2.
It’s a perl script, so you would go to your web server log in the shell, and type “perl feedreaders.pl logfilename”
or if you like just “tail -99999 blogfilename | perl feedreaders.pl -” because you only need to scan a couple of days worth of logs to get the figures.
Here are some notes:
- I take advantage of the fact that most blog aggregators now report how many people they are aggregating for. There is no standard but I have put in code to match the common patterns.
- I identify common RSS feed URLs, as well as the most common “main feed” names. If you have other feeds that it doesn’t pick up on, it’s easy to add them to the list at the start of the program.
- A reader has to fetch the feed or home page multiple times from the same IP to count
- On the other hand, people who change IPs regularly will count multiple times. People behind caches may count just once all together.
- I try to eliminate fetches from the most common non-RSS-aggregating spiders
- Based on my experiences, Google Reader and Bloglines are the most popular aggregators, then NewsGator.
- At least one aggregator identifies as Mozilla, custom code tags it.
- It also counts people who fetch your non-RSS blog page multiple times as readers.
- Programs that don’t say they handle multiple users get grouped among the singles.
- Programs with only a few fetches are not counted
I invite my 1146 main blog readers to give it a whirl. (The 53 readers of the new Battlestar blog feed won’t see this notice, nor the 72 reading the comments.
Submitted by brad on Sun, 2008-04-06 17:07.
Recently, while keynoting the Freedom 2 Connect conference in Washington, I spoke about some of my ideas for fiber networks being built from the ground up. For example, I hope for the day when cheap kits can be bought at local stores to fiber up your block by running fiber through the back yards, in some cases literally burying the fiber in the “grass roots.”
Doc Searls, while he was listening to the talk made up a clever term — “Glass Roots” to describe this, and other movements to deploy fiber bottom up, without waiting for telcos and city governments. Any time you can deploy a technology without permission and red tape, it quickly zooms ahead of other technology. Backyard fiber, — combined with cheaper, mass produced free-space-optics or gigabit EHF radio equipment to bridge blocks together across streets or make links to hilltops — could provide the bandwidth we want without waiting.
Because let’s face it. While wireless ISPs sound great and are indeed great for serving some types of customers, right now real bandwidth requires a wire or glass fiber in the ground, and that means monopoly telcos and cable companies as well as the hassles of city government. We want our gigabits (forget megabits) and we want them now.
There are other elements to this Glass Roots movement, though usually with city involvement. Several small towns have put in fiber based ISPs with good success. My friend Brewster Kahle, from the Internet Archive, has brought 100 megabit service to housing projects in San Francisco using some city-laid fiber and the Archive’s bandwidth. You go, Brewster.
Brough Turner has the right idea. We should get dark fiber under our streets, and lots of it, installed and leased by a company that is only in the fiber business, and not in the business of selling you video or phone service or internet. While this company might get a franchise, the important difference is that the franchised monopoly would not light the fiber. Instead, anybody could lease a fiber from their house to a major switching point, and light it any way they want. Darth Vader would tell us “you don’t understand the power of the dark fiber.”
Why is that important? While fiber and wire are basic, the technologies to “light them up” run on Moore’s law. They get obsolete very quickly. Instead of monopoly rents and long cost-plus amortization tables, you want lots of turnover in the actual electronics found at the ends. You want the option to get the latest stuff, which is usually faster and cheaper than the stuff from 2 years ago. Lots faster and lots cheaper.
If you get a lot of free market competition on what lights those endpoints, it gets even better. The result is plenty of choice in how you light it and who you get connectivity from. And that eliminates all the issues around network neutrality or walled gardens. The investment in the dark fiber can probably be amortized over a decade or two, which is long enough.
One might argue the monopoly should even just be at the level of a conduit which it’s easy to drag other things like fiber or wire through. And indeed, whoever does bury pipes under the streets should expect to pull other wires before too long. But having monopoly lockdown at any level above the glass is what slows down the advance of broadband. Get rid of that lockdown, and the real glass roots revolution can begin.
Submitted by brad on Mon, 2008-03-03 16:15.
Over the weekend I was at the [BIL conference]http://www.bilconference.com, a barcamp/unconference style justaposition on the very expensive TED conference. I gave a few talks, including one on self driving cars, privacy and AI issues.
The conference, being free, was at a small community center. This location did not have internet. Various methods were possible to provide internet. The easiest are routers which can take cellular network EVDO cards and offer an 802.11 access point. That works most places, but is not able to handle many people, and may or may not violate some terms of service. However, in just about all these locations there are locations very nearby with broadband internet which can be used, including hotels, businesses and even some private homes. But how to get the access in quickly?
What would be useful would be an “instant internet kit” with all you need to take an internet connection (or two) a modest distance over wireless. This kit would be packed up and available via courier to events that want internet access on just a couple of days notice.
What would you put in the kit? read more »
Submitted by brad on Fri, 2008-02-29 02:10.
As our devices get more and more complex, configuring them gets harder and harder. And for members of the non-tech-savvy public, close to impossible.
Here’s an answer: Develop a simple browser plug-in for all platforms that can connect a USB peripheral to a TCP socket back to the server where the plugin page came from. (This is how flash and Java applets work, in fact this could be added to flash or Java.)
Once activated, the remote server would be able to talk to the device like its USB master, sending and receiving data from it and talking other USB protocol commands. And that means it could do any configuration or setup you might like to do, under the control of a web application that has access to the full UI toolset that web applications have. You could upload new firmware into devices that can accept that, re-flash configuration, read configuration — do anything the host computer can do.
As a result, for any new electronics device you buy — camera, TV remote control, clock, TV, DVD player, digital picture frame, phone, toy, car, appliance etc. — you could now set it up with a nice rich web interface, or somebody else could help you set it up. It would work on any computer — Mac, Linux, Windows and more, and the web UIs would improve and be reprogrammed with time. No software install needed, other than the plug-in. Technicians could remotely diagnose problems and fix them in just about anything.
So there is of course one big question — security. Of course, the plug-in would never give a remote server access to a USB device without providing a special, not-in-browser prompt for the user to confirm the grant of access, with appropriate warnings. Certain devices might be very hard to give access to, such as USB hard drives, the mouse, the keyboard etc. In fact, any device which has a driver in the OS and is mounted by it would need extra confirmation (though that would make it harder to have devices that effectively look like standard USB flash drives into which basic config is simply read and written.)
One simple security technique would be to insist the device be hot plugged during the session. Ie. the plugin would only talk to USB devices that were not plugged in when the page was loaded, and then were plugged in as the app was running. The plugin would not allow constant reloading of the page to trick it on this.
For added security, smarter devices could insist on an authentication protocol with the server. Thus the USB device would send a challenge, which the server would sign/hash with its secret key, and the USB device could then check that using a public key to confirm its talking to its manufacturer. (This however stops 3rd parties from making better configuration tools, so it has its downsides.) It could also be arranged that only devices that exhibit a standard tag in their identification would allow remote control, so standard computer peripherals would not allow this. And the plugin could even maintain and update a list of vendors and items which do or don’t want to allow this.
There are probably some other security issues to resolve. However, should we resolve this it could result in a revolution of configuring consumer electronics, as finally everything would get a big screen, full mouse and keyboard web UI. (Non portable devices like cars and TVs would require a wireless laptop to make this work, but many people have that. Alternately they could use bluetooth, and the plugin could have a similar mode for working with paired bluetooth devices. Again, doing nothing without a strong user confirmation.)
This works because basic USB chips are very cheap now. Adding a small bit of flash to your electronics device and a mini-USB socket that can read and write the flash would add only a small amount to the cost of most items — nothing to many of them, as they already have it. Whatever new toy you buy, you could set it up on the web, and if the company provides a high level of service, you could speak to a tech support agent who could help you set it up right there.
Submitted by brad on Tue, 2008-02-19 18:43.
I’m a director of BitTorrent Inc. (though not speaking for it) and so the recent debate about P2P applications and ISPs has been interesting to me. Comcast has tried to block off BitTorrent traffic by detecting it and severing certain P2P connections by forging TCP reset packets. Some want net neutrality legislation to stop such nasty activity, others want to embrace it. Brett Glass, who runs a wireless ISP, has become a vocal public opponent of P2P.
Some base their opposition on the fact that since BitTorrent is the best software for publishing large files, it does get used by copyright infringers a fair bit. But some just don’t like the concept at all. Let’s examine the issues.
A broadband connection consists of an upstream and downstream section. In the beginning, this was always symmetric, you had the same capacity up as down. Even today, big customers like universities and companies buy things like T-1 lines that give 1.5 megabits in each direction. ISPs almost always buy equal sized pipes to and from their peers.
With aDSL, the single phone wire is multiplexed so that you get much less upstream than downstream. A common circuit will give 1.5mbps down and say 256kb up — a 6 to 1 ratio. Because cable systems weren’t designed for 2 way data, they have it worse. They can give a lot down, but they share the upstream over a large block of customers under the existing DOCSIS system. They also will offer upstream on near the 6 to 1 ratio but unlike the DSL companies, there isn’t a fixed line there. read more »
Submitted by brad on Mon, 2008-02-11 11:08.
Fast internet access at home has spoiled me. Like Manfred Macx in Tourist I feel like I’ve lost my glasses when I’m a tourist. I get annoyed that I can’t quickly and easily get at all the information that’s out there.
I would gladly rent the ultimate tourist mobile device. A large GPS equipped PDA (and also a cell phone for tourists roaming from other countries or from CDMA vs. GSM) that has everything. Every database that can be had on geo-data for the region I’m walking. It has mobile data service of course but also just pre-caches the region I’m in.
Not just the maps and the lists of tourist-related items like restaurants. I want reviews of those restaurants and ratings and even the menus, so I can easily ask “Where’s a the best place in the $15/plate range near here” and similar questions. I don’t just want every hotel in a town (not just the ones in the popular databases) I want their recently updated price offers. And with the data connection, I want something like Wotif for the hotels tied into the computer reservation networks.
I don’t just want to know where the museum is, I want all of its literature. I want its internal map, with all of the placards translated into my language. Indeed, I want just about everything I need to read in a geolocation translated into my language.
And I want opinions on everything, from travel writers, tourists and locals. I want every single major travel book on the area loaded and ready and searchable. (Because I will be searching I want this to be bigger than a typical PDA/phone and have a moderately usable keyboard, or a really big touchscreen keyboard.)
I want it to have a decent camera, both in case I forget to bring mine with me, but for something grander. I want to be able to photograph any sign, any menu, and have it upload the photo to a system that OCRs the text and translates it for me. This is no longer science fiction — decent camera based OCR is available, and while translation software still has its hiccups it’s starting to get decent. In fact, as this gets better, the need for a database of signs at locations becomes less. Of course it should also be able to let locals type messages for me on it which it translates.
It should be trainable to my voice as well, so I can enter text with speech recognition instead of typing. Both for using the device, and saying things that are translated for locals, either to the screen or output from today’s quality text to speech systems. This will get better as the translation software gets better. In some cases, the processing may be done in the cloud to save battery on my device. But as I’ve noted the normal portability requirements on this device are not the same as for my everyday PDA. I don’t mind if this is big and a bit heavy, sized more like a Kindle than an iPhone.
It should be able to take me on walking and driving tours, of course.
And finally, at additional cost, it should connect me to a person, via voice or IM, who can help me. That can be a travel agent to book me a room of course, but it can also be a local expert — somebody who perhaps even works sometimes as a tourist guide. Earlier I wrote of the ability to call a local expert where people with local expertise would register, and when they were online, they could receive calls, billed by the minute. Your device would know where you were, and might well connect you with somebody living one street over who speaks your language and can tell you things you want to know about the area.
Now some of the things I have described are expensive, though as such a device became popular the economies of scale kick in for popular tourist areas. But I’m imagining tourists paying $20 to $30 a day for such a device. Rented 2/3 of the year, that’s $5,000 to $7,000 of revenue in a single year — enough to pay for the things I describe — every travel guide, every database, high volume data service and more. And I want the real thing, not the advertising-biased false information found in typical tourist guides or the “I’m afraid to be critical of anything” information generated by local tourist bureaus.
Why would I pay so much? Travel costs for a party of tourists are an order of magnitude higher than this. I think it would be a rare day that such a device didn’t save you more than this by finding you better food at a better price, savings on hotels and more. And it would save you time. If you are paying $200 to $400/day to travel, including your airfare, your hours are precious. You want to spend them seeing the best things for your taste — not wondering where things are. Saving you an hour of futzing pays for the device.
With scale, it could come down under $10/day, making it crazy not to get it. In fact, locals would start to want some of these databases.
Of course, UI is paramount. You must not have to spend the time you save trying to figure out the UI of the device. That is non-trivial, but doable for a budget like this.
Submitted by brad on Thu, 2008-01-31 22:59.
eBay has announced sellers will no longer be able to leave negative feedback for buyers. This remarkably simple change has caused a lot of consternation. Sellers are upset. Should they be?
While it seems to be an even-steven sort of thing, what is the purpose of feedback for buyers, other than noting if they pay promptly? (eBay will still allow sellers to mark non-paying buyers.) Sellers say they need it to have the power to give negative feedback to buyers who are too demanding, who complain about things that were clearly stated in listings and so on. But what it means in reality is the ability to give revenge feedback as a way to stop buyers from leaving negatives. The vast bulk of sellers don’t leave feedback first, even after the buyer has discharged 99% of his duties just fine.
Fear of revenge feedback was hurting the eBay system. It stopped a lot of justly deserved negative feedback. Buyers came to know this, and know that a seller with a 96% positive rating is actually a poor seller in many cases. Whatever happens on the new system, buyers will also come to notice it. Sellers will get more negatives but they will all get more negatives. What matters is your percentile more than your percentage. In fact, good sellers may get a better chance to stand out in the revenge free world, because they will get fewer negatives than the bad sellers who were avoiding negatives by threat of revenge.
As such, the only sellers who should be that afraid are ones who think they will get more negatives than average.
To help, eBay should consider showing feedback scores before and after the change as well as total. By not counting feedback that’s over a year old they will effectively be doing that within a year, of course.
There were many options for elimination of revenge feedback. This one was one of the simplest, which is perhaps why eBay went for it. I would tweak a bit, and also take a look at a buyer’s profile and how often they leave negative feedback as a fraction of transactions. In effect, make a negative from a buyer who leaves lots and lots of negatives count less than one who never leaves negatives. Put simply, you could give a buyer some number, like 10 negatives per 100 transactions. If they do more than that, their negatives are reduced, so that if they do 20 negatives, each one only counts as a half. That’s more complex but helps sellers avoid worrying about very pesky buyers.
Feedback on buyers was always a bit dubious. After all, while you can cancel bids, it’s hard to pick your winner based on their feedback level. If your winner has a lousy buyer reptutation, there is not normally much you can do — just sit and hope for funds.
If eBay wants to get really bold, they could go a step further and make feedback mandatory for all buyers. (ie. your account gets disabled if you have too many feedbacks not left older than 40 days.) This would make feedback numbers much more trustable by other buyers, though the lack of fear of revenge should do most of this. eBay doesn’t want to go too far. It likes high reputations, they grease the wheels of commerce that eBay feeds on.
One thing potentially lost here is something that never seemed to happen anyway. I always felt that if the seller had very low reputation (few transactions) and the buyer had a strong positive reputation, then the order of who goes first should change. Ie. the seller should ship before payment, and the buyer pay after receipt and satisfaction. But nobody ever goes for that and they will do so less often. A nice idea might be that if a seller offers this, this opens up the buyer to getting negative feedback again, and the seller would not offer it to buyers with bad feedback.
Submitted by brad on Tue, 2008-01-29 10:21.
A couple of weeks ago many wrote about the mistakes of spock which made us call them the “evil spock” for the way they had you mass mail your friends by fooling you into thinking they were already users of Spock.
The newest company to make a similar mistake is called NotchUp. I am loathe to discuss their business, because this means they get publicity for being bad actors, but it involves companies paying candidates for the chance to interview them rather than just giving all the fees to the headhunters. (Something that could only work in a boom market, I expect.) But in this case, some of the fees go to the headhunters, of course, and in a particularly nasty turn, 10% of them go to the “friend” who “invited” you to sign up.
When I get a bunch of invites for something brand new in a short period, it’s either something really hot, or something fishy. In this case it’s the latter. And one person suggests they didn’t authorize NotchUp to email their entire linked-in contact list so there may be something really fishy.
Here are some of the mistakes:
- The offering of affiliate fees to spam your friends, effectively an Amway style marketing system, has been pernicious for some time. While this should be strongly discouraged, I am not calling for its total prohibition, but it should never be secret. Every such message should contain a note explaining the financial incentive.
- The ad comes with your friend’s name on it, but the reply address is a dummy “invite@notchup” which I presume doesn’t work. Any site that does this sort of mailing should put in the friend’s real e-mail, so I can complain to them.
- The ad comes as a combined HTML and plain text message. Which would be good except the plain text part is just “Go read the HTML part.” Seriously. Boy is that evil.
- The site contains no “contact us” information for users who have issues. Their FAQ is all about signing up.
- The site has no “opt out” to stop my friends from doing these mass mailings to me. These are not particularly useful, because I have many email addresses and in fact whole domains that come to me, but they are better than nothing.
- It may have some of these things if I sign up. Of course as somebody who wants to opt-out, I hardly want to create an account just to do that. A few other sites have had this flaw. (I have no idea if you can opt out by signing up, I presume it does give you the ability to at least not get mailings because you have already been fished by your friend.)
Whether their headhunting model sounds interesting or not, the company’s practices seem slimy enough that I would wait for a nicer competitor to come along if you want to get headhunted this way.
Submitted by brad on Tue, 2008-01-15 13:10.
Bruce Schneier has made a fuss by writing about how he leaves his wireless internet open. As a well regarded security expect, how can he do this. You’ll see many arguments for and against in his posting. I’ll expand on one of mine.
Part of Bruce’s argument is one I express different. I sometimes say “Firewalls are a hoax.” They are the wrong choice for security, but we sell them as a good choice. Oddly, however, this very fact does make them a valid choice. I will explain the contradiction.
Firewalls, I should say, are a form of network security — creating an internal network which is “trusted” and protected from the outside world. In an obscure way, encrypting your wireless net is in this class of security. Note that the “firewall” programs that run on PCs are not network firewalls so they are generally not in this class of security, though they are called Firewalls.
The right way to do things, in the ideal world, is to secure each PC, and to have that PC encrypt its traffic end-to-end with all the sites it communicates with. If you do this, you have almost no need for firewalls or encryption on the network. This is important because in many cases, the idea that your internal network is trustable is a dangerous one. That’s because many networks are populated with insecure consumer computers which frequently get infected with malware (viruses, trojans etc.) They can get infected because they are laptops that visit exposed networks they are not secured well enough for — because you thought you could get away with less on the home net — or because their owner is tricked into downloading malware, or going to a web site that exploits a browser bug, etc.
Once a local computer is infected, your trusted local net betrays you, as the malware now gets to take advantage of all that trust.
We don’t live in that ideal world. The same insecurity these consumer computers (and yes, I mean Windows but other OSs are not immune) have makes them unsuitable for general exposure. The firewall industry gets to sell firewalls because the workstations are so insecure.
In the real world, virus/trojan attacks are the most common. Up to 30% of PCs are “botted” — taken over by malware and acting as zombies under the control of some distant master. A significant number are just plain compromised in other ways, though botting seems the most popular motive today for taking control of systems. The volume of attacks coming in via outsiders sniffing or connecting to your wireless network is insignificant in comparison, I think research would show.
And sadly, while we would like all web traffic to be HTTPS and all E-mail to be secured over TLS, this is just not an option. Most web servers don’t over encrypted versions, and even the ones that do get rarely used because the UI was not set up correctly for it. (Ideally, http should have been designed so that you don’t have to put your encryption desires into the URL — https vs. http — so that it could be negotiated for each connection. Even then, it would be hard to do this though identity certificates could make it happen.)
So we must surf the web in the open, or at best through an encrypted tunnel to a proxy that surfs in the open. So this does call for encrypting one’s wifi. However, again, the number of people sniffing private homes wifi is tiny in comparison to the other threats.
One of the factors supporting Bruce’s choice is that most security continues to have bad UI. The computer and security industries regularly vastly underestimate the importance of good UI. The hard truth is that good security with bad (hard to use) UI simply doesn’t get deployed very much unless you force it and force it hard. This suggests that lesser security with good UI can actually deliver more real world results than better security with bad UI.
For encrypting networks, the UI is poor. Different vendors use different passphrase algorithms to input keys. For many devices (phones, digital picture frames etc.) even entering a passphrase is difficult. We’re starting to see some better UI but it’s slow to deploy and for now it is no surprise that people want to leave their nets open, both for their own devices, and to give access to guests in their home or office.
To my mind the ideal UI is a device tries to connect to the network, and the AP or a computer flashes a light that says that one, and exactly one device is asking to join the net. You then push a button to confirm that device. Also good is the ability to allow arbitrary devices to connect in a secured channel but with no special ability to route packets to one another or into general devices. A full configuration has an internal net (with routing), guest devices that can’t route to the internal net or to other guests, and host devices which can be seen by guests but not the outside world.
Oddly, as I said at the start, the choices we make affect the value of the choices. Because NATs and firewalls provide some security, people (and vendors) allow the computers behind these NATs and firewalls to be insecure in a way they never would or could if the NATs and firewalls weren’t there. This in turn makes the NATs and firewalls worthwhile. And yes, random attacks from outside will always be more probable than attacks from the inside from compromised machines, and they will be more probable than attacks from neighbours. So it’s not as simple as we like. However, computers are going to roam more and more. My PDA has wifi and roams. It also has EVDO and some day those networks will open and need more endpoint security.
So is Bruce right or wrong? Both. The real world risk of what he’s doing isn’t great. It’s not zero, either. The real question is whether the UI penalties of an encrypted network are worse than the risk. And that decision varies from person to person. Better UI and protocol design could mostly eliminate the tradeoff, which is the real lesson.
Submitted by brad on Thu, 2007-11-29 22:26.
If you have bought a home router or access point, you know it comes by default listening to some NAT based IP address, and the setup guide tells the user to type "http://192.168.1.1" or similar into their browser.
Instead, these companies should define a domain, like "setup.linksys.com" that points to a page that redirects to that IP address. In addition, the box, before it is set up, should have a mini DHCP server and DNS server that returns the right address for that domain for people who just plug a PC into the box. (I guess it could return that address for any domain you type in if the box is not configured,n ot just the official one.)
This would serve several purposes. The instructions to the unskilled user become less cryptic. Just plug your PC into the box, boot it and type this easy to remember name into the browser.
If the user is more sophisticated and changes the address of the router, a cookie could be set so the redirect goes to the valid address, but of course if the cookie is lost the user will have to remember, but that's always true. And the user who does not use DHCP from the router will also have to use the numeric address, so it must be printed as an alternative for such folks. But one value of the whole thing is that if it got standardized, it would make it easy to figure out the address for a box if you know the brand. The domain could and should be printed on it. Along with the default password (which should then be changed of course.)