Internet

Even ASIC miners of Bitcoins face security threats

Last month I wrote about paradoxes involving bitcoin and other cryptocurrency mining. In particular, I pointed out that while many people are designing alternative coins so that they are hard to mine with ASICs — and thus can be more democratically mined by people’s ordinary computers or GPUs — this generates a problem. If mining is done on ordinary computers, it becomes worthwhile to break into ordinary computers and steal their resources for mining. This has been happening, even with low powered NAS box computers which nobody would ever bother to mine on if they had to pay for the computer and its electricity. The attacker pays nothing, so any mining capacity is good.

Almost any. In Bitcoin, ASIC mining is so productive that it’s largely a waste of time to mine with ordinary CPUs even if you get them for free, since there is always some minor risk in stealing computer time. While ordinary computers are very hard to secure, dedicated ASIC mining rigs are very simple special purpose computers, and you can probably secure them.

But in a recently revealed attack thieves stole bitcoins from miners by attacking not the ASIC mining rigs, but their internet connections. The rigs may be simple, but the computers they flow their data through, and the big network routers, are less so. Using BGP redirection, it is suspected, the thieves just connected the mining rigs to a different mining pool than the one they thought they joined. And so they worked away, mining hard, and sometimes winning the bitcoin lottery, not for their chosen pool, but the thieves’ pool.

It’s not hard to imagine fixes for this particular attack. Pools and rigs can authenticate more strongly, and pools can also work to keep themselves more secure.

But we are shown one of the flaws of almost all digital money systems. If your computer can make serious money just by computing, or it can spend money on your behalf without need for a 2nd factor authentication, then it becomes very worthwhile for people to compromise your system and steal your computer time or your digital money. Bitcoin makes this even worse by making transactions irrevocable and anonymous. For many uses, those are features, but they are also bugs.

For the spending half, there is much effort in the community to build more secure wallets that can’t just spend your money if somebody takes over your computer. They rely on using multiple keys, and keeping at least one key in a more secure, even offline computer. Doing this is very hard, or rather doing it with a pleasant and happy user interface is super hard. If you’re going to compete with PayPal it’s a challenge. If somebody breaks into my PayPal account and transfers away the money there, I can go to PayPal and they can reverse those transactions, possibly even help track down the thieves. It’s bad news if a merchant was scammed but very good news for me.

One could design alternate currencies with chargebacks or refundability, but Bitcoin is quite deliberate in its choice not to have those. It was designed to be like cash. The issue is that while you could probably get away keeping your cash in your mattress and keeping a secure house, this is a world where somebody can build robots that can go into all the houses it can find and pull the cash out of the mattresses without anybody seeing.

The tide of surveys gets worse -- "would you please rate our survey?"

Five years ago, I posted a rant about the excess of customer service surveys we’re all being exposed to. You can’t do any transaction these days, it seems, without being asked to do a survey on how you liked it. We get so many surveys that we now just reject these requests unless we have some particular problem we want to complain about — in other words, we’re back to what we had with self-selected complaints. The value of surveys is now largely destroyed, and perversely, as the response rates drop and the utility diminishes, that just pushes some companies to push even harder on getting feedback, creating a death spiral.

A great example of this death spiral came a few weeks ago when I rode in an Uber and the driver had a number of problems. So this time I filled out the form to rate the driver and leave comments. Uber’s service department is diligent, and actually read it, and wrote me back to ask for more details and suggestions, which I gave.

That was followed up with:

Hi Brad Templeton,

We’d love to hear what you think of our customer service. It will only take a second, we promise. This feedback will allow us to make sure you always receive the best possible customer service experience in future.

If you were satisfied in how we handled your query, simply click this link.

If you weren’t satisfied in how we handled your ticket, simply click this link.

A survey on my satisfaction with the survey process! Ok, to give Uber some kudos, I will note:

  • They really did try to make this one simple, just click a link. Though one wonders, had I clicked I was unsatisfied, would there have been more inquiry? Of course I was unsatisfied — because they sent yet another survey. The service was actually fine.
  • At least they addressed me as “Hi Brad Templeton.” That’s way better than “Dear Brad” like the computer sending the message pretending it’s on a first-name basis with me. Though the correct salutation should be “Dear Customer” to let me know that it is not a personally written message for me. The ability to fill in people’s names in form letters stopped being impressive or looking personal in the 1970s.

This survey-on-a-survey is nice and short, but many of the surveys I get are astoundingly long. They must be designed, one imagines, to make sure nobody who values their time ever fully responds.

Why does this happen? Because we’ve become so thrilled at the ability to get high-volume feedback from customers that people feel it is a primary job function to get that feedback. If that’s your job, then you focus on measuring everything you can, without thinking about how the measurement (and over-measurement) affects the market, the customers and the very things you are try to measure. Heisenberg could teach these folks a lesson.

To work, surveys must be done on a small sample of the population, chosen in a manner to eliminate bias. Once chosen, major efforts should be made to assure people who are chosen do complete the surveys, which means you have to be able to truthfully tell them they are part of a small sample. Problem is, nobody is going to believe that when your colleagues are sending a dozen other surveys a day. It’s like over-use of antibiotics. All the other doctors are over-prescribing and so they stop working for you, even if you’re good.

The only way to stop this is to bring the hammer down from above. People higher up, with a focus on the whole customer experience, must limit the feedback efforts, and marketing professionals need to be taught hard in school and continuing education just why there are only so many they can do.

Reflections on the 25th anniversary of ClariNet and the dot-com

25 years ago, on June 8, 1989, I announced to the world my new company ClariNet, which offered for sale an electronic newspaper delivered over the internet. This has the distinction, as far as I know, of being the first business created to use the internet as a platform, what we usually call a “dot-com” company.

I know it was the first because up until that time, the internet’s backbone was run by the National Science Foundation and it had a policy disallowing commercial use of the network. In building ClariNet, I found a way to hack around those rules and sell the service. Later, the rules would be relaxed and the flood of dot-coms came on a path of history that changed the world.

A quarter of a century seems like an infinite amount of time in internet-years. Five years ago, for the 20th anniversary, I decided to write up this history of the company, how I came to found it, and the times in which it was founded.

Read The history of ClariNet.com and the dawn of internet based business

There’s not a great deal to add in the 5 years since that prior anniversary.

  • Since then, USENET’s death has become more complete. I no longer use it, and porn, spam and binaries dominate it now. Even RSS, which was USENET’s successor — oddly with some inferiorities — has begun to fall from favour.
  • The last remnants of ClariNet, if they exist at Yellowbrix, are hard to find, though that company exists and continues to sell similar services.
  • Social media themselves are showing signs of shrinking. Publishing and discussing among large groups just doesn’t scale past a certain point and people are shrinking their circles rather than widening them.
  • We also just saw the 25th anniversary of the Web itself a few months ago, or at least its draft design document. ClariNet’s announcement in June was just that — work had been underway for many months before that, and product would not ship until later in the summer.

Many readers of this blog will not have seen this history before, and 25 years is enough of an anniversary to make it worth re-issuing. There is more than just the history of ClariNet in there. You will also find the history of other early internet business, my own personal industry history that put me in the right place at the right time with these early intentions, and some anecdotes from ClariNet’s life and times.

The endgame for Bitcoin

Bitcoin is hot-hot-hot, but today I want to talk about how it ends. Earlier, I predicted a variety of possible fates for Bitcoin ranging from taking over the entire M1 money supply to complete collapse, but the most probable one, in my view, is that Bitcoin is eventually supplanted by one or more successor digital currencies which win in the marketplace. I think that successor will also itself be supplanted, and that this might continue for some time. I want to talk about not just why that might happen, but also how it may take place.

Nobody thinks Bitcoin is perfect, and no digital currency (DigiC) is likely to satisfy everybody. Some of the flaws are seen as flaws by most people, but many of its facets are seen as features by some, and flaws by others. The anonymity of addresses, the public nature of the transactions, the irrevocable transactions, the fixed supply, the mining system, the resistance to control by governments — there are parties that love these and hate these.

Bitcoin’s most remarkable achievement, so far, is the demonstration that a digital currency with no intrinsic value or backer/market maker can work and get a serious valuation. Bitcoin argues — and for now demonstrates — that you can have a money that people will accept only because they know they can get others to accept it with no reliance on a government’s credit or the useful physical properties of a metal. The price of a bitcoin today is pretty clearly the result of speculative bubble investment, but that it sustains a price at all is a revelation.

Bitcoins have their value because they are scarce. That scarcity is written into the code — in the regulated speed of mining, and in the fixed limit on coins. There will only be so many bitcoins, and this gives you confidence in their value, unlike say, Zimbabwe 100 trillion dollar notes. This fixed limit is often criticised because it will be strongly deflationary over time, and some more traditional economic theory feels there are serious problems with a deflationary currency. People resist spending it because holding it is better than spending it, among other things.

Altcoins

While bitcoins have this scarcity, digital currencies as a group do not. You can always create another digital currency. And many people have. While Bitcoin is the largest, there are many “altcoins,” a few of which (such as Ripple, Litecoin and even the satirical currency Dogecoin) have serious total market capitalizations of tens or hundreds of millions of dollars(1). Some of these altcoins are simply Bitcoin or minor modifications of the Bitcoin protocol with a different blockchain or group of participants, others have more serious differences, such as alternate forms of mining. Ripple is considerably different. New Altcoins will emerge from time to time, presumably forever.

What makes one digital coin better than another? Obviously a crucial element is who will accept the coin in exchange for goods, services or other types of currency. The leading coin (Bitcoin) is accepted at more stores which gives it a competitive advantage.

If one is using digital currency simply as a medium — changing dollars to bitcoins to immediately buy something with bitcoins at a store, then it doesn’t matter a great deal which DigiC you use, or what its price is, as long as it is not extremely volatile. (You may be interested in other attributes, like speed of transaction and revocation, along with security, ease of use and other factors.) If you wish to hold the DigC you care about appreciation, inflation and deflation, as well as the risk of collapse. These factors are affected as well by the “cost” of the DigiC.

The cost of a digital currency

I will advance that every currency has a cost which affects its value. For fiat currency like dollars, all new dollars go to the government, and every newly printed dollar devalues all the other dollars, and overprinting creates clear inflation.  read more »

Birth of the World Wide Web

Yesterday, I was interviewed for the public radio program Marketplace and as is normal, 30 minutes come down to 30 seconds. So I wanted to add some commentary to that story.

As you are no doubt hearing today, it was 25 years ago that Tim Berners-Lee first developed his draft proposal for an internet based hypertext system to tie together all the internet’s protocols: E-mail, USENET, FTP, Gopher, Telnet and a potential new protocol (HTTP) to serve up those hypertext pages. He didn’t call it the web then, and the first web tools were not written for a while, and wouldn’t make it to the outside world until 1991, but this was the germ of a system that changed the internet and the world. The first wave of public attention came when the UIUC’s supercomputing center released a graphical browser called Mosaic in 1993 and CERN declared the web protocols non-proprietary. Mosaic’s main author went on to start Mozilla/Netscape, which turned into the Firefox browser you may be reading this with.

As the radio piece explains, many people are confused as to what the difference is between the internet and the web. (They also are unsure what a browser is, or how the web is distinct even from Google sometimes.) To most, the internet was an overnight success — an overnight success that had been developing for over 20 years.

I don’t want to diminish the importance of the web, or TimBL’s contribution to it. He writes a guest editorial on the Google blog today where he lays out a similar message. The web integrated many concepts from deeper internet history.

Prior to the web, several systems emerged to let you use the internet’s resources. Mailing lists were the first seat of community on the internet, starting with Dave Farber’s MSGGROUP in the 70s. In the early 80s, that seat of community moved to USENET. USENET was serial, rather than browsed, but it taught lessons about having a giant network with nobody owning it or being in control.

The large collection of FTP servers were indexed by the Archie search engine, the first internet search engine from McGill University. Greater excitement came from the Gopher protocol from the U. of Minnesota, which allowed you to browse a tree of menus, moving from site to site, being taken to pages, files, local search resources and more all over the internet.

The web was not based on menus, though. It took the concept of hypertext; the ability to put links into documents that point at other documents. Hypertext concepts go back all the way to Vannevar Bush’s famous “Memex” but the man most known for popularizing it was Ted Nelson, who wrote the popular book Comptuer Lib. Ted tried hard for decades to commercialize hypertext and saw his Project Xanadu system as the vision for the future computerized world. In Xanadu, links were to specific points in other documents, were bi-directional and also allowed for copyright ownership and billing — I could link in text from your document and you got paid when people paid to read my document. Hypertext was the base of Apple’s “Hypercard” and a few other non-networked systems.

So did TimBL just combine hypertext with internet protocols to make a revolution? One important difference with the web was that the links were one-way and the system was non-proprietary. Anybody could join the system, anybody could link to anybody, and no permission or money were needed. Embracing the internet’s philosophy of open protocols, while others had built more closed systems, this was a tool that everybody could jump aboard.

Another key difference, which allowed WWW to quickly supplant gopher, was counter-intuitive. Gopher used menus and thus was structured. Structure enables several useful things, but it’s hard to maintain and limits other things you can do. Hypertext is unstructured and produces a giant morass, what we math nerds would call a big directed graph. This “writer friendly” approach was easy to add to, in spite of the lack of plan and the many broken links.

The Web was a superset of Gopher, but by being less structured it was more powerful. This lesson would be taught several times in the future, as Yahoo’s structure menus, which made billions for its founders, were supplanted by unstructured text search from Lycos, Alta Vista and eventually Google. Wikipedia’s anybody-can-contribute approach devoured the old world of encyclopedias.

For the real explosion into the public consciousness, though, the role of Mosaic is quite important. TimBL did envision the inclusion of graphics — I remember him excitedly showing me an early version of Mosaic in 1992 he was playing with — but at the time most of us used USENET, gopher and the very early Web using text browsers, and more to the point, we liked it that way. The inclusion of graphics into web pages was mostly superfluous and slowed things down, making it harder, not easier to get to the meat of what we wanted. The broader public doesn’t see it that way, and found Mosaic to be their gateway into the internet. In addition, many companies and content producers would not be satisfied with publishing online until they could make it look the way they wanted it to look. Graphical browsers allowed for that, but at the time, people were much more interested in the new PDF format which let you publish a document to look just like paper than in the HTML format where you didn’t control the margins, fonts or stylistic elements.

(The HTML specification’s history is one of a war between those who believe you should specify the meaning of the structural elements in your documents and let the browser figure out the best way to present those, and those who want tight control to produce a specific vision. CSS has settled some of that war, but it continues to this day.)

Nobody owned the web, and while Tim is not poor, it was others like Marc Andreesen, Jerry Yang & Dave Filo who would become the early billionaires from it. The web was the internet’s inflection point, when so many powerful trends came together and reached a form that allowed the world to embrace it. (In addition, it was necessary that the Moore’s law curves governing the price of computing and networking were also reaching the level needed to give these technologies to the public.)

25 years ago, I was busy working on the code for ClariNet, which would become the first business founded on the internet when I announced it in June — I will post an update on that 25th anniversary later this year.

More about stolen bitcoins

Yesterday, I wrote about stolen bitcoins and the issues around a database of stolen coins. The issue is very complex, so today I will add some follow-up issues.

When stolen property changes hands (innocently) the law says that nobody in the chain had authority to transfer title to that property. Let’s assume that the law accepts bitcoins as property, and bitcoin transactions as denoting transfer of title, (as well as possession/control) to it. So with a stolen bitcoin, the final recipient is required on the law to return possession of the coin to its rightful owner, the victim of the theft. However, that recipient is also now entitled to demand back whatever they paid for the bitcoin, and so on down the line, all the way to the thief. With anonymous transactions, that’s a tall order, though most real world transactions are not that anonymous.

This is complicated by the fact that almost all Bitcoin transactions mix coins together. A Bitcoin “wallet” doesn’t hold bitcoins, rather it holds addresses which were the outputs of earlier transactions, and those outputs were amounts of bitcoin. When you want to do a new transaction, you do two things:

  1. You gather together enough addresses in your wallet which hold outputs of prior transactions, which together add up to as much as you plan to spend, and almost always a bit more.
  2. You write a transaction that lists all those old outputs as “inputs” and then has a series of outputs, which are the addresses of the recipients of the transaction.

There are typically 3 (or more) outputs on a transaction:

  1. The person you’re paying. The output is set to be the amount you’re paying
  2. Yourself. The output is the “change” from the transaction since the inputs probably didn’t add up exactly to the amount you’re paying.
  3. Any amount left over — normally small and sometimes zero — which does not have a specific output, but is given as a transaction fee to the miner who put your transaction into the Bitcoin ledger (blockchain.)

They can be more complex, but the vast majority work like this. While normally you pay the “change” back to yourself, the address for the change can be any new random address, and nothing in the ledger connects it to you.

So as you can see, a transaction might combine a ton of inputs, some of which are clean, untainted coins, some of which are tainted, and some of which are mixed. After coins have been through a lot of transactions, the mix can be very complex. Not so complex as the computers can’t deal with it and calculate a precise fraction of the total coin that was tainted, but much too complex for humans to wish to worry about.

A thief will want to mix up their coins as quickly as possible, and there are a variety of ways to do that.

Right now, the people who bought coins at Mt.Gox (or those who sent them there to buy other currency) are the main victims of this heist. They thought they had a balance there, and its gone. Many of them bought these coins at lower prices, and so their loss is not nearly as high as the total suggests, but they are deservedly upset.

Unfortunately, if the law does right by them and recovers their stolen property, it is likely that might come from the whole Bitcoin owning and using community, because of the fact that everybody in the chain is liable. Of particular concern are the merchants who are taking bitcoin on their web sites. Let’s speculate on the typical path of a stolen coin that’s been around for a while:

  • It left Mt.Gox for cash, sold by the thief, and a speculator simply held onto the coins. That’s the “easy” one, the person who now has stolen coins has to find the thief and get their money back. Not too likely, but legally clear.
  • It left Mt.Gox and was used in a series of transactions, ending up with one where somebody bought an item from a web store using bitcoin.
  • With almost all stores, the merchant system takes all bitcoin received and sells it for dollars that day. Somebody else — usually a bitcoin speculator — paid dollars for that bitcoin that day, and the chain continues.

There is the potential here for a lot of hassle. The store learns they sold partially tainted bitcoins. The speculator wants and is entitled to getting a portion of her money back, and the store is an easy target to go after. The store now has to go after their customer for the missing money. The store also probably knows who their customer is. The customer may have less knowledge of where her bitcoins came from.

This is a huge hassle for the store, and might very well lead to stores reversing their decisions to accept bitcoin. If 6% of all bitcoins are stolen, as the Mt.Gox heist alleges, most transactions are tainted. 6% is an amount worth recovering for many, and it’s probably all the profit at a typical web store. Worse, the number of stolen coins may be closer to 15% of all the circulating bitcoins, certainly something worth recovering on many transactions.

The “sinking taint” approach

Previously, I suggested a rule. The rule was that if a transaction merges various inputs which are variously reported as stolen (tainted) and not, then the total percentage be calculated, and the first outputs receive all the tainting, and the latter outputs (including the transaction fee, last of all) be marked clear. One of the outputs would remain partial unless the transaction was designed to avoid this. There is no inherent rule that the “change” comes last, it is just a custom, and it would probably be reversed, so that as much of the tainted fraction remains in the change as possible, and the paid amount is as clean as possible. Recipients would want to insist on that.

This allows the creation of a special transaction that people could do with themselves on discovering they have coin that is reported stolen. The transaction would split the coin precisely into one or more purely tainted outputs, and one or more fully clean outputs. Recipients would likely refuse bitcoin with any taint on it at all, and so holders of bitcoin would be forced to do these dividing transactions. (They might have to do them again if new theft reports come on coin that they own.) People would end up doing various combinations of these transactions to protect their privacy and not publicly correlate all their coin.

Tainted transaction fees?

The above system makes the transaction fee clean if any of the coin in the transaction is clean. If this is not done, miners might not accept such transactions. On the other hand, there is an argument that it would be good if miners refused even partially tainted transactions, other than the ones above used to divide the stolen coins from the clean. There would need to be a rule that allows a transaction to be declared a splitting transaction which pays its fees from the clean part. In this case, as soon as coins had any taint at all, they would become unspendable in the legit markets and it would be necessary to split them. They would still be spendable with people who did not accept this system, or in some underground markets, but they would probably convert to other currencies at a discount.

This works better if there is agreement on the database of tainted coins, but that’s unlikely. As such, miners would decide what databases to use. Anything in the database used by a significant portion of the miners would make those coins difficult to spend and thus prime for splitting. However, if they are clean in the view of a significant fraction of the miners, they will enter the blockchain eventually.

This is a lot of complexity, much more than anybody in the Bitcoin community wants. The issue is that if the law gets involved, there is a world of pain in store for the system, and merchants, if a large fraction of all circulating coins are reported as stolen in a police report, even a Japanese police report.

What if somebody steals a bitcoin?

Bitcoin has seen a lot of chaos in the last few months, including being banned in several countries, the fall of the Silk Road, and biggest of all, the collapse of Mt. Gox, which was for much of Bitcoin’s early history, the largest (and only major) exchange between regular currencies and bitcoins. Most early “investors” in bitcoin bought there, and if they didn’t move their coins out, they now greatly regret it.

I’ve been quite impressed by the ability of the bitcoin system to withstand these problems. Each has caused major “sell” days but it has bounced back each time. This is impressive because nothing underlies bitcoins other than the expectation that you will be able to use them into the future and that others will take them.

It is claimed (though doubted by some) that most of Mt.Gox’s bitcoins — 750,000 of them or over $400M — were stolen in some way, either through thieves exploiting a bug or some other means. If true, this is one of the largest heists in history. There are several other stories of theft out there as well. Because bitcoin transactions can’t be reversed, and there is no central organization to complain to, theft is a real issue for bitcoin. If you leave your bitcoin keys on your networked devices, and people get in, they can transfer all your coins away, and there is no recourse.

Or is there?

If you sell something and are paid in stolen money, there is bad news for you, the recipient of the money. If this is discovered, the original owner gets the money back. You are out of luck for having received stolen property. You might even be suspected of being involved, but even if you are entirely innocent, you still lose.

All bitcoin transactions are public, but the identities of the parties are obscured. If your bitcoins are stolen, you can stand up and declare they were stolen. More than that, unless the thief wiped all your backups, you can 99.9% prove that you were, at least in the past, the owner of the allegedly stolen coins. Should society accept bitcoins as money or property, you would be able to file a police report on the theft, and identify the exact coin fragments stolen, and prove they were yours, once. We would even know “where” they are today, or see every time they are spent and know who they went to, or rather, know the random number address that owns them now in the bitcoin system. You still own them, under the law, but in the system they are at some other address.

That random address is not inherently linked to this un-owner, but as the coins are spent and re-spent, they will probably find their way to a non-anonymous party, like a retailer, from whom you could claim them back. Retailers, exchanges and other legitimate parties would not want this, they don’t want to take stolen coins and lose their money. (Clever recipients generate a new address for every transaction, but others use publicly known addresses.)

Tainted coin database?

It’s possible, not even that difficult, to create a database of “tainted” coins. If such a database existed, people accepting coins could check if the source transaction coins are in that database. If there, they might reject the coins or even report the sender. I say “reject” because you normally don’t know what coins you are getting until the transaction is published, and if the other party publishes it, the coins are now yours. You can refuse to do your end of the transaction (ie. not hand over the purchased goods) or even publish a transaction “refunding” the coins back to the sender. It’s also possible to imagine that the miners could refuse to enter a transaction involving tainted coins into the blockchain. (For one thing, if the coins are stolen, they won’t get their transaction fees.) However, as long as some miner comes along willing to enter it, it will be recorded, though other miners could refuse to accept that block as legit.  read more »

End the redirect wrapper on links

A lot of sites, most notably search engines like Google, like to rewrite all the links on their pages. So search for this page and instead of http://ideas.4brad.com, the link Google gives you is http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=short-string&url=http%3A%2F%2Fideas.4brad.com%2F&ei=med-string&usg=huge-string&bvm=short-string or similar. (I have redacted the actual codes.)

What’s happening is that when you click on the link, you really go to Google. Google records what you clicked on and other parameters related to the search so they can study just how people use their search engine, what they click on and when. It’s a reasonable thing for them to want to study, though a potential privacy invasion.

Because each click goes through Google, your clicks are slowed down. Because Google has such huge resources, and is almost never down, you usually don’t notice it, though even with Google you will see the delay on slow links, like mobile GPRS and Edge connections. It also means you can’t easily cut and paste links from search results.

Other sites are not as good. They sometimes noticeably slow own your click. Worse, they sometimes break it. For example, on my phone, when I click on links in LinkedIn messages, as well as Facebook ones, which are also redirected, it doesn’t work if I’m not currently logged in to those sites. Due to some bad code, it also wants to send the link to the mobile apps of these sites, which is not what I want. (The one for LinkedIn is particularly broken, as it doesn’t seem to know where the app is, and sends me to the Play store to install it even though it is already installed.)

In other words, these links break the web from time to time. They can also interfere with spiders. On the plus side, they can be set to protect your privacy by hiding data in the REFERER field from the target web site. For sites that have been identified ad malicious, they can provide a warning.

To fix this, sites can change all their links to be javascript. The link can be a real target, and associated onClick javascript can also send a web hit back to the server with the logging info.

A better solution would be to push use of the “ping” attribute in the HTML spec, and allow links to have both an href to the target, and another URL which gets invoked when the link is clicked. In the background, this would not slow down your click, or break it. Browsers could also elect to block it, which the sites might not like but is good for users. Links to malicious sites could be treated differently if that’s part of the service. There would also be no need to fake the status window when moving the mouse over the link, as must be done with redirects.

Let’s say no to all these redirects.

Satoshi, is now the time to consider donating lots of bitcoin to charity?

I don’t know who the person or people are who, under the name Satoshi Nakamoto, created the Bitcoin system. The creator(s) want to keep their privacy, and given the ideology behind Bitcoin, that’s not too surprising.

There can only be 21 million bitcoins. It is commonly speculated that Satoshi did much of the early mining, and owns between 1 million and 1.5 million unspent bitcoins. Today, thanks in part to a speculative bubble, bitcoins are selling for $800, and have been north of $1,000. In other words, Satoshi has near a billion dollars worth of bitcoin. Many feel that this is not an unreasonable thing, that a great reward should go to Satoshi for creating such a useful system.

For Satoshi, the problem is that it’s very difficult to spend more than a small portion of this block, possibly ever. Bitcoin addresses are generally anonymous, but all transactions are public. Things are a bit different for the first million bitcoins, which went only to the earliest adopters. People know those addresses, and the ones that remain unspent are commonly believed to be Satoshi’s. If Satoshi starts spending them in any serious volume, it will be noticed and will be news.

The fate of Bitcoin

Whether Bitcoin becomes a stable currency in the future or not, today few would deny it is not stable, and undergoing speculative bubbles. Some think that because nothing backs the value of bitcoins, it will never become stable, but others are optimistic. Regardless of that, today the value of a bitcoin is fragile. The news that “Satoshi is selling his bitcoins!” would trigger panic selling, and that’s bad news in any bubble.

If Satoshi could sell, it is hard to work out exactly when the time to sell would be. Bitcoin has several possible long term fates:

  1. It could become the world’s dominant form of money. If it replaced all of the “M1” money supply in the world (cash and very liquid deposits) a bitcoin could be worth $1 million each!
  2. It could compete with other currencies (digital and fiat) for that role. If it captured 1% of world money supply, it might be $10,000 a coin. While there is a limit on the number of bitcoins, the limit on the number of cryptocurrencies is unknown, and as bitcoin prices and fees increase, competition is to be expected.
  3. It could be replaced by one or more successors of superior design, with some ability to exchange during a modest window, and then drifting down to minimal value
  4. It could collapse entirely and quickly in the face of government opposition, competition and other factors during its bubble phase.

My personal prediction is #3 — that several successor currencies will arise which fix issues with Bitcoin, with exchange possible for a while. However, just as bitcoins had their sudden rushes and bubbles, so will this exchange rate, and as momentum moves into this currency it could move very fast. Unlike exchanges that trade bitcoins for dollars, inter-cryptocurrency exchanges will be fast (though the settlement times of the currencies will slow things down.) It could be even worse if the word got out that “Satoshi is trading his coins for [Foo]Coin” as that could cause complete collapse of Bitcoin.

Perhaps he could move some coins through randomizing services that scramble the identity association, but moving the early coins to such a system would be seen as selling them.  read more »

No, we don't want much more Fedex and UPS on Dec 24

A big story this Christmas was a huge surge in the use of rush shipping in the last 2 days before Christmas. Huge numbers of people signed up for Amazon Prime, and other merchants started discounting 2 day and overnight shipping to get those last minute sales. In turn, a lot of stuff didn’t get delivered on time, making angry customers and offers of apology discounts from merchants. This was characterized as a “first world problem” by many outside the game, of course.

When I shop, I am usually travelling outside the US and so I have to get stuff even before the 24th, and I’ve had stuff I left to the last day not delivered several times, so I know to avoid doing it. Some packages are not going to make it, and this should be expected — even desired.

While it makes sense to increase the infrastructure a bit as online shopping grows in popularity, you don’t want to go nuts at Christmas. If you need to build your infrastructure to handle every Christmas gift, you have to build it too big, and you pay for that through higher prices the rest of the year. Shippers need to figure out their real capacity, and everybody needs to plan based on it.

The failure this season was not a failure of the delivery system. Rather it was a failure of either the shippers to tell the merchants what their capacity was, and/or a failure of the merchants to communicate to customers that too much was being shipped and not everybody could be promised Dec 24 delivery.

The obvious way to fix this is first to have the shippers get a solid handle on their capacity for the various types of shipping to the various destinations. They can also identify the bottlenecks and widen them a modest amount.

The next thing is for the merchants to know just how much shipping they can buy. There can either be a live spot market — so the merchant web sites just stop offering the delivery promise when the capacity is reached, or merchants could even attempt to pre-contract for capacity, paying for it whether they need it or not (or reselling it if they know they won’t need it.) Merchants should be building their own forecasts about available capacity and querying shippers for updates on just how much more is left. Capacity isn’t a fixed thing — it depends on the size of packages and where they are going and many other things — but this is a problem computers can handle.

Finally, the shippers and the merchants can start increasing the price of the rush shipping so that demand and supply match. This can be based on accurate forecasts, or just live data. As Dec 23rd wears on, the price of next-day shipping will keep going up and up so only the serious buy it. Of course, this might reveal just how keen some people are to get items, and justify having more capacity in years to come. Indeed, as the price goes up, it may make sense for Amazon to say, “Listen, we’re just going to buy this for you at your local Wal-Mart, it will be waiting for you there.” Wal-Mart surely won’t mind that.

There are also some tricks to increase capacity. For example, most people would probably tolerate having to pick up items at a retail location — FedEx and UPS and the USPS of course have tons of those — especially if it is the only option or offers a serious discount over surge priced home delivery. (This is not as good for sending gifts to remote locations.) Temporarily contracted depots could also be used. You want to streamline these depots, as lots of people will be coming in, so you want some nice system where people bring in a bar code and everything is optimized to get them out the door with the right package quickly.

All of this will push people to shop and ship a little earlier, smoothing out the rush, and avoiding having to design the system for one peak day. I have always found it remarkable that most stores and malls have giant parking lots (back in the brick and mortar world) which are only filled in December. It’s such a waste — but something robocars will fix in the future.

Delivery to the wrong address

I had a missed delivery myself this year. In this case it was on December 14th because I went home early, and I had the gifts arriving 2 days before I left. But oddly, I got the note that the package had been delivered at 6pm — but it wasn’t. Both UPS and Amazon had very little set up to handle this. Amazon’s system insists you wait at least a day to complain about this, which was no help to me. I could have used that day to replace the items if I were sure it wasn’t coming.

After I left, the package showed up on my porch on Sunday. UPS does not operate Sunday so it seems pretty likely they had left the package with a neighbour who was perhaps away for a few days. I presume the neighbour eventually came and dropped off the package but they left no note. (Of course I wish they had done it right away — replacing the gifts in Canada cost me a bunch extra.)

Amazon had already given a refund — fairly good service there — and so I just had UPS return the package as undelivered which costs me nothing, so that all worked out, except the scramble and the extra cost of replacing the items.

I don’t know how often this happens — it’s in the Amazon FAQ so it must be often enough — but there are some obvious fixes. The UPS driver’s wand, which scans the package on delivery, should record more data, including any location from a GPS in the wand or the truck, but perhaps more easily the MACs and signal strengths of any WIFI nodes visible when the package was scanned.

That information would have both allowed UPS to say, “OK, that’s odd, this doesn’t match where the package should be going” right when it was scanned, or it would have allowed me to figure out where it went and get it right away.

You’re probably wondering, didn’t I just imagine it was stolen? I did consider that possible, though in my safe neighbourhood it doesn’t appear to be a real danger. Somebody following UPS trucks at Christmas time to steal gifts would be very Grinchey, not to say it doesn’t happen. In safe neighbourhoods, UPS and Fedex routinely just leave packages at the door. Not actually signed for, I presume they just eat the loss the rare times they are stolen, or perhaps the merchant does. It’s small enough shrinkage that the system handles it.

Having secure open wifi (Death to wifi login part 2)

In part 1 I outlined the many problems caused by wifi login pages that hijack your browser (“captive portals”) and how to improve things.

Today I want to discuss the sad state of having security in WIFI in most of the setups used today.

Almost all open WIFI networks are simply “in the clear.” That means, however you got on, your traffic is readable by anybody, and can be interfered with as well, since random users near you can inject fake packets or pretend to be the access point. Any security you have on such a network depends on securing your outdoing connections. The most secure way to do this is to have a VPN (virtual private network) and many corporations run these and insist their employees use them. VPNs do several things:

  • Encrypt your traffic
  • Send all the traffic through the same proxy, so sniffers can’t even see who else you are talking to
  • Put you on the “inside” of corporate networks, behind firewalls. (This has its own risks.)

VPNs have downsides. They are hard to set up. If you are not using a corporate VPN, and want a decent one, you typically have to pay a 3rd party provider at least $50/year. If your VPN router is not in the same geographic region as you are, all your traffic is sent to somewhere remote first, adding latency and in some cases reducing bandwidth. Doing voice or video calls over a VPN can be quite impractical — some VPNs are all TCP without the UDP needed for that, and extra latency is always a killer. Also, there is the risk your VPN provider could be snooping on you — it actually can make it much easier to snoop on you (by tapping the outbound pipe of your VPN provider) than to follow you everywhere to tap where you are.

If you don’t have a VPN, you want to try to use encrypted protocols for all you do. At a minimum, if you use POP/IMAP E-mail, it should be configured to only get and receive mail over TLS encrypted channels. In fact, my own IMAP server doesn’t even accept connections in the clear to make sure nobody is tempted to use one. For your web traffic, use sites in https mode as much as possible, and use EFF’s plugin https everywhere to make your browser switch to https wherever it can.  read more »

Canada to stop urban mail home delivery, but fails to abolish snail-mail

Here in Canada, a hot political issue (other than disgust with Rob Ford) is the recent plan by Canada Post to stop home delivery in cities. My initial reaction was, “Wow, I wish we could get that in the USA!” but it turns out all they are doing is making people go to neighbourhood mailboxes to get their mail. For many years, people in new developments have had to do this — they install a big giant mailbox out on the street, and you get a key to get your mail. You normally don’t walk further than the end of your block. However, this will save a lot of work — and eliminate a lot of jobs, which also has people upset.

But let me go back to my original reaction — I want to see home letter delivery abolished.

Why? All I, and most other people get by mail are:

  • Junk mail (the vast bulk of the mail.)
  • One or two magazines
  • Bills and communications from companies that refuse to switch to all-electronic communication
  • Official notices (from governments who refuse to switch to all-electronic communication)
  • Cheques from companies who refuse to do direct deposit (see note below.)
  • Parcels (lots of these, though many more from UPS/Fedex/etc.)
  • A tiny and dwindling number of personal cards and letters. Perhaps 2-3 personal xmas cards.

The abolition of general mail delivery would force all those parties who refuse to do electronic communication to switch to it. The concept of an official e-mail address would arise. We would also need to see a better e-cheque service, something priced like a cheque (ie. not paypal which takes 2% or more) and as easy to use (ACH is not there yet.) This would force it into existing if you could not mail a cheque.

A replacement for registered mail would need to arise — that is what is needed for legal service. Putting that into e-mail is doable though challenging, as it requires adding money to e-mail, because you want people to have to pay to use it so that you don’t get it all the time.

And of course, parcel service would continue. And people who really want to send a letter could send it via parcel service, but not for sub-dollar first class mail prices.

Magazines would have to go all-electronic. Some may not see the world ready for that, but I think the time is very near. Today, one can make cheap large tablets in the 14 to 17 inch size that would be great for magazines. They would be too heavy to handhold (though possibly if they had no batteries and used a small cord they could be light enough for that) but they could easily be held on laps and tables and replace the magazine.

Few would mourn the death of junk mail, though it might lead to more spam in e-mail boxes until that’s under control. Senders of junk mail (notably politicians) might mourn it.

So the only sad thing would be the loss of the dwindling supply of personal letters. People getting married could use the parcel companies or go electronic. Thank-you notes would go electronic, making Miss Manners spin in her grave, but spin she eventually will. Truth is, the parcel companies would probably start up a basic letter service priced higher than 1st class mail but less than their most basic parcel. The more addresses you can share the cost of a truck on, the better — until the deliverbots arrive, at least. This is not easy, though. The postal service got to use the economies of delivering several letters a day to your house, and this could pay for a person to walk the street with a bag full, while the parcel companies use trucks.

We all know this day is coming. The question is, can we do better if we force it, and shut down letter delivery sooner rather than later?

Death to the Wifi login page (part 1)

It’s the bane of the wanderer. A large fraction of open Wifi access points don’t connect you to the internet, but instead want you to login somehow. They do this by redirecting (hijacking) any attempt to fetch a web page to a login or terms page, where you either have to enter credentials, or just click to say you agree to the terms of service. A few make you watch an ad. It’s sometimes called a captive portal.

I’m going to contend that these hijack screens are breaking a lot of things, and probably not doing anybody — including portal owners — any good.

The terms of service generally get you to declare you will be a good actor. You won’t spam or do anything illegal. You won’t download pirated content or join torrents of such content. You waive rights to sue the portal. Sometimes you have to pay money or show you are a hotel guest or have an access card.

These screens are a huge inconvenience, and often worse than that. All sorts of things go wrong when they are in place:

  • Until you do the login with the browser, your other apps, like e-Mail, don’t work though it looks like internet is there.
  • With devices that don’t have keyboards, like Google Glass, you can’t use the network at all!
  • Some redirect you from the link you wanted, and don’t pass you on to that link when you are logged in, you have to type it in again.
  • If you go to a secure URL (https) some of them attempt an insecure redirect and cause browser security warnings. They look like a hijack because they are a hijack! This trains people to be more tolerant of browser security warnings, and breaks tools that try to improve your security and stop more malicious hijacks properly.
  • Some for “security” block the remembering of credentials, making it hard to login every time.
  • Really bad ones time-out quickly, and make you repeat the login process every time you suspend your laptop, and worse, every time you turn off and turn on your phone — making the network almost unusable. Almost all require re-login one or two times a day — still very annoying.
  • Every so often the login systems are broken on mobile browsers, locking out those devices.

A lot of headaches. And one can perhaps understand the need for this when you must pay for the network or only authorized users are allowed in, though WPA passwords are much better for that because they need only one-time setup and also offer security on the wireless connection.

With all this pain, the question the world needs to answer is, “is it worth it?” What is the value of this hijack and “I agree” terms page? Nobody reads the terms, and people who connect, and would ignore the terms to spam or do other bad things, will happily agree to them and ignore them, and they will do so anonymously leaving no way to punish them for violating the terms. This is not to say that certain entities have not desired to actually find users of open Wifi networks and try to enforce terms on them, but this is extremely rare and almost certainly not desirable to most access point operators.

There are thus just a few remaining purposes for the hijack screen.

Charging money

If you want to charge money, you might need a login screen. I don’t deny the right of a provider to ask for money, but there are different ways to do it. There are a variety of aggregator networks (Such as Boingo and FON) which will handle billing. They have already installed an app on the user’s device which allows it to authenticate and handle billing (mostly) seamlessly for the user. The very common skype application is one of these, and people pay from their skype credit accounts. Of course, you may not like Skype’s rates or the cut it takes, so this may not be enough.

You might also want to consider why you are charging the money. If bandwidth is very expensive, I can see it, but it’s not been uncommon to find some sites like cafes saying they charge — I kid you not — because the whole system including the charging gateway — is expensive to run. A cheap free gateway would have been much more affordable. Many operators decide that it’s worth it to offer it free, since it draws people in to restaurants, cafes and hotels. Cheap hotels usually give free Wifi — only expensive hotels put on fat charges.

It could be that your real goal is just to get attention…

Letting them know who provided the Wifi

I’ve seen a number of gateways that primarily seem to exist just to let you know who provided the gateway. Very rarely (I’ve mostly seen this at airports) they will make you watch a short ad to get your free access. They break a lot of stuff to do this. The SSID name is another way to tell them, though of course it’s not nearly as satisfactory.

Reducing the amount of usage

There is a risk that fully open networks will get overused by guests, and often thanklessly, too. You may be afraid your neighbours will realize they don’t need to buy internet at all, and can just use your open network. Here, making it hard to use and broken is a feature, not a bug. If you have to go through the hijack every so often it’s a minor burden to cafe patrons but a bigger annoyance to overusing neighbours. Those neighbours can play tricks, like using programs that do automatic processing of hijack gateways, but not too many do. They can also change their MAC addresses to get past restrictions based on that. You can do MAC limiting without a hijack screen, and it’s a great way to do it, possibly saving the hijack for after they reach the limit, not using it at the start. Clever abusers can change their MACs, though again most people don’t.

Covering your ass

The large number of complex terms of service suggest that people believe, or have been told, that it is essential they keep themselves covered in case a user of open Wifi does something bad, such as spamming or violating copyrights or even nastier stuff. They figure that if they made them agree to a terms-of-service that forbade this, this absolves them of any responsibility for the bad actions, and even, just maybe, offers a way to go after the unwanted guest.

Turns out that there is much less need to cover your ass in this situation, at least in the USA. You aren’t liable for coypright infringement by your guests if you did not encourage it. Thanks to the DMCA and CDA rules, you are probably not liable for a lot of other stuff these unwanted guests might do.

I am interested to hear reports from anybody of how they used the fact that Wifi guests had to agree to terms of service to protect themselves in an actual legal action. I have not heard of any, and I suspect there are few. It would be a great shame to confirm that everybody is breaking their networks in hope of a protection that’s actually meaningless.

It is true that you can get in real world trouble for what your unwanted guests do. If they violate copyrights, you might be the one getting the nasty letter from the copyright holder. The fact that you are not actually liable may not be much comfort when you are faced with taking the time and cost to point that out. Often these lawsuits come with offers to settle for less than the cost of consulting a lawyer on the matter. Naturally, those interested in violating copyrights are unlikely to be all that worried that they clicked on a contract that promised they wouldn’t. This is just a risk of an open network.

Likewise, if they send spam over your network, you may find yourself on spam-blocking blacklists who don’t care that it wasn’t you who did the spamming. Those vigilante groups run by their own rules. Again, the contract isn’t much protection. You may instead want to look to technical measures, including throttling the use of certain ports or bandwidth limits on guests. (It is better if you can throttle rather than cut off, since your guests probably do need to send e-Mail, just not thousands of them.)

Towards a protocol of open guest WIFI

How could we do this better? In part two I talk about how to have a secure open WIFI and the problems in doing that. Part three will talk about how to make it easy to connect to any of these networks automatically.

Going beyond the vacation program with dynamic status

I’m back from Burning Man, and this year, for the first time in a while, we didn’t get internet up in our camp, so I only did occasional email checks while wandering other places. And thus, of course, there are many hundred messages backed up in my box to get to. I will look at the most important but some will just be ignored or discarded.

We all know it’s getting harder and harder to deal with email backlog after travel, even connected travel. If you don’t check in it gets even worse. Vacation autoreplies can help a little, but I think they are no longer enough.

Some years ago a friend tried something radical. He had is autoreply say that he was away for 2 months, and could not possibly handle the email upon his return. He said that thus the email you had sent had been discarded. You were told that if it was still important when he returned that you should send it again then. His correspondents were completely furious at the temerity of this action, though it has a lot of attractions. They had taken the time to write an email, and to have it discarded and left in their hands to resend seemed rude. (I believe the reply included a copy of the email at least.)

Worse, because we are always connected, vacation replies sometimes lie. People are scanning their email, responding to the most important ones if they can, even though a vacation autoreply was sent. And so we always hope for that.

I think the time has come for an extra internet protocol as a companion to mail. When you type an E-mail address into your mail client, it should be able to query a server that handles information for that domain — something like an MX record — and query it about the email that is about to be written, including the sender address and recipient address, and possibly a priority. If the recipient is in a vacation mode or other do not disturb mode, the sender would be told immediately, before writing the e-mail. They would have the option of not writing it, writing it for delivery at the designated date in the future, or writing it with various tags about its urgency in case the recipient is doing some checking of mail.

This could be an LDAP derived protocol or something else. Indeed it could be combined, when trusted, with directory lookup and autocomplete directory services. It’s not easy because often (with things like MX) the server that handles mail for a user may not have a strong link to the user in order to serve this data. In the old e-mail regime of store and forward, live connections were not expected. Still, I think it can be done, and it would not be a mandatory thing.

There are some security and privacy implications here that are challenging:

  • Spammers will try to use this information to confirm addresses or hunt for them
  • This lets the recipient know if somebody just typed in their name to send mail, and when they did so, and thus how long they took to write a mail, or if they aborted one. To avoid this, the directory servers could be trusted 3rd parties.
  • This provides a reliable IP address for the sender’s client, or at least a proxy acting for the sender.
  • It could be misused to build a general database of many people’s vacation status, invading their privacy, unless there are tools to prevent broad spidering of this sort.

Mail servers would remember who queried, and in fact it might be encouraged to include a header in the email that came from the query, to officially tie them together. This would allow clients to know who queried and who did not, giving priority to messages which came from people who queried and acted upon the result (for example waiting to send) over those who just sent mail without checking. Users could get codes that would allow them to declare the message higher (or lower) priority that would not be available to those who just did plain SMTP.

Mailing lists might also make use of this data, and the response could tell mailing lists what the user wants to do, including temporarily unsubscribing until a given date, or asking for a digest of threads to be sent upon return, or other useful stuff. Responsible corporate bulk mailers could also accept that you don’t want customer satisfaction surveys or useful coupon offers during your vacation and just not send them. Ok, I’m dreaming on that one, perhaps.

For security, it could be that only past correspondents could do this query, or only users with some amount of authentication. Anonymous email and mail from strangers would still be possible, but not with a pre-query. The response could also be sent back via a special email that servers know to intercept, so it can’t be used to gain information that would not be gained by mailing a person today. (You could get a report of people who queried you and never mailed you when not on vacation.)

We might see some features in mailers, like a pop-up in your mailers that says, “Brad just started writing you a message” the way instant messaging programs do. I am not sure this is a good idea, but it would happen. Readers: what other consequences do you see happening?

Kayak sailing "san juan islands"

Yahoo announced that in a few days they will shut down the altavista web site. This has prompted a few posts on the history of internet search, to which I will add an anecdote.

The first internet search engine predated the “web” and was called Archie search engine. Archie (an archive search) was basic by today’s standards. The main protocol for getting files on the internet in those days was FTP. Many sites ran an open FTP server, which you could connect to and download files from. If you had files or software to share with people, you put it up on an FTP server, in particular one that allowed anonymous login to get public files. The Archie team (from Montreal) built a tool to go to all the open servers, read their indexes and generate a database. You could then search, and get a pointer to all the places you could get a file. It was hugely popular for the day.

(You will probably note that this is almost exactly the way Napster worked, the only difference being that Napster was a bit more sophisticated and people used it to share files that were copyrighted. FTP servers had copyrighted material, but mostly they had open source software and documents.)

Around the same time, a lot of folks were building full-text search engines for use on large collections of documents. You could find these on private databases around the world, and the WAIS protocol was developed by Brewster Kahle to make a standardized interface to text search and his own text search tools.

Not long after the web started to grow, Fuzzy Mauldin at CMU made Lycos which was a full-text search engine applied to documents gathered from the web. The ability to search the web generated much attention, and a few other competing spiders and search engines appeared. Everybody had a favourite. (To add to my long list of missed opportunities, in April of 95 I wrote a few notes to Fuzzy looking to get his spider index so we could sort web pages based on how many incoming links they had. Nothing ever came of that but as you may know that concept later had some value. :-) And I also turned down a $4M offer from Lycos to buy ClariNet (which would have turned into $40M when their stock shot up in the bubble. Sigh.)

In 1995, for many people that favourite changed to Alta Vista, a new search engine from Digital Equipment Corp. DEC was a huge name at the time, the biggest name in minicomputers, and it was just losing the Unix crown to Sun. The team at DEC put a lot of computing power into Alta Vista, and so it had two useful attributes. First, they spidered a lot more pages, and thus were more likely to find stuff. They were also fast compared to most of the other engines. In a precursor to other rapid turnarounds in the internet business, you could switch your favourite search engine in a heartbeat and many did. It was big and fast due to DEC putting a lot of fancy computer hardware on it, and DEC eventually justified the money they were spending on it (there was no revenue for search in those days) by saying it showed off just how powerful DEC’s computers with big address spaces were. Indeed the limits of Alta Vista were the limits of the architecture, using the 64-bit Alpha to address 130gb of RAM and 500gb of disk — huge for the day.

On Alta Vista’s home page, they gave you a sample query to type in the search box, to show you how to use it. That query was:

kayak sailing “san juan islands”

Indeed, if you typed that, you got a nice array of pages which talked about kayaking up in the San Juan islands, tour operators, etc. — just what you wanted to get from a query.

My devious mind wondered, “what if I put up a page on my own web site with this as the title?” I created the Kayak Sailing “San Juan Islands” home page on the rec.humor.funny site, which was already a very popular site in those days. (Indeed it’s around 1995 that RHF fell behind Yahoo as the most widely read thing on the internet, but that refers to the USENET group, not the page.)

You will note as you look at the page that it contains the words in the title and headers, and repeated many times in invisible comments. In those days the search engines were ranking higher simply based on where words were, and if they were repeated many times. So I gave it a whirl. This was an early attempt at what is now called “black hat search engine optimization” though I was doing it for fun, rather than nefarious gain.

The results didn’t change though. Alta Vista relied on huge computer power, but it only rebuilt the index by hand. It would be a month or more before Alta Vista recalculated its index. One day I went to type in the query and bingo — there was my page on the first page of search results. Along with a dozen other people who had tried the same thing, and a few pages that were articles writing about Alta Vista and giving the example query, or which were copying its search page which of course had that string.

More to the point, not a single item on the results page was about actual Kayaking! The sample query was ruined, though the results were quite amusing. Not long after, Alta Vista changed the example to Pizza “deep dish” Chicago and of course I added it to my page as well. So not much longer after that AV switched to showing different examples from a rotating and changing collection so people could not play this game any more.

While Alta Vista ruled Search, in spite of efforts from Infoseek, Inktomi/Hotbot and others, we all know that a few years later, Google was born at Stanford, and it proved again how quickly people could switch to a new favourite search engine, and lives under that fear (but with great success) to this day. And Google’s dominance turned SEO into a giant industry.

A Bitcoin Analogy

Bitcoin is having its first “15 minutes” with the recent bubble and crash, but Bitcoin is pretty hard to understand, so I’ve produced this analogy to give people a deeper understanding of what’s going on.

It begins with a group of folks who take a different view on several attributes of conventional “fiat” money. It’s not backed by any physical commodity, just faith in the government and central bank which issues it. In fact, it’s really backed by the fact that other people believe it’s valuable, and you can trade reliably with them using it. You can’t go to the US treasury with your dollars and get very much directly, though you must pay your US tax bill with them. If a “fiat” currency faces trouble, you are depending on the strength of the backing government to do “stuff” to prevent that collapse. Central banks in turn get a lot of control over the currency, and in particular they can print more of it any time they think the market will stomach such printing — and sometimes even when it can’t — and they can regulate commerce and invade privacy on large transactions. Their ability to set interest rates and print more money is both a bug (that has sometimes caused horrible inflation) and a feature, as that inflation can be brought under control and deflation can be prevented.

The creators of Bitcoin wanted to build a system without many of these flaws of fiat money, without central control, without anybody who could control the currency or print it as they wish. They wanted an anonymous, privacy protecting currency. In addition, they knew an open digital currency would be very efficient, with transactions costing effectively nothing — which is a pretty big deal when you see Visa and Mastercard able to sustain taking 2% of transactions, and banks taking a smaller but still real cut.

With those goals in mind, they considered the fact that even the fiat currencies largely have value because everybody agrees they have value, and the value of the government backing is at the very least, debatable. They suggested that one might make a currency whose only value came from that group consensus and its useful technical features. That’s still a very debatable topic, but for now there are enough people willing to support it that the experiment is underway. Most are aware there is considerable risk.

Update: I’ve grown less fond of this analogy and am working up a superior one, closer to the reality but still easy to understand.

Wordcoin

Bitcoins — the digital money that has value only because enough people agree it does — are themselves just very large special numbers. To explain this I am going to lay out an imperfect analogy using words and describe “wordcoin” as it might exist in the pre-computer era. The goal is to help the less technical understand some of the mechanisms of a digital crypto-based currency, and thus be better able to join the debate about them.  read more »

V2V and connected car part 3: Broadcast data

Earlier in part one I examined why it’s hard to make a networked technology based on random encounters. In part two I explored how V2V might be better achieved by doing things phone-to-phone.

For this third part of the series on connected cars and V2V I want to look at the potential for broadcast data and other wide area networking.

Today, the main thing that “connected car” means in reality is cell phone connectivity. That began with “telematics” — systems such as OnStar but has grown to using data networks to provide apps in cars. The ITS community hoped that DSRC would provide data service to cars, and this would be one reason for people to deploy it, but the cellular networks took that over very quickly. Unlike DSRC which is, as the name says, short range, the longer range of cellular data means you are connected most of the time, and all of the time in some places, and people will accept nothing less.

I believe there is a potential niche for broadcast data to mobile devices and cars. This would be a high-power shared channel. One obvious way to implement it would be to use a spare TV channel, and use the new ATSC-M/H mobile standard. ATSC provides about 19 megabits. Because TV channels can be broadcast with very high power transmitters, they reach almost everywhere in a large region around the transmitter. For broadcast data, that’s good.

Today we use the broadcast spectrum for radio and TV. Turns out that this makes sense for very popular items, but it’s a waste for homes, and largely a waste for music — people are quite satisfied instead with getting music and podcasts that are pre-downloaded when their device is connected to wifi or cellular. The amount of data we need live is pretty small — generally news, traffic and sports. (Call in talk shows need to be live but their audiences are not super large.)

A nice broadcast channel could transmit a lot of interest to cars.

  • Timing and phase information on all traffic signals in the broadcast zone.
  • Traffic data, highly detailed
  • Alerts about problems, stalled vehicles and other anomalies.
  • News and other special alerts — you could fit quite a few voice-quality station streams into one 19 megabit channel.
  • Differential GPS correction data, and even supplemental GPS signals.

The latency of the broadcast would be very low of course, but what about the latency of uploaded signals? This turns out to not be a problem for traffic lights because they don’t change suddenly on a few milliseconds notice, even if an emergency vehicle is sending them a command to change. If you know the signal is going to change 2 seconds in advance, you can transmit the time of the change over a long latency channel. If need be, a surprise change can even be delayed until the ACK is seen on the broadcast channel, to within certain limits. Most emergency changes have many seconds before the light needs to change.

Stalled car warnings also don’t need low latency. If a car finds itself getting stalled on the road, it can send a report of this over the cellular modem that’s already inside so many cars (or over the driver’s phone.) This may take a few seconds to get into the broadcast stream, but then it will be instantly received. A stalled car is a problem that lasts minutes, you don’t need to learn about it in the first few milliseconds.

Indeed, this approach can even be more effective. Because of the higher power of the radios involved, information can travel between vehicles in places where line of sight communications would not work, or would actually only work later than the server-relayed signal. This is even possible in the “classic” DSRC example of a car running a red light. While a line of sight communication of this is the fastest way to send it, the main time we want this is on blind corners, where LoS may have problems. This is a perfect time for those longer range, higher power communications on the longer waves.

Most phones don’t have ATSC-M/H and neither do cars. But receiver chips for this are cheap and getting cheaper, and it’s a consumer technology that would not be hard to deploy. However, this sort of broadcast standard could also be done in the cellular bands, at some cost in bandwidth for them.

19 megabits is actually a lot, and since traffic incidents and light changes are few, a fair bit of bandwidth would be left over. It could be sold to companies who want a cheaper way to update phones and cars with more proprietary data, including map changes, their own private traffic and so on. Anybody with a lot of customers might fight this more efficient. Very popular videos and audio streams for mobile devices could also use the extra bandwidth. If only a few people want something, point to point is the answer, but once something is wanted by many, broadcast can be the way to go.

What else might make sense to broadcast to cars and mobile phones in a city? While I’m not keen to take away some of the nice whitespaces, there are many places with lots of spare channels if designed correctly.

Solving V2V Part 2: Make it Phone to Phone

Last week, I began in part 1 by examining the difficulty of creating a new network system in cars when you can only network with people you randomly encounter on the road. I contend that nobody has had success in making a new networked technology when faced with this hurdle.

This has been compounded by the fact that the radio spectrum at 5.9ghz which was intended for use in short range communications (DSRC) from cars is going to be instead released as unlicenced spectrum, like the WiFi bands. I think this is a very good thing for the world, since unlicenced spectrum has generated an unprecedented radio revolution and been hugely beneficial for everybody.

But surprisingly it might be something good for car communications too. The people in the ITS community certainly don’t think so. They’re shocked, and see this as a massive setback. They’ve invested huge amounts of efforts and careers into the DSRC and V2V concepts, and see it all as being taken away or seriously impeded. But here’s why it might be the best thing to ever happen to V2V.

The innovation in mobile devices and wireless protocols of the last 1-2 decades is a shining example to all technology. Compare today’s mobile handsets with 10 years ago, when the Treo was just starting to make people think about smartphones. (Go back a couple more years and there weren’t any smartphones at all.) Every year there are huge strides in hardware and software, and as a result, people are happily throwing away perfectly working phones every 2 years (or less) to get the latest, even without subsidies. Compare that to the electronics in cars. There is little in your car that wasn’t planned many years ago, and usually nothing changes over the 15-20 year life of the car. Car vendors are just now toying with the idea of field upgrades and over-the-air upgrades.

Car vendors love to sell you fancy electronics for your central column. They can get thousands of dollars for the packages — packages that often don’t do as much as a $300 phone and get obsolete quickly. But customers have had enough, and are now forcing the vendors to give up on owning that online experience in the car and ceding it to the phone. They’re even getting ready to cede their “telematics” (things like OnStar) to customer phones.

I propose this: Move all the connected car (V2V, V2I etc.) goals into the personal mobile device. Forget about the mandate in cars.

The car mandate would have started getting deployed late in this decade. And it would have been another decade before deployment got seriously useful, and another decade until deployment was over 90%. In that period, new developments would have made all the decisions of the 2010s wrong and obsolete. In that same period, personal mobile devices would have gone through a dozen complete generations of new technology. Can there be any debate about which approach would win?  read more »

The importance of serial media vs. sampled and Google Reader

The blogging world was stunned by the recent announcement by Google that it will be shutting down Google reader later this year. Due to my consulting relationship with Google I won’t comment too much on their reasoning, though I will note that I believe it’s possible the majority of regular readers of this blog, and many others, come via Google reader so this shutdown has a potential large effect here. Of particular note is Google’s statement that usage of Reader has been in decline, and that social media platforms have become the way to reach readers.

The effectiveness of those platforms is strong. I have certainly noticed that when I make blog posts and put up updates about them on Google Plus and Facebook, it is common that more people will comment on the social network than comment here on the blog. It’s easy, and indeed more social. People tend to comment in the community in which they encounter an article, even though in theory the most visibility should be at the root article, where people go from all origins.

However, I want to talk a bit about online publishing history, including USENET and RSS, and the importance of concepts within them. In 2004 I first commented on the idea of serial vs. browsed media, and later expanded this taxonomy to include sampled media such as Twitter and social media in the mix. I now identify the following important elements of an online medium:

  • Is it browsed, serial or to be sampled?
  • Is there a core concept of new messages vs. already-read messages?
  • If serial or sampled, is it presented in chronological order or sorted by some metric of importance?
  • Is it designed to make it easy to write and post or easy to read and consume?

Online media began with E-mail and the mailing list in the 60s and 70s, with the 70s seeing the expansion to online message boards including Plato, BBSs, Compuserve and USENET. E-mail is a serial medium. In a serial medium, messages have a chronological order, and there is a concept of messages that are “read” and “unread.” A good serial reader, at a minimum, has a way to present only the unread messages, typically in chronological order. You can thus process messages as they came, and when you are done with them, they move out of your view.

E-mail largely is used to read messages one-at-a-time, but the online message boards, notably USENET, advanced this with the idea of move messages from read to unread in bulk. A typical USENET reader presents the subject lines of all threads with new or unread messages. The user selects which ones to read — almost never all of them — and after this is done, all the messages, even those that were not actually read, are marked as read and not normally shown again. While it is generally expected that you will read all the messages in your personal inbox one by one, with message streams it is expected you will only read those of particular interest, though this depends on the volume.

Echos of this can be found in older media. With the newspaper, almost nobody would read every story, though you would skim all the headlines. Once done, the newspaper was discarded, even the stories that were skipped over. Magazines were similar but being less frequent, more stories would be actually read.

USENET newsreaders were the best at handling this mode of reading. The earliest ones had keyboard interfaces that allowed touch typists to process many thousands of new items in just a few minutes, glancing over headlines, picking stories and then reading them. My favourite was TRN, based on RN by Perl creator Larry Wall and enhanced by Wayne Davison (whom I hired at ClariNet in part because of his work on that.) To my great surprise, even as the USENET readers faded, no new tool emerged capable of handling a large volume of messages as quickly.

In fact, the 1990s saw a switch for most to browsed media. Most web message boards were quite poor and slow to use, many did not even do the most fundamental thing of remembering what you had read and offering a “what’s new for me?” view. In reaction to the rise of browsed media, people wishing to publish serially developed RSS. RSS was a bit of a kludge, in that your reader had to regularly poll every site to see if something was new, but outside of mailing lists, it became the most usable way to track serial feeds. In time, people also learned to like doing this online, using tools like Bloglines (which became the leader and then foolishly shut down for a few months) and Google Reader (which also became the leader and now is shutting down.) Online feed readers allow you to roam from device to device and read your feeds, and people like that.  read more »

Mesh networking when the cell network fails

Interesting article about a new plan for mesh networking Android phones if the cell network fails. I point this out because of another blog post of mine from 2005 on a related proposal from Klein Gilhousen that he was pushing after Katrina.

The wifi mesh has the problem that wifi range is not going to get much better then 30-40m, and so you need a very serious density of phones to get a real mesh going, especially to route IP as this plan wishes to. Klein’s plan was to have the phones mesh over the wireless bands that were going unusued when the cell networks were dead (or absent in the wilderness.) The problem with his plan was that phone tranceivers tend to not be able to transmit and receive on the same bands, they need a cell tower. He proposed new generations of phones be modified to allow that.

But it hasn’t happened, in spite of being an obviously valuable thing in disasters. Sure there are some interference issues at the edges of legitimate cell nets, but they could be worked out. Cell phones are almost exclusively sold via carriers in the many countries, including the USA. They haven’t felt it a priority to push for phones that can work without carriers.

I suspect trying to route voice or full IP is also a mistake, especially for a Katrina like situation. There the older network technologies of the world, designed for very intermittent connectivity, make some sense. A network designed to send short text messages, a “short message service” if you will, using mesh principles combined with store and forward could make sure texts got to and from a lot of places. You might throw in small photos so trapped people could do things like send photos of wounds to doctors.

Today’s phones have huge amounts of memory. Phones with gigabytes of flash could store tens to hundreds of millions of passing (compressed and encrypted) texts until work got out that a text had been delivered. Texts could hop during brief connections, and airplanes, blimps and drones could fly overhead doing brief data syncs with people on the ground. (You would not send every text to every phone, but every phone would know how many hops it has been recently from the outside, and you could send always upstream.) A combination of cell protocols when far and wifi when close (or to those airplanes) could get decent volumes of data moving.

Phones would know if they were on their own batteries, or plugged into a car or other power source, and the ones with power would advertise they can route long term. It would not be perfect but it would be much better than what we have now.

But the real lament is that, as fast as the pace of change is in some fields of mobile, here we are 7.5 years after Katrina, having seen several other disasters that wiped out cell nets, and nothing much has changed.

Syndicate content