You are here

The dark ages of lost data are over


For much of history, we've used removable media for backup. We've used tapes of various types, floppy disks, disk cartridges, and burnable optical disks. We take the removable media and keep a copy offsite if we're good, but otherwise they sit for a few decades until they can't be read, either because they degraded or we can't find a reader for the medium any more.

But I now declare this era over. Disk drives are so cheap -- 25 cents/gb and falling, that it no longer makes sense to do backups to anything but hard disks. We may use external USB drives that are removable, but at this point our backups are not offline, they are online. Thanks to the internet, I even do offsite backup to live storage. I sync up over the internet at night, and if I get too many changes (like after an OS install, or a new crop of photos) I write the changes to a removable hard disk and carry it over to the offsite hard disk.

Of course, these hard drives will fail, perhaps even faster than CD-roms or floppies. But the key factor is that the storage is online rather than offline, and each new disk is 2 to 3 times larger than the one it replaced. What this means is that as we change out our disks, we just copy our old online archives to our new online disk. By constantly moving the data to newer and newer media -- and storing it redundantly with online, offsite backup, the data are protected from the death that removable media eventually suffer. So long as disks keep getting bigger and cheaper, we won't lose anything, except by beng lazy. And soon, our systems will get more automated at this, so it's hard to set up a computer that isn't backed up online and remotely. We may still lose things because we lose encryption keys, but it won't be for media.

Thus, oddly, the period of the latter part of the 20th century will be a sort of "dark ages" to future data archaeologists. Those disks will be lost. The media may be around, but you will have to do a lot of work to recover them -- manual work. However, data from the early 21st onward will be there unless it was actively deleted or encrypted.

Of course this has good and bad consequences. Good for historians. Perhaps not so good for privacy.


Not to mention google is backing up everything you do . . . .

It surprises me that you view the hard disks as a blessing and not a curse for the preservation of data. Sure they work if one is willing to actively propagate information forward. What happens when one stops? Within a few decades the archives die, and are not very likely to be recovered by the historians (especially if the recovery is happening centuries in the future).

Why may one stop being able to propagate the data forward? There can be plenty of reasons. The civilization as we know it may collapse (not die out, but collapse); we may run out of the (cheap) resources to produce more and more storage media; or more pragmatically the people who currently find the data valuable may die, which does not mean that the data will not be valuable to historians millenia later.

If we can ever hope to escape these dark ages, what we need is something that is able to withstand the test of time on its own. Unfortunately, I don't really know of anything besides paper and stone. Do you?

I will agree there is a problem in the event of collapse of civilization. Certainly old hard disk motors will not survive too long. I don't know what the lifetimes are for the magnetic recordings but it's probably not great.

Alas, you probably will never find a market for a product just because it is likely to retain information beyond the buyer's lifetime. Except to the long now and archival folks. What are the lifetimes of compact flash and similar storage?

How about political dissidents and political prisoners? The data that they are able to retain are of prime interest to historians 50-100 years after the events, and are oftentimes the only way for us to get any insight into the era. It's feasible to imagine that their hard drives outlive them, it's much harder to imagine that those hard drives are readable 50 years later when somebody can get to them without being prosecuted. Imagine if the 1930s of the Soviet Union happened in the digital era, and you'll see what I'm trying to say.

I once thought that microfilm/microfiche would never die because unlike digital media, it can last over 100 years. And to comfort those who worry about a collapse of civilization, information on microfilm can be retrieved using only simple enlargement techniques.


Add new comment

Subscribe to Comments for "The dark ages of lost data are over"