Brad Ideas - Making RAID easier - Comments

Not reliable enough

brad — Thu, 05 Jun 2008 23:22:27 -0700

Disks don’t fail very often, but it’s still too often. I’ve had 4 or 5 fail — eventually — in the last 20 years or so. Doesn’t sound like a lot, but when there’s important data on the disk, even one is too many. Failed drives have cost me more in disk recovery fees than I have spent on 10 drives, and more than that in lost time. And more than that in lost data which was not recovered.

So of course, you should make backups. But there would be no drive recovery companies if people always did that. Tools to automate backup, especially offsite, would be better than RAID, but until this becomes the norm, there’s a need for more reliability. My proposal is to make something that’s simple, in fact in raid-1 mode it’s just as simple as a regular drive until one of the units fails, at which point it beeps at you and you go get a replacement to put in. (No need for hot swap.)

Power is a concern, though as I said, I don’t mind having one piece of drive electronics. Turns out a two-platter drive does take a fair bit more power than a 1-platter drive for the motor, so it’s not twice the power, but it is more. 3-disk raid certainly is more power.

Also good would be better SMART support, so the OS is very vocal with you about failing drives and getting you to replace them — and making it easy to do so.

wrong assumptions about disks and users

gpshead — Thu, 05 Jun 2008 21:47:25 -0700

Too much of what you claim is just plain wrong. Drives are vulnerable? Drive vendors control what their customers want? The benefits of doubling or tripling one of the most expensive components in a computer outweigh the costs? RAID needs hardware acceleration to be fast? false, false, false, false & false.

The drive industry does exactly what customers demand. They demand cheap reliability. They get it. If there were demand for more reliability at the disk level it would already be done.

Cost: A disk costs between $50-200. When an entire computer costs $350, doubling or tripling the cost of the disk ain't gonna happen. It does not bring benefit equal to its cost to 99% of computer users. The other 1% are more likely to read this blog and have already configured a bazillion RAIDs in the past.

Windows is more likely to fail and trash itself and the system requiring a reinstall or restore equivalent than the hard drive is to fail. Users are used to this (complacent). A RAID does not protect against the most common failure: software. Backups do. Most computers also lack ECC RAM which means memory errors go undetected and happily get written to disk. RAID does not protect against that.

Power consumption: A 3.5" drive consumes ~10W. Thats an additional $1-2/month for always on computers. A 2-3 pack of drives can not be made any more energy efficient than a single drive could. Don't advocate RAID and then toss out the R, A and I.

Customer service/support costs (assuming you're suggesting the HPs and Dells of the world should do this): More drives == more parts that can fail. Even if the computer isn't broken when one drive has failed it still needs to be serviced by someone somehow.

Value of data: If your data is valuable you make backups. period. otherwise you really don't value your data. A raid isn't a backup. Many people undervalue their data and learn the hard way. Many raid users don't do backups and learn the hard way.

My home server has a two disk RAID1 and ECC ram. I do it to prevent painful time consumed by a reinstall/configure/restore. Its also low power (35w at max cpu load) and virtually silent. I'm not a normal user.

USB and 100mbit just won't cut it

brad — Wed, 28 May 2008 18:26:54 -0700

USB (and certainly 100 mbit ether) won’t cut it for your main disk. They are fine for remote servers for files that do not need to be accessed quickly, such as video. I would not want it for photo storage. USB seems to only pull about 30 megabytes/second. A 3 disk RAID-5 should be able to give you about 140 megabytes/second sustained read, a 4-disk even more.

And that is too much even for gigabit ether, and would saturate the gigE while doing so.

An external box using eSata would make sense. The 3 gigabits there should keep you happy. Such a box could then also have gigE for access by other computers.

However, most people don’t want to have an external box for their disks.

Now, one reasonable architecture is a regular disk in the PC, and regular mirroring to a NAS box. In that case you are fine with the ethernet, even at 100mbps. However, you should also be looking into offsite.

One product that might make sense is a NAS box which not only provides reliable storage and mirror of the fast disk in each PC, but also which can be “paired” with a friend’s similar NAS box in a remote location. The two boxes would then do offsite backup for one another, automatically. Using an efficient incremental algorithm. Of course, your data would be encrypted on the friend’s box with a key stored in a 3rd place, as well as on your home box.

In such a system, there would be a special disk allocation system, mounted on two drives. New files on the server would be stored twice, once on each disk. Files from local PCs would be stored only once (there’s already a copy on the PC.) Over time, new files would be mirrored to the offsite box. Once mirrored, they need only be stored once, freeing up storage.

Thus the only thing stored twice would be files not yet sent offsite. If, due to internet connection limits, this grew too large, users would have the option of connecting a USB drive and writing the delta to it, and physically moving it to the offsite location, which need not be too far away — just not so far that a major catastrophe would not get both.

In this circumstance you could survive any disk failure, though physical travel would be needed for failures on the server for fast response. And you could survive your house burning down, too.

Drobo

Jay Cuthrell — Wed, 28 May 2008 16:14:30 -0700

I still like the concept of the Drobo. Folks that I know with them seem very content to use them. These are the same folks that deal with storage for a living too so it isn't that they don't want to roll their own... they just choose a simpler path.

green?

Moz — Wed, 28 May 2008 14:18:25 -0700

Jess, we don't have air conditioning and we have a total of 10 light bulbs in the apartment, all compact fluoros and rarely do we run them all at the same time. Our major electricity use is the fridge, and after that it's computers. Since we mostly cook with gas the electric stove doesn't use much power either. While we're on that topic, our transport emissions mostly come from the food we buy, since we all ride bikes and occasionally use public transport (yes, we carefully selected an apartment where that is possible. It didn't even cost more, because the average person doesn't yet consider it important).

AW: green?

jess — Wed, 28 May 2008 10:44:00 -0700

it's nice that you worry so much about using your laptop instead of a regular PC and monitor, but let's be honest, the energy you use to power your pc is probably pretty low in comparison to other devices (lights, AC, and so) on actually contribute.

as to brad's post. i definitely think it sucks big time you cannot buy and isntall new components for laptops as easily as for regular pc's. ia m sure that somehow this would be technically possible, but then the companies would sell fewer new laptops...

Seen some of those

brad — Wed, 28 May 2008 10:40:13 -0700

Even bought one of those multi-bay units but it didn’t fit well in my case.

There are 3.5” disks that are less than 5.25 inches long, and thus will fit sideways in a 5.25” bay. They are just slightly longer this way than many CD-rom drives so should fit in most systems. The electronics add very slightly to the length, and the connectors add a bunch, but in this case I am proposing different electronics and thus no connector issue. And there is room for a low-profile laptop style DVD drive on top.

This is useful because you get one unit you can slot into a PC’s DVD bay, and get Raid-1 and DVD all in one box. I will contend that every PC needs this. Add another drive for RAID-5, though probably with some software assistance for the conversion of the raid-1 to a raid-5.

The goal is to make it cheap and easy. It’s only cheap of course if this does become standard and everybody is buying it.

With today’s multi-core processors, I like software raid but as yet it’s not nearly simple enough for most computer users. Hell, it’s too much work for most advanced admins. An ideal would be hardware raid with full software instrumentation so the software can control it, configure it etc. but doesn’t have to do so except for complex operations (like convert Raid-1 to Raid-5).

For servers on all the time, power is a factor. Here it’s $8 per watt for a device on for 3 years. An 8w drive uses $64 of power, a raid controller even more. That does make RAID power-expensive, since the drives cost only a bit more than that. (750gb for $99 last weekend at Fry’s) This makes RAID-6 or frankly even 4-disk Raid-5 seem less practical for personal machines.

green?

Moz — Wed, 28 May 2008 03:37:49 -0700

My approach to greening the system is to turn it off.

A great deal of what I use the computer for can be done using a second hand ex-corporate laptop, so I do that in the mornings and after work, then once I've had dinner I crank up the big machine. Although most of the power actually goes into dual 20" monitors rather than the PC. According to the UPS the monitors run about 50W each on ~80% brightness (to the nearest 3W) while the PC uses about 60W when I'm browsing and 80W when the RAID disks are all spun up (more when defragging, about 230W when defragging all disks and running the Prime CPU torture test).

The laptop uses about 20W. It has just enough disk to run Windows and uTorrent, enough RAM that when I have to I can run PhotoShop, but mostly it just idles along.

So 20W to browse... 200W to play C-Evo :)

Variations abound already...

Moz — Wed, 28 May 2008 03:31:07 -0700

I've seen units that stack 4 3.5" disks into 3 5" bays, but I think your maths is dodgy if you expect to get them into two. With 2.5" disks that's not a problem size wise, just cost wise. But then, for general use most users can get by with ~.5TB from a few of the current 200GB-ish 2.5" drives.

There are increasingly many external NAS boxes that do RAID5, some of them quite well. For most users these make more sense, especially then genuine NAS ones that allow multiple users to connect rather than the cheaper USB-only ones (why anyone would buy a disk where filling it takes a week I do not know... but people do the same with USB keys).

I think the better solution is those two combined - run a pair of small-ish disks mirrored by your motherboard plus an external big disk. That way a single disk (or mobo) failure does not kill you, and even if the PC dies you will probably still have the external media disk in usable condition. With NAS, I would probably go for some sort of near-live backup, probably daily replication to a second copy of the live array. Buying two NAS boxes sounds expensive until you look at the cost of reacquiring 2TB or more of data (even just scrounging recorded TV shows off friends will take a while).

There are also increasingly affordable PCIe RAID cards that don't completely suck, and even a few motherboards that do it too. When I bought my current PC and extra $50 or so got me a "super deluxe" motherboard with RAID1 (and claytons RAID) built in, and while it chokes at about 50MB/s I still use it to mirror a pair of drives for booting. These days a mobo with 8 SATA ports is not uncommon, and I believe some of them even have dedicated RAID processors (although with quad core that may not be so necessary any more).

At the higher end, I like quiet PCs so I bought a big case with slow fans and lots of bays (P180), plus an expensive RAID card (Areca 4 port). That let me start with 4x250GB disks then upgrade some time later to 4x750GB (tripling disk capacity means I can copy the old array onto one disk, build a new array with the other three and copy onto it, then add the fourth disk, so I always have two copies of the data). Then when 1TB disks came out I bought a couple of 2TB external backup units (two 1TB disks in claytons RAID) and a FireWire 800 card so they ran a little faster.

My lessons from that experience are mostly to do with buying excess capacity in non-upgradeable components - I should have spent 50% more and got an 8 port card so that I can just keep adding disks. Areca offer RAID6 even on the four port cards, but with 8 ports it makes sense to bulk up a 4 disk RAID5 to 6-8 disk RAID6 once drive prices drop (if you have the bays for it).

Oh, and TrueCrypt currently sucks when combined with RAID because it serialises access. But I do not want 2TB of every file I've got falling into the hands of some random burglar or sneak thief. 2TB via USB2 is more than two days, with FireWire800 it's under a day. That $50 card was a bargain :) When I bought the first 2TB disk the first thing I did was copy my current system across to it... it took a very long time. The second one was much faster with the better connection. And much cheaper, thanks to an online pricing error... $450 instead of the $800 it should have been or the $990 I paid 3 months before that.

Making RAID easier

brad — Tue, 27 May 2008 20:49:04 -0700

Hard disks fail. If you prepared properly, you have a backup, or you swap out disks when they first start reporting problems. If you prepare really well you have offsite backup (which is getting easier and easier to do over the internet.)

One way to protect yourself from disk failures is RAID, especially RAID-5. With RAID, several disks act together as one. The simplest protecting RAID, RAID-1, just has 2 disks which work in parallel, known as mirroring. Everything you write is copied to both. If one fails, you still have the other, with all your data. It’s good, but twice as expensive.

RAID-5 is cleverer. It uses 3 or more disks, and uses error correction techniques so that you can store, for example, 2 disks worth of data on 3 disks. So it’s only 50% more expensive. RAID-5 can be done with many more disks — for example with 5 disks you get 4 disks worth of data, and it’s only 25% more expensive. However, having 5 disks is beyond most systems and has its own secret risk — if 2 of the 5 disks fail at once — and this does happen — you lose all 4 disks worth of data, not just 2 disks worth. (RAID-6 for really large arrays of disks, survives 2 failures but not 3.)

Now most people who put in RAID do it for more than data protection. After all, good sysadmins are doing regular backups. They do it because with RAID, the computer doesn’t even stop when a disk fails. You connect up a new disk live to the computer (which you can do with some systems) and it is recreated from the working disks, and you never miss a beat. This is pretty important with a major server.

But RAID has value to those who are not in the 99.99% uptime community. Those who are not good at doing manual backups, but who want to be protected from the inevitable disk failures. Today it is hard to set up, or expensive, or both. There are some external boxes like the “readynas” that make it reasonably easy for external disks, but they don’t have the bandwidth to be your full time disks.

RAID-5 on old IDE systems was hard, they usually could truly talk to only 2 disks at a time. The new SATA bus is much better, as many motherboards have 4 connectors, though soon one will be required by blu-ray drives.