For many years, I have been using RAID for my home storage. With RAID (and its cousins) everything is stored redundantly so that if any disk drive fails, you don't lose your data, and in fact your system doesn't even go down. This can come at a cost of anywhere from about 25% to 50% of your disk space (but disk is cheap) and it also often increases disk performance. Some years ago I wrote about how disk drives should be sold in form factors designed for easy RAID in every PC, and I still believe that.

Studies have shown that if you leave USB sticks on the ground outside an office building, 60% of them will get picked up and plugged into a computer in the building. If you put the company logo on the sticks, closer to 90% of them will get picked up and plugged in.

We all know how annoying it is that today's much faster computers take such a long time to boot, and OS developers are working on speeding it up. Some time ago I proposed a defragmenter that notice what blocks were read in what order at boot and put the contiguous on the disk. I was told that experiments with this had not had much success, but more recently I read reports of how the latest Linux distributions will boot as much a 3 times faster on solid state disks as on rotating ones. There are some SSDs with performance that high (and higher) but typical ones range more in the 120 mb/second rate, better than 80 mb/second HDDs but getting more wins from the complete lack of latency.

However, today I want to consider something which is a large portion of the boot time, namely the power-on-self-test or "POST." This is what the BIOS does before it gets ready to load the real OS. It's important, but on many systems is quite slow.

I propose an effort to make the POST multitask with the loading of the real OS. Particularly on dual-core systems, this would be done by having one core do the POST and other BIOS (after testing all the cores of course) and other cores be used by the OS for loading. There are ways to do all this with one core I will discuss below, but this one is simple and almost all new computers have multiple cores.

Of course, the OS has to know it's not supposed to touch certain hardware until after the BIOS is done initializing it and testing it. And so, there should be BIOS APIs to allow the OS to ask about this and get events as BIOS operations conclude. The OS, until given ownership of the screen, would output its status updates to the screen via a BIOS call. In fact, it would do that with all hardware, though the screen, keyboard and primary hard disk are the main items. When I say the OS, I actually mean both the bootloader that loads the OS and the OS itself once it is handed off to.

Next, the BIOS should, as soon as it has identified that the primary boot hard disks are ready, begin transferring data from the boot area into RAM. Almost all machines have far more RAM than they need to boot, and so pre-loading all blocks needed for boot into a cache, done in optimal order, will mean that by the time the OS kernal takes over, many of the disk blocks it will want to read will already be sitting in ram. Ideally, as I noted, the blocks should have been pre-stored in contiguous zones on the disk by an algorithm that watched the prior boots to see what was accessed and when.

Indeed, if there are multiple drives, the pre-loader could be configured to read from all of them, in a striping approach. Done properly, a freshly booted computer, once the drives had spun up, would start reading the few hundred megabytes of files it needs to boot from multiple drives into ram. All of this should be doable in just a few seconds on RAID style machines, where 3 disks striped can deliver 200mb/second or more of disk read performance. But even on a single drive, it should be quick. It would begin with the OS kernel and boot files, but then pre-cache all the pages from files used in typical boots. For any special new files, only a few seeks will be required after this is done.

Most programs that ask for a password will put in a delay if you get it wrong. They do this to stop password crackers from quickly trying lots of passwords. The delay makes brute force attacks impossible, in theory.

First, let me introduce a new blog topic, Sysadmin where I will cover computer system administration and OS design issues, notably in Linux and related systems.

My goal is to reduce the nightmare that is system administration and upgrading.

One step that goes partway in my plan would be a special software system that would build for a user a specialized operating system "package" or set of packages. This magic package would, when applied to a virgin distribution of the operating system, convert it into the customized form that the user likes.

The program would work from a modified system, and a copy of a map (with timestamps and hashes) of the original virgin OS from which the user began. First, it would note what packages the user had installed, and declare dependencies for these packages. Thus, installing this magic package would cause the installation of all the packages the user likes, and all that they depend on.

In order to do this well, it would try to determine which packages the user actually used (with access or file change times) and perhaps consider making two different dependency setups -- one for the core packages that are frequently used, and another for packages that were probably just tried and never used. A GUI to help users sort packages into those classes would be handy. It must also determine that those packages are still available, dealing with potential conflicts and name change concerns. Right now, most package managers insist that all dependencies be available or they will abort the entire install. To get around this, many of the packages might well be listed as "recommended" rather than required, or options to allow install of the package with missing 1st level (but not 2nd level) dependencies would be used.

As part of my series on the horrors of modern system administration and upgrading, let me propose the need for a universal API, over all operating systems, for accessing data from, and some control of the package management system.

There have been many efforts in the past to standardize programming APIs within all the unix-like operating systems, some of them extending into MS Windows, such as Posix. Posix is a bit small to write very complex programs fully portably but it's a start. Any such API can make your portability easier if it can't make it trivial the way it's supposed to.

But there has been little effort to standardize the next level, machine administration and configuration. Today a large part of that is done with the package manager. Indeed, the package manager is the soul (and curse) of most major OS distributions. One of the biggest answers to "what's the difference between debian and Fedora" is "dpkg and apt, vs. rpm and yum." (Yes you can, and I do, use apt with rpm.)

Now the truth is that from a user perspective, these package managers don't actually look very different. They all install and remove packages by name, perform upgrades, handle dependencies etc. Add-ons like apt and GUI package managers help users search and auto-install all dependencies. To the user, the most common requests are to find and install a package, and to upgrade it or the system.

One of my current peeves is just how much time we spend maintaining and upgrading computer operating systems, even as ordinary users. The workload for this is unacceptably high, though it's not as though people are unaware of the problem.

Right now I'm updating one system to the beta of the new Ubuntu Feisty Fawn. (Ubuntu is the Linux distro I currently recommend.) They have done some work on building a single upgrader, which is good, but I was shocked to see an old problem resurface. In a 2 hour upgrade process, it asked me questions it didn't need to ask me, and worse, it asked them at different times in the process.

I've ranted before about just how hard it has become to configure and administer computers. And there are services where you can hire sysadmins to help you, primarily aimed at novice users.

But we advanced users often need help today, too. Mostly when we run into problems we go to message boards, or do web searches and find advice on what to do. And once we get good on a package we can generally fix problems with it in no time.

Virtualizer technology, that lets you create a virtual machine in which to run another "guest" operating system on top of your own, seems to have arrived. It's common for servers (for security) and for testing, as well as things like running Windows on linux or a Mac. There are several good free ones. One, kvm, is built into the lastest Linux kernel (2.6.20). Microsoft offers their own.

Last week I wrote about linux's problems with dependencies and upgrades and promised some suggestions this week.

There are a couple of ideas here to be stolen from (sacrilige) windows which could be a start here, though they aren't my long term solution.

Microsoft takes a different approach to updates, which consists of little patches and big service packs. The service packs integrate a lot of changes, including major changes, into one upgrade. They are not very frequent, and in some ways akin to the major distribution releases of systems like Ubuntu (but not its parent Debian ), Fedora Core and SuSE.

Installing a service pack is certainly not without risks, but the very particular combination of new libraries and changed apps in a service pack is extensively tested together, as is also the case for a major revision of a linux distribution. Generally installing one of these packs has been a safe procedure. Most windows programs also do not use hand-edited configuration files for local changes, and so don't suffer from the upgrade problems associated with this particular technique nearly as much.

We all spend far too much of our time doing sysadmin. I'm upgrading and it's as usual far more work than it should be. I have a long term plan for this but right now I want to talk about one of Linux's greatest flaws -- the dependencies in the major distributions.