What's Chris been doing?

As part of project 171 I’m trying to work out how we might more easily upgrade the firmware and bios on our servers. This post is an attempt to make sense of what I’ve done so far and to work out what to do next.

Ideally a firmware upgrade should just be a matter of using handy Dell or HP software to identify and install any required upgrades. However until now that hasn’t really worked out very well for us, and we’ve ended up with some fairly convoluted workarounds, for example this.

Since most of our servers are currently Dells, I’ll deal with them first.

Dell’s big bundle of software for monitoring and maintaining servers is called OMSA, short for OpenManage Server Administrator. We were somewhat frustrated with it for years because although it looked quite useful, it was installed in a way which we judged to be incompatible with our management standards: instead of being installed cleanly and removably using a conventional Linux RPM the OMSA files seemed to be dropped all over the filesystem in unpredictable and sometimes disruptive places, so without a lot of work we couldn’t tell precisely what the consequences of OMSA installation were and how to properly control it.

The package updates also seemed a little more unpredictable than what we were used to dealing with: instead of RPMs they came in the form of executable scripts with built-in payloads.

The web downloader, the main alternative way of obtaining Dell updates, also had its own charming idiosyncrasies, such as assuming that the user was using a Windows computer.

The good news is that Dell has seen the RPM light, and in a big way.

OMSA, from version 6.5 onwards, is now packaged in a collection of several dozen RPMs, with proper dependencies, making it possible to install just the required parts of it, to easily find out what has been installed, and to have some confidence that OMSA installation and removal can be done without disrupting other software.

Dell Firmware and BIOS updates for Linux are now packaged in RPMs.

There’s also Dell software which identifies what firmware versions your server has, what versions it should have, and offers to find the necessary RPMs for you and perform the upgrades.

All of these wonderful things are available in a Dell RPM repository which is compatible with package tools such as yum and which can even be mirrored using rsync. The repository is documented here and situated here.

On my test server I’ve enabled yum access to the Dell repository (though not yet using LCFG, though that should be easily doable), installed OMSA, started it running and tried a few sample commands. So far it seems fairly straightforward to get it running on a server. We may find it useful. A summary of what it offers would be helpful; I shall have another look at it and at the documentation, which can be got from here.

There are a few worries and niggles.

One is that although the documentation site offers information on OMSA version 7.0, the RPM repository only goes up to OMSA version 6.5.3. Is it just lagging behind or has it been abandoned in some way? Will OMSA 7.0 appear in the RPM repository at some point or should I believe the OMSA 7.0 readme which claims that Dell OpenManage System Management software, including Server Administrator, is available only on the “Dell Systems Management Tools and Documentation” DVD? Or is the current version of OMSA still officially 6.5, as claimed on the Dell techcenter wiki? In which case what is the status of OMSA 7.0?

Another area with questions over it is firmware updates, my current main interest. These were once handled by OMSA, but at some point the functionality was split out into separate software.

The Dell tool of choice for handling firmware updates seems to be OpenManage Essentials, but this appears to be available only for Windows operating systems. At the moment firmware-tools seems to be the tool of choice for Linux firmware updates. I have no idea if it is still supported for Linux, if OME is one day going to replace it for Linux, or whether there is official or dependable Dell support for it.

Anyway, I installed firmware-tools following Dell’s instructions and like OMSA I found it easy to install (thanks Dell). It inventoried my system (told me what firmware and BIOS versions my server was running) and assessed it for updates (told me what firmware and BIOS versions it should be running). I then tried running it in “yes, do the updates for me” mode, but this was where it went wrong. Here’s the output from the update_firmware program.

% update_firmware --yes

Running system inventory...

Searching storage directory for available BIOS updates...
Checking BIOS - a00
        Available: dell_dup_componentid_00159 - a05
        Found Update: dell_dup_componentid_00159 - a05
Checking SAS5ira Controller 0 Firmware - 00.06.50.00.06.06.00.02
        Available: pci_firmware(ven_0x1000_dev_0x0054_subven_0x1028_subdev_0x1f09) - 00.10.51.00.06.12.05.00
        Found Update: pci_firmware(ven_0x1000_dev_0x0054_subven_0x1028_subdev_0x1f09) - 00.10.51.00.06.12.05.00
Checking ATLAS10K5_073SAS Firmware - bp00
        Did not find a newer package to install that meets all installation checks.
Checking System BIOS for PowerEdge 860 - A00
        Did not find a newer package to install that meets all installation checks.

Found firmware which needs to be updated.

Running updates...
\       Installing dell_dup_componentid_00159 - a05Installation failed for package: dell_dup_componentid_00159 - a05
aborting update...

The error message from the low-level command was:

Enough contiguous physical memory is not available to perform the BIOS update running under this Operating System. Reboot and try again.

The server in question has 2G of memory. The same error message was produced whether update_firmware was run in multi-user mode or in single-user mode, and whether or not it was run just after rebooting.

I haven’t tried locating and installing the RPMs which it mentioned.

I have come across an interesting blog post though which suggests that I may be on a hiding to nothing here, and suggesting some other things to look at.

Now for HP. It’s been supporting well-behaved RPMs for rather longer than Dell has, I think. Its repository is called the HP Software Delivery Repository or SDR for short. It’s written up here.

The SDR doesn’t seem to support Scientific Linux. It does support RHEL and (here and there?) CentOS.

HP’s management software is called ProLiant Support Pack or PSP for short. Or has that been replaced now with Service Pack for ProLiant, or SPP for short? It would appear that SPP supports at least RHEL and SUSE so it’s worth taking a closer look at.

Disappointingly, there doesn’t seem to be a mailing list which deals with HP PSP or SPP issues.

The HP repository can be mirrored by rsync.

After finding out this lot I’m left with lots of uncertainty and lots of possible avenues to explore, and I haven’t even mentioned other possibilities such as getting our servers to automatically look for and report memory errors in the IPMI log.

Do you have any experience in this area, or any observations or knowledge or clarifications to pass on? Any hints about which area might be the most promising to look at? Please get in touch if you do.

Finally, to reward you for reading this far here’s a magnificent picture of a purple heron, similar to the ones I saw on holiday in India last winter. Apparently they’re very rare indeed here but in India we saw them every day, lurking in the fields near where we were staying. They have extraordinarily long and flexible necks and tend to stand for long periods totally immobile, resembling weirdly twisted lengths of wood, until they suddenly strike at prey. The picture and its attribution and more information are available from here.