So, it’s time for the famous “chicken-and-the-egg”-problem again. We have some SuperMicro-nodes that has been experiencing hardware-problems. After a long discussion with SM, they supposedly have figured out the root-cause of the problem, and has released a new BIOS-upgrade. That means we need to upgrade quite a lot of nodes.

Since we have no clue to find the nodes we want to upgrade, by just looking at the hostnames, we need to find it some other way. We know they have 2x CPU’s with 12 cores each, so, the following should list the nodes we need;

root@portal-ecs1:/etc/dsh/group# for host in $(cat prodcluster); do if ping -c1 -w1 $host|grep -qi "bytes from"; then ssh $host 'if [ `cat /proc/cpuinfo|grep -i "model name"|wc -l` -eq 24 ]; then hostname; fi'; fi; done
cn006
cn007
cn008
cn009
cn011
cn012
cn013
cn020
cn021
cn022
cn023
cn024
cn025
cn026
cn027
cngpu000
cngpu001
cngpu002
cngpu003
cngpu004
cngpu005
cngpu006
cngpu007
cngpu008
cngpu009
cngpu010
cngpu011
cngpu012
cngpu013
cngpu014
cngpu015
cngpu016
cngpu017
cngpu018
cngpu019
cngpu020
cngpu021
cngpu022
cngpu023
cngpu024
cngpu025
cngpu026
cngpu027
cngpu028
cngpu029
cngpu030
cngpu031
cngpu032
cngpu033

49 in total.

The problem, however, is that, even though we live in 2011, you need to use DOS to upgrade it. Fair enough. But what about the built-in BIOS-upgrade in BMC/IPMI? Well, the latter actually bricks the node, as, at some point, the BMC/IPMI goes through the BIOS, and hence, when upgrading the BIOS, it looses connectivity to itself some way. Brilliant. So, back to DOS. The nodes, obviously, has no floppy-disks, so, we need to use a CD. They don’t have a CD either, so you’ll either have to use a USB-stick, a USB-CD-ROM, or BMC/IPMI’s built-in virtual CD-ROM, where you can mount .iso-files from a SMB-share. Quite nifty. Except that, so far, we haven’t found a CD-ROM driver for DOS that accepts the virtual CD-ROM. So, then we can’t access the BIOS-upgrade software. Great. What about using the BMC/IPMI’s built-in virtual floppy-disk? That would have been a great idea, except that it’s limited to 1.44MB. Guess what? The new BIOS-firmware is 2.1MB. Wohooo!

So, for the moment we’re somewhat stuck. We’ll be looking into using a USB-stick, and maybe get it to work that way. I guess it’s all about finding a driver that accepts the virtual virtual virtual floppy-disk virtualized as a CD-ROM on a USB-stick, or something. Hahaha.