CERN

Hello, mgmt

So, I switched the management-network over to the new IP-layout today. First, I had to reconfigure all the old management-switches (Netgear-switches from World War 2). I decided to wipe the old configuration completely, and then start from scratch. They have a web-interface, and of course telnet. However, the telnet is a so-called CMI (Command Menu Interface), so you have to navigate through the menus to get to the actual CLI. After applying a new configuration, with new IP’s, proper VLAN-assignment, and all that, I configured ‘sw-mgmt’ accordingly.

Since I couldn’t reset the BMC/CHARM-cards after moving them to the new network, there are only two possible ways to actually get them to request new IPs or, possibly three, as some of the BMC might support “localhost”-connections from the host itself using ipmitool, but this is untested, and wouldn’t work for the CHARM-cards anyhow); 1) wait for their lease-time to wear out (28800 seconds = 8 hours), 2) turn the cluster completely off (by unplugging their power-cable), and back on again. We’re obviously going for the first alternative. If we, after 8 hours+, still have missing BMC/CHARM-cards, we’ll investigate those manually.

Since I only have static leases in the dhcpd-config, it’s fairly easy to spot DHCPDISCOVERs from unknown nodes;

root@ns0:~# cat /var/log/syslog|grep -i "10.162.64.1"|grep -i "no free leases"|grep -vi "via eth0"|perl -wple 's|\s+| |g'|cut -d' ' -f8,10|sort|uniq|perl -wple 's|:$||i'
00:02:c9:11:34:22 10.162.64.1
00:02:c9:11:34:b6 10.162.64.1
00:02:c9:11:39:12 10.162.64.1
00:02:c9:11:39:1e 10.162.64.1
00:02:c9:11:39:32 10.162.64.1
00:02:c9:11:39:42 10.162.64.1
00:02:c9:11:39:44 10.162.64.1
00:26:b0:d9:fc:0a 10.162.64.1
00:30:48:c9:87:d1 10.162.64.1
00:30:48:c9:8c:74 10.162.64.1
00:30:48:c9:8d:ce 10.162.64.1
00:30:48:c9:8f:a5 10.162.64.1
00:30:48:f2:0a:82 10.162.64.1
00:40:dc:fa:00:0a 10.162.64.1
00:40:dc:fa:00:17 10.162.64.1
00:40:dc:fa:00:1b 10.162.64.1
00:40:dc:fa:00:20 10.162.64.1
00:40:dc:fa:00:2d 10.162.64.1
00:40:dc:fa:00:34 10.162.64.1
00:40:dc:fa:00:55 10.162.64.1
00:40:dc:fa:00:57 10.162.64.1
00:40:dc:fa:00:5d 10.162.64.1
00:40:dc:fa:00:60 10.162.64.1
00:40:dc:fa:00:63 10.162.64.1
00:40:dc:fa:00:68 10.162.64.1
00:40:dc:fa:00:69 10.162.64.1
00:40:dc:fa:00:83 10.162.64.1
00:40:dc:fa:00:88 10.162.64.1
00:40:dc:fa:00:8b 10.162.64.1
00:40:dc:fa:00:bf 10.162.64.1
00:40:dc:fa:00:c0 10.162.64.1
00:40:dc:fa:00:c6 10.162.64.1
00:40:dc:fa:00:c9 10.162.64.1
00:40:dc:fa:00:cd 10.162.64.1
00:40:dc:fa:00:ce 10.162.64.1
00:40:dc:fa:00:cf 10.162.64.1
00:40:dc:fa:00:d4 10.162.64.1
00:40:dc:fa:00:dd 10.162.64.1
00:40:dc:fa:00:de 10.162.64.1

So, let’s figure out what those nodes are (by using the old LDAP-database);

joachim@keklolwtf: ~/Documents/CERN/Temp/Hostname-stuff $ for host in $(cat no-free-leases_10feb); do mac=`echo $host|cut -d'#' -f1`; network=`echo $host|cut -d'#' -f2`; old_hostname=`cat ldap.ldif|grep -i -A1 "$mac"|tail -1|perl -wple 's|dhcpOption: host-name||gi,s|"||gi,s| ||gi'`; new_hostname=`cat dhcpd-10.162.*|grep -i "$mac"|cut -d' ' -f2`; echo -e "$old_hostname -> $new_hostname\t$network\t$mac"; done
swibt1 -> 	10.162.64.1	00:02:c9:11:34:22
swibt0 -> 	10.162.64.1	00:02:c9:11:34:b6
swiby07 -> 	10.162.64.1	00:02:c9:11:39:12
swibt2 -> 	10.162.64.1	00:02:c9:11:39:1e
swiby09 -> 	10.162.64.1	00:02:c9:11:39:32
swibt3 -> 	10.162.64.1	00:02:c9:11:39:42
swibt4 -> 	10.162.64.1	00:02:c9:11:39:44
 -> 	10.162.64.1	00:26:b0:d9:fc:0a
dev0-bmc -> 	10.162.64.1	00:30:48:c9:87:d1
dev1-bmc -> 	10.162.64.1	00:30:48:c9:8c:74
cntpca096-bmc -> 	10.162.64.1	00:30:48:c9:8d:ce
ms1-bmc -> 	10.162.64.1	00:30:48:c9:8f:a5
ms0-bmc -> 	10.162.64.1	00:30:48:f2:0a:82
portal-dcs0-charm -> 	10.162.64.1	00:40:dc:fa:00:0a
 -> 	10.162.64.1	00:40:dc:fa:00:17
 -> 	10.162.64.1	00:40:dc:fa:00:1b
 -> 	10.162.64.1	00:40:dc:fa:00:20
 -> 	10.162.64.1	00:40:dc:fa:00:2d
 -> 	10.162.64.1	00:40:dc:fa:00:34
dev3-charm -> 	10.162.64.1	00:40:dc:fa:00:55
feptriggerdet-charm -> 	10.162.64.1	00:40:dc:fa:00:57
portal-ecs1-charm -> 	10.162.64.1	00:40:dc:fa:00:5d
 -> 	10.162.64.1	00:40:dc:fa:00:60
ms2-charm -> 	10.162.64.1	00:40:dc:fa:00:63
portal-ecs0-charm -> 	10.162.64.1	00:40:dc:fa:00:68
ms3-charm -> 	10.162.64.1	00:40:dc:fa:00:69
 -> 	10.162.64.1	00:40:dc:fa:00:83
dev2-charm -> 	10.162.64.1	00:40:dc:fa:00:88
 -> 	10.162.64.1	00:40:dc:fa:00:8b
 -> 	10.162.64.1	00:40:dc:fa:00:bf
 -> 	10.162.64.1	00:40:dc:fa:00:c0
 -> 	10.162.64.1	00:40:dc:fa:00:c6
 -> 	10.162.64.1	00:40:dc:fa:00:c9
 -> 	10.162.64.1	00:40:dc:fa:00:cd
 -> 	10.162.64.1	00:40:dc:fa:00:ce
 -> 	10.162.64.1	00:40:dc:fa:00:cf
portal-gui2-charm -> 	10.162.64.1	00:40:dc:fa:00:d4
 -> 	10.162.64.1	00:40:dc:fa:00:dd
 -> 	10.162.64.1	00:40:dc:fa:00:de

We see that all these hosts doesn’t have an entry in our current config (obviously, because it’s the no-free-leases from dhcpd), but we also see that we actually have requests coming from nodes that didn’t exist in the LDAP-database — these are “totally” unknown nodes, and has to be traced through the switches using their mac-tables.

I can find the unknown MAC-addresses this way;

joachim@keklolwtf: ~/Documents/CERN/Temp/Hostname-stuff $ for host in $(cat no-free-leases_10feb); do mac=`echo $host|cut -d'#' -f1`; network=`echo $host|cut -d'#' -f2`; old_hostname=`cat ldap.ldif|grep -i -A1 "$mac"|tail -1|perl -wple 's|dhcpOption: host-name||gi,s|"||gi,s| ||gi'`; if [ "$old_hostname" == "" ]; then echo -e "$network\t$mac"; fi; done
10.162.64.1	00:26:b0:d9:fc:0a
10.162.64.1	00:40:dc:fa:00:17
10.162.64.1	00:40:dc:fa:00:1b
10.162.64.1	00:40:dc:fa:00:20
10.162.64.1	00:40:dc:fa:00:2d
10.162.64.1	00:40:dc:fa:00:34
10.162.64.1	00:40:dc:fa:00:60
10.162.64.1	00:40:dc:fa:00:83
10.162.64.1	00:40:dc:fa:00:8b
10.162.64.1	00:40:dc:fa:00:bf
10.162.64.1	00:40:dc:fa:00:c0
10.162.64.1	00:40:dc:fa:00:c6
10.162.64.1	00:40:dc:fa:00:c9
10.162.64.1	00:40:dc:fa:00:cd
10.162.64.1	00:40:dc:fa:00:ce
10.162.64.1	00:40:dc:fa:00:cf
10.162.64.1	00:40:dc:fa:00:dd
10.162.64.1	00:40:dc:fa:00:de

Wish me happy hunting! :D

Next, I can find out all successful DHCPOFFERS;

root@ns0:~# cat /var/log/syslog|grep -i "10.162.64.1"|grep -ivE "(no free leases|ignored)"|grep -i "DHCPOFFER"|perl -wple 's|\s+| |g'|cut -d' ' -f8|sort|uniq
10.162.64.100
10.162.64.102
10.162.64.103
10.162.64.109
10.162.64.11
10.162.64.110
10.162.64.111
10.162.64.113
10.162.64.114
10.162.64.115
10.162.64.116
10.162.64.117
10.162.64.118
10.162.64.119
10.162.64.12
10.162.64.120
10.162.64.121
10.162.64.129
10.162.64.13
10.162.64.130
10.162.64.131
10.162.64.132
10.162.64.133
10.162.64.134
10.162.64.135
10.162.64.136
10.162.64.137
10.162.64.14
10.162.64.15
10.162.64.155
10.162.64.156
10.162.64.157
10.162.64.158
10.162.64.159
10.162.64.16
10.162.64.160
10.162.64.161
10.162.64.162
10.162.64.163
10.162.64.164
10.162.64.165
10.162.64.166
10.162.64.167
10.162.64.168
10.162.64.169
10.162.64.170
10.162.64.171
10.162.64.172
10.162.64.173
10.162.64.174
10.162.64.175
10.162.64.176
10.162.64.177
10.162.64.178
10.162.64.179
10.162.64.180
10.162.64.181
10.162.64.182
10.162.64.183
10.162.64.184
10.162.64.185
10.162.64.186
10.162.64.187
10.162.64.188
10.162.64.189
10.162.64.190
10.162.64.191
10.162.64.192
10.162.64.193
10.162.64.194
10.162.64.195
10.162.64.196
10.162.64.197
10.162.64.198
10.162.64.199
10.162.64.200
10.162.64.201
10.162.64.202
10.162.64.203
10.162.64.204
10.162.64.205
10.162.64.206
10.162.64.207
10.162.64.209
10.162.64.210
10.162.64.211
10.162.64.212
10.162.64.214
10.162.64.25
10.162.64.26
10.162.64.27
10.162.64.28
10.162.64.29
10.162.64.30
10.162.64.39
10.162.64.40
10.162.64.41
10.162.64.42
10.162.64.43
10.162.64.44
10.162.64.45
10.162.64.46
10.162.64.47
10.162.64.48
10.162.64.49
10.162.64.50
10.162.64.51
10.162.64.52
10.162.64.53
10.162.64.54
10.162.64.55
10.162.64.56
10.162.64.57
10.162.64.58
10.162.64.59
10.162.64.60
10.162.64.61
10.162.64.62
10.162.64.63
10.162.64.98
10.162.64.99
10.162.72.15
10.162.72.16
10.162.72.19
10.162.72.20
10.162.72.31
10.162.80.10
10.162.80.102
10.162.80.11
10.162.80.112
10.162.80.12
10.162.80.121
10.162.80.122
10.162.80.131
10.162.80.132
10.162.80.140
10.162.80.150
10.162.80.151
10.162.80.152
10.162.80.153
10.162.80.154
10.162.80.155
10.162.80.20
10.162.80.21
10.162.80.30
10.162.80.31
10.162.82.10
10.162.82.11

And to show the hostnames as well;

root@ns0:~# for ip in $(cat /var/log/syslog|grep -i "10.162.64.1"|grep -ivE "(no free leases|ignored)"|grep -i "DHCPOFFER"|perl -wple 's|\s+| |g'|cut -d' ' -f8|sort|uniq); do hostname=`dig -x $ip +short|cut -d'.' -f1`; echo "$hostname ($ip)"; done
fepdimutrk2-charm (10.162.64.100)
fepdimutrk4-charm (10.162.64.102)
fepdimutrk5-charm (10.162.64.103)
fepfmdaccorde-charm (10.162.64.109)
cn000-charm (10.162.64.11)
fephltout0-charm (10.162.64.110)
fephltout1-charm (10.162.64.111)
fephmpid0-charm (10.162.64.113)
fephmpid1-charm (10.162.64.114)
fephmpid2-charm (10.162.64.115)
fephmpid3-charm (10.162.64.116)
fepphos0-charm (10.162.64.117)
fepphos1-charm (10.162.64.118)
fepphos2-charm (10.162.64.119)
cn001-charm (10.162.64.12)
fepphos3-charm (10.162.64.120)
fepphos4-charm (10.162.64.121)
fepspd0-charm (10.162.64.129)
cn002-bmc (10.162.64.13)
fepspd1-charm (10.162.64.130)
fepspd2-charm (10.162.64.131)
fepspd3-charm (10.162.64.132)
fepspd4-charm (10.162.64.133)
fepssd0-charm (10.162.64.134)
fepssd1-charm (10.162.64.135)
fepssd2-charm (10.162.64.136)
fepssd3-charm (10.162.64.137)
cn003-bmc (10.162.64.14)
cn004-bmc (10.162.64.15)
feptpcai00-charm (10.162.64.155)
feptpcai02-charm (10.162.64.156)
feptpcai04-charm (10.162.64.157)
feptpcai06-charm (10.162.64.158)
feptpcai08-charm (10.162.64.159)
cn005-bmc (10.162.64.16)
feptpcai10-charm (10.162.64.160)
feptpcai12-charm (10.162.64.161)
feptpcai14-charm (10.162.64.162)
feptpcai16-charm (10.162.64.163)
feptpcao00-charm (10.162.64.164)
feptpcao01-charm (10.162.64.165)
feptpcao02-charm (10.162.64.166)
feptpcao03-charm (10.162.64.167)
feptpcao04-charm (10.162.64.168)
feptpcao05-charm (10.162.64.169)
feptpcao06-charm (10.162.64.170)
feptpcao07-charm (10.162.64.171)
feptpcao08-charm (10.162.64.172)
feptpcao09-charm (10.162.64.173)
feptpcao10-charm (10.162.64.174)
feptpcao11-charm (10.162.64.175)
feptpcao12-charm (10.162.64.176)
feptpcao13-charm (10.162.64.177)
feptpcao14-charm (10.162.64.178)
feptpcao15-charm (10.162.64.179)
feptpcao16-charm (10.162.64.180)
feptpcao17-charm (10.162.64.181)
feptpcci00-charm (10.162.64.182)
feptpcci02-charm (10.162.64.183)
feptpcci04-charm (10.162.64.184)
feptpcci06-charm (10.162.64.185)
feptpcci08-charm (10.162.64.186)
feptpcci10-charm (10.162.64.187)
feptpcci12-charm (10.162.64.188)
feptpcci14-charm (10.162.64.189)
feptpcci16-charm (10.162.64.190)
feptpcco00-charm (10.162.64.191)
feptpcco01-charm (10.162.64.192)
feptpcco02-charm (10.162.64.193)
feptpcco03-charm (10.162.64.194)
feptpcco04-charm (10.162.64.195)
feptpcco05-charm (10.162.64.196)
feptpcco06-charm (10.162.64.197)
feptpcco07-charm (10.162.64.198)
feptpcco08-charm (10.162.64.199)
feptpcco09-charm (10.162.64.200)
feptpcco10-charm (10.162.64.201)
feptpcco11-charm (10.162.64.202)
feptpcco12-charm (10.162.64.203)
feptpcco13-charm (10.162.64.204)
feptpcco14-charm (10.162.64.205)
feptpcco15-charm (10.162.64.206)
feptpcco16-charm (10.162.64.207)
feptrd00-charm (10.162.64.209)
feptrd04-charm (10.162.64.210)
feptrd08-charm (10.162.64.211)
feptrd10-charm (10.162.64.212)
feptriggerdet-charm (10.162.64.214)
cn014-bmc (10.162.64.25)
cn015-bmc (10.162.64.26)
cn016-bmc (10.162.64.27)
cn017-bmc (10.162.64.28)
cn018-bmc (10.162.64.29)
cn019-bmc (10.162.64.30)
cn028-bmc (10.162.64.39)
cn029-bmc (10.162.64.40)
cn030-bmc (10.162.64.41)
cn031-bmc (10.162.64.42)
cn032-charm (10.162.64.43)
cn033-charm (10.162.64.44)
cn034-bmc (10.162.64.45)
cn035-bmc (10.162.64.46)
cn036-bmc (10.162.64.47)
cn037-bmc (10.162.64.48)
cn038-charm (10.162.64.49)
cn039-charm (10.162.64.50)
cn040-bmc (10.162.64.51)
cn041-bmc (10.162.64.52)
cn042-bmc (10.162.64.53)
cn043-bmc (10.162.64.54)
cn044-bmc (10.162.64.55)
cn045-bmc (10.162.64.56)
cn046-bmc (10.162.64.57)
cn047-bmc (10.162.64.58)
cn048-bmc (10.162.64.59)
cn049-bmc (10.162.64.60)
cn050-bmc (10.162.64.61)
cn051-bmc (10.162.64.62)
cn052-charm (10.162.64.63)
fepdimutrg-charm (10.162.64.98)
fepdimutrk1-charm (10.162.64.99)
cndev004-charm (10.162.72.15)
cndev005-charm (10.162.72.16)
fepdev10-charm (10.162.72.19)
fepdev11-charm (10.162.72.20)
msdev0-bmc (10.162.72.31)
cooling0-charm (10.162.80.10)
portal-ecs1-bmc (10.162.80.102)
ns0-bmc (10.162.80.11)
portal-dcs1-bmc (10.162.80.112)
ns1-bmc (10.162.80.12)
portal-vobox0-charm (10.162.80.121)
portal-vobox1-charm (10.162.80.122)
portal-gui0-charm (10.162.80.131)
portal-gui1-charm (10.162.80.132)
hotspare0-bmc (10.162.80.140)
dfs0-charm (10.162.80.150)
dfs1-charm (10.162.80.151)
dfs2-charm (10.162.80.152)
dfs3-charm (10.162.80.153)
dfs4-charm (10.162.80.154)
dfs5-charm (10.162.80.155)
db0-bmc (10.162.80.20)
db1-bmc (10.162.80.21)
mon0-bmc (10.162.80.30)
mon1-bmc (10.162.80.31)
vhost0-charm (10.162.82.10)
vhost1-charm (10.162.82.11)