So, I switched the management-network over to the new IP-layout today. First, I had to reconfigure all the old management-switches (Netgear-switches from World War 2). I decided to wipe the old configuration completely, and then start from scratch. They have a web-interface, and of course telnet. However, the telnet is a so-called CMI (Command Menu Interface), so you have to navigate through the menus to get to the actual CLI. After applying a new configuration, with new IP’s, proper VLAN-assignment, and all that, I configured ‘sw-mgmt’ accordingly.
Since I couldn’t reset the BMC/CHARM-cards after moving them to the new network, there are only two possible ways to actually get them to request new IPs or, possibly three, as some of the BMC might support “localhost”-connections from the host itself using ipmitool, but this is untested, and wouldn’t work for the CHARM-cards anyhow); 1) wait for their lease-time to wear out (28800 seconds = 8 hours), 2) turn the cluster completely off (by unplugging their power-cable), and back on again. We’re obviously going for the first alternative. If we, after 8 hours+, still have missing BMC/CHARM-cards, we’ll investigate those manually.
Since I only have static leases in the dhcpd-config, it’s fairly easy to spot DHCPDISCOVERs from unknown nodes;
root@ns0:~# cat /var/log/syslog|grep -i "10.162.64.1"|grep -i "no free leases"|grep -vi "via eth0"|perl -wple 's|\s+| |g'|cut -d' ' -f8,10|sort|uniq|perl -wple 's|:$||i' 00:02:c9:11:34:22 10.162.64.1 00:02:c9:11:34:b6 10.162.64.1 00:02:c9:11:39:12 10.162.64.1 00:02:c9:11:39:1e 10.162.64.1 00:02:c9:11:39:32 10.162.64.1 00:02:c9:11:39:42 10.162.64.1 00:02:c9:11:39:44 10.162.64.1 00:26:b0:d9:fc:0a 10.162.64.1 00:30:48:c9:87:d1 10.162.64.1 00:30:48:c9:8c:74 10.162.64.1 00:30:48:c9:8d:ce 10.162.64.1 00:30:48:c9:8f:a5 10.162.64.1 00:30:48:f2:0a:82 10.162.64.1 00:40:dc:fa:00:0a 10.162.64.1 00:40:dc:fa:00:17 10.162.64.1 00:40:dc:fa:00:1b 10.162.64.1 00:40:dc:fa:00:20 10.162.64.1 00:40:dc:fa:00:2d 10.162.64.1 00:40:dc:fa:00:34 10.162.64.1 00:40:dc:fa:00:55 10.162.64.1 00:40:dc:fa:00:57 10.162.64.1 00:40:dc:fa:00:5d 10.162.64.1 00:40:dc:fa:00:60 10.162.64.1 00:40:dc:fa:00:63 10.162.64.1 00:40:dc:fa:00:68 10.162.64.1 00:40:dc:fa:00:69 10.162.64.1 00:40:dc:fa:00:83 10.162.64.1 00:40:dc:fa:00:88 10.162.64.1 00:40:dc:fa:00:8b 10.162.64.1 00:40:dc:fa:00:bf 10.162.64.1 00:40:dc:fa:00:c0 10.162.64.1 00:40:dc:fa:00:c6 10.162.64.1 00:40:dc:fa:00:c9 10.162.64.1 00:40:dc:fa:00:cd 10.162.64.1 00:40:dc:fa:00:ce 10.162.64.1 00:40:dc:fa:00:cf 10.162.64.1 00:40:dc:fa:00:d4 10.162.64.1 00:40:dc:fa:00:dd 10.162.64.1 00:40:dc:fa:00:de 10.162.64.1 |
So, let’s figure out what those nodes are (by using the old LDAP-database);
joachim@keklolwtf: ~/Documents/CERN/Temp/Hostname-stuff $ for host in $(cat no-free-leases_10feb); do mac=`echo $host|cut -d'#' -f1`; network=`echo $host|cut -d'#' -f2`; old_hostname=`cat ldap.ldif|grep -i -A1 "$mac"|tail -1|perl -wple 's|dhcpOption: host-name||gi,s|"||gi,s| ||gi'`; new_hostname=`cat dhcpd-10.162.*|grep -i "$mac"|cut -d' ' -f2`; echo -e "$old_hostname -> $new_hostname\t$network\t$mac"; done swibt1 -> 10.162.64.1 00:02:c9:11:34:22 swibt0 -> 10.162.64.1 00:02:c9:11:34:b6 swiby07 -> 10.162.64.1 00:02:c9:11:39:12 swibt2 -> 10.162.64.1 00:02:c9:11:39:1e swiby09 -> 10.162.64.1 00:02:c9:11:39:32 swibt3 -> 10.162.64.1 00:02:c9:11:39:42 swibt4 -> 10.162.64.1 00:02:c9:11:39:44 -> 10.162.64.1 00:26:b0:d9:fc:0a dev0-bmc -> 10.162.64.1 00:30:48:c9:87:d1 dev1-bmc -> 10.162.64.1 00:30:48:c9:8c:74 cntpca096-bmc -> 10.162.64.1 00:30:48:c9:8d:ce ms1-bmc -> 10.162.64.1 00:30:48:c9:8f:a5 ms0-bmc -> 10.162.64.1 00:30:48:f2:0a:82 portal-dcs0-charm -> 10.162.64.1 00:40:dc:fa:00:0a -> 10.162.64.1 00:40:dc:fa:00:17 -> 10.162.64.1 00:40:dc:fa:00:1b -> 10.162.64.1 00:40:dc:fa:00:20 -> 10.162.64.1 00:40:dc:fa:00:2d -> 10.162.64.1 00:40:dc:fa:00:34 dev3-charm -> 10.162.64.1 00:40:dc:fa:00:55 feptriggerdet-charm -> 10.162.64.1 00:40:dc:fa:00:57 portal-ecs1-charm -> 10.162.64.1 00:40:dc:fa:00:5d -> 10.162.64.1 00:40:dc:fa:00:60 ms2-charm -> 10.162.64.1 00:40:dc:fa:00:63 portal-ecs0-charm -> 10.162.64.1 00:40:dc:fa:00:68 ms3-charm -> 10.162.64.1 00:40:dc:fa:00:69 -> 10.162.64.1 00:40:dc:fa:00:83 dev2-charm -> 10.162.64.1 00:40:dc:fa:00:88 -> 10.162.64.1 00:40:dc:fa:00:8b -> 10.162.64.1 00:40:dc:fa:00:bf -> 10.162.64.1 00:40:dc:fa:00:c0 -> 10.162.64.1 00:40:dc:fa:00:c6 -> 10.162.64.1 00:40:dc:fa:00:c9 -> 10.162.64.1 00:40:dc:fa:00:cd -> 10.162.64.1 00:40:dc:fa:00:ce -> 10.162.64.1 00:40:dc:fa:00:cf portal-gui2-charm -> 10.162.64.1 00:40:dc:fa:00:d4 -> 10.162.64.1 00:40:dc:fa:00:dd -> 10.162.64.1 00:40:dc:fa:00:de |
We see that all these hosts doesn’t have an entry in our current config (obviously, because it’s the no-free-leases from dhcpd), but we also see that we actually have requests coming from nodes that didn’t exist in the LDAP-database — these are “totally” unknown nodes, and has to be traced through the switches using their mac-tables.
I can find the unknown MAC-addresses this way;
joachim@keklolwtf: ~/Documents/CERN/Temp/Hostname-stuff $ for host in $(cat no-free-leases_10feb); do mac=`echo $host|cut -d'#' -f1`; network=`echo $host|cut -d'#' -f2`; old_hostname=`cat ldap.ldif|grep -i -A1 "$mac"|tail -1|perl -wple 's|dhcpOption: host-name||gi,s|"||gi,s| ||gi'`; if [ "$old_hostname" == "" ]; then echo -e "$network\t$mac"; fi; done 10.162.64.1 00:26:b0:d9:fc:0a 10.162.64.1 00:40:dc:fa:00:17 10.162.64.1 00:40:dc:fa:00:1b 10.162.64.1 00:40:dc:fa:00:20 10.162.64.1 00:40:dc:fa:00:2d 10.162.64.1 00:40:dc:fa:00:34 10.162.64.1 00:40:dc:fa:00:60 10.162.64.1 00:40:dc:fa:00:83 10.162.64.1 00:40:dc:fa:00:8b 10.162.64.1 00:40:dc:fa:00:bf 10.162.64.1 00:40:dc:fa:00:c0 10.162.64.1 00:40:dc:fa:00:c6 10.162.64.1 00:40:dc:fa:00:c9 10.162.64.1 00:40:dc:fa:00:cd 10.162.64.1 00:40:dc:fa:00:ce 10.162.64.1 00:40:dc:fa:00:cf 10.162.64.1 00:40:dc:fa:00:dd 10.162.64.1 00:40:dc:fa:00:de |
Wish me happy hunting! :D
Next, I can find out all successful DHCPOFFERS;
root@ns0:~# cat /var/log/syslog|grep -i "10.162.64.1"|grep -ivE "(no free leases|ignored)"|grep -i "DHCPOFFER"|perl -wple 's|\s+| |g'|cut -d' ' -f8|sort|uniq 10.162.64.100 10.162.64.102 10.162.64.103 10.162.64.109 10.162.64.11 10.162.64.110 10.162.64.111 10.162.64.113 10.162.64.114 10.162.64.115 10.162.64.116 10.162.64.117 10.162.64.118 10.162.64.119 10.162.64.12 10.162.64.120 10.162.64.121 10.162.64.129 10.162.64.13 10.162.64.130 10.162.64.131 10.162.64.132 10.162.64.133 10.162.64.134 10.162.64.135 10.162.64.136 10.162.64.137 10.162.64.14 10.162.64.15 10.162.64.155 10.162.64.156 10.162.64.157 10.162.64.158 10.162.64.159 10.162.64.16 10.162.64.160 10.162.64.161 10.162.64.162 10.162.64.163 10.162.64.164 10.162.64.165 10.162.64.166 10.162.64.167 10.162.64.168 10.162.64.169 10.162.64.170 10.162.64.171 10.162.64.172 10.162.64.173 10.162.64.174 10.162.64.175 10.162.64.176 10.162.64.177 10.162.64.178 10.162.64.179 10.162.64.180 10.162.64.181 10.162.64.182 10.162.64.183 10.162.64.184 10.162.64.185 10.162.64.186 10.162.64.187 10.162.64.188 10.162.64.189 10.162.64.190 10.162.64.191 10.162.64.192 10.162.64.193 10.162.64.194 10.162.64.195 10.162.64.196 10.162.64.197 10.162.64.198 10.162.64.199 10.162.64.200 10.162.64.201 10.162.64.202 10.162.64.203 10.162.64.204 10.162.64.205 10.162.64.206 10.162.64.207 10.162.64.209 10.162.64.210 10.162.64.211 10.162.64.212 10.162.64.214 10.162.64.25 10.162.64.26 10.162.64.27 10.162.64.28 10.162.64.29 10.162.64.30 10.162.64.39 10.162.64.40 10.162.64.41 10.162.64.42 10.162.64.43 10.162.64.44 10.162.64.45 10.162.64.46 10.162.64.47 10.162.64.48 10.162.64.49 10.162.64.50 10.162.64.51 10.162.64.52 10.162.64.53 10.162.64.54 10.162.64.55 10.162.64.56 10.162.64.57 10.162.64.58 10.162.64.59 10.162.64.60 10.162.64.61 10.162.64.62 10.162.64.63 10.162.64.98 10.162.64.99 10.162.72.15 10.162.72.16 10.162.72.19 10.162.72.20 10.162.72.31 10.162.80.10 10.162.80.102 10.162.80.11 10.162.80.112 10.162.80.12 10.162.80.121 10.162.80.122 10.162.80.131 10.162.80.132 10.162.80.140 10.162.80.150 10.162.80.151 10.162.80.152 10.162.80.153 10.162.80.154 10.162.80.155 10.162.80.20 10.162.80.21 10.162.80.30 10.162.80.31 10.162.82.10 10.162.82.11 |
And to show the hostnames as well;
root@ns0:~# for ip in $(cat /var/log/syslog|grep -i "10.162.64.1"|grep -ivE "(no free leases|ignored)"|grep -i "DHCPOFFER"|perl -wple 's|\s+| |g'|cut -d' ' -f8|sort|uniq); do hostname=`dig -x $ip +short|cut -d'.' -f1`; echo "$hostname ($ip)"; done fepdimutrk2-charm (10.162.64.100) fepdimutrk4-charm (10.162.64.102) fepdimutrk5-charm (10.162.64.103) fepfmdaccorde-charm (10.162.64.109) cn000-charm (10.162.64.11) fephltout0-charm (10.162.64.110) fephltout1-charm (10.162.64.111) fephmpid0-charm (10.162.64.113) fephmpid1-charm (10.162.64.114) fephmpid2-charm (10.162.64.115) fephmpid3-charm (10.162.64.116) fepphos0-charm (10.162.64.117) fepphos1-charm (10.162.64.118) fepphos2-charm (10.162.64.119) cn001-charm (10.162.64.12) fepphos3-charm (10.162.64.120) fepphos4-charm (10.162.64.121) fepspd0-charm (10.162.64.129) cn002-bmc (10.162.64.13) fepspd1-charm (10.162.64.130) fepspd2-charm (10.162.64.131) fepspd3-charm (10.162.64.132) fepspd4-charm (10.162.64.133) fepssd0-charm (10.162.64.134) fepssd1-charm (10.162.64.135) fepssd2-charm (10.162.64.136) fepssd3-charm (10.162.64.137) cn003-bmc (10.162.64.14) cn004-bmc (10.162.64.15) feptpcai00-charm (10.162.64.155) feptpcai02-charm (10.162.64.156) feptpcai04-charm (10.162.64.157) feptpcai06-charm (10.162.64.158) feptpcai08-charm (10.162.64.159) cn005-bmc (10.162.64.16) feptpcai10-charm (10.162.64.160) feptpcai12-charm (10.162.64.161) feptpcai14-charm (10.162.64.162) feptpcai16-charm (10.162.64.163) feptpcao00-charm (10.162.64.164) feptpcao01-charm (10.162.64.165) feptpcao02-charm (10.162.64.166) feptpcao03-charm (10.162.64.167) feptpcao04-charm (10.162.64.168) feptpcao05-charm (10.162.64.169) feptpcao06-charm (10.162.64.170) feptpcao07-charm (10.162.64.171) feptpcao08-charm (10.162.64.172) feptpcao09-charm (10.162.64.173) feptpcao10-charm (10.162.64.174) feptpcao11-charm (10.162.64.175) feptpcao12-charm (10.162.64.176) feptpcao13-charm (10.162.64.177) feptpcao14-charm (10.162.64.178) feptpcao15-charm (10.162.64.179) feptpcao16-charm (10.162.64.180) feptpcao17-charm (10.162.64.181) feptpcci00-charm (10.162.64.182) feptpcci02-charm (10.162.64.183) feptpcci04-charm (10.162.64.184) feptpcci06-charm (10.162.64.185) feptpcci08-charm (10.162.64.186) feptpcci10-charm (10.162.64.187) feptpcci12-charm (10.162.64.188) feptpcci14-charm (10.162.64.189) feptpcci16-charm (10.162.64.190) feptpcco00-charm (10.162.64.191) feptpcco01-charm (10.162.64.192) feptpcco02-charm (10.162.64.193) feptpcco03-charm (10.162.64.194) feptpcco04-charm (10.162.64.195) feptpcco05-charm (10.162.64.196) feptpcco06-charm (10.162.64.197) feptpcco07-charm (10.162.64.198) feptpcco08-charm (10.162.64.199) feptpcco09-charm (10.162.64.200) feptpcco10-charm (10.162.64.201) feptpcco11-charm (10.162.64.202) feptpcco12-charm (10.162.64.203) feptpcco13-charm (10.162.64.204) feptpcco14-charm (10.162.64.205) feptpcco15-charm (10.162.64.206) feptpcco16-charm (10.162.64.207) feptrd00-charm (10.162.64.209) feptrd04-charm (10.162.64.210) feptrd08-charm (10.162.64.211) feptrd10-charm (10.162.64.212) feptriggerdet-charm (10.162.64.214) cn014-bmc (10.162.64.25) cn015-bmc (10.162.64.26) cn016-bmc (10.162.64.27) cn017-bmc (10.162.64.28) cn018-bmc (10.162.64.29) cn019-bmc (10.162.64.30) cn028-bmc (10.162.64.39) cn029-bmc (10.162.64.40) cn030-bmc (10.162.64.41) cn031-bmc (10.162.64.42) cn032-charm (10.162.64.43) cn033-charm (10.162.64.44) cn034-bmc (10.162.64.45) cn035-bmc (10.162.64.46) cn036-bmc (10.162.64.47) cn037-bmc (10.162.64.48) cn038-charm (10.162.64.49) cn039-charm (10.162.64.50) cn040-bmc (10.162.64.51) cn041-bmc (10.162.64.52) cn042-bmc (10.162.64.53) cn043-bmc (10.162.64.54) cn044-bmc (10.162.64.55) cn045-bmc (10.162.64.56) cn046-bmc (10.162.64.57) cn047-bmc (10.162.64.58) cn048-bmc (10.162.64.59) cn049-bmc (10.162.64.60) cn050-bmc (10.162.64.61) cn051-bmc (10.162.64.62) cn052-charm (10.162.64.63) fepdimutrg-charm (10.162.64.98) fepdimutrk1-charm (10.162.64.99) cndev004-charm (10.162.72.15) cndev005-charm (10.162.72.16) fepdev10-charm (10.162.72.19) fepdev11-charm (10.162.72.20) msdev0-bmc (10.162.72.31) cooling0-charm (10.162.80.10) portal-ecs1-bmc (10.162.80.102) ns0-bmc (10.162.80.11) portal-dcs1-bmc (10.162.80.112) ns1-bmc (10.162.80.12) portal-vobox0-charm (10.162.80.121) portal-vobox1-charm (10.162.80.122) portal-gui0-charm (10.162.80.131) portal-gui1-charm (10.162.80.132) hotspare0-bmc (10.162.80.140) dfs0-charm (10.162.80.150) dfs1-charm (10.162.80.151) dfs2-charm (10.162.80.152) dfs3-charm (10.162.80.153) dfs4-charm (10.162.80.154) dfs5-charm (10.162.80.155) db0-bmc (10.162.80.20) db1-bmc (10.162.80.21) mon0-bmc (10.162.80.30) mon1-bmc (10.162.80.31) vhost0-charm (10.162.82.10) vhost1-charm (10.162.82.11) |