So today I went to spin up a new VM for development use. It wouldn’t get an IP address, I saw the DHCP request on the DHCP server, and saw an offer go out but it was never received.  I dug through, and it seemed like this was just happening on one VLAN since everything else was OK.

Did I mention everything else was already running?

Did I mention if I had a trap collector with an alarm board that I would know what had happened almost immediately and been able to pinpoint the issue before I even saw the effects?

No? Well, now I have.

Let’s just say that I spent over an hour digging, running tcpdump on various interfaces, then finally hit the switches. I noticed there was only one port in the port channel on the Dell 5224 access switch when there should have been two down to the distribution switch. Odd but I thought inconsequential (at the time).

I got into the Cisco switch and saw MAC flaps (TRAPPABLE) all over the place with Po2. Odd again. The Dell switch must be to blame, so I go back to it and shut the port that’s not in the LACP port channel but should be. Things improve. Have I mentioned that I’d unplugged that fiber a week ago and only recently got a new one to plug back in?

I spend some time trying to get both ports in the port channel to no avail. I finally look at the config and notice the VLAN allowed config is slightly off (one is missing from eth 1/23), so I shut both the ports on the Cisco side as Dell won’t let you change interface configs while it’s part of a port channel and this was just faster — I reset the eth 1/23 config to match eth 1/24, and voila both ports came up.

But things were even worse now, barely any MACs were seen in ‘show mac address-table’ on my 3550-12 from Po2. And they were all on VLAN 1. Ugh. I shut the interfaces again and reset some more of the configuration on the Dell switch. I pray. (I don’t really pray). I bring the interfaces back up and all is good. The VM gets its IP address and everything is right in the world.

I really hate the Dell configurations. If I hated this switch before it’d be an understatement, and it’s only given me more of a reason to want to smash it with a hammer today. It’s mainly due to me not being familiar with them, but their configs aren’t as intuitive as I’d like.