Not a good day. And it didn’t even start good. I got a message from Erik that the Internet connection was down. I hurried over to check the pertinent server and found out that the disk was full. I noticed that the messages log was HUGE so I truncated it and restarted services. Presto! Problem solved. Or so I thought.
A few moments later, Erik told me that in addition to the Internet connection, DHCP clients were unable to acquire addresses. I checked the messages logs and a found repetitive stream of DHCPDISCOVERs, DHCPOFFERs, DHCPREQUESTs, and DHCPACKs: It was the reason why the messages log grew so big! DHCP clients were trying over and over to get an IP but couldn’t. I tried everything I (and Google) can think of and more.
All the while and even through working lunch, people were calling asking why there’s a problem and worse, telling me that it shouldn’t happen (duh!), and generally whining. I hate it when people do that. Can’t they see I was doing something?!? Can’t they just let the crisis pass then whine later?
Worse, by the end of the day, I wasn’t able to fix the problem. Instead we worked around it and just manually configured all the DHCP clients with IPs. What a pain!