I run a small rack of servers at home. It is where I test everything before it goes near a customer. In early September, the storage server kept losing its network connection for a few minutes. It happened twice that I knew of, on September 9 and September 11, and both times the nightly backups that copy my other machines onto it failed.
Network drops that come and go usually get blamed on hardware: a tired network card, a bad cable. The easy next step would have been to order parts.
Looking one layer down
Servers like mine have a small management computer built in, separate from the main system. It keeps its own logs about power, temperature and every network port going up or down. Nobody had ever asked it anything, and when I did, it had a lot to say.
First, the timing. Both drops happened between 3:35 and 3:40 in the morning. On September 9, every port on every card dropped together for about three minutes. On September 11, only the one port wired straight to my router dropped, for about a minute. A failing card does not keep a schedule like that.
Second, a surprise: both servers had been running on only one of their two power supplies since mid-August, about a month without their backup power path. The cause was a change to the battery backup unit they share, not two broken parts. It had nothing to do with the drops, but I would never have known without looking.
The real cause
My router and network switch were set to install updates automatically, at 3 in the morning. Comparing when each device last restarted with the server's log lined up exactly: the switch rebooted during the first drop, the router during the second. And 3 a.m. was also when my backups ran. The equipment was doing what I had told it to do, at the worst possible time.
The fix cost nothing. I moved the automatic updates to once a week on Saturday morning, away from the backup window. Now I watch the logs for another early-morning drop over the next few weeks before calling it solved. The network card stays.
What this means for you
- If something breaks at the same time every time, ask what else is scheduled then: updates, backups, a timer on the router.
- Check the dates before buying parts. Many "broken" devices are just rebooting to update.
- Stuff you set up once and forgot about is still running. It is worth a look once in a while.
If your Wi-Fi drops every night or your computer restarts on its own, I can help you find out why. If you run your own servers at home and want to learn to read these clues yourself, that is what homelab coaching is for.