๐Ÿ’ฅ Today I broke my lab… and learned a lesson in the process. ๐Ÿ˜…

This morning, I connected to my home network over my wireguard tunnel to make some experimental changes to my proxmox host.

Things went well, until they didn’t. I reconfigured the management interface for a project and suddenly, my entire home network went dark.

No access. No response. No turning back… yet.

I figured, “no problem Iโ€™ll just fix it when I get homeโ€. Of course, the rollback didnโ€™t go as planned, and thatโ€™s when the real troubleshooting began.

I dove deep:

๐Ÿ” Scrubbed my VXLAN configs (OVS interfaces)
๐Ÿ” Rebooted EVERYTHING.
๐Ÿ” Dug through /etc/hosts on Proxmox
๐Ÿ” Double-checked all of my bridge interfaces.
๐Ÿ” Ensured /etc/network/interfaces and /etc/resolv.conf were configured properly, etc..

Of course at this point the kiddos needed my attention, so in between feeding them, playing with them, and wiping theirโ€ฆ โ€œtearsโ€ lol, I found little bits of time to keep working on a solution.

Still nothing.

After hours of spiraling through potential worst-case scenarios, I finally looked at my pfSense firewallโ€™s default gateway.

There it was, the WAN gateway had picked up the IP assigned to the gateway on my Wi-Fi network (which is in a subnet separate my main network).

Because I use IP-Passthrough and DHCP on my WAN interface to stay resilient against ISP changes, the system had gone rogue.. it picked up the wrong interface for it’s gateway, and dropped everything.

I fixed the WAN gateway and like magic, the network sprang back to life.

Lessons learned:

1. Never overlook the basics, even when you think their covered. (looking at you, default gateway)

2. Dynamic environments require extra attention to how components resolve IPs and routes, especially when you have a lot of virtualized appliances thrown into the mix.

3. Donโ€™t be afraid to break things, just be prepared for real-world chaos when you do ๐Ÿ˜‰


This is why I love working in IT, and working in my lab. Never stop learning, every misstep is an opportunity to understand your systems a little better.

hashtag#homelab hashtag#networking hashtag#Proxmox hashtag#pfSense hashtag#WireGuard hashtag#ITLife

Leave a Reply