The cold came to Texas and the internet froze 🧊

I woke up Sunday morning with projects to work on, things to do, and found I could do practically none of it because my internet was down 🤦.

But that's not what bothered me.

What bothered me was that I also could not access any of my locally running self-hosted services.

Now technically, I could access these services if I typed in their exact IP Addresses and port numbers. For example, https://myservice.domain.com did not work, but http://192.168.1.96:1234 worked just fine.

But I wasn't satisfied with that, I don't have all the ports memorized and surely there should be a more user friendly way to access my services locally, right?

👷‍♀ alright, let's get to work

The first step is to always narrow down the problem. So let's focus on why I can't access a single service https://traggo.mydomain.com. This service is very simple, and to my knowledge attempts to makes zero external network calls, so this should limit variables while testing.

🧠 Initial thoughts:

1. Using https://traggo.domain.com requires two things
   1. The DNS record to be resolved by an external service
   2. My home server to be reachable

Ok, this makes sense, the internet is down which means both of these steps will fail. I can't reach out to any DNS providers to resolve the domain and even if I could, the request could not be responded to by my server.

💡 Idea:

1. Resolve the DNS entry locally on my network instead of reaching out to the external provider

I run a local instance of AdGuard Home which is responsible for serving DNS requests. Most of these requests it proxies out to public DNS providers but it also applies it's own filtering and rewrite rules to enable blocking traffic to certain domains like trackers, advertisers, and data collectors.

In simple terms, my network setup should look something like this:

In addition to filtering DNS lookups, AdGuardHome also supports rewrite rules, which simply means we can override what the DNS provider told us and use a different IP Address for any domain we like.

For my problem, the idea is to setup a local rewrite rule that effectively says all calls to traggo.domain.com should be resolved to 192.168.1.96, my home server.

Here's how things will look in this configuration:

So I wire it all up and..... nothing, didn't work. Still can't access https://traggo.domain.com from my browser.

🧘‍♀️ Deeper thoughts:

1. My services are configured behind a reverse proxy that listens on ports :6080 and :6443 (for reasons)
2. My router normally handles the port forwarding of :80 -> :6080 and :443 -> :6443
3. Perhaps, when AdGuard applies the rewrite rule, since this is all internal traffic, the port forwarding is not happening at the router, so my requests are ultimately going to 192.168.1.96:443 instead of 192.168.1.96:6443

💡 Idea:

1. Let's try changing my reverse proxy to listen on the standard :80 and :443

Tested it out, but, this still didn't work.

And neither did the next idea.

Or the next hack.

Or the next one.

Or the one after that.

😞 I give up

Hey 👋, sorry for the tense change, we've got present-Bailey here.... I've sunk 6+ hours (on and off) into investigating this, we're into my second day without internet, and I still haven't figured out how to access my local services using their domain names...

I'm frustrated by this, but have some takeaways from the experience:

1. I don't really understand how networks work, at least not beyond the surface level information, this limits my ability to come up with creative ideas
2. I don't know how to effectively trace or debug network traffic
3. This is impossible for me to solve without Google. I actually did have some access to the internet using mobile data on my phone, but even so I still felt hamstrung

🦤 but wait there's more....

So I'm sitting here fiddling with the mermaid diagram's above, feeling sorry for myself, and musing over point #2 from the "I give up" section. I decide to go poking through my Router's admin page again, remembering that there are some network debugging tools there.

Here I find that the router has a tool for running nslookup. NSLookup should atleast let me sanity check that the router is resolving my domains to the correct IP Address.

I morosely plug in traggo.domain.com and click "Diagnose".

I look at the output.... I look again.... that's the wrong IP Address.

Not only is the domain resolving to my Public IP address (instead of 192.168.1.96), but I can see that it's using my ISP's DNS server to do it. 🤯

This is interesting 🤔, I know I have my router configured to use AdGuard for DNS so why would it be trying to reach out to my ISP?

I poke around a bit more on my router and find this setting: WAN - Connect to DNS Server Automatically: Yes, and I also notice for the first time that my WAN and DHCP DNS settings are different.

💡 Idea:

1. Set WAN - Connect to DNS Server Automatically: No
2. Update Router WAN settings to explicitly use AdGuardHome for DNS

This is an exciting idea, it means my original ideas were probably correct, but this one misconfigured setting was causing me to bypass AdGuard and go to the ISP DNS server instead.

Let's make the changes.

Testing.....

annnnnnnd.....

My internet comes back online 🤦.

No not, the service I'm trying to test, I mean literally after 16hrs without internet, it magically starts working right this minute.

But that won't stop me. We've come too far. I leap from my computer. I run to the networking closet.

I stare my modem's solid blue connectivity lights right in the face, and....

I unplug it like the mad-woman I am 😈.

Lo and behold, it works!

Victory is mine.