I'll just provide my own example: my homelab consists of 6 Kubernetes nodes placed across the country. Some differ by ISP, some are placed in different cities, one is hosted on a cloud provider. Basically it's a very cheap variant of geo-replicating my workloads.

Two of these nodes are visible from the Internet and have a static IP address; one node also has an IPv6 address. Each node hosts an authoritative DNS server (CoreDNS) for my personal domain pootis.network; and the .network TLD has glue records which point to IPs of these two nodes. This is a classic "self-hosted DNS" scenario.

Here's an excerpt from my zonefile so you can understand the setup better:

$ORIGIN pootis.network.
$TTL 300

@       SOA     ns1.pootis.network. admin.pootis.network. (
  2026082001
  1200
  300
  1209600
  300
)

; Nameservers and glue records
@       NS      ns1.pootis.network.
@       NS      ns2.pootis.network.
ns1     A       178.44.116.85
ns2     A       91.219.150.30
ns2     AAAA    2a06:dd00:1:4::4189

This 5-record block (NS/A/AAAA) is mirrored into the .network zone by my domain registrar (plus DS for DNSSEC but that's another thing).

As such, my DNS becomes fully independent - and, in theory, if one of my externally-facing nodes breaks, let's say ns1, then DNS resolvers all over the world (forwarders, recursive, and such) will fall back to ns2, and everything will keep working. Kubernetes will also reorganize the pod placement so all my workloads are available again after a slight downtime.

That would have been great, if it worked as described, but apparently, after one nameserver in my zone fails, then the resolvers... just give up? Let's say ns1 failed but ns2 is working. The parent zone still points to both nameservers. My external resource records (websites and other stuff) at this point would have already been auto-reconfigured by a custom k8s controller to point to the IP addresses of the node that hosts ns2. Simplifying: the entire world basically sees this after ns1 fails and after TTL caches expire:

; all of this has very low TTL, 5 minutes or so

@       NS      ns1.pootis.network. ; from .network 
@       NS      ns2.pootis.network. ; from .network

ns1     A       178.44.116.85 ; broken. Either from .network glue or from my auth DNS
ns2     A       91.219.150.30 ; either from .network glue or from my auth DNS
ns2     AAAA    2a06:dd00:1:4::4189 ; same

; my-website     A       178.44.116.85 ; does not appear because ns1 is broken- my LB already removed it from the set
my-website     A       91.219.150.30 ; fronted by a pair of CNAMEs due to loadbalancing but still
my-website     AAAA    2a06:dd00:1:4::4189 ; same

But even if I query 1.1.1.1 directly for my-website's record, it just doesn't work most of the time because the resolver pins itself to ns1 which is currently failing, or it selects ns1 and does not even care to try ns2.

To be precise: some resolver implementations DO fall back to ns2 as expected, but most of them just pin themselves to ns1 and then outright refuse to resolve the records in my zone.

And there's actually no reasonable way out, as far as I can see:

• moving my DNS infra somewhere else (CloudFlare, for example) is unacceptable since I would like for my homelab to be as independent as practically possible;
• anycasting, or running a fully-fledged BGP AS is also impossible because that costs a lot of money and I'd like for my homelab to fit into a $10/month budget with room to spare;
• "live-patching" the NS and glue records in the parent zone (.network), to keep up with the set of my working nodes, is possible, but very unwieldy and somewhat hard to accomplish.

There's a lot of custom machinery that keeps my workloads running and accessible after a node failure, but all of this becomes completely moot when authoritative DNS is the bottleneck.

Has anyone been running a similar stack and encountered this problem? I'm aware that the answer is usually "host your DNS at CloudFlare" or "use the registrar's DNS infra" but still...

Not sure if it helps but I'm using BunnyDNS from Slovenia for $1/month. I would like to host my own DNS servers but I'm not at that level yet.

Put HE (dns.he.net) in your back pocket - they run a free DNS service which includes slave (do we still call it that?) service so that you can add their resolver as an ultimate failover / failback if your self hosted DNS takes a dive for whatever reason things break in our world.

Generally, hosting your own DNS "should" mean having a primary and secondary on separate AS networks, or at least different provider routes or locations. But if you're doing all this from home, where is the second location for DR purposes... exactly? I sure don't have one, power goes out here everything goes dark - in the old days everyone used to do this (share out tertiary DNS to each other) but it's become a thing of the past in today's world. HE can help fill that gap.

To your point - I use Gcore DNS and Bunny DNS for my stuff, as I mainly use online VPSes and don't run racks of hardware at the house. The only things I physically selfhost at home are a pair of old laptops acting as storage servers which backup all my bytes and bits from online to some SSDs.

But even if I query 1.1.1.1 directly for my-website's record, it just doesn't work most of the time because the resolver pins itself to ns1 which is currently failing, or it selects ns1 and does not even care to try ns2.

That sounds bad. They really shouldn't do that.

moving my DNS infra somewhere else (Cloudflare, for example)

And it would also mean this isn't a solution either. If the problem is on the resolvers' side, moving to a different DNS hoster would not change anything. Hm. And the other two potential solutions you've listed would be extremely overkill for this use case...

Personally, I only host one authoritative nameserver and do redundancy by using some free secondary DNS services (although a friend has also recently set up an authoritative nameserver, so maybe we'll host secondary DNS for each other in the future). I haven't encountered this issue before, but my server also hasn't been down a lot.

There's a lot of custom machinery that keeps my workloads running and accessible after a node failure, but all of this becomes completely moot when authoritative DNS is the bottleneck.

If it's only necessary for your own stuff, maybe you should set up a local DNS resolver that works correctly which your services can use?

(do we still call it that?)

Primary and secondary is what it is usually called today. I know both Microsoft and Cloudflare does at least.
https://www.cloudflare.com/learning/dns/glossary/primary-secondary-dns/

I actually already have a local DNS resolver... well, sort of: most of my cluster's internal services, such as Vaultwarden and other stuff, are available only over an in-cluster IKEv2 VPN.

What I mean is, I have to connect to the VPN first, which pushes a Configuration Payload to my IKEv2 client, and that payload contains a DNS server IP, and that DNS server (exposed by a Kubernetes Service) serves a custom "internal" zone (.int.pootis.network), and proxies everything else either to k8s coredns, or to upstream DNS forwarders.

Okay, that explanation may have been somewhat complicated (it sounded simpler in my head) but the end result is that I can resolve and reach stuff like lemmy.int.pootis.network (my Lemmy frontend) only while connected to the in-cluster VPN. It is completely hidden from external users (no CT log record, no external DNS record, and only proxyable by internal Traefik instance).

The issue only occurs with my externally-facing DNS, which is needed to, for example, to reach my Lemmy backend instance (lemmy.pootis.network) and the pictrs deployment.
But yeah, internal DNS works perfectly for me and it easily survives a node failure.

Basically my internal DNS zonefile is configured like this (it should be easier to read than my explanation):

coredns_config:
  zoneFiles:
    - filename: int.pootis.network.zone
      domain: int.pootis.network
      contents: |
        $ORIGIN int.pootis.network.
        $TTL 300

        @       SOA     ns.int.pootis.network. admin.pootis.network. (
          2026082001
          1200
          300
          1209600
          300
        )

        @       NS      ns.int.pootis.network.
        ns      AAAA    {{ k8s_dns_ipv6 }}
        ns      A       {{ k8s_dns_ipv4 }}

        vault     CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        cinny     CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        grafana   CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        stalwart  CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        lemmy     CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.
        mail      CNAME   stalwart.stalwart.svc.{{ k8s_cluster_domain }}.
        webmail   CNAME   internaltraefik.internaltraefik.svc.{{ k8s_cluster_domain }}.

        ; ACME challenges
        ; int.pootis.network + *.int.pootis.network
        _acme-challenge      CNAME 1ca12d2c-8034-4a09-a940-dc59bd4f38d8.acme-dns.pootis.network.
        ; mail.int.pootis.network
        _acme-challenge.mail CNAME d175b66d-909f-46ba-8572-636ebc235d4c.acme-dns.pootis.network.

Interesting, that's much more sophisticated than my setup (though to be fair, I host my services on a rented VPS instead of home)!

The issue only occurs with my externally-facing DNS, which is needed to, for example, to reach my Lemmy backend instance (lemmy.pootis.network) and the pictrs deployment. But yeah, internal DNS works perfectly for me and it easily survives a node failure.

Could still set up a recursive resolver for external DNS, but I guess that wouldn't help other people who use a bad resolver. Tricky situation.

Yeah, the issue is that people's resolvers are wildly different and I guess some of them do not cater to weird and wacky self-hosting setups (most people just use anycast highly-available DNS, usually provided by a registrar or by some other company).

I had a hunch that resolvers break because one of my nodes does not have a public IPv6 address: meaning,

• node2: externally-facing node, I host it from my home, IP 178.44.116.85, acts as NS ns1. It has IPv6 connectivity but no external IPv6;
• node4: a cloud VPS that I rent. Externally-facing, with IPv4 91.219.150.30 and IPv6 2a06:dd00:1:4::4189, acts as NS ns2.

If node4 decides to break, then ns2 becomes unavailable - but since resolvers can see that ns1 does not have an IPv4 address, and since everyone knows that IPv6 is better because 6 is greater than 4, then these resolvers would always prefer ns2, even though it's clearly down.

This could be solved by asking my ISP to set up a static IPv6 prefix for me (they already provide static IPv4), except my ISP does not do that, unfortunately. I could also rent another VPS (so I would have two VPS'es, giving me some resilience from downtimes) but that means extra $$$... anyway, I guess I should probably test the IPv6-preference hypothesis first and then act on the results.

With multiple DNS servers and zone transfers.

Yeah unfortunately you're depending on proper behavior from other people's systems, which is almost never a safe thing to do.

I'm sure there's some magic you can do to only have one ns backed by two servers, but the easiest fix would be to randomize the order that the ns records are presented. It would reduce, but not eliminate, the failure you describe. If ns1 is down but ns2 is listed first, even a poorly-behaved resolver should be able to resolve your stuff.

It’s going to be something stupid like “ns1 doesn’t have an AAAA listed and so therefore all queries fall back to v4 only, and because ns2 has an AAAA then it is ignored to prevent responses in v6 breaking clients because those queries now only support v4 responses”

At this point I pretty much think this may be the exact case - and it may work in reverse: "since ns2 has AAAA listed then it's an IPv6-capable deployment and we are allowed to fall back to IPv4, but since ns1 lacks the AAAA record, then we won't even consider falling back to ns1 if ns2 is down".

Or maybe another scenario:

• let's check ns2's AAAA record
• oh it's down! Let's check ns2's A record
• it's down too! Let's switch to ns1...
• ... but at that time the DNS resolution timeout expires, since we took too long resolving the records; and on the second try, no one remembers that ns2 was down, repeating the cycle

BGP anycast person here. If you have any presence in the region RIPE operates in then the pricing is within the homelab reach. ASN and a block of /48 would be about 70 EUR/year.

Alternatively, something like route64 would happily tunnel you IPs they announce for about 2 EUR/month.

I don't know of any resolvers that simply fail after not getting responses from a single name server. Please name and shame!

That does seem to be a good solution, thank you for the recommendation!

Running an AS and obtaining a /48 through a sponsoring LIR seems to cost about $150/year in my country, but hardly anyone (except hosting providers and large companies) does this, since self-hosting, especially more complicated stuff, isn't really that popular here; people are mostly uninformed that it even exists.

But route64 is apparently completely free (donations are welcome); they provide a /56 PA-like IPv6 block carved out of their PI, and they also handle BGP stuff, and I don't even have to pay for any of this - overall, a great choice, considering my constraints. Anycast to multiple tunnels costs money (maybe that's what you meant with 2 EUR/month?) but that's pretty much lunch money so it would be OK with me.

May I ask how you got this started? Hosting your own ASN, as far as I understand only works if my ISP would actually route traffic to my ASN, right? I am thinking about getting into IPv6 for self-hosting and I could ask my ISP to change my current setup (I only have IPv4, but a public one, without CGNAT) and I do not trust them that they mess this up. So my preferred way would be to not touch anything on the ISP side and host my own ASN and find a Sponsor for a /48 Block. I still cannot wrap my head around this.

Just that I understand you correctly: You got your ASN and /48 block from RIPE (or a Sponsor I assume) and you host your own AS? Or is the AS hosted by someone else? If the latter, I wonder how traffic can find to your home or to your Server locations.

Nothing of this would work without getting in touch with my ISP, and I fear the usual resedential IPs will not care.

First on how to get an ASN: you can buy it for reasonably cheap from a LIR. Some will even toss a free /48 with that. Happy to offer names in private so that there's no advertising. Expect a budget quoted above.

Once you have an ASN, you need to get an upstream - actually two as RIPE mandates at least two (otherwise why’d you need an ASN). Some LIRs would offer transit with ASN purchase. You can upstream via your ISP, if they allow you to (that's very rare). Another option is a tunnel (there are free and paid ones) or a VM somewhere (some cloud providers offer to set up bgp with VMs they host). Generally, free ones are enough for basic stuff. Not much bandwidth and oftentimes IPv6 only, but you don’t pay anything either. Besides, you can ask around in various network related chats. Practically, I can offer ip transit with some marginally low burstable bandwidth, and that’s pretty common. You can look/ask around https://discord.gg/ipv6 for example.

For getting ASN to your homelab you’re looking at a tunnel option, most probably. Great if you have static ipv4 - allows you to use more common tunnels, but is still doable with a floating IP (e.g. check bgptunnel).

Thank you very much for your reply. So the bottomline is, that when bound to a residential ISP, we most likely have no chance to utilize our own AS and IPv6 Prefix without the help of a 3rd party (ASN peering aside, which is needed of course, no matter what). So it's usually involving tunnels to get it up and running. That solves the mystery somewhat for me and I think I know in which direction I have to do my research.

I think I will start prototyping with route64 (which you mentioned above), but if I get this up and running, I would appreciate it if you could PM me the LIRs you have on your list. If I can wrap my head around this, if the fees are still in the ballpark you mentioned above, that sounds very reasonable to use this in my Homelab and gather experience with IPv6 and AS.

Thanks again for your help :-)

Since you're using Kubernetes, have you looked at external-dns? In my home lab, I use external-dns to take Ingress and Service resources and register them into my internal DNS zone. I use the etcd backend. external-dns registers zones in etcd, then CoreDNS queries from etcd. As soon as I create a new Ingress, it's automatically registered and pointing to my Traefik service.

https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/coredns-etcd.md