It Is Always DNS

"It's always DNS" is the oldest joke in operations, and like the good
jokes it is repeated because it keeps being true. Something breaks, an
hour disappears into the wrong layer, and the root cause turns out to
be a name that resolved to the wrong thing, or did not resolve, or
resolved differently depending on who asked. This post is about why DNS
earns the blame so reliably, and how to stop guessing and actually
prove where it broke.

First, What DNS Actually Does

DNS turns a name into an address. You type blog.c0xl.ch, but the
network only routes to numbers, so before anything else can happen,
something has to look up the address behind that name. That lookup is
not one step. It is a chain, and knowing the chain is what makes the
rest of this post make sense.

Here is the full journey the first time a name is looked up:

   You ask for  blog.c0xl.ch
        |
        v
   +-------------------+     "Do you have it cached?"
   |  Your computer    |     If yes -> done instantly.
   |  (OS resolver)    |     If no  -> ask the resolver.
   +-------------------+
        |
        v
   +-------------------+     Your ISP's or a public resolver
   | Recursive resolver|     (like 1.1.1.1 or 8.8.8.8).
   | (1.1.1.1, 8.8.8.8)|     It does the legwork below for you.
   +-------------------+
        |
        |   asks, in order, if not already cached:
        |
        v
   +-------------------+     "Where is .ch handled?"
   |   Root servers    | -->  points to the TLD servers
   +-------------------+
        |
        v
   +-------------------+     "Which nameservers are
   |  TLD servers (.ch)| -->  authoritative for c0xl.ch?"
   +-------------------+
        |
        v
   +-------------------+     THE source of truth.
   |  Authoritative    | -->  "blog.c0xl.ch is 203.0.113.10"
   |  nameserver       |
   +-------------------+
        |
        v
   Answer travels back up, and everyone caches it on the way.

The last line is the whole story of this post. The answer does not
just come back. Every layer it passes through keeps a copy.

Why DNS Gets Blamed

DNS gets blamed because it sits underneath everything and fails in ways
that do not look like DNS.

Almost every network operation starts with that name-to-address lookup.
Your app connects to a database by hostname, pulls an image from a
registry by hostname, calls an API by hostname. When the name resolves
wrong, the failure shows up as the thing that came after: a connection
timeout, a TLS error, a 502, an app that hangs on startup. You go debug
the database, the proxy, the certificate, the application, and the
actual problem was one layer below all of them, in the step so routine
nobody thinks to check it.

And DNS fails quietly. A wrong answer is still an answer. The lookup
succeeds, returns an IP, and the IP is simply the old one, or someone
else's, or an internal address that does not exist anymore. Nothing
errors. The system does exactly what it was told, and what it was told
was wrong. That is the worst kind of failure to chase, because every
layer reports healthy while the whole thing is broken.

The Real Reason: Caching Everywhere

The single biggest source of DNS confusion is that the answer is cached
in more places than anyone keeps track of, and each cache expires on
its own schedule.

Every DNS record has a TTL, a time-to-live in seconds, that says how
long a cached answer stays valid. Think of it as a "use by" timer
stamped on the answer. That sounds simple until you count how many
independent caches sit between a name and your actual program:

  The SAME lookup can be answered from any of these,
  and each keeps its own copy with its own timer:
 
  [ App's own memory ]   <- e.g. a Java (JVM) app caches for its whole life
        |
  [ Service mesh sidecar ]  <- Envoy etc. has its own DNS cache
        |
  [ Kubernetes NodeLocal + CoreDNS ]  <- two more layers, in a cluster
        |
  [ Operating system ]   <- systemd-resolved / nscd on Linux
        |
  [ Recursive resolver ]  <- your ISP or 1.1.1.1 / 8.8.8.8
        |
  [ Authoritative server ]  <- the real source of truth
 
  Each box above can hand back a DIFFERENT answer, because
  each one's copy expires at a different moment.

So when you change a record and it "does not propagate," nothing is
propagating anywhere. There is no magic push that goes out and updates
the world. Every one of those caches is just holding the old answer
until its own timer runs out, and they all started counting at
different times. Two machines, or even two programs on one machine,
genuinely see different answers for the same name at the same moment,
and both are behaving correctly. This is not a bug. It is the design.
Misreading it as a bug is how people burn an afternoon changing a
record over and over expecting it to take effect faster, which does
nothing, because the stale caches do not care how many times you edit
the source.

The Propagation Trap

The most common self-inflicted version, and the one a beginner hits
first:

You have a record with a long TTL, say a day, because it points at
something stable. Then you need to move that thing urgently. You change
the IP and discover every resolver on earth still has the old address
cached for up to a day, and keeps sending traffic to a server that is
already gone.

  09:00  You change the IP at the source (authoritative).
  09:00  But the old answer had TTL = 1 day, cached at 08:00...
         so caches keep serving the OLD ip until ~08:00 tomorrow.
 
  Result: your change is "live" but the world does not see it
          for up to 24 hours. Editing it again does not help.

The fix has to happen before the change, not after. Lower the TTL to
sixty seconds well ahead of any planned move, wait for the old long TTL
to age out everywhere, then make the change, and clients pick it up
within a minute. Raise the TTL again once things are stable. The TTL is
a promise about how long future answers may be cached; lowering it does
not erase answers already handed out under the old value. If you only
think about TTL at the moment you need to change something, you are
already too late, and you wait out the day.

One extra cruelty: some ISPs and mobile carriers override your TTL and
enforce a minimum of their own. Your sixty-second TTL becomes an hour
on their network no matter what you set, so "propagation" can drag on
certain networks regardless of how careful you were.

How to Actually Prove It

The reason DNS debugging feels like guessing is that people run one
lookup against their own computer and read whatever cached answer comes
back, which tells them nothing about where the chain actually broke.
The whole skill is asking a specific server and reading what it says.
The tool for this is dig. A few moves settle almost everything.

Ask a specific resolver instead of your default. Query a couple of
public resolvers directly and compare:

  dig @1.1.1.1 blog.c0xl.ch +short
  dig @8.8.8.8 blog.c0xl.ch +short

If they disagree, you are watching caches expire at different rates,
which means the record changed recently and you are looking at
propagation, not a misconfiguration. If they agree and the answer is
wrong, the problem is upstream at the source.

Ask the authoritative server directly. This is the move that separates
"still waiting for caches to expire" from "the zone itself is wrong."
Find the nameservers responsible for the zone, then ask one of them
straight, with no caching in between:

  dig NS c0xl.ch +short              # who is authoritative?
  dig @<that-nameserver> blog.c0xl.ch +short   # ask it directly

If the authoritative answer is correct but public resolvers still serve
the old value, you are only waiting on TTL and there is nothing to fix.
If the authoritative answer itself is wrong, you found the actual bug,
and no amount of waiting will fix it.

Trace the whole chain. This walks from the root down to the
authoritative server, showing every hop:

  dig blog.c0xl.ch +trace

It is how you catch the subtle ones: a nameserver that does not answer
for a zone it was delegated, or a hop that times out.

Read the status word. Every answer carries one, and it points straight
at the kind of problem:

  NOERROR   -> it resolved. (The value may still be wrong, but DNS worked.)
  NXDOMAIN  -> the name does not exist (or its non-existence is cached).
  SERVFAIL  -> a server in the chain errored or could not be reached.

NXDOMAIN and SERVFAIL point at completely different causes, and knowing
which you got stops you looking in the wrong place. (Yes, even "does not
exist" gets cached, so a name can keep saying NXDOMAIN for a while after
you create it.)

When dig sees the right answer but your app does not. This one
confuses everyone, and it is the in-process cache almost every time.
dig asks the resolver fresh; your long-running application asked once
at startup and cached the result inside itself. The JVM is the classic
offender. The lookup chain is fine. The application is holding a stale
answer and will keep holding it until you restart it.

The Checklist That Saves the Afternoon

When something is broken and DNS is a suspect, before you go deep on the
database or the proxy or the certificate:

  1. dig @1.1.1.1 <name> +short   -> is the answer what you expect?
        wrong?  -> go to step 2
        right?  -> go to step 3
 
  2. dig NS <zone> +short, then dig @<nameserver> <name>
        authoritative right, resolvers wrong -> just wait on TTL
        authoritative wrong -> fix the zone, that is the bug
 
  3. name resolves right everywhere but the app still fails?
        -> suspect an in-process / container cache, restart it
 
  4. still stuck? test from a totally different network (phone hotspot)
        works there, not on your network -> your resolver or ISP

Four checks, a couple of minutes, and you either rule DNS out or you
have proven exactly where it broke. That is the difference between "it's
always DNS" as a shrug and as a diagnosis.

The Point

DNS gets blamed because it underlies every connection, fails as some
other layer's error, and returns wrong answers without ever reporting
one. The confusion is almost always caching: many independent caches,
each with its own timer, each expiring on its own clock, so different
observers correctly see different answers, and editing the source does
not hurry the stale ones along.

Stop guessing. Ask a specific resolver, ask the authoritative server
directly, read whether you got SERVFAIL or NXDOMAIN, and remember your
application keeps its own cache that dig does not share. Do that and
DNS stops being the thing that mysteriously ruins your afternoon and
becomes the thing you rule in or out in two minutes. It is still always
DNS. You can just prove it now.