Rendered at 09:35:26 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
quuxplusone 8 hours ago [-]
From my read of the article, this isn't a problem "in DNS." OP runs an uptime monitoring service that purports to check whether DNS can resolve your domain — but today OP learned that because he's hitting his ISP's recursive resolver, he doesn't notice downtime until the TTL of the previous response expires.
Solution (which everyone else does, and OP has now implemented): don't use your ISP's recursive resolver! Run your own instance of bind9 or whatever, with the cache disabled. Or it seems like `dig +trace` would probably work, too.
"Cached resources remain visible for their whole TTL, even if the original becomes unreachable or changes" seems like one of the very first principles someone should learn when deciding to go into business selling an uptime monitoring service.
It's not a "ghost domain," it's a Time-To-Live field!
toast0 3 hours ago [-]
If you disable caching on your recursive resolver, I guess that works...
But I think there is a real issue if caching recursive resolvers don't revalidate delegations. If .com tells you the example.com nameserver is ns1.example.com with some TTL, and ns1.example.com tells you its the nameserver for example.com every time you ask for www.example.com, you should still check in with the .com nameservers ever TTL to validate.
I'm not surprised the default behavior doesn't do that, because when I migrated a high profile domain to new nameservers, we still saw requests to the old servers for more than a month... but it's still ugh, recursive DNS should do better!
whatthesmack 8 hours ago [-]
I get where you're coming from, but I think the part you're missing is that caching is not optional (per the "with the cache disabled" comment). If you're running a service like theirs, you need to do caching to some degree.
My understanding of their implementation is that it does caching on a per-worker basis with lower TTLs, so it balances caching with accuracy, but doesn't "fix" visibility to the problem either. However, it narrows the window such that their method of monitoring will very likely expose a pulled domain sooner.
wahern 6 hours ago [-]
Caching is always optional. If you want sophisticated control over caching, or just no caching at all, you can do recursive lookups directly. Some existing DNS libraries can do recursive lookups directly, including libunbound. Ideally you would do caching of TLD nameserver addresses so you're not hammering the root servers for each query. But otherwise I don't think it's necessarily a big deal for a monitoring service to query the .de nameservers every time it checks example.de. If you had a monitor check every 5 minutes for example.de, it's only a 12x load factor between querying .de directly every check versus once per hour. Unless you're AWS Route53, that's not much of a difference relative to the traffic TLD servers handle even if you were checking tens of thousands of domains. Clamping your cache at 1 hour is probably already a much greater load factor. The .de nameservers return a TTL of 1 day for the example.de NS RR sets, so that's already a 24x factor right there. At that point you're clearly giving the .de nameservers the middle finger and relying on your own judgement of what it means to play nice.
johnhtodd 9 hours ago [-]
One approach to solving this for a very limited set of intervals is to actually block namespace that has been removed at the registry level. There is a paper on this from Raffaele Sommese:
Quad9 (9.9.9.9) consumes this feed from U Twente of "just deleted" names, as most of them are malicious, and blocking them even if they are NOT malicious causes zero harm. Currently, this is only names that are very short-lived, so may not catch the longer intervals where names are deleted and become ghosts.
Another model using something similar would be to specifically clear those "just-deleted" name cached entries out of the recursive resolver, but that is expensive. Also, with blocking instead of removal it is possible to get high-level metrics on how often those are being abused where NXDOMAIN tracking is not measured in the same dimensions.
(disclaimer: I work for Quad9)
winstonwinston 8 hours ago [-]
Technically every domain is a ‘ghost’ domain until TTL expires and NS RRs are usually cached for very long.
Solution (which everyone else does, and OP has now implemented): don't use your ISP's recursive resolver! Run your own instance of bind9 or whatever, with the cache disabled. Or it seems like `dig +trace` would probably work, too.
"Cached resources remain visible for their whole TTL, even if the original becomes unreachable or changes" seems like one of the very first principles someone should learn when deciding to go into business selling an uptime monitoring service.
It's not a "ghost domain," it's a Time-To-Live field!
But I think there is a real issue if caching recursive resolvers don't revalidate delegations. If .com tells you the example.com nameserver is ns1.example.com with some TTL, and ns1.example.com tells you its the nameserver for example.com every time you ask for www.example.com, you should still check in with the .com nameservers ever TTL to validate.
I'm not surprised the default behavior doesn't do that, because when I migrated a high profile domain to new nameservers, we still saw requests to the old servers for more than a month... but it's still ugh, recursive DNS should do better!
My understanding of their implementation is that it does caching on a per-worker basis with lower TTLs, so it balances caching with accuracy, but doesn't "fix" visibility to the problem either. However, it narrows the window such that their method of monitoring will very likely expose a pulled domain sooner.
https://static.sched.com/hosted_files/icann83/5b/Rafaelle%20...
Quad9 (9.9.9.9) consumes this feed from U Twente of "just deleted" names, as most of them are malicious, and blocking them even if they are NOT malicious causes zero harm. Currently, this is only names that are very short-lived, so may not catch the longer intervals where names are deleted and become ghosts.
Another model using something similar would be to specifically clear those "just-deleted" name cached entries out of the recursive resolver, but that is expensive. Also, with blocking instead of removal it is possible to get high-level metrics on how often those are being abused where NXDOMAIN tracking is not measured in the same dimensions.
(disclaimer: I work for Quad9)