RFC 6471 and "listing the Internet" as a punishment

[Martijn Grooten]

It was nice to see the RFC being published. Good work.

Then I came across this:

http://blog.vamsoft.com/2012/01/24/ub-black-uribl-com-url-blacklist-started-to-block-everything/

(Vamsoft ORF is a spam-filter.) Basically uribl.com was returning 127.0.0.1 to _all_ queries from nameservers that are sending high volumes (presumably without paying for it) as some kind of punishment. http://uribl.com/ confirms that.

Now, as Vamsoft mentions, it is not a good idea to use third-party nameservers on a server you're making DNS requests from. (Although, unlike openDNS, Google's nameservers do return NXDOMAIN when they can't resolve a domain.) Moreover, it does seem Google's nameservers are now getting REFUSED as a response to any uribl.com request. I was just wondering whether the RFC says anything about this kind of behaviour ('listing' everything as a punishment). To my reading it doesn't.

Martijn.

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

[Emanuele Balla (aka Skull)]

On 1/24/12 4:07 PM, Martijn Grooten wrote:
> It was nice to see the RFC being published. Good work.
>
> Then I came across this:
>
> http://blog.vamsoft.com/2012/01/24/ub-black-uribl-com-url-blacklist-started-to-block-everything/
>
> (Vamsoft ORF is a spam-filter.) Basically uribl.com was returning
> 127.0.0.1 to _all_ queries from nameservers that are sending high
> volumes (presumably without paying for it) as some kind of
> punishment. http://uribl.com/ confirms that.
>
> Now, as Vamsoft mentions, it is not a good idea to use third-party
> nameservers on a server you're making DNS requests from. (Although,
> unlike openDNS, Google's nameservers do return NXDOMAIN when they
> can't resolve a domain.) Moreover, it does seem Google's nameservers
> are now getting REFUSED as a response to any uribl.com request. I was
> just wondering whether the RFC says anything about this kind of
> behaviour ('listing' everything as a punishment). To my reading it
> doesn't.
>
> Martijn.

In truth there's
in point 3.3:

« Note: In Section 3.4, it is noted that some DNSBLs have shut down in
such a way to list all of the Internet. Further, in Section 3.5,
DNSBL operators MUST NOT list 127.0.0.1. Therefore, a positive
listing for 127.0.0.1 SHOULD indicate that the DNSBL has started
listing the world and is non-functional. »

and again, in point 3.5:

« A functioning DNSBL MUST NOT list 127.0.0.1. There are a number of
mail server implementations that do not cope with this well, and many
will use a positive response for 127.0.0.1 as an indication that the
DNSBL is shut down and listing the entire Internet.»


That is not clearly against "listing everything as a punishment", but
means uribl.com is technically "non-functional"... ;-)


--
Paranoia is a disease unto itself. And may I add: the person standing
next to you may not be who they appear to be, so take precaution.
-----------------------------------------------------------------------------
http://bofhskull.wordpress.com/

[David Romerstein]

On 1/24/12 10:19 AM, Emanuele Balla (aka Skull) wrote:
> In truth there's
> in point 3.3:
>
> « Note: In Section 3.4, it is noted that some DNSBLs have shut down in
> such a way to list all of the Internet. Further, in Section 3.5,
> DNSBL operators MUST NOT list 127.0.0.1. Therefore, a positive
> listing for 127.0.0.1 SHOULD indicate that the DNSBL has started
> listing the world and is non-functional. »
>
> and again, in point 3.5:
>
> « A functioning DNSBL MUST NOT list 127.0.0.1. There are a number of
> mail server implementations that do not cope with this well, and many
> will use a positive response for 127.0.0.1 as an indication that the
> DNSBL is shut down and listing the entire Internet.»

"Listing 127.0.0.1" is not the same as "returning 127.0.0.1".

-- D

[Emanuele Balla (aka Skull)]

On 1/24/12 4:29 PM, David Romerstein wrote:
> On 1/24/12 10:19 AM, Emanuele Balla (aka Skull) wrote:
>> In truth there's
>> in point 3.3:
>>
>> « Note: In Section 3.4, it is noted that some DNSBLs have shut down in
>> such a way to list all of the Internet. Further, in Section 3.5,
>> DNSBL operators MUST NOT list 127.0.0.1. Therefore, a positive
>> listing for 127.0.0.1 SHOULD indicate that the DNSBL has started
>> listing the world and is non-functional. »
>>
>> and again, in point 3.5:
>>
>> « A functioning DNSBL MUST NOT list 127.0.0.1. There are a number of
>> mail server implementations that do not cope with this well, and many
>> will use a positive response for 127.0.0.1 as an indication that the
>> DNSBL is shut down and listing the entire Internet.»
>
> "Listing 127.0.0.1" is not the same as "returning 127.0.0.1".

Damm. FAIL. :-D


--
Paranoia is a disease unto itself. And may I add: the person standing
next to you may not be who they appear to be, so take precaution.
-----------------------------------------------------------------------------
http://bofhskull.wordpress.com/

[Rich Kulawiec]

On Tue, Jan 24, 2012 at 04:19:00PM +0100, Emanuele Balla (aka Skull) wrote:
> and again, in point 3.5:
>
> ? A functioning DNSBL MUST NOT list 127.0.0.1. There are a number of
> mail server implementations that do not cope with this well, and many
> will use a positive response for 127.0.0.1 as an indication that the
> DNSBL is shut down and listing the entire Internet.?
>
> That is not clearly against "listing everything as a punishment", but
> means uribl.com is technically "non-functional"... ;-)

I may well be misreading this report AND the RFC (coffee level: alarmingly
low) but it appears to me that this DNSBL is not listing "127.0.0.1",
but is returning 127.0.0.1 in response to queries for "example.com"
(and all other domains) when those queries are issued from certain hosts.
(And I presume those hosts are the set which have issued excessive
and/or unwelcome queries in the opinions of the DNSBL's operators.)

---rsk

[Emanuele Balla (aka Skull)]

On 1/24/12 4:35 PM, Rich Kulawiec wrote:
> On Tue, Jan 24, 2012 at 04:19:00PM +0100, Emanuele Balla (aka Skull) wrote:
>> and again, in point 3.5:
>>
>> ? A functioning DNSBL MUST NOT list 127.0.0.1. There are a number of
>> mail server implementations that do not cope with this well, and many
>> will use a positive response for 127.0.0.1 as an indication that the
>> DNSBL is shut down and listing the entire Internet.?
>>
>> That is not clearly against "listing everything as a punishment", but
>> means uribl.com is technically "non-functional"... ;-)
>
> I may well be misreading this report AND the RFC (coffee level: alarmingly
> low) but it appears to me that this DNSBL is not listing "127.0.0.1",
> but is returning 127.0.0.1 in response to queries for "example.com"
> (and all other domains) when those queries are issued from certain hosts.
> (And I presume those hosts are the set which have issued excessive
> and/or unwelcome queries in the opinions of the DNSBL's operators.)

You (and David) are absolutely right: I simply messed up the whole
point. ;-)


The only point that directly applies, here, is IMHO 3.3:

«
3.3. DNSBLs SHOULD Provide Operational Flags

Most IP address-based DNSBLs follow a convention of query entries for
IP addresses in 127.0.0.0/8 (127.0.0.0-127.255.255.255) to provide
online indication of whether the DNSBL is operational. Many, if not
most, DNSBLs arrange to have a query of 127.0.0.2 return an A record
(usually 127.0.0.2) indicating that the IP address is listed. This
appears to be a de facto standard indicating that the DNSBL is
operating correctly. See [RFC5782] for more details on DNSBL test
entries.

If this indicator is missing (query of 127.0.0.2 returns NXDOMAIN),
or any query returns an A record outside of 127.0.0.0/8, the DNSBL
should be considered non-functional.
»

Somehow, this seems to allow URIBL to do what it does: they state that
returning 127.0.0.1 means you're an undesired client and you have the
chance to trap that flag (with the right software, at least)...


On the other side, there's point 3.4:

«
The DNSBL operator MUST issue impending shutdown warnings (on the
DNSBL web site, appropriate mailing lists, newsgroups, vendor
newsletters, etc.), and indicate that the DNSBL is inoperative using
the signaling given in Section 3.3.
[...]

The shutdown procedure should have the following properties:

1. MUST NOT list the entire Internet
»


One could argue they're not shutting down (just listing the Internet
:-D) so the point does not apply, but IMHO the underlying concept is.

After all, they're trying to force the offending customers to change
their configurations, just as during shutdowns...


--
Paranoia is a disease unto itself. And may I add: the person standing
next to you may not be who they appear to be, so take precaution.
-----------------------------------------------------------------------------
http://bofhskull.wordpress.com/

[d***@chaosreigns.com]

As I tried to say in the past, having a value to return for all
queries from a DNS server that has been deemed abusive is *useful* to
black/whitelist providers. Enough that it's looking like it'll be done
whether the ASRG likes it or not. If you'd prefer something other than
127.0.0.1 to be used, document it somewhere.

Also, as the linked article said, "...the 127.0.0.1 response indicates
that uribl.com does not accept any queries from the DNS server".
SpamAssassin had this handled as URIBL defined, no false positives
resulted.

--
"Go forth, and be excellent to one another." - http://www.jhuger.com/fredski.php
http://www.ChaosReigns.com

[David Romerstein]

On 1/24/12 1:23 PM, ***@chaosreigns.com wrote:
> As I tried to say in the past, having a value to return for all
> queries from a DNS server that has been deemed abusive is *useful* to
> black/whitelist providers. Enough that it's looking like it'll be done
> whether the ASRG likes it or not. If you'd prefer something other than
> 127.0.0.1 to be used, document it somewhere.

Following with RFC 6471, would it be possible to do a split zone
(abusive/non-abusive), sending abusive IPs to do their loopups from IP
addresses in TEST-NET RFC-5735 addresses?

-- D

[John Levine]

>Following with RFC 6471, would it be possible to do a split zone
>(abusive/non-abusive), sending abusive IPs to do their loopups from IP
>addresses in TEST-NET RFC-5735 addresses?

In principle, although it's a little tricky since the setup is usually
something like this:

(at TLD)
example.net NS foo.example.net <-- main name server that does split horizon

(at foo.example.net)
dnsbl.example.net NS rbldnsd.example.net <-- the server that's getting hammered

So the server that's getting overloaded has to tell the server above
what lies to tell.

R's,
John

[Emanuele Balla (aka Skull)]

On 1/24/12 7:23 PM, ***@chaosreigns.com wrote:
> As I tried to say in the past, having a value to return for all
> queries from a DNS server that has been deemed abusive is *useful* to
> black/whitelist providers. Enough that it's looking like it'll be done
> whether the ASRG likes it or not. If you'd prefer something other than
> 127.0.0.1 to be used, document it somewhere.

I fully agree with you, FWIW...


> Also, as the linked article said, "...the 127.0.0.1 response indicates
> that uribl.com does not accept any queries from the DNS server".
> SpamAssassin had this handled as URIBL defined, no false positives
> resulted.

Yes, and somehow that's the point: SW (like spamassassin) that deal with
return values correctly, will not encounter FPs but this means it also
gives the BL operator no advantage.

While any other return value outside 127/8, while more opportune,
probably will affect bad implementations like 127.0.0.1 or any other code.


IMHO there's no winner here...

[Derek Diget]

On Jan 24, 2012 at 20:50 +0100, Emanuele Balla (aka Skull) wrote:
=>On 1/24/12 7:23 PM, ***@chaosreigns.com wrote:
=>> As I tried to say in the past, having a value to return for all
=>> queries from a DNS server that has been deemed abusive is *useful* to
=>> black/whitelist providers. Enough that it's looking like it'll be done
=>> whether the ASRG likes it or not. If you'd prefer something other than
=>> 127.0.0.1 to be used, document it somewhere.
=>
=>I fully agree with you, FWIW...
=>
=>
=>> Also, as the linked article said, "...the 127.0.0.1 response indicates
=>> that uribl.com does not accept any queries from the DNS server".
=>> SpamAssassin had this handled as URIBL defined, no false positives
=>> resulted.
=>
=>Yes, and somehow that's the point: SW (like spamassassin) that deal with
=>return values correctly, will not encounter FPs but this means it also
=>gives the BL operator no advantage.
=>
=>While any other return value outside 127/8, while more opportune,
=>probably will affect bad implementations like 127.0.0.1 or any other code.


ASRG List,

See the SpamAssassin bugs ids related to this issue from just a few
weeks ago.

Bug 6724 - DNSxL returning purposefully wrong answers as part of
Anti-Abuse / Free for Some Policies
<https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6724>


Bug 6728 - DNSBLs need a way to turn off queries based on BLOCKED rules
triggering <https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6728>



--
***********************************************************************
Derek Diget Office of Information Technology
Western Michigan University - Kalamazoo Michigan USA - www.wmich.edu/
***********************************************************************

[Chris Lewis]

On 12-01-24 01:23 PM, ***@chaosreigns.com wrote:
> As I tried to say in the past, having a value to return for all
> queries from a DNS server that has been deemed abusive is *useful* to
> black/whitelist providers. Enough that it's looking like it'll be done
> whether the ASRG likes it or not. If you'd prefer something other than
> 127.0.0.1 to be used, document it somewhere.

As I've said more than once, there are two methods of doing it
documented already that are preferable to listing the world:

- return TEST NET DNS server addresses, OR
- return something outside of 127.0.0.0/8

The first one has the advantage of doing something useful EVEN IF the
DNSBL client has never heard of RFC6471.

[David Romerstein]

On 1/24/12 10:07 AM, Martijn Grooten wrote:
> I was just wondering whether the RFC says anything about this kind of behaviour ('listing' everything as a punishment). To my reading it doesn't.

Only thing in the RFC close to this is 3.4, regarding shutting down
gracefully (MUST NOT list the entire Internet).

Listing the world for folks overloading your system is unlikely to have
the effect that you want, and is most likely going to impact folks who
have no say in the configuration of the receiving mail server.

-- D, who was just investigating a spate of URIBL-related blocks
yesterday...

[Dave Warren]

On 1/24/2012 7:28 AM, David Romerstein wrote:
> Listing the world for folks overloading your system is unlikely to
> have the effect that you want, and is most likely going to impact
> folks who have no say in the configuration of the receiving mail server.

It will, however, impact folks who know how to get in touch with whoever
does have a say in the configuration of said mail server.

It's not spectacularly ideal, but lacking in some way to tell the
software to reconfigure itself, it's sometimes the only way to get
attention.

--
Dave Warren, CEO
Hire A Hit Consulting Services
http://ca.linkedin.com/in/davejwarren

[John R. Levine]

>Listing the world for folks overloading your system is unlikely to have
>the effect that you want, and is most likely going to impact folks who
>have no say in the configuration of the receiving mail server.

You may be right, but I have to have some sympathy for BL operators who
are getting bombed by clueless misconfigurations.

Last month the abuse.net lookup stopped working, and after poking
around, I saw that there was an enormous stream of A record queries
from Marc Perkel's spam filtering company. They were clearly stupid,
since you need to do a TXT lookup to find out what the abuse.net
contacts are, and he wasn't doing any of those, apparently under the
misconception that abuse.net is some kind of BL. I've had similar
problems in the past with systems in Brazil that bombed
korea.services.net. Although I agree that it is anti-social to list
the world for every query, I'm less sure about individual query
sources that are mishehaving.

In the case of abuse.net, the A records are all non-zero anyway (the
value is the number of TXT records it'll return, to help debug clients
that have trouble with multiple TXT records), so I experimented for
a while with different return values to try to get his attention,
then finally gave up and put in a packet filter.

If it becomes more of a problem, my main DNS servers can do split horizon
DNS, so it'd probably be more effective to return NS records pointing
to loopback addresses or the like.

R's,
John

[David Romerstein]

On 1/24/12 1:50 PM, John R. Levine wrote:
>> Listing the world for folks overloading your system is unlikely to have
>> the effect that you want, and is most likely going to impact folks who
>> have no say in the configuration of the receiving mail server.
>
> You may be right, but I have to have some sympathy for BL operators who
> are getting bombed by clueless misconfigurations.

I do not, in any way, disagree with this.

I'm just wondering if there's a middle ground that can stop BL operators
from being abused by (willfully or unwillfully) clueless folks without
severely impacting innocent users. Something more than "just sit back
and take all those stupid queries" and less than "return a response code
that could indicate a listing for every one of those stupid queries".

-- D

[Steve Atkins]

On Jan 24, 2012, at 10:59 AM, David Romerstein wrote:

> On 1/24/12 1:50 PM, John R. Levine wrote:
>>> Listing the world for folks overloading your system is unlikely to have
>>> the effect that you want, and is most likely going to impact folks who
>>> have no say in the configuration of the receiving mail server.
>>
>> You may be right, but I have to have some sympathy for BL operators who
>> are getting bombed by clueless misconfigurations.
>
> I do not, in any way, disagree with this.
>
> I'm just wondering if there's a middle ground that can stop BL operators from being abused by (willfully or unwillfully) clueless folks without severely impacting innocent users. Something more than "just sit back and take all those stupid queries" and less than "return a response code that could indicate a listing for every one of those stupid queries".

The innocent users are the people who are abusing the blacklist and those who send email to them. And most of them are using blacklist data for scoring, not for blocking. Listing the world doesn't usually affect them much at all.

There really isn't much of a middle ground. If you have non-broken software querying the blacklist then there aren't any problems - it'll shut down gracefully when the blacklist goes away. But if the software querying the blacklist doesn't do that (and almost everything deployed is broken in that way) then you really only have three options as a blacklist operator:

1. List nothing
2. List everything
3. List things at random

(1) leads to no change, you get to keep fielding bogus DNS requests until the end of time.
(2) causes immediate change at abusers who are using the blacklist to block email, and maximises the chance of someone using the data as part of a scoring based system noticing
(3) is worse than either, as it will potentially cause some mail to be lost but is much less likely to cause people to stop using the list

This isn't a trivial problem, nor a trivial amount of traffic. In the past, I've had service overages of >$2000/mo due to massive DNSBL traffic to cbl.abuseat.com (which isn't a DNSBL, so returns "not listed" for every query).

I'm currently eating quite amazing amounts of misconfigured CBL lookup traffic (hundreds of different ways to misspell cbl.abuseat.org) - and configuring my zones to list everything still doesn't stop the traffic. I've played with long TTL delegations to non-routable addresses and suchlike, and it doesn't have much effect either. At some point I'll probably start running a "stunt" server rather than powerdns for that zone, and be more creative with how I handle the abusive queries.

Cheers,
Steve

[SM]

At 07:07 24-01-2012, Martijn Grooten wrote:
>(Vamsoft ORF is a spam-filter.) Basically uribl.com was returning
>127.0.0.1 to _all_ queries from nameservers that are sending high
>volumes (presumably without paying for it) as some kind of
>punishment. http://uribl.com/ confirms that.

"After investigating this further, it seems the affected ORF users
all use Google public DNS servers for the queries (or use such servers
as forwarders in their local DNS configuration)."

Anyone using open recursive DNS servers or their ISP's DNS server for
DNSBL queries is asking for trouble. The listing is to get the
attention of the sender. It's "antisocial".

Regards,
-sm

[Dave Warren]

On 1/25/2012 10:37 AM, SM wrote:
> At 07:07 24-01-2012, Martijn Grooten wrote:
>> (Vamsoft ORF is a spam-filter.) Basically uribl.com was returning
>> 127.0.0.1 to _all_ queries from nameservers that are sending high
>> volumes (presumably without paying for it) as some kind of
>> punishment. http://uribl.com/ confirms that.
>
> "After investigating this further, it seems the affected ORF users
> all use Google public DNS servers for the queries (or use such servers
> as forwarders in their local DNS configuration)."
>
> Anyone using open recursive DNS servers or their ISP's DNS server for
> DNSBL queries is asking for trouble. The listing is to get the
> attention of the sender. It's "antisocial".

I have to wonder, if a DNSBL were being operated entirely on a free
basis (rather than a freemium model), might it not be better to take
advantage of the large caches out there rather than having thousands of
individual servers performing the same mundane set of lookups individually?

Obviously this negates the DNSBL's ability to try and pull cash out of
the larger entities, but purely from a resource management point of
view, if someone wants to offer a front-line cache for free, surely that
should reduce load.

--
Dave Warren, CEO
Hire A Hit Consulting Services
http://ca.linkedin.com/in/davejwarren

[SM]

At 11:23 25-01-2012, Dave Warren wrote:
>I have to wonder, if a DNSBL were being operated entirely on a free
>basis (rather than a freemium model), might it not be better to take
>advantage of the large caches out there rather than having thousands
>of individual servers performing the same mundane set of lookups individually?

There are no large caches out there. That's not how DNS works in my opinion.

>Obviously this negates the DNSBL's ability to try and pull cash out
>of the larger entities, but purely from a resource management point
>of view, if someone wants to offer a front-line cache for free,
>surely that should reduce load.

It is expensive to provide such a cache for free. Why would someone
want to offer such a service for free?

Regards,
-sm

[Douglas Otis]

On 1/25/12 12:11 PM, SM wrote:
> At 11:23 25-01-2012, Dave Warren wrote:
>> I have to wonder, if a DNSBL were being operated entirely on a free
>> basis (rather than a freemium model), might it not be better to take
>> advantage of the large caches out there rather than having thousands
>> of individual servers performing the same mundane set of lookups
>> individually?
>
> There are no large caches out there. That's not how DNS works in my
> opinion.
Dear SM,

Many providers offer non-authoritative DNS servers. SOPA and PIPA bills
attempt to regulate which domains are to be excluded, and ignore
authoritative DNS traffic not cached by providers. Of course, only
accepting results from authoritative servers would reduce performance.
Infrequently requested records might be dropped prior to expiry to
liberate cache.
>> Obviously this negates the DNSBL's ability to try and pull cash out
>> of the larger entities, but purely from a resource management point
>> of view, if someone wants to offer a front-line cache for free,
>> surely that should reduce load.
>
> It is expensive to provide such a cache for free. Why would someone
> want to offer such a service for free?
Use of these caching services generate valuable information and reduce
network traffic between peers.

Regards,
Doug Otis

[SM]

Hi Doug,
At 13:12 25-01-2012, Douglas Otis wrote:
>Many providers offer non-authoritative DNS servers. SOPA and PIPA
>bills attempt to regulate which domains are to be excluded, and
>ignore authoritative DNS traffic not

I do not have any opinion about SOPA and PIPA bills.

BTW, do you have an opinion about Pippa? :-)

>Use of these caching services generate valuable information and
>reduce network traffic between peers.

Could you elaborate on the "generate valuable information"?

Are there any examples of DNS "caching services" set up to reduce
network traffic and why it is being done?

Regards,
-sm

[Paul Smith]

On 25/01/2012 22:57, SM wrote:
>
>
> Are there any examples of DNS "caching services" set up to reduce
> network traffic and why it is being done?

Hang on - isn't that the whole point of DNS caching? If people didn't
care about network traffic, no one would cache anything, they'd just
resolve the addresses every time. Apart from increased network traffic,
resolving every time would be much better, as there'd be no stale data.

So, pretty much every caching DNS server is there to reduce network
traffic. Isn't it?

[Brendan Hide]

On 01/26/12 01:26, Paul Smith wrote:
> On 25/01/2012 22:57, SM wrote:
>>
>>
>> Are there any examples of DNS "caching services" set up to reduce
>> network traffic and why it is being done?
>
> Hang on - isn't that the whole point of DNS caching? If people didn't
> care about network traffic, no one would cache anything, they'd just
> resolve the addresses every time. Apart from increased network
> traffic, resolving every time would be much better, as there'd be no
> stale data.
>
> So, pretty much every caching DNS server is there to reduce network
> traffic. Isn't it?
Pretty much every ISP has caching DNS servers *partly* for this exact
reason, yes.

The other reason is speed. Cached entries result in less latency when
the authoritative DNS server is on the other side of the world. :)

--
__________
Brendan Hide
Mobile: [phone]
Mobile: ***@swiftspirit.co.za (plain text only please)
Web Africa - Internet Business Solutions
http://www.webafrica.co.za/?AFF1E97

[John Levine]

>So, pretty much every caching DNS server is there to reduce network
>traffic. Isn't it?

Except for the "pretty much" part, yes.

R's,
John

[Steve Atkins]

On Jan 25, 2012, at 3:26 PM, Paul Smith wrote:

> On 25/01/2012 22:57, SM wrote:
>>
>>
>> Are there any examples of DNS "caching services" set up to reduce network traffic and why it is being done?
>
> Hang on - isn't that the whole point of DNS caching? If people didn't care about network traffic, no one would cache anything, they'd just resolve the addresses every time. Apart from increased network traffic, resolving every time would be much better, as there'd be no stale data.
>
> So, pretty much every caching DNS server is there to reduce network traffic. Isn't it?

Partly, yes. It also reduces latency significantly (helped by very high cache hit rates near the root of the tree).

(Queries to DNSBLs and similar trees - e.g. in-addr.arpa - do damage that somewhat, by creating a large number of different queries few of which are reused, hence tending to evict higher value records from the cache. But that's orthogonal to what we're discussing here, really.)

Cheers,
Steve

[Dave Warren]

(I'm a bit late getting back to this, my apologies)

On 1/25/2012 4:08 PM, Steve Atkins wrote:

> (Queries to DNSBLs and similar trees - e.g. in-addr.arpa - do damage that somewhat, by creating a large number of different queries few of which are reused, hence tending to evict higher value records from the cache. But that's orthogonal to what we're discussing here, really.)

And, one might argue, an artifact of poor cache expiry policies. At
least in my version of an ideal world, I'd want to keep records in the
cache based on frequency of use over nearly anything else (within the
TTL lifetime, of course)

However, in the context of DNSBLs, you may well have the same problem as
in-addr.arpa in that there are a lot of records that will have limited
cache re-use. Still, if a DNSBL is overloaded, increasing TTLs and
encouraging (rather than discouraging or prohibiting) use of public
caches would probably decrease load on the DNSBL servers.

For example, with a DNSBL negative-caching at, say, 150 seconds, my
servers check Gmail's outbound IPs for DNSBL listings, on average, every
180 seconds or so. Were I and 10 of my best friends running similarly
sized mail servers to start querying 8.8.8.8 instead of using our own
internal resolvers, a DNSBL might see one hit every 150 seconds instead
of 1 every 18 seconds (10 every 180 seconds).

Now that being said, as a matter of practice I wouldn't suggest we start
suggesting mail server operators start using Google's public DNS as
their primary DNS. However, the reality of it is that the majority of
people hit by "listing the Internet for over-quota usage" policies were
using shared (or public) DNS resolvers, most weren't actually hitting
any sort of limit due to their own traffic.

Obviously if a DNSBL keeps their TTLs (positive and negative) too low
then aggregating queries does little good, and there does need to be
some level of responsiveness. However, if a DNSBL recommends a hourly or
daily rsync for rsync users, that might suggest a starting point for TTLs.

At the end of the day though, it's not about stopping abuse or people
hammering the DNSBLs, but rather, it's about making it more convenient
for larger players to pay money for a valuable service. That's not
really unfair, and the freemium model is always a complicated one with
potential holes for abuse, but it's disingenuous to declare that
listing-the-internet is the only way to cut down query volume.

--
Dave Warren, CEO
Hire A Hit Consulting Services
http://ca.linkedin.com/in/davejwarren

[John Levine]

>It is expensive to provide such a cache for free. Why would someone
>want to offer such a service for free?

Perhaps you might ask Google why they run 8.8.8.8 and 8.8.4.4.

I run a free DNSBL, korea.services.net. The public caches have
never been a problem. The people I have to block hammer on it
directly from their own clients. The query streams suggest that
they aren't caches, since they tend to ask the same questions
over and over.

R's,
John

[SM]

At 16:02 25-01-2012, John Levine wrote:
>Perhaps you might ask Google why they run 8.8.8.8 and 8.8.4.4.

Vertical integration? :-)

Regards,
-sm

[Douglas Otis]

On 1/25/12 4:33 PM, SM wrote:
> At 16:02 25-01-2012, John Levine wrote:
>> Perhaps you might ask Google why they run 8.8.8.8 and 8.8.4.4.
>
> Vertical integration? :-)
Dear SM,

To support touted performance feature, chrome aggressively resolves
links in advance. This strategy makes chrome sensitive to DNS
performance. Cache in their recursive resolvers is kept current ahead
of requests. People addicted to speed. ;^)

Regards,
Doug Otis

[Dave Warren]

On 1/26/2012 10:27 AM, Douglas Otis wrote:
> To support touted performance feature, chrome aggressively resolves
> links in advance. This strategy makes chrome sensitive to DNS
> performance. Cache in their recursive resolvers is kept current ahead
> of requests. People addicted to speed. ;^)

I might be being argumentative for it's own sake (sorry), but doesn't
this make Chrome less sensitive to DNS performance rather than more
sensitive?

With a browser that doesn't pre-cache DNS lookups, the user is aware of
DNS latency every time they click a link and with every resource that
the browser loads from a different domain. Conversely, with caching, in
most cases pre-caching every link on a page will take longer than it
takes the user to find a link and click on it even if the DNS cache is
extremely slow.

Now I'd agree that faster DNS servers makes a noticeable difference when
browsing since many websites load content from a dozen or more hostname,
but I'd argue that Chrome's precaching makes it less sensitive to slow
DNS queries.

--
Dave Warren, CEO
Hire A Hit Consulting Services
http://ca.linkedin.com/in/davejwarren

[Douglas Otis]

On 1/27/12 4:51 PM, Dave Warren wrote:
> On 1/26/2012 10:27 AM, Douglas Otis wrote:
>> To support touted performance feature, chrome aggressively resolves
>> links in advance. This strategy makes chrome sensitive to DNS
>> performance. Cache in their recursive resolvers is kept current
>> ahead of requests. People addicted to speed. ;^)
>
> I might be being argumentative for it's own sake (sorry), but doesn't
> this make Chrome less sensitive to DNS performance rather than more
> sensitive?
>
> With a browser that doesn't pre-cache DNS lookups, the user is aware
> of DNS latency every time they click a link and with every resource
> that the browser loads from a different domain. Conversely, with
> caching, in most cases pre-caching every link on a page will take
> longer than it takes the user to find a link and click on it even if
> the DNS cache is extremely slow.
>
> Now I'd agree that faster DNS servers makes a noticeable difference
> when browsing since many websites load content from a dozen or more
> hostname, but I'd argue that Chrome's precaching makes it less
> sensitive to slow DNS queries.
Dear Dave,

When DNS transactions attempt to include all possible choices, necessary
transactions are at a greater risk of being queued, rather than being
ready in advance. Chrome offers a switch setting to disable this
behavior that may actually result in reduced performance.

Regards,
Doug Otis