spam down?

[Michael Thomas]

There was a little side box in the current Economist that spam was
down from 80+% to 67% and credited it to, among other things
"sophisticated authentication" which I assume means DKIM and SPF.

First is there actual evidence that spam is on the wane? And if so,
does it actually have to due in part with authentication? I'd be
ecstatic to hear that the latter was true, but correlation is not
causation.

Mike

[Martijn Grooten]

Post by Michael Thomas
First is there actual evidence that spam is on the wane?

I believe there is. Measuring spam is tricky, and different sources may use different definitions and methods, but all agree that the global volume of spam has declined over the past four years.

Note that a reduction from 80% to 67% would mean the volume of spam has halved, rather than reduced by 13%. I believe most sources claim that reduction, since late 2008 (the McColo shutdown), is even bigger than that.

Post by Michael Thomas
And if so,
does it actually have to due in part with authentication? I'd be
ecstatic to hear that the latter was true, but correlation is not
causation.

I think it has little to do with that, but that it's mostly because botnets are able to spew out a lot less than they used to. The graph used here shows a strong correlation between drops in the volume of spam and big takedowns:

http://krebsonsecurity.com/2013/01/spam-volumes-past-present-global-local/

It's good to keep in mind that the spam "that's not being sent anymore" was relatively easy to block. See for instance:

http://www.lightbluetouchpaper.org/2009/07/17/how-much-did-shutting-down-mccolo-help/

Martijn.

________________________________

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

[Chris Lewis]

Post by Michael Thomas
There was a little side box in the current Economist that spam was
down from 80+% to 67% and credited it to, among other things
"sophisticated authentication" which I assume means DKIM and SPF.
First is there actual evidence that spam is on the wane? And if so,
does it actually have to due in part with authentication? I'd be
ecstatic to hear that the latter was true, but correlation is not
causation.

In the wane ... how? Is the real question.

Absolute volumes have indeed changed, as this graph (and many others) show:

http://cbl.abuseat.org/totalflow.html

but that doesn't tell the whole story.

The reality is that authentication (we're talking DKIM/SPF/DMARC) has
relatively little effect. They're pretty easy to make irrelevant.

There are fewer bot families than there used to be. Bot takedowns have
made major inroads. Still, there are a couple left that can dwarf what
we've seen before _if_ it was attractive to fire them off. Kelihos and
Festi are bigger than Rustock or Srizbi ever were. The defenses we have
for bots are well-developed and widely-deployed. The ROI has declined
markedly, so the bot armies are often left idle.

What we're seeing instead, is an evolution from the massive
scatter-gunning of a Rustock infecting a home computer, to that of
compromised servers, compromised user accounts etc. These are harder to
deal with, harder to stop, harder to filter.

So, while there are fewer spams in the Internet, I strongly suspect that
more of them are getting through.

Spammers may not be spamming as much but they are spamming "better".

[Rob McEwen]

Post by Chris Lewis
What we're seeing instead, is an evolution from the massive
scatter-gunning of a Rustock infecting a home computer, to that of
compromised servers, compromised user accounts etc. These are harder to
deal with, harder to stop, harder to filter.
So, while there are fewer spams in the Internet, I strongly suspect that
more of them are getting through.

EXACTLY!!! Along those lines, there has been an uptick in hijacked
domains where, instead of the spammer buying their own domain, they
break through a hoster's security (or obtained the FTP credentials), and
then they install their spammy scripts or pages. Then, when they send
out their spams, the domains are not so easily blacklist-able because
the various URI or domain blacklists often skip listing these due to the
false-positive-prevention-filters preventing such listings. In other
words, the same legitimacy or "good reputation" which would cause a URI
blacklist's engine to purposely NOT blacklist innocent decoy domains...
often give these hijacked domains a free pass, too.

Therefore, over at invaluement.com, we made recent improvements to our
ivmURI blacklist to allow us to now more surgically target many of these
hijacked domains, yet without lessening our protections against
blacklisting innocent "decoy" domains.

FOR EXAMPLE... The following is a list of about 2,500 domains which are
CURRENTLY hijacked with "live" spammy URLs present:

http://dnsbl.invaluement.com/urls-hijacked-by-spammers-Jan-29-2013.zip

Actually, the number of such hijacked domains blacklisted by invaluement
is much larger, but we narrowed it down in THAT example list to only
those domains NOT currently blacklisted by either SURBL or Spamhaus's
DBL list... to make it more interesting! See the included "notes" text
file for more details.

PS - as the notes file mentions, please don't throw these into manual
local blacklists since many of these sites will fix their problems and
then get removed from ivmURI. These generally shouldn't be permanently
blacklisted. Again, see the included "notes" file for more information.

[Dotzero]

I'm late to the party but as opinionated as ever....

Post by Chris Lewis

Post by Michael Thomas
There was a little side box in the current Economist that spam was
down from 80+% to 67% and credited it to, among other things
"sophisticated authentication" which I assume means DKIM and SPF.
First is there actual evidence that spam is on the wane? And if so,
does it actually have to due in part with authentication? I'd be
ecstatic to hear that the latter was true, but correlation is not
causation.

In the wane ... how? Is the real question.
http://cbl.abuseat.org/totalflow.html
but that doesn't tell the whole story.

Agreed

Post by Chris Lewis
The reality is that authentication (we're talking DKIM/SPF/DMARC) has
relatively little effect. They're pretty easy to make irrelevant.

I think it depends on what you mean by "relatively little effect".
From my perspective - given the current statof adoption - it may not
have an effect on the overall ecosystem but it is certainly pushing
the bad guys from abusing (sending) domains that are implementing
strong email auth efforts to ones that are not. My comment is a
generalization but I see it with the domains I work with and I think
those who watch abuse against financials see similar behavior. The bad
guys still test but at the end of the day it is about ROI for them as
much as it is for a legitimate business.

It would be interesting to see (I don't have the data) if there is any
kind of shift from sending spam targeting accounts at mailbox
providers that validate to targeting (preferentially) accounts at
mailbox providers that don't.

Post by Chris Lewis
There are fewer bot families than there used to be. Bot takedowns have
made major inroads. Still, there are a couple left that can dwarf what
we've seen before _if_ it was attractive to fire them off. Kelihos and
Festi are bigger than Rustock or Srizbi ever were. The defenses we have
for bots are well-developed and widely-deployed. The ROI has declined
markedly, so the bot armies are often left idle.

True. It may also be true that the bot armies are being put to other uses.

Post by Chris Lewis
What we're seeing instead, is an evolution from the massive
scatter-gunning of a Rustock infecting a home computer, to that of
compromised servers, compromised user accounts etc. These are harder to
deal with, harder to stop, harder to filter.

"We" should certainly be blocking on malicious URLs even if they are
at otherwise legitimate sites. And if legitimate sites show a pattern
of not addressing their problems then they should be blocked as well.
This is no different than the open relay problem. I've had my share of
issues over the years but I think most folks would say that I pay
attention and deal with problems expeditiously.

Post by Chris Lewis
So, while there are fewer spams in the Internet, I strongly suspect that
more of them are getting through.

I think it varies by mailbox provider.

Post by Chris Lewis
Spammers may not be spamming as much but they are spamming "better".

Darwin was right.

[Chris Lewis]

Post by Dotzero
I think it depends on what you mean by "relatively little effect".

Post by Dotzero
From my perspective - given the current statof adoption - it may not

have an effect on the overall ecosystem but it is certainly pushing
the bad guys from abusing (sending) domains that are implementing
strong email auth efforts to ones that are not.

If that were true, I wouldn't be seeing millions of paypal, linkedin,
et. al. impersonations a day. But I do.

Validation is so irrelevant that the spammers impersonate sites when
it's clearly unnecessary. They use their facebook impersonation
templates to send out pill spam for crissakes. If validation was making
a difference, the ROI would suffer. I can only guess it isn't.

The reality is that you don't have to forge the From/sender/helo et. al.
to successfully impersonate any domain. Especially with the mail
readers oh-so-carefully _not_ showing you the actual email address.

Post by Dotzero
It would be interesting to see (I don't have the data) if there is any
kind of shift from sending spam targeting accounts at mailbox
providers that validate to targeting (preferentially) accounts at
mailbox providers that don't.

Most spoofers are already bypassing validation. So why would it matter
to them whether the mailbox provider is validating or not?

[Dotzero]

Post by Chris Lewis

Post by Dotzero
I think it depends on what you mean by "relatively little effect".

Post by Dotzero
From my perspective - given the current statof adoption - it may not

have an effect on the overall ecosystem but it is certainly pushing
the bad guys from abusing (sending) domains that are implementing
strong email auth efforts to ones that are not.

If that were true, I wouldn't be seeing millions of paypal, linkedin,
et. al. impersonations a day. But I do.

No difference in the nature/behavior in the spamming? I can't speak
for other brands in terms of the effect of email
authentication/validation but I have seen it make a difference for our
brands/domains. I know anecdotally that other brands have said the
same. This is why I wrote that "it depends on what you mean by
relatively little effect". I think we all know that spam/phishing is
all about the social engineer. That means there will be some amount of
friction pushing bad folks away from high value targets they wish to
leverage. When I've looked in the past I've seen differentiation in
abuse comparing financials that are aggressive in fighting abuse vs
those that are less clueful.

Post by Chris Lewis
Validation is so irrelevant that the spammers impersonate sites when
it's clearly unnecessary. They use their facebook impersonation
templates to send out pill spam for crissakes. If validation was making
a difference, the ROI would suffer. I can only guess it isn't.
The reality is that you don't have to forge the From/sender/helo et. al.
to successfully impersonate any domain. Especially with the mail
readers oh-so-carefully _not_ showing you the actual email address.

You are assuming that the place of email auth is at the MUA and let
the recipient figure it out. That IS an epic fail. And I agree with
you that showiing the display name and hiding the email address is
suboptimal.

Post by Chris Lewis

Post by Dotzero
It would be interesting to see (I don't have the data) if there is any
kind of shift from sending spam targeting accounts at mailbox
providers that validate to targeting (preferentially) accounts at
mailbox providers that don't.

Most spoofers are already bypassing validation. So why would it matter
to them whether the mailbox provider is validating or not?

So I take it you aren't a fan of email authentication at all. I think
we'll have to agree to disagree.

Mike

[Alessandro Vesely]

Post by Chris Lewis
Validation is so irrelevant that the spammers impersonate sites when
it's clearly unnecessary. They use their facebook impersonation
templates to send out pill spam for crissakes. If validation was making
a difference, the ROI would suffer. I can only guess it isn't.

That seems to be true, sadly.

Looking at myself, the reason why I don't use authentication results
is because so few messages bear any. If they covered a significant
percentage of messages (and spam), I'd use the data in those A-R
header fields to send complaints, implementing RFC 6650, according to
the discussions we already had.

A way to enlarge the covered base is to add more authentication
methods. For example, "dnswl" would be quite similar to "vbr",
"iprev" is already defined, "rdap" is a promising new feature. By
adding those, I'd expect it will become fairly difficult to find
messages with no authentication at all. Correct?

[Barry Shein]

It depends a lot on how and particularly where spam is measured.

Most naive measurements, I'd say 99% of what we hear about, only care
about what hits an end-user's screen.

By that time it's had to run a gauntlet of blocks at sources, blocks
at ISPs, and whatever spam filters are in the user's MUA.

So the report of reduction could just as well mean that any one, or
all, of those are improving.

Or of course that less spam is being sent out.

Those of us more involved in the infrastructure of the internet tend
to be at least as concerned with the volume of attempted deliveries of
spam. The gates may hold back the barbarians, but you still have
barbarians at the gates!

Perhaps that leads to some useful taxonomy since the model is pretty
straightforward.

Volume of spam sourced, volume of spam which leaves the network it's
sourced from, volume received at providers, volume delivered to a
mailbox, and volume seen by an end-user.

Those are all different measurements and each interesting in its own
way.