Some statistics on SPF and spam

[Martijn Grooten]

I had promised to produce some stats on SPF and spam.

Over the Christmas holidays, I sent over 60k spam messages through 21 spam filters in the spam-filter test I run regularly. I checked the SPF status of the messages and measured how many filters failed to block each message.

Here are the results:
SPF fail: 3171 emails, on average missed by 0.24 filters (out of 21) with a standard deviation of 0.04.
SPF pass: 8106 emails, avg 0.93, stddev 0.23
SPF softfail: 8672 emails, avg 0.45, stddev 0.09
SPF neutral: 13466 emails, avg 0.34, stddev 0.04
SPF none: 26938 emails, avg 0.43, stddev 0.06

A neater table and a graph can be found here: http://www.virusbtn.com/news/2013/02_04.xml

Now correlation doesn't imply causation and there are good reasons why the relationship here may not causal, but let's for a moment we assume it is.

This means that if you're a spammer, failing SPF isn't a good idea, while making sure your emails pass SPF means you're more likely to see your messages delivered, but you by no means get a free ride to users' inboxes.

If you find a 'clever' way to avoid failing SPF by using a domain with no SPF record, there is only a small improvement in your delivery rates.

Martijn.


________________________________

Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.

[d***@chaosreigns.com]

I didn't see the discussion where you promised to produce this, but I think
the problem is how much non-spam also fails SPF.
MSECS SPAM% HAM% S/O RANK SCORE NAME WHO/AGE
0 0.0236 0.9635 0.024 0.15 0.00 SPF_FAIL
0 0.0383 0.3059 0.111 0.27 0.00 SPF_SOFTFAIL

Way more non-spam is failing than spam.

Catching spam is easy. Doing so without excessive false positives is
what's hard.

Post by Martijn Grooten
I had promised to produce some stats on SPF and spam.
Over the Christmas holidays, I sent over 60k spam messages through 21 spam filters in the spam-filter test I run regularly. I checked the SPF status of the messages and measured how many filters failed to block each message.
SPF fail: 3171 emails, on average missed by 0.24 filters (out of 21) with a standard deviation of 0.04.
SPF pass: 8106 emails, avg 0.93, stddev 0.23
SPF softfail: 8672 emails, avg 0.45, stddev 0.09
SPF neutral: 13466 emails, avg 0.34, stddev 0.04
SPF none: 26938 emails, avg 0.43, stddev 0.06
A neater table and a graph can be found here: http://www.virusbtn.com/news/2013/02_04.xml
Now correlation doesn't imply causation and there are good reasons why the relationship here may not causal, but let's for a moment we assume it is.
This means that if you're a spammer, failing SPF isn't a good idea, while making sure your emails pass SPF means you're more likely to see your messages delivered, but you by no means get a free ride to users' inboxes.
If you find a 'clever' way to avoid failing SPF by using a domain with no SPF record, there is only a small improvement in your delivery rates.
Martijn.
________________________________
Virus Bulletin Ltd, The Pentagon, Abingdon, OX14 3YP, England.
Company Reg No: 2388295. VAT Reg No: GB 532 5598 33.
_______________________________________________
Asrg mailing list
http://www.irtf.org/mailman/listinfo/asrg

[Dave Warren]

Post by d***@chaosreigns.com
I didn't see the discussion where you promised to produce this, but I think
the problem is how much non-spam also fails SPF.
MSECS SPAM% HAM% S/O RANK SCORE NAME WHO/AGE
0 0.0236 0.9635 0.024 0.15 0.00 SPF_FAIL
0 0.0383 0.3059 0.111 0.27 0.00 SPF_SOFTFAIL
Way more non-spam is failing than spam.

If you just use SPF for positive scoring and never for negative scoring
or blocking then that's okay.

I'm also note suggesting that SPF or DKIM or similar alone is sufficient
for positive scoring, but when combined with a local whitelist, I can
aggressively whitelist companies that we do business with without having
to worry about a spammer spoofing a whitelisted major corporation.

When the company starts sending mail from a non-listed IP, they don't
get the benefit of whitelisting, but nothing else "breaks", so there's
no harm done.

Post by d***@chaosreigns.com
Catching spam is easy. Doing so without excessive false positives is
what's hard.

Amen.

I guarantee you that I can block every single spam, 100% of the time, no
questions asked, as long as one of the unasked questions is the false
positive percentage.