Discussion:
FTC tells us what is SPAM (NO! what is commercial!)
(too old to reply)
Amir Herzberg
2004-12-19 08:52:51 UTC
Permalink
>>FTC Issues Final Rule Defining What Constitutes a "Commercial Electronic
>>Mail Message"

> This has vanishingly little to do with defining what spam is.

That's correct, and indeed I think the rule does not claim to define
what is spam. Instead, it defines several classes of often-undesirable
e-mail (commercial, etc.).

If e-mail belonging to such often-undesirable classes would have
included an appropriate `warning` category label, this could provide
very good mechanism for users and providers to filter out undesirable
messages (based on user's and/or providers policies).

One possible (and imho very useful) definition of spam is `e-mail
belonging ot one of the often-undesirable categories, but without an
appropriate label`. The FCC rule may help define these categories, and
technical mechanisms, and potentially complementary legal measures, can
enforce appropriate labeling. See
http://www.cs.biu.ac.il/~herzbea//Papers/ecommerce/Spam.htm for a simple
cryptographic protocol to enforce correct labeling for content
selection (filtering).

Best, Amir Herzberg


They're
> defining what messages are subject to the CAN SPAM act, which
> regulates all commercial e-mail, solicited and unsolicited.
>
> I don't think much of the rules that CAN SPAM defines, but don't shoot
> the messenger. The Congress directed the FTC to regulate some kinds
> of email, and the FTC is clarifying what mail that is.
>
> Regards,
> John Levine, ***@taugh.com, Taughannock Networks, Trumansburg NY
> http://www.taugh.com
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 16 Dec 2004 13:13:29 -0800 (PST)
> From: William Leibzon <***@completewhois.com>
> Subject: Re: [Asrg] FTC tells us what is SPAM
> To: John Levine <***@johnlevine.com>
> Cc: ***@ietf.org
> Message-ID:
> <***@cwhois1.completewhois.com>
> Content-Type: text/plain; charset="us-ascii"
>
> On Thu, 16 Dec 2004, John Levine wrote:
>
>
>>>FTC Issues Final Rule Defining What Constitutes a "Commercial Electronic
>>>Mail Message"
>>>
>>> Notice Includes Criteria For Determining the "Primary Purpose" of an
>>> E-Mail Message
>>
>>This has vanishingly little to do with defining what spam is. They're
>>defining what messages are subject to the CAN SPAM act, which
>>regulates all commercial e-mail, solicited and unsolicited.
>>
>>I don't think much of the rules that CAN SPAM defines, but don't shoot
>>the messenger. The Congress directed the FTC to regulate some kinds
>>of email, and the FTC is clarifying what mail that is.
>
>
> I agree. I've very little faith in the "You can spam all you want" act by
> congress. It seems to have allowed large spammers to tell ISPs that they
> are not doing anything illegal because they are CAN-SPAM compliant (which
> might or might not be true, but verifying takes long time) and if ISPs do
> threated a disconnect then they begin to use lawyers to keep them up longer.
>
> Nevertheless that Congress clarified what consititutes a COMMERCIAL EMAIL
> is very important because that directly related to UCE type of spam and
> commercial part is very often what makes a difference if deciding how to
> react to bulk sender on your net.
>
> ---
> William Leibzon
> mailto: ***@completewhois.com
> Anti-Spam and Email Security Research Worksite:
> http://www.elan.net/~william/emailsecurity/
> Whois & DNS Network Investigation Tools:
> http://www.completewhois.com
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: smime.p7s
> Type: application/x-pkcs7-signature
> Size: 1118 bytes
> Desc: S/MIME Cryptographic Signature
> Url : http://www1.ietf.org/pipermail/asrg/attachments/20041216/8c673d47/smime-0001.bin
>
> ------------------------------
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg
>
>
> End of Asrg Digest, Vol 8, Issue 14
> ***********************************
>
> .
>
Matthew Elvey
2004-12-20 06:48:27 UTC
Permalink
>
> One possible (and imho very useful) definition of spam is `e-mail
> belonging ot one of the often-undesirable categories, but without an
> appropriate label`. The FCC rule may help define these categories, and
> technical mechanisms, and potentially complementary legal measures,
> can enforce appropriate labeling. See
> http://www.cs.biu.ac.il/~herzbea//Papers/ecommerce/Spam.htm for a
> simple cryptographic protocol to enforce correct labeling for content
> selection (filtering).

Proposals in the form of I-Ds (produced using xml2rfc, perhaps) are
encouraged; please consider providing one.

The original RBL started out as a list of IPs from which to shun ALL
traffic. I think it was a mistake (given hindsight) to have switched to
just shunning email traffic from IPs that were the source of abuse. As
we solve the email spam problem, spammers switch to other media (spim,
wikispam,...) so we should be prepared.
John Levine
2004-12-20 18:13:36 UTC
Permalink
>> One possible (and imho very useful) definition of spam ...

Please keep in mind that attempts to create a definition of spam are
specifically excluded from the ASRG's area of interest.

Taxonomies of objectionable mail might be interesting, though.

Regards,
John Levine, ***@taugh.com, Taughannock Networks, Trumansburg NY
http://www.taugh.com
Laird Breyer
2004-12-21 00:52:59 UTC
Permalink
On Dec 20 2004, John Levine wrote:
>
> Please keep in mind that attempts to create a definition of spam are
> specifically excluded from the ASRG's area of interest.

This is intriguing. While it isn't clear that having a specific
definition of spam is actually useful for blocking it, this policy does
have implications on the solution set.

Would it be accurate to say that ASRG is interested in Z-mail
prevention solutions where Z-mail is a label for some set of messages,
and Z-mail can be taken to be spam?

If that's the case, then any solution of interest would also apply to
other labeling problems, e.g. ham-prevention. Furthermore, such a
restriction on the solution set would likely exclude some economic
solutions, as these solutions don't usually make sense for
ham-prevention say, and any solution the ASRG would be interested in
would have to work for Z-mail, whatever Z-mail turns out to be.

What am I missing here?

--
Laird Breyer.
John Levine
2004-12-21 02:28:14 UTC
Permalink
>Would it be accurate to say that ASRG is interested in Z-mail
>prevention solutions where Z-mail is a label for some set of
>messages, and Z-mail can be taken to be spam?

Not really. That still implies that there's a fixed set of mail
that ASRG is trying to stamp out. I presume you have looked at
the charter at http://www.irtf.org/charters/asrg.html.

My position is that any anti-spam project has to identify the flavor
of spam that the project is attempting to address, but I don't expect
projects all to address the same thing.

Pluralistically,
John
Laird Breyer
2004-12-21 05:15:51 UTC
Permalink
On Dec 21 2004, John Levine wrote:
> >Would it be accurate to say that ASRG is interested in Z-mail
> >prevention solutions where Z-mail is a label for some set of
> >messages, and Z-mail can be taken to be spam?
>
> Not really. That still implies that there's a fixed set of mail
> that ASRG is trying to stamp out. I presume you have looked at
> the charter at http://www.irtf.org/charters/asrg.html.
>

The charter does emphasise that the ASRG is looking for well
specified problems, and makes it clear that a multitude of
techniques to tackle them is desirable. As such, doesn't that
encourage attempts to define spam?

The original message you responded to suggested a possible
definition, followed by a link to some protocol which takes
that definition as a basis. I guess I don't see what prompted
you to warn against attempting to define spam. Could you explain?

> My position is that any anti-spam project has to identify the flavor
> of spam that the project is attempting to address, but I don't expect
> projects all to address the same thing.

I agree with that position. Perhaps I read too much into your comment.

--
Laird Breyer.
Bill Cole
2004-12-21 13:45:19 UTC
Permalink
At 3:15 PM +1000 12/21/04, Laird Breyer wrote:
>On Dec 21 2004, John Levine wrote:
>> >Would it be accurate to say that ASRG is interested in Z-mail
>> >prevention solutions where Z-mail is a label for some set of
>> >messages, and Z-mail can be taken to be spam?
>>
>> Not really. That still implies that there's a fixed set of mail
>> that ASRG is trying to stamp out. I presume you have looked at
>> the charter at http://www.irtf.org/charters/asrg.html.
>>
>
>The charter does emphasise that the ASRG is looking for well
>specified problems, and makes it clear that a multitude of
>techniques to tackle them is desirable. As such, doesn't that
>encourage attempts to define spam?

Probably, which is why it needs to be expressly excluded.

This is a relatively young forum for discussions about spam. One
lesson from other places is that there is no point in defining what
is or is not spam because even if there were to be an agreement of
some sort to settle on some useful definition, there will be some
people who want to shun other sorts of mail or some people who think
some subset of spam should not be shunned or both. If one defines
'spam' in some way that is technically discernable with any sort of
useful reliability, so much unsolicited bulk email will be excluded
that practically no one will take that definition seriously. In the
end, debates over what is or is not spam always devolve into
scholastic hair-splitting over things like the nature of permission
and the meaning of 'bulk' or 'broadcast' and so on.

It seems less hopeless to approach the issue of controlling spam in
the same sort of ways that have already proven effective in the
field, and attack subsets of spam that lend themselves to
programmatic detection, in order to narrow the ranges of behavior
that are useful for spammers. A great example is intentional
selective slowing of mail, either by extended response delays in SMTP
or explicit 4xx errors for a 'first try' of a particular piece of
mail. Such tactics do not impede things like the Verisign spam I got
last week, because Verisign hires a class of spammer that tries to
operate above-ground and with some attention to the norms of SMTP and
have their own facilities for sending mail. It DOES impede the
Goodyear and Kraft spam, because those fine upstanding companies
contract with spammers who act as if SMTP runs on a one-way channel.
Simply requiring all SMTP clients to implement certain bits of SMTP
sanely stops a lot of spam, but it doesn't stop all spam and once in
a while it stops mail which no one would call spam. It is absurd to
define spam as mail which is identified by some set of technical
tricks, but existing practice makes it pretty clear that one can do a
fairly effective and precise job of controlling spam by selecting the
right set of technical tricks.



--
Bill Cole
***@scconsult.com
Walter Dnes
2004-12-21 04:46:52 UTC
Permalink
On Tue, Dec 21, 2004 at 10:52:59AM +1000, Laird Breyer wrote

> This is intriguing. While it isn't clear that having a specific
> definition of spam is actually useful for blocking it, this policy does
> have implications on the solution set.
>
> Would it be accurate to say that ASRG is interested in Z-mail
> prevention solutions where Z-mail is a label for some set of messages,
> and Z-mail can be taken to be spam?
>
> If that's the case, then any solution of interest would also apply to
> other labeling problems, e.g. ham-prevention. Furthermore, such a
> restriction on the solution set would likely exclude some economic
> solutions, as these solutions don't usually make sense for
> ham-prevention say, and any solution the ASRG would be interested in
> would have to work for Z-mail, whatever Z-mail turns out to be.
>
> What am I missing here?

The thing that you may be missing is that "one size does not fit all".
One man's spam is another man's ham. My idea of spam is different from
someone else's. That's why end-user-control of some sort is so
desirable, so that end-users could stop (block/reject/filter/whatever)
the email that *THEY* don't want. Rather than "blocking spam", the
problem is better stated as "blocking unwanted email". Each person has
their own set of wants.

There is another, legal, problem with "blocking spam". The word
"spam" has a lot of negative baggage and spammers^H^H^H^H^H^H^H^H email
marketers with lawyers will hit back with lawsuits claiming "It's not
spam, it's ooga-booga" (substitute euphemism-du-jour for spam). A good
parallel is the current spyware controversy, where Claria has sued anti-
spyware program publisherss, claiming to have been libelled/slandered by
being labelled as "spyware" rather than "adware". Their lawsuit does
not currently affect "adware removal" programs. A similar potential
problem exists in the spam arena. Rather than a "spam-blocking" program
labelling something as spam and blocking it, we should have an
"unwanted-email-blocking program". When the spammer's lawyer screams
"It's not spam, it's ooga-booga. Stop blocking my client's emails with
your 'spam-blocking' program", we should respond that it's really an
'unwanted-email-blocking-program'. And that our customer has indicated
that he doesn't want your email.

--
Walter Dnes <***@waltdnes.org>
An infinite number of monkeys pounding away on keyboards will
eventually produce a report showing that Windows is more secure,
and has a lower TCO, than linux.
Laird Breyer
2004-12-21 05:37:54 UTC
Permalink
On Dec 20 2004, Walter Dnes wrote:
> On Tue, Dec 21, 2004 at 10:52:59AM +1000, Laird Breyer wrote
>
> > This is intriguing. While it isn't clear that having a specific
> > definition of spam is actually useful for blocking it, this policy does
> > have implications on the solution set.
> >
> > Would it be accurate to say that ASRG is interested in Z-mail
> > prevention solutions where Z-mail is a label for some set of messages,
> > and Z-mail can be taken to be spam?
> >
> > If that's the case, then any solution of interest would also apply to
> > other labeling problems, e.g. ham-prevention. Furthermore, such a
> > restriction on the solution set would likely exclude some economic
> > solutions, as these solutions don't usually make sense for
> > ham-prevention say, and any solution the ASRG would be interested in
> > would have to work for Z-mail, whatever Z-mail turns out to be.
> >
> > What am I missing here?
>
> The thing that you may be missing is that "one size does not fit all".
> One man's spam is another man's ham. My idea of spam is different from
> someone else's. That's why end-user-control of some sort is so
> desirable, so that end-users could stop (block/reject/filter/whatever)
> the email that *THEY* don't want. Rather than "blocking spam", the
> problem is better stated as "blocking unwanted email". Each person has
> their own set of wants.

Not necessarily. Some people do not have the luxury of defining spam
as that which they don't want. For example, children whose mail is
monitored, or employees whose mail is filtered centrally.

There are some classes of solutions which don't allow end-user
control. For example, in proposals where postage or a bond is
required for sending mail, the recipient is only indirectly involved
with the definition of spam. Instead, such a user can indicate a price
for accepting mail, but if the sender is rich or at least has access
to some other source of funding, then the recipient must accept the
spam together with some form of compensation.

--
Laird Breyer.
Barry Shein
2004-12-21 22:35:52 UTC
Permalink
The idea of complete end-user control is a utopian ideal which died a
few years ago when spammers' zombie bots began hitting ISPs by the
thousands simultaneously.

At this point there are only two choices, either block at the ingress
(e.g., blocking network blocks and similar), or begin charging around
$1000/month for e-mail accounts to pay for the bandwidth and hardware
necessary to keep up with the unfettered flow (essentially, a
dedicated server with a dedicated T1 per mailbox.)

Anyone who says otherwise, notably EFF, is just ignorant.

--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Seth Breidbart
2004-12-21 23:21:07 UTC
Permalink
Barry Shein <***@world.std.com> wrote:

> The idea of complete end-user control is a utopian ideal which died
> a few years ago when spammers' zombie bots began hitting ISPs by the
> thousands simultaneously.
>
> At this point there are only two choices, either block at the
> ingress (e.g., blocking network blocks and similar), or begin
> charging around $1000/month for e-mail accounts to pay for the
> bandwidth and hardware necessary to keep up with the unfettered flow
> (essentially, a dedicated server with a dedicated T1 per mailbox.)
>
> Anyone who says otherwise, notably EFF, is just ignorant.

What about counterexamples, such as Panix?

Seth
Barry Shein
2004-12-22 01:59:47 UTC
Permalink
I have no idea what panix does.

On December 21, 2004 at 18:21 ***@panix.com (Seth Breidbart) wrote:
> Barry Shein <***@world.std.com> wrote:
>
> > The idea of complete end-user control is a utopian ideal which died
> > a few years ago when spammers' zombie bots began hitting ISPs by the
> > thousands simultaneously.
> >
> > At this point there are only two choices, either block at the
> > ingress (e.g., blocking network blocks and similar), or begin
> > charging around $1000/month for e-mail accounts to pay for the
> > bandwidth and hardware necessary to keep up with the unfettered flow
> > (essentially, a dedicated server with a dedicated T1 per mailbox.)
> >
> > Anyone who says otherwise, notably EFF, is just ignorant.
>
> What about counterexamples, such as Panix?
>
> Seth
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg
Seth Breidbart
2004-12-22 02:09:41 UTC
Permalink
Barry Shein <***@world.std.com> top-posted:

> I have no idea what panix does.

In response to your claim that ISPs have to either do ingress blocking
or charge $1000/month, I offered panix as a counterexample. Ingress
blocking is a user option. Everything past it is, too; stuff is made
available to make things like spamassassin easy to configure, but it's
all up to the user to decide what to do. Email accounts, with shell,
Usenet, etc. cost $100/year, two orders of magnitude below your claim.

Seth
Barry Shein
2004-12-22 02:23:56 UTC
Permalink
So you're saying panix doesn't block any, e.g., broadband IP pools or
similar, at a system level?

Do you work at panix or otherwise have authoritative info on this? I'm
not asking that to be baiting, but oftentimes it's not the most public
information.

On December 21, 2004 at 21:09 ***@panix.com (Seth Breidbart) wrote:
> Barry Shein <***@world.std.com> top-posted:
>
> > I have no idea what panix does.
>
> In response to your claim that ISPs have to either do ingress blocking
> or charge $1000/month, I offered panix as a counterexample. Ingress
> blocking is a user option. Everything past it is, too; stuff is made
> available to make things like spamassassin easy to configure, but it's
> all up to the user to decide what to do. Email accounts, with shell,
> Usenet, etc. cost $100/year, two orders of magnitude below your claim.
>
> Seth

--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Seth Breidbart
2004-12-31 02:56:04 UTC
Permalink
Barry Shein <***@world.std.com> top-posted:

> So you're saying panix doesn't block any, e.g., broadband IP pools or
> similar, at a system level?

As far as I know, yes. It's a user option.

> Do you work at panix or otherwise have authoritative info on this?

No, I just have the information they provided all their users when
they created that option. But I know I get email from IPs that are on
various blocklists, because I see that in the logs (spamassassin
reports which blocklists).

Seth
william(at)elan.net
2004-12-21 23:51:53 UTC
Permalink
It should be ISP's responsibility to help other ISPs and as such most
appropriate way to deal with zombies is for ISPs to indicate which of
their ip blocks are used by broadband end-users and which are real mail
servers. If majority of ISPs did that, that would effectively cut the
zombie spam only to be within ISP's own network, which I'm sure they'd
deal with a lot more promptly (and if they don't its really their
problem and their users who have to suffer but nobody elses).

On Tue, 21 Dec 2004, Barry Shein wrote:

> The idea of complete end-user control is a utopian ideal which died a
> few years ago when spammers' zombie bots began hitting ISPs by the
> thousands simultaneously.
>
> At this point there are only two choices, either block at the ingress
> (e.g., blocking network blocks and similar), or begin charging around
> $1000/month for e-mail accounts to pay for the bandwidth and hardware
> necessary to keep up with the unfettered flow (essentially, a
> dedicated server with a dedicated T1 per mailbox.)
>
> Anyone who says otherwise, notably EFF, is just ignorant.
>
>

--
William Leibzon
Elan Networks
***@elan.net
Barry Shein
2004-12-22 00:11:12 UTC
Permalink
So what you're saying is that anyone with a broadband connection
should be prevented from delivering email directly.

I don't disagree, in the current milieu, but I'm sure there are some
who would disagree.

On December 21, 2004 at 15:51 ***@elan.net (william(at)elan.net) wrote:
>
> It should be ISP's responsibility to help other ISPs and as such most
> appropriate way to deal with zombies is for ISPs to indicate which of
> their ip blocks are used by broadband end-users and which are real mail
> servers. If majority of ISPs did that, that would effectively cut the
> zombie spam only to be within ISP's own network, which I'm sure they'd
> deal with a lot more promptly (and if they don't its really their
> problem and their users who have to suffer but nobody elses).
>
> On Tue, 21 Dec 2004, Barry Shein wrote:
>
> > The idea of complete end-user control is a utopian ideal which died a
> > few years ago when spammers' zombie bots began hitting ISPs by the
> > thousands simultaneously.
> >
> > At this point there are only two choices, either block at the ingress
> > (e.g., blocking network blocks and similar), or begin charging around
> > $1000/month for e-mail accounts to pay for the bandwidth and hardware
> > necessary to keep up with the unfettered flow (essentially, a
> > dedicated server with a dedicated T1 per mailbox.)
> >
> > Anyone who says otherwise, notably EFF, is just ignorant.
> >
> >
>
> --
> William Leibzon
> Elan Networks
> ***@elan.net
>

--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
william(at)elan.net
2004-12-22 01:18:43 UTC
Permalink
On Tue, 21 Dec 2004, Barry Shein wrote:

> So what you're saying is that anyone with a broadband connection
> should be prevented from delivering email directly.
>
> I don't disagree, in the current milieu, but I'm sure there are some
> who would disagree.

Everyone who has not agree to be responsible for consequences equvalent to
having user run its own mail server. I take it that those users with windows
who ended up with zombied machines don't have a clue about this and I
certainly don't want them anywhere near my mail system. A responsible
tech user with his own linux box is another matter entirely but I believe
those users can make arrangement with their ISPs to get static ip and have
that ip included as valid smtp transmission initiating entity.

> On December 21, 2004 at 15:51 ***@elan.net (william(at)elan.net) wrote:
> >
> > It should be ISP's responsibility to help other ISPs and as such most
> > appropriate way to deal with zombies is for ISPs to indicate which of
> > their ip blocks are used by broadband end-users and which are real mail
> > servers. If majority of ISPs did that, that would effectively cut the
> > zombie spam only to be within ISP's own network, which I'm sure they'd
> > deal with a lot more promptly (and if they don't its really their
> > problem and their users who have to suffer but nobody elses).
> >
> > On Tue, 21 Dec 2004, Barry Shein wrote:
> >
> > > The idea of complete end-user control is a utopian ideal which died a
> > > few years ago when spammers' zombie bots began hitting ISPs by the
> > > thousands simultaneously.
> > >
> > > At this point there are only two choices, either block at the ingress
> > > (e.g., blocking network blocks and similar), or begin charging around
> > > $1000/month for e-mail accounts to pay for the bandwidth and hardware
> > > necessary to keep up with the unfettered flow (essentially, a
> > > dedicated server with a dedicated T1 per mailbox.)
> > >
> > > Anyone who says otherwise, notably EFF, is just ignorant.
> > --
> > William Leibzon
> > Elan Networks
> > ***@elan.net
--
William Leibzon
Elan Networks
***@elan.net
Seth Breidbart
2004-12-22 01:05:15 UTC
Permalink
Barry Shein <***@world.std.com> top-posted:

> So what you're saying is that anyone with a broadband connection
> should be prevented from delivering email directly.

Actually, he wrote that ISPs should indicate which IP addresses are in
use by customers, and which by mailservers. It's up to anybody else
to use that information as they choose; it isn't a Port 25 block.

> I don't disagree, in the current milieu, but I'm sure there are some
> who would disagree.

Many disagree with blocking Port 25. I haven't seen anyone disagree
with the providing of accurate information.

Seth
Barry Shein
2004-12-22 02:11:37 UTC
Permalink
On December 21, 2004 at 20:05 ***@panix.com (Seth Breidbart) wrote:
> Barry Shein <***@world.std.com> top-posted:
>
> > So what you're saying is that anyone with a broadband connection
> > should be prevented from delivering email directly.
>
> Actually, he wrote that ISPs should indicate which IP addresses are in
> use by customers, and which by mailservers. It's up to anybody else
> to use that information as they choose; it isn't a Port 25 block.

You're just detailing that distinction to us because you think it
makes you look intelligent? I don't understand why you bother.

> > I don't disagree, in the current milieu, but I'm sure there are some
> > who would disagree.
>
> Many disagree with blocking Port 25. I haven't seen anyone disagree
> with the providing of accurate information.

Are you trying to be helpful, or do you just like to type?


--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Seth Breidbart
2004-12-31 02:54:08 UTC
Permalink
Barry Shein <***@world.std.com> wrote:
> On December 21, 2004 at 20:05 ***@panix.com (Seth Breidbart) wrote:
> > Barry Shein <***@world.std.com> top-posted:
> >
> > > So what you're saying is that anyone with a broadband connection
> > > should be prevented from delivering email directly.
> >
> > Actually, he wrote that ISPs should indicate which IP addresses are in
> > use by customers, and which by mailservers. It's up to anybody else
> > to use that information as they choose; it isn't a Port 25 block.
>
> You're just detailing that distinction to us because you think it
> makes you look intelligent? I don't understand why you bother.

No, I'm specifying the distinction because blocking Port 25 is
objected to, for what can be considered good reason. (The customer
says he paid for Internet access, not blocked Internet access.)

> > Many disagree with blocking Port 25. I haven't seen anyone disagree
> > with the providing of accurate information.
>
> Are you trying to be helpful, or do you just like to type?

The former. And you?

Seth
Barry Shein
2004-12-31 03:24:01 UTC
Permalink
I'm sorry, this repartee' has exceeded its shelf expiration date.

-b

On December 30, 2004 at 21:54 ***@panix.com (Seth Breidbart) wrote:
> Barry Shein <***@world.std.com> wrote:
> > On December 21, 2004 at 20:05 ***@panix.com (Seth Breidbart) wrote:
> > > Barry Shein <***@world.std.com> top-posted:
> > >
> > > > So what you're saying is that anyone with a broadband connection
> > > > should be prevented from delivering email directly.
> > >
> > > Actually, he wrote that ISPs should indicate which IP addresses are in
> > > use by customers, and which by mailservers. It's up to anybody else
> > > to use that information as they choose; it isn't a Port 25 block.
> >
> > You're just detailing that distinction to us because you think it
> > makes you look intelligent? I don't understand why you bother.
>
> No, I'm specifying the distinction because blocking Port 25 is
> objected to, for what can be considered good reason. (The customer
> says he paid for Internet access, not blocked Internet access.)
>
> > > Many disagree with blocking Port 25. I haven't seen anyone disagree
> > > with the providing of accurate information.
> >
> > Are you trying to be helpful, or do you just like to type?
>
> The former. And you?
>
> Seth
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg
Laird Breyer
2004-12-21 23:36:51 UTC
Permalink
On Dec 21 2004, Barry Shein wrote:
>
> At this point there are only two choices, either block at the ingress
> (e.g., blocking network blocks and similar), or begin charging around
> $1000/month for e-mail accounts to pay for the bandwidth and hardware
> necessary to keep up with the unfettered flow (essentially, a
> dedicated server with a dedicated T1 per mailbox.)
>

Since you're an ISP, could you indicate roughly the numbers you are
talking about (per user if you like)? How many inbound raw mails per
day? How much total mail data (megabytes) per day? How much total
traffic of all types per day?

I don't know if the list already has discussed those kinds of numbers,
but I'd like to get an idea of the current order of magnitude of the
problem, from the provider's POV. If others have numbers to share for
ISPs, let's hear them too.

--
Laird Breyer.
Devdas Bhagat
2004-12-22 00:32:34 UTC
Permalink
On 22/12/04 09:36 +1000, Laird Breyer wrote:
> On Dec 21 2004, Barry Shein wrote:
> >
> > At this point there are only two choices, either block at the ingress
> > (e.g., blocking network blocks and similar), or begin charging around
> > $1000/month for e-mail accounts to pay for the bandwidth and hardware
> > necessary to keep up with the unfettered flow (essentially, a
> > dedicated server with a dedicated T1 per mailbox.)
> >
>
> Since you're an ISP, could you indicate roughly the numbers you are
> talking about (per user if you like)? How many inbound raw mails per
> day? How much total mail data (megabytes) per day? How much total
> traffic of all types per day?
>
> I don't know if the list already has discussed those kinds of numbers,
> but I'd like to get an idea of the current order of magnitude of the
> problem, from the provider's POV. If others have numbers to share for
> ISPs, let's hear them too.

August numbers for work:
http://nixcartel.org/~devdas/minute.png

I'll try and get the new numbers out and displayed on the official
website.

Devdas Bhagat
Barry Shein
2004-12-22 01:58:45 UTC
Permalink
I don't know since we block so much at the ingress, such as vast
swaths of broadband pools which have been a problem, etc.

It's difficult to measure what you block.

I will say that another huge part of the problem is the stuff no one
ever sees which is the bazillions of user unknowns as the spammers use
various dictionary type attacks. There's far more user unknown
attempted deliveries than other deliveries.

So, even if you knew what went (or would go) into peoples' mailboxes,
opening the floodgates (AKA accepting EFF's point of view) would also
necessitate sifting through all that spew, increasing costs
significantly.

Someone would have to pay for that also, and removing those blocks is
the only way to allow what's called "end-user control" since you don't
know it's for a non-existant user until you let them begin delivery.


On December 22, 2004 at 09:36 ***@lbreyer.com (Laird Breyer) wrote:
> On Dec 21 2004, Barry Shein wrote:
> >
> > At this point there are only two choices, either block at the ingress
> > (e.g., blocking network blocks and similar), or begin charging around
> > $1000/month for e-mail accounts to pay for the bandwidth and hardware
> > necessary to keep up with the unfettered flow (essentially, a
> > dedicated server with a dedicated T1 per mailbox.)
> >
>
> Since you're an ISP, could you indicate roughly the numbers you are
> talking about (per user if you like)? How many inbound raw mails per
> day? How much total mail data (megabytes) per day? How much total
> traffic of all types per day?
>
> I don't know if the list already has discussed those kinds of numbers,
> but I'd like to get an idea of the current order of magnitude of the
> problem, from the provider's POV. If others have numbers to share for
> ISPs, let's hear them too.
>
> --
> Laird Breyer.
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg

--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Laird Breyer
2004-12-22 04:09:40 UTC
Permalink
On Dec 21 2004, Barry Shein wrote:

> It's difficult to measure what you block.

I think it's enough to discuss actual messages accepted for delivery or
filtering by the SMTP servers. See below.

>
> I will say that another huge part of the problem is the stuff no one
> ever sees which is the bazillions of user unknowns as the spammers use
> various dictionary type attacks. There's far more user unknown
> attempted deliveries than other deliveries.
>

The dictionary attacks use up incoming bandwidth, and processing or
storage.

Without knowing the numbers, it's hard to know if the bandwidth used
is a large fraction of incoming capacity, but I doubt it.

However, the processing involved is in principle fully scalable.
There's only the need to exchange a request which can be denied by the
SMTP server for nonexistent users. So the full email is only
transmitted if the mail is accepted.

Figuring out whether a recipient address represents a real user is a
fast operation in principle, ie no matter how many valid user
addresses, a yes/no needs O(1) operations with the right algorithms.

So from a scalability POV, I'd expect the costs of spam to be mainly
proportional to those messages which are sent to a valid user.

> So, even if you knew what went (or would go) into peoples' mailboxes,
> opening the floodgates (AKA accepting EFF's point of view) would also
> necessitate sifting through all that spew, increasing costs
> significantly.

Interestingly, Google started this fad of offering 1 gigabyte
mailboxes. I don't know if they crunched the numbers and figured it's
economically worthwhile, or whether they just have IPO money to burn.

--
Laird Breyer.
Barry Shein
2004-12-22 23:04:13 UTC
Permalink
On December 22, 2004 at 14:09 ***@lbreyer.com (Laird Breyer) wrote:
>
> Figuring out whether a recipient address represents a real user is a
> fast operation in principle, ie no matter how many valid user
> addresses, a yes/no needs O(1) operations with the right algorithms.
>
> So from a scalability POV, I'd expect the costs of spam to be mainly
> proportional to those messages which are sent to a valid user.
>

In my not inconsiderable experience as a system's load, measured in
requests per second, increases the response time per request tends to
follow an exponential.

So, your comments are only true if the hardware and software expands
at some rate to match the request rate. It doesn't have to be linearly
expanded, but when you hit a knee in the curve you either move that
knee (i.e., expand resources) or suffer the consequences (exponential
growth in response time.)

Where I'm using the word "exponential" colloquially to mean
"super-linear" if that bothers anyone, "hockey-sticks" might be a
better verb.

To bring this back to earth, spammers' behavior is such that, barring
response (blocking them, killing them, increasing resources, pulling
the plug on your internet connection, etc) they will find the knee in
that curve.

I am not speaking hypothetically, I am speaking from my not
inconsiderable...

--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Laird Breyer
2004-12-23 00:58:06 UTC
Permalink
On Dec 22 2004, Barry Shein wrote:
>
> So, your comments are only true if the hardware and software expands
> at some rate to match the request rate. It doesn't have to be linearly
> expanded, but when you hit a knee in the curve you either move that
> knee (i.e., expand resources) or suffer the consequences (exponential
> growth in response time.)

That's true, but you aren't seriously suggesting that ISPs won't upgrade
their equipment over time? The internet hasn't finished growing yet,
and service providers are middle men, so their incoming traffic
is bound to grow.

It's perfectly reasonable (don't you think?) to expect an ISP to
deploy resources proportional to their number of users, which brings
us back to the question of the mail request rate per legitimate user
and how it can be eventually limited(*). Some real numbers have
already been offered on the list.

(*) why not (still) consider mail requests for non-users? Verifying
whether an address corresponds to a local user is cheap in principle
nevertheless. Yes, you'll be seeing very many incoming socket
connections, but you don't need to set up a full SMTP service
connection until you already know that the recipient is valid. Now
cheap connections together with Moore's law ought(comment?) to handle
the increase in requests over time at a constant upgrade cost for the
front end.

>
> To bring this back to earth, spammers' behavior is such that, barring
> response (blocking them, killing them, increasing resources, pulling
> the plug on your internet connection, etc) they will find the knee in
> that curve.

That's a lot of different responses, with different costs and
benefits. How can we get an overview without numbers? Or is the idea
that ASRG offers a grab bag of solutions and implementors silently
pick and choose what works for them without giving feedback? I don't
have a preference, just wondering how it's supposed to work.

--
Laird Breyer.
Barry Shein
2004-12-23 03:56:50 UTC
Permalink
On December 23, 2004 at 10:58 ***@lbreyer.com (Laird Breyer) wrote:
> On Dec 22 2004, Barry Shein wrote:
> >
> > So, your comments are only true if the hardware and software expands
> > at some rate to match the request rate. It doesn't have to be linearly
> > expanded, but when you hit a knee in the curve you either move that
> > knee (i.e., expand resources) or suffer the consequences (exponential
> > growth in response time.)
>
> That's true, but you aren't seriously suggesting that ISPs won't upgrade
> their equipment over time? The internet hasn't finished growing yet,
> and service providers are middle men, so their incoming traffic
> is bound to grow.

Yes, but upgrading equipment just to handle spam load is a bad thing,
like any crime no one wants to pay for it yet it must be paid for
nonetheless.

The increased resource pressure from spammers isn't just "in the
noise", it's quite significant. Right now I think what we're seeing is
a mixture of upgrading anyhow and trying to adjust, and just putting
more bread crumbs in the hamburger (slower mail delivery, fooling
around with filters/blocks that are questionable, etc.)

> It's perfectly reasonable (don't you think?) to expect an ISP to
> deploy resources proportional to their number of users, which brings
> us back to the question of the mail request rate per legitimate user
> and how it can be eventually limited(*). Some real numbers have
> already been offered on the list.

I don't think the "per legitimate user" is a realistic measure any
longer since some huge percentage, I'll guess more than half, is to
unknown users and other mischief like endless probes etc.

> (*) why not (still) consider mail requests for non-users? Verifying
> whether an address corresponds to a local user is cheap in principle
> nevertheless.

It's not all that cheap. Remember there are aliases and mailing lists
and so on to deal with.

Cheap is a relative thing. Don't sell "cheaper" (than a full delivery)
as "cheap".

I've been hit by 1500 zombie pc's simultaneously pumping the same
exact spam at us.

How cheap does cheap have to be to deal with an attack like that?

Yes, you'll be seeing very many incoming socket
> connections, but you don't need to set up a full SMTP service
> connection until you already know that the recipient is valid.

I don't get you, SMTP is SMTP, until I talk some SMTP with them I
don't know who their intended recipient is.

Now
> cheap connections together with Moore's law ought(comment?) to handle
> the increase in requests over time at a constant upgrade cost for the
> front end.

My experience sez that spammers seem to follow a parkinsonian law of
resource exploitation which means if you get more resources they'll
use more resources.

Otherwise how could they be such a nuisance to garageshopisp.com and
AOL simultaneously?

I don't think Moore's law covers it, not by a long shot.

Moore's law mostly applies to processing power (specifically,
transistor density.)

Disk speed and network bandwidth costs, two very critical factors in
mail server scaling, don't go up anything like Moore's law.

>
> >
> > To bring this back to earth, spammers' behavior is such that, barring
> > response (blocking them, killing them, increasing resources, pulling
> > the plug on your internet connection, etc) they will find the knee in
> > that curve.
>
> That's a lot of different responses, with different costs and
> benefits. How can we get an overview without numbers? Or is the idea
> that ASRG offers a grab bag of solutions and implementors silently
> pick and choose what works for them without giving feedback? I don't
> have a preference, just wondering how it's supposed to work.
>

ASRG is a research group.

Anxious as we may be for a solution I still think we're quite a ways
away from any meeting of the minds on what the problem is or, perhaps
put better, what a solution might look like.


--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Laird Breyer
2004-12-23 06:04:39 UTC
Permalink
On Dec 22 2004, Barry Shein wrote:
>
> The increased resource pressure from spammers isn't just "in the
> noise", it's quite significant. Right now I think what we're seeing is
> a mixture of upgrading anyhow and trying to adjust, and just putting
> more bread crumbs in the hamburger (slower mail delivery, fooling
> around with filters/blocks that are questionable, etc.)
>

I don't deny it's significant. I'm just trying to help clear up the issues.

> I don't think the "per legitimate user" is a realistic measure any
> longer since some huge percentage, I'll guess more than half, is to
> unknown users and other mischief like endless probes etc.
>

I've got a back of the envelope scheme below to address that specific
problem. Purely theoretical, but I really do think that issue can be
discounted.

> Cheap is a relative thing. Don't sell "cheaper" (than a full delivery)
> as "cheap".

I'm talking about really cheap. I'll give a rough idea below of what I
have in mind. Note also that der Mouse already implemented something
of that nature as described in his post, but probably not the way I'm
going to describe it.

PROBLEM:
Say you have one million valid email addresses. Each address is between
10 and 20 characters long. The problem is to cheaply discard inbound
SMTP connections with invalid addresses in the RCPT command.

WHAT TO DO:
Use a dedicated proxy in front of your SMTP servers. This proxy keeps
all the addresses in memory, either hashed or set up as a trie.
This guarantees that given an address string, figuring if it's valid
is literally a few machine instructions, independent of the number of
valid addresses. You can reload those lists once an hour say, and
you can precompute aliases if you like, whatever.

The proxy has these addresses in memory, but its real job is to accept
socket connections. An SMTP server, once the client has connected,
typically waits for HELO, MAIL, RCPT, DATA followed by the mail
message.

The proxy server is not going to be a full SMTP server. Conceptually,
it's going to wait for HELO, MAIL, RCPT, and at that point check if
the address is valid. If valid, the proxy opens a real SMTP connection
to your SMTP servers and replays the HELO, MAIL, RCPT commands, then
transparently forwards the SMTP conversation. If the address was
invalid, there's no need to bother the SMTP servers, just return an
error and do whatever the protocol allows to close the connection.

So your proxy code really doesn't need to be very smart at all, and
the processing load for each attempted connection (up until the RCPT)
will be really cheap: buffer space for a couple of lines of input to
be replayed, and a very small number of instructions to check the
address, and after that it's mindless copying of bytes. So ten or
twenty megabytes for the full list of addresses, and a couple of K for
buffering each open socket, if you're paranoid about ultra-long SMTP
commands.

The bottleneck will be how many simultaneous open sockets your server
can handle over time. That'll depend on your kernel/hardware limits and
the time you have to wait until a socket is reusable.

Either way, let's take Devdas Bhagat's statistics

http://nixcartel.org/~devdas/minute.png

I don't know if these numbers include invalid users or not, but let's
say 60K invalid requests per minute, and another 10K valid requests.
That's one thousand rejected requests per second, and less than 200
long lived requests. If you wait 30 seconds until you can reuse a
socket, then you'll need about 35,000 socket addresses in total in
this case. Or if you have fewer, you can use several proxies in
parallel.

>
> My experience sez that spammers seem to follow a parkinsonian law of
> resource exploitation which means if you get more resources they'll
> use more resources.

Probably. One problem at a time.

> Moore's law mostly applies to processing power (specifically,
> transistor density.)
>
> Disk speed and network bandwidth costs, two very critical factors in
> mail server scaling, don't go up anything like Moore's law.

You'll note that the proxy scenario I painted needs no disk space, but
does need good I/O hardware and a typical PC's worth of RAM.

>
> ASRG is a research group.
>
> Anxious as we may be for a solution I still think we're quite a ways
> away from any meeting of the minds on what the problem is or, perhaps
> put better, what a solution might look like.
>

Good point. However, I do think some real numbers once in a while help
focus the issues.

BTW, the proxy scenario I gave above isn't intended to be a ready
solution, rather it's an argument to prop up my claim that handling
the invalid requests can be handled scalably.

--
Laird Breyer.
der Mouse
2004-12-23 06:44:19 UTC
Permalink
> Note also that der Mouse already implemented something of that nature
> as described in his post, but probably not the way I'm going to
> describe it.

[much snippage -dM]
> Use a dedicated proxy in front of your SMTP servers. This proxy
> keeps all the addresses in memory, either hashed or set up as a trie.

> The proxy server is not going to be a full SMTP server.
> Conceptually, it's going to wait for HELO, MAIL, RCPT, and at that
> point check if the address is valid. If valid, the proxy opens a real
> SMTP connection to your SMTP servers and replays the HELO, MAIL, RCPT
> commands, then transparently forwards the SMTP conversation.

This is..._almost exactly_ what I did. The only ways in which it isn't
are (a) that it HELOs as itself, rather than relaying the HELO from the
client, and (b) it inserts a Received: header at the beginning of the
data (if the conversation gets to that point).

> BTW, the proxy scenario I gave above isn't intended to be a ready
> solution, rather it's an argument to prop up my claim that handling
> the invalid requests can be handled scalably.

It certainly was a solution to a very real practical problem when I
wrote it; their mailserver went from tottering on the brink of total
collapse to fully usable in the length of time it took the MX record
changes to work their way through the world's DNS caches. Not a really
large ISP - your transaction counts were high by orders of magnitude
compared to them - but not a mom-&-pop either.

Anyone who would like a copy of the code is welcome to it (minus a few
site-specific fragments that really need to be written per-site, like
checking that the domain portion of the address is right). Drop me a
line and if I get lots of requests I'll put it up for ftp. No
guarantees, of course, but I figure at worst someone may steal some
ideas from it.

/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Devdas Bhagat
2004-12-23 14:43:53 UTC
Permalink
On 23/12/04 16:04 +1000, Laird Breyer wrote:
<snip>
> Either way, let's take Devdas Bhagat's statistics
>
> http://nixcartel.org/~devdas/minute.png
>
> I don't know if these numbers include invalid users or not, but let's
> say 60K invalid requests per minute, and another 10K valid requests.

The breakup is very roughly 25% invalid users, 25% DNSBLs and 50% local
IP blocks.
And the invalid user query is a database hit (40 million+ users and
those keep changing very rapidly).

Devdas Bhagat
Laird Breyer
2004-12-24 09:14:58 UTC
Permalink
On Dec 23 2004, Devdas Bhagat wrote:
> On 23/12/04 16:04 +1000, Laird Breyer wrote:
> <snip>
> > Either way, let's take Devdas Bhagat's statistics
> >
> > http://nixcartel.org/~devdas/minute.png
> >
> > I don't know if these numbers include invalid users or not, but let's
> > say 60K invalid requests per minute, and another 10K valid requests.
>
> The breakup is very roughly 25% invalid users, 25% DNSBLs and 50% local
> IP blocks.
> And the invalid user query is a database hit (40 million+ users and
> those keep changing very rapidly).

Right, 15K invalid database hits per minute. What I'm proposing needs a fairly
static list, as that's what is needed to build a quick lookup table (here
quick means you can find out your answer in n character comparisons and jumps
, where n is the number of characters in the email address, independent of the
total number of users. This would also work for IP addresses).

While your 40 million user database has constant activity, I presume the
vast majority of usernames have a long lifetime, a few days at least.
So if you build username snapshots at fixed intervals, then users would have
to wait a little for their new address to become "live" etc.



--
Laird Breyer.
Seth Breidbart
2004-12-24 16:05:48 UTC
Permalink
Laird Breyer <***@lbreyer.com> wrote:

> While your 40 million user database has constant activity, I presume
> the vast majority of usernames have a long lifetime, a few days at
> least. So if you build username snapshots at fixed intervals, then
> users would have to wait a little for their new address to become
> "live" etc.

If you're willing to suffer a _few_ false positives on the filtering,
and do a little more arithmetic, you can do better. Construct some
number of hash functions whose output is an address that holds 1 bit.
Hash each email address with each function, and turn the pointed-at
bit on. When email comes in, apply all the hash functions and check
the pointed-at bits. If they're all on, pass the message through.

A new user gets turned on immediately, by turning on the necessary
bits in the table. A deleted user doesn't go away until the next full
rebuild. There's a small chance (depends on hash table size, number
of hash functions, number of valid email addresses) of a random
address being accepted due to collisions, but this can be made
acceptably small. (Using a parameterized set of hash functions allows
them to be changed every rebuild, to avoid the chance that a commonly
attacked bad address passes.)

Seth
Barry Shein
2004-12-23 20:07:21 UTC
Permalink
On December 23, 2004 at 16:04 ***@lbreyer.com (Laird Breyer) wrote:
> BTW, the proxy scenario I gave above isn't intended to be a ready
> solution, rather it's an argument to prop up my claim that handling
> the invalid requests can be handled scalably.

All I get from it is that an in-memory hash or other fast lookup
algorithm, if it's not being used already, might speed up response
time assuming that your model improves on the resources which are
actually strapped.

That doesn't address scalability, it just shifts the knee of the curve
much like a faster CPU or more memory might.

Scalability has to address:

a) The critical resource(s)
b) Permutational effects

Where (b) is often subtle and significant. For example, if you use N
servers you have to load balance between them. How does that
decision-making scale as N increases. It's often somewhere between
O(N^2) and O(N!).

At any rate, as fascinating as some might find it I sincerely don't
believe the problem with spam, even at the ingress, is going to be
solved or even ameliorated for very long by some improvement in
recipient validity processing.

It's like someone breaking your windows unchallenged and someone says
I know, let's find cheaper ways to make window panes!

It helps no doubt but somehow misses the point, including whether the
cost is in the panes or installing the panes, etc.

--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Laird Breyer
2004-12-24 01:57:16 UTC
Permalink
On Dec 23 2004, Barry Shein wrote:
>
> On December 23, 2004 at 16:04 ***@lbreyer.com (Laird Breyer) wrote:
> > BTW, the proxy scenario I gave above isn't intended to be a ready
> > solution, rather it's an argument to prop up my claim that handling
> > the invalid requests can be handled scalably.
>
> All I get from it is that an in-memory hash or other fast lookup
> algorithm, if it's not being used already, might speed up response
> time assuming that your model improves on the resources which are
> actually strapped.

I'm sure this sort of thing is being used already where needed.
However, it addresses scalability because each handled request uses up
a fixed amount of resources (bounded number of processing
instructions, fixed memory, I/O channels) independent of the number of
users or the number of queries. So there's not going to be a slow down
with size. Moreover, it's clearly a parallelizable concept.

>
> That doesn't address scalability, it just shifts the knee of the curve
> much like a faster CPU or more memory might.

Yes, it shifts the burden of spawning full-fledged SMTP sessions onto
the subset of valid requests. The question is whether the resulting
cost of handling an invalid request is so insignificant with off the
shelf hardware to allow orders of magnitude more requests than your
current full fledged SMTP. I believe it is in this case.

So we don't need to argue that invalid requests are as difficult to
address as spam sent to valid users, which is all I'm trying to
convince you of. Put differently, so long as (invalid user) requests
are a constant couple orders of magnitude more than (valid user)
requests, they are a pure optimization problem. If however the invalid
requests do grow exponentially compared to the valid requests, the
above won't work.

>
> Scalability has to address:
>
> a) The critical resource(s)
> b) Permutational effects
>
> Where (b) is often subtle and significant. For example, if you use N
> servers you have to load balance between them. How does that
> decision-making scale as N increases. It's often somewhere between
> O(N^2) and O(N!).

Come on, for practical purposes you're not looking for the optimum,
just a local optimum so the big oh isn't as bad as you suggest.

The load involved in censoring invalid user SMTP requests is going to
be purely I/O bound, since the picture I painted clearly won't tax the
CPU. Each system can handle so many simultaneous socket connections,
and each such connection is going to cost a nearly identical amount of
resources as any other connection.

So I doubt you'll need complex load balancing. Just pick a server at
random for each request. This is different from full fledged SMTP
because with SMTP, the total resources needed will vary according to
the input, so the resultant load varies too. You will still need to load
balance your real SMTP servers.

>
> At any rate, as fascinating as some might find it I sincerely don't
> believe the problem with spam, even at the ingress, is going to be
> solved or even ameliorated for very long by some improvement in
> recipient validity processing.
>
> It's like someone breaking your windows unchallenged and someone says
> I know, let's find cheaper ways to make window panes!
>
> It helps no doubt but somehow misses the point, including whether the
> cost is in the panes or installing the panes, etc.

Well, if you're going to start with window analogies, I've got one too ;-)

You're saying we can't just worry about someone breaking the windows,
we also need to worry about those who try and fail, because they leave
scratches. I'm saying make your windows scratch proof (cheaply) and then
only bother with people who actually break the windows. There's enough
window-breakers to keep us occupied.

--
Laird Breyer.
der Mouse
2004-12-24 20:44:56 UTC
Permalink
>> That doesn't address scalability, it just shifts the knee of the
>> curve much like [sic] a faster CPU or more memory might.
> Yes, it shifts the burden of spawning full-fledged SMTP sessions onto
> the subset of valid requests. The question is whether the resulting
> cost of handling an invalid request is so insignificant with off the
> shelf hardware to allow orders of magnitude more requests than your
> current full fledged SMTP. I believe it is in this case.

It certainly was so when I implemented such a thing.

/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
der Mouse
2004-12-22 05:27:14 UTC
Permalink
> Since you're an ISP, could you indicate roughly the numbers you are
> talking about (per user if you like)? [...]

I'm not bzs, and the numbers I have aren't quite the ones you ask for,
but they're harder numbers than I've usually seen cited. :-)

One of my tasks at $DAYJOB has been to deal with the crippling load
caused by spam blowback. (Yes, crippling. It had gotten to the point
where they were having to shut off all incoming SMTP during peak
periods just to keep the mailserver up at all.) I wrote a small shim
layer whose job was to turn away unknown local-parts very cheaply.

The reason this is relevant is that I had it keep stats. Thus, I know
what percentage of the RCPTs offered were rejected - ie, were totally
invalid local-parts.

At the peak of the blowback, when I first put it in place, it was
running about one in two hundred - 99.5% of the RCPTs offered were
invalid. (These days, it's much better; it cruises along at only
something like 75% - I just now took a data sample, and while it was
too small to be really all that significant, I have no reason to think
it was unrepresentative, and it showed 93 bad and 32 good. This
matches with my own memory of the sort of numbers I've seen when I've
looked at them in the past.)

That's just totally invalid local-parts. Some amount of that is
doubtless non-spam (a very small amount - see below); there is also
doubtless a significant quantity of spam among the mail aimed at
existing local-parts. I did arrange to capture some of the rejected
traffic, and when I looked, it was approximately all bounce blowback
from spam - other messages were notable by their rarity, and most of
*them* were non-bounced spam.

/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Barry Shein
2004-12-22 23:55:45 UTC
Permalink
This is a good note: Read it again!


On December 22, 2004 at 00:27 ***@Rodents.Montreal.QC.CA (der Mouse) wrote:
> > Since you're an ISP, could you indicate roughly the numbers you are
> > talking about (per user if you like)? [...]
>
> I'm not bzs, and the numbers I have aren't quite the ones you ask for,
> but they're harder numbers than I've usually seen cited. :-)
>
> One of my tasks at $DAYJOB has been to deal with the crippling load
> caused by spam blowback. (Yes, crippling. It had gotten to the point
> where they were having to shut off all incoming SMTP during peak
> periods just to keep the mailserver up at all.) I wrote a small shim
> layer whose job was to turn away unknown local-parts very cheaply.
>
> The reason this is relevant is that I had it keep stats. Thus, I know
> what percentage of the RCPTs offered were rejected - ie, were totally
> invalid local-parts.
>
> At the peak of the blowback, when I first put it in place, it was
> running about one in two hundred - 99.5% of the RCPTs offered were
> invalid. (These days, it's much better; it cruises along at only
> something like 75% - I just now took a data sample, and while it was
> too small to be really all that significant, I have no reason to think
> it was unrepresentative, and it showed 93 bad and 32 good. This
> matches with my own memory of the sort of numbers I've seen when I've
> looked at them in the past.)
>
> That's just totally invalid local-parts. Some amount of that is
> doubtless non-spam (a very small amount - see below); there is also
> doubtless a significant quantity of spam among the mail aimed at
> existing local-parts. I did arrange to capture some of the rejected
> traffic, and when I looked, it was approximately all bounce blowback
> from spam - other messages were notable by their rarity, and most of
> *them* were non-bounced spam.
>
> /~\ The ASCII der Mouse
> \ / Ribbon Campaign
> X Against HTML ***@rodents.montreal.qc.ca
> / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg

--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Laird Breyer
2004-12-23 01:26:00 UTC
Permalink
On Dec 22 2004, Barry Shein wrote:
>
>
> This is a good note: Read it again!
>

I agree, it's an excellent note. I find this particularly interesting:

> > existing local-parts. I did arrange to capture some of the rejected
> > traffic, and when I looked, it was approximately all bounce blowback
> > from spam - other messages were notable by their rarity, and most of
> > *them* were non-bounced spam.

It's particularly interesting because it suggests that the bulk of
those invalid requests are protocol related, not directly related to
the spammers' creativity.

--
Laird Breyer.
Walter Dnes
2004-12-22 05:21:33 UTC
Permalink
On Tue, Dec 21, 2004 at 03:37:54PM +1000, Laird Breyer wrote

> Not necessarily. Some people do not have the luxury of defining spam
> as that which they don't want. For example, children whose mail is
> monitored, or employees whose mail is filtered centrally.

The "client" in this case is the entity that pays for the inbox. In
the case of children, minors can't legally make contracts anyway. It
has to be approved by their parent. And the parent is the one paying
for the inbox as well.

My email inbox at work is no more my property, than is the the
computer I use at work. It is paid for, owned, and controlled by my
employer. I use it in my capacity as an agent of my employer.

--
Walter Dnes <***@waltdnes.org>
An infinite number of monkeys pounding away on keyboards will
eventually produce a report showing that Windows is more secure,
and has a lower TCO, than linux.
Seth Breidbart
2004-12-22 20:15:40 UTC
Permalink
(Responses to multiple messages included.)

"Walter Dnes" <***@waltdnes.org> wrote:
> On Tue, Dec 21, 2004 at 10:52:59AM +1000, Laird Breyer wrote

>> Would it be accurate to say that ASRG is interested in Z-mail
>> prevention solutions where Z-mail is a label for some set of messages,
>> and Z-mail can be taken to be spam?
. . .
> The thing that you may be missing is that "one size does not fit all".
> One man's spam is another man's ham.

More important, I think, is that having a single definition of spam
works against solutions.

Somebody has a system that works well against Type X spam, and wants
to improve it. Arguing that it isn't good because it ignores Type Y
spam just wastes effort.

Then somebody else comes up with a system that works OK against Type Y
spam. If they both spend their time arguing over whether "spam" means
"Type X" or "Type Y" that's time they're not spending improving (or
combining) their systems.

> Rather than a "spam-blocking" program labelling something as spam
> and blocking it, we should have an "unwanted-email-blocking
> program". When the spammer's lawyer screams "It's not spam, it's
> ooga-booga. Stop blocking my client's emails with your
> 'spam-blocking' program", we should respond that it's really an
> 'unwanted-email-blocking-program'. And that our customer has
> indicated that he doesn't want your email.

That doesn't work: the spammer will get a shill to sign up with your
ISP and claim he _wants_ the spam.

Laird Breyer <***@lbreyer.com> wrote:
> On Dec 21 2004, John Levine wrote:

> The charter does emphasise that the ASRG is looking for well
> specified problems, and makes it clear that a multitude of
> techniques to tackle them is desirable. As such, doesn't that
> encourage attempts to define spam?

No, that discourages them, because having one blessed definition works
against having multiple techniques each of which is most effective
against a different part of the problem.

> The original message you responded to suggested a possible
> definition, followed by a link to some protocol which takes
> that definition as a basis. I guess I don't see what prompted
> you to warn against attempting to define spam. Could you explain?
>
>> My position is that any anti-spam project has to identify the flavor
>> of spam that the project is attempting to address, but I don't expect
>> projects all to address the same thing.
>
> I agree with that position.

But having one definition works against that.

***@terabites.com wrote:

> Of course, as far as the recipient, THEIR decision about what is
> spam and what isn't, and what they want to receive and what they
> don't, is the ONLY opinion which matters in the final analysis.

Providing your goal is to market your services to them, yes.

> This is yet another reason why a fine-grained permissions-based
> approach, where the *recipient* can decide what E-mails they do and
> do not want to receive (based on who the sender is, and what the
> mail contains) is so decidedly the right way to pursue this problem.

It is _a_ right way. It is not the only one.

> Especially with threats of attorneys et al, there is NO feasible
> way for a spammer to try to sue a recipient who has chosen to refuse
> to accept, or read, the mail a spammer has tried to jam into the
> recipient's mailbox.

You might have noticed that spammers don't limit themselves to
"feasible" lawsuits.

***@terabites.com wrote:
> On 22 Dec 2004, John Levine <***@johnlevine.com> wrote:

>>>This is yet another reason why a fine-grained permissions-based
>>>approach, where the *recipient* can decide what E-mails they do and
>>>do not want to receive (based on who the sender is, and what the mail
>>>contains) is so decidedly the right way to pursue this problem.
>
>>So long as you are willing to spend an unlimited amount of money on an
>>e-mail infrastructure that accepts and stores terabytes of spam,
>>almost all of which will be discarded unread,
>
> You are presuming that (1) spamming will continue to be lucrative;

No, just that it will continue to be done.

> that (2) an approach such as I envision will reduce neither the
> numeric number of spam messages nor their average size;

or at least leave those numbers large.

> and (3) that spammers will continue to be able
> to recruit zombie spambot armies to do their mailings for them.

That's certainly likely for the foreseeable future.

> First off, I believe that a fine-grained permissions list (with
> permissions based on who messages come from, along with what the
> nature of the contents of those messages are) and which by default
> will not allow either "large", HTML or attachments from untrusted
> senders, together will virtually eliminate E-mail as an effective
> vector for recruiting spambot zombie armies (it will in fact
> virtually eliminate the efficacy of sending worms and viruses in
> E-mail messages).

In order for it to eliminate zombies, it has to be implemented by
_everybody else_. That's not going to happen.

> Secondly, making HTML-burdened E-mail acceptance CONTINGENT upon the
> sender being whitelisted by the recipient

You're assuming that you can tell whether or not the _recipient's_ MUA
will attempt to interpret a message as HTML.

> Third, making spam filtering more effective and harder to defeat or
> evade will dramatically reduce the payback to spammers, and the
> payback is what motivates spamming in the first place.

For some spammers, maybe. Many others sell their services, and the
worldwide shortage of suckers is not expected soon.

>>and a fabuously complex
>>spam filter control panel that almost nobody will use,
>
> Oh, that's TRULY rubbish. While obviously it would be CONCEIVABLE
> to implement such a filter in a stupid and clumsy way, a reasonable
> implementation could make this VERY user-friendly (far more
> user-friendly, in fact, than typical "security permissions" for NTFS
> file systems).

Prove it. Come up with a reasonable implementation that my mother can
handle.

>>ISPs tell me that when they have crummy filters that leak a lot of
>>spam, people are constantly asking to be able to tune the filters.
>
> The fact is that users who are able to simply and easily control
> THEIR OWN spam filtering, using techniques which are understandable
> and logical, are less likely to require as much ISP support.

As spammers learn to evade those controls, the ISP has to upgrade
them.

>>If you're in the United States, please consider them in relation to 47
>>USC 230 and section 8(c) of CAN SPAM.
>
> I don't think that's relevant to what we're talking about here (at
> least not what *I* am talking about). I'm not talking about
> companies, individuals, or governments suing spammers, I'm talking
> about spammers (or those whose behavior might arguably look like
> spamming) suing (or threatening to sue) ISPs.

Did you read those? They provide _immunity for ISPs_ for good-faith
spam filtering.

***@terabites.com wrote:

>> The user has still made a clear decision - one to delegate their right
>> to accept/refuse to the ISP. As long as the ISP's contract makes this
>> clear, then I believe there's no difference in the 2 cases.
>
> Sounds good, until the "spammer" can find a case (even just ONE)
> where the intended recipient actually WANTED the E-mail in question,
> and said user didn't feel they had in fact granted their ISP the
> 'right' to MIS-BLOCK mail that the user actually wanted.

Then the ISP show the contract the user agreed to, and the law
granting it immunity.

> Now, perhaps you can come up with a contract/TOS that has enough
> waivers and disclaimers in it,

And every ISP already has; have you looked at any? (I have, and they
all say the ISP can do what it wants.)

> but I still think that the "spammer"/"mailer"
> will probably have adequate claim to bring it up in court, SOMEWHERE.

I don't care what the laws in Spamzania say.

>> The user can exercise choice by switching to an ISP which allows greater
>> control by its users (and as John has pointed out, probably charges a
>> premium to cover the costs of offering that control).
>
> I think you're being far, far too presumptuous about the
> practicality of switching to a different ISP. Maybe you haven't
> been paying attention, but the ISP world has been consolidating (and
> especially if we're talking about broadband type services... okay,
> yes, dialup ISPs are more plentiful).

Mail Service Providers are growing. They don't have to provide
Internet access; some people prefer buying unbundled services.

> So if I didn't like Comcast's terms (and I don't, for example, like
> their policy of blocking port 25, which would enable me to send mail
> through my own domain provider's SMTP server) what, exactly, are my
> choices for switching to a different ISP?

$100/year for a Panix account with shell, mail, Usenet, etc.

$0/year and up for email accounts with Outblaze, gmail, Yahoo, . . .

> Meanwhile, even if one presumes that I'm therefore going to use
> Comcast for my connectivity, the fact that Comcast insists on
> blocking port 25 pretty well precludes me from signing up with a
> different and more-satisfactory "ISP", at least for outgoing
> SMTP-type mail service.

A decent one uses "SUBMIT".

Seth
Walter Dnes
2004-12-23 00:40:13 UTC
Permalink
On Wed, Dec 22, 2004 at 03:15:40PM -0500, Seth Breidbart wrote

> > Rather than a "spam-blocking" program labelling something as spam
> > and blocking it, we should have an "unwanted-email-blocking
> > program". When the spammer's lawyer screams "It's not spam, it's
> > ooga-booga. Stop blocking my client's emails with your
> > 'spam-blocking' program", we should respond that it's really an
> > 'unwanted-email-blocking-program'. And that our customer has
> > indicated that he doesn't want your email.
>
> That doesn't work: the spammer will get a shill to sign up with your
> ISP and claim he _wants_ the spam.

I have a remote inbox at an ISP that has hacked up Qmail to allow
*END-USER-CONFIGURED* rulesets that block email *AT THE SMTP STAGE*.
Each user can have their own custom filters. As a matter of fact, each
of the 10 email accounts that a user is allowed, can have different
rulesets. I host my email domain with them. My waltdnes ID is heavily
guarded. My postmaster and abuse ID's are unfiltered, unless I run into
a really snotty spammer.

As I said earlier, one size does not fit all. The goal should be
end-user control, even if it's only an "on-off" switch for spam-blocking.
A serendipitous side-effect is that it defeats the shill-signs-up-at-ISP
stunt.

> > Especially with threats of attorneys et al, there is NO feasible
> > way for a spammer to try to sue a recipient who has chosen to refuse
> > to accept, or read, the mail a spammer has tried to jam into the
> > recipient's mailbox.
>
> You might have noticed that spammers don't limit themselves to
> "feasible" lawsuits.

In that case, why are you participating in this forum? You might get
sued for contributing to tortious interference with a spammer's income.

--
Walter Dnes <***@waltdnes.org>
An infinite number of monkeys pounding away on keyboards will
eventually produce a report showing that Windows is more secure,
and has a lower TCO, than linux.
Paul Lambert
2004-12-20 18:33:36 UTC
Permalink
Just a taxonomy of mail would also be interesting ... many of the
interesting issues with preventing spam result from the many types of
e-mail messages: errors, first time contacts, forwarding, mail lists,
etc.

Paul

> -----Original Message-----
> From: asrg-***@ietf.org [mailto:asrg-***@ietf.org] On
> Behalf Of John Levine
> Sent: Monday, December 20, 2004 10:14 AM
> To: ***@ietf.org
> Cc: ***@elvey.com
> Subject: Re: [Asrg] SICS
>
> >> One possible (and imho very useful) definition of spam ...
>
> Please keep in mind that attempts to create a definition of
> spam are specifically excluded from the ASRG's area of interest.
>
> Taxonomies of objectionable mail might be interesting, though.
>
> Regards,
> John Levine, ***@taugh.com, Taughannock Networks,
> Trumansburg NY http://www.taugh.com
Just a tax
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg
>
Hannigan, Martin
2004-12-22 00:37:15 UTC
Permalink
The ISP's cooperate. Going after the zombies is, for the
most part, an ineffective approach to the situation. Search
and destroy of the controllers is more effective i.e.
1 controller = 100K downed bots. (example)

There's a ton of work going on behind the scenes.

-M<

--
Martin Hannigan (c) 617-388-2663
VeriSign, Inc. (w) 703-948-7018
Network Engineer IV Operations & Infrastructure
***@verisign.com



> -----Original Message-----
> From: asrg-***@ietf.org [mailto:asrg-***@ietf.org]On Behalf Of
> william(at)elan.net
> Sent: Tuesday, December 21, 2004 6:52 PM
> To: Barry Shein
> Cc: ASRG list
> Subject: Re: [Asrg] SICS
>
>
>
> It should be ISP's responsibility to help other ISPs and as such most
> appropriate way to deal with zombies is for ISPs to indicate which of
> their ip blocks are used by broadband end-users and which are
> real mail
> servers. If majority of ISPs did that, that would effectively cut the
> zombie spam only to be within ISP's own network, which I'm sure they'd
> deal with a lot more promptly (and if they don't its really their
> problem and their users who have to suffer but nobody elses).
>
> On Tue, 21 Dec 2004, Barry Shein wrote:
>
> > The idea of complete end-user control is a utopian ideal
> which died a
> > few years ago when spammers' zombie bots began hitting ISPs by the
> > thousands simultaneously.
> >
> > At this point there are only two choices, either block at
> the ingress
> > (e.g., blocking network blocks and similar), or begin
> charging around
> > $1000/month for e-mail accounts to pay for the bandwidth
> and hardware
> > necessary to keep up with the unfettered flow (essentially, a
> > dedicated server with a dedicated T1 per mailbox.)
> >
> > Anyone who says otherwise, notably EFF, is just ignorant.
> >
> >
>
> --
> William Leibzon
> Elan Networks
> ***@elan.net
>
>
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg
>
william(at)elan.net
2004-12-22 01:38:25 UTC
Permalink
On Tue, 21 Dec 2004, Hannigan, Martin wrote:
>
> The ISP's cooperate. Going after the zombies is, for the
> most part, an ineffective approach to the situation.

I'm not talking about reactive approach - I'm talking about prevention of
this in the first place. All that is necessary is that ISPs agree to share
in a standard way a list of host they believe to be responsible enough to
freely participate in SMTP transactions on their own. This cuts down list
of possible zombie targets to very few machines run by users who are
already likely to have security mechanism that prevents their system from
being taken over.

> Search and destroy of the controllers is more effective i.e.
> 1 controller = 100K downed bots. (example)
> There's a ton of work going on behind the scenes.

It is certainly good that this is going on, I've been involved in couple
of these "search and destroy" missions myself. But this is all work after
the fact when we should be trying to research ways to prevent the occurance
of the problem in the first place. In other words, would you prefer to
face possiblity of being sick with a smallpox rather then the world
having choosen to immunize everyone against it some time ago which
effectively got rid of the problem?

--
William Leibzon
Elan Networks
***@elan.net
Eric Brunner-Williams in Portland Maine
2004-12-21 21:30:35 UTC
Permalink
> ...
> the fact when we should be trying to research ways to prevent the occurance
> of the problem in the first place. In other words, would you prefer to
> face possiblity of being sick with a smallpox rather then the world
> having choosen to immunize everyone against it some time ago which
> effectively got rid of the problem?

Variola was effectively quarentened until about 1521. It wasn't an issue
in the New World population until infected immigration populations were
introduced from the Old World.
Barry Shein
2004-12-22 02:19:46 UTC
Permalink
Yes, but the very assumption that anyone would "take down" a zombie PC
(block, disable, whatever) belies the "end-user control" notion of
spam.

For all I know there's someone out there who wants that zombie to spew
at them, maybe they're doing research or something. And, by EFF's
recommendations, it would be morally repugnant for an ISP to thus
interfere.

So although it might be a slightly different case, it's the same exact
issue.

You just think no one is crazy enough to defend zombie PCs, and I'd
like to think you're correct. But as far as I can tell EFF is that
crazy.

And if that's not what they're trying to say in that white paper then
they should disabuse it and admit that under current conditions the
distinction they make is at best purely academic (i.e., there may
exist at least one instance which fits their profile, but whether it's
representative or even definable remains an open question), and at
worst purely idiotic.


On December 21, 2004 at 17:38 ***@elan.net (william(at)elan.net) wrote:
>
> On Tue, 21 Dec 2004, Hannigan, Martin wrote:
> >
> > The ISP's cooperate. Going after the zombies is, for the
> > most part, an ineffective approach to the situation.
>
> I'm not talking about reactive approach - I'm talking about prevention of
> this in the first place. All that is necessary is that ISPs agree to share
> in a standard way a list of host they believe to be responsible enough to
> freely participate in SMTP transactions on their own. This cuts down list
> of possible zombie targets to very few machines run by users who are
> already likely to have security mechanism that prevents their system from
> being taken over.
>
> > Search and destroy of the controllers is more effective i.e.
> > 1 controller = 100K downed bots. (example)
> > There's a ton of work going on behind the scenes.
>
> It is certainly good that this is going on, I've been involved in couple
> of these "search and destroy" missions myself. But this is all work after
> the fact when we should be trying to research ways to prevent the occurance
> of the problem in the first place. In other words, would you prefer to
> face possiblity of being sick with a smallpox rather then the world
> having choosen to immunize everyone against it some time ago which
> effectively got rid of the problem?
>
> --
> William Leibzon
> Elan Networks
> ***@elan.net
>
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg
william(at)elan.net
2004-12-22 03:29:50 UTC
Permalink
On Tue, 21 Dec 2004, Barry Shein wrote:

> Yes, but the very assumption that anyone would "take down" a zombie PC
> (block, disable, whatever) belies the "end-user control" notion of
> spam.
>
> For all I know there's someone out there who wants that zombie to spew
> at them, maybe they're doing research or something. And, by EFF's
> recommendations, it would be morally repugnant for an ISP to thus
> interfere.

Who interfered? If somebody wants to accept spam from zombie pc - they can
go ahead (ok, they may have to tell their ISP that they want to do it or
may have to run their own mail server with different set of policies).

> So although it might be a slightly different case, it's the same exact
> issue.
>
> You just think no one is crazy enough to defend zombie PCs, and I'd
> like to think you're correct. But as far as I can tell EFF is that
> crazy.

Well since they "that crazy" to hold a view it as that anybody has the
right to send then any kind forged and spoofed of email and its their
responsibility to (re)view every one of those emails manually to determine
if its something they wanted to receive, perhaps we need to run a small
test campaign of their convictions - how about all the spamtraps instead
of sending zombie spam to /dev/null instead start forwarding and [re]forged
those email to EFF! :) So anybody knows list of their officers who are good
audience to read about what email traffic originates from zombied machines?

Perhaps even a screensaver could be created to help in this process ... :)

--
William Leibzon
Elan Networks
***@elan.net
John Levine
2004-12-22 03:35:11 UTC
Permalink
>You just think no one is crazy enough to defend zombie PCs, and I'd
>like to think you're correct. But as far as I can tell EFF is that
>crazy.

I had lunch with the EFF rep who spoke at the FTC authentication
conference about a month ago. She told me that she hand sorts 2000
spams a day from her mailbox. Apparently she thinks this is a
productive use of her time. Go figure.

Regards,
John Levine, ***@taugh.com, Taughannock Networks, Trumansburg NY
http://www.taugh.com
Eric Brunner-Williams in Portland Maine
2004-12-21 21:12:33 UTC
Permalink
> Search and destroy of the controllers is more effective
> i.e. 1 controller = 100K downed bots. (example)

But ... the acquisition cost or re-acquisition cost or rendezvous cost
is zero, and the controller may simply be the current lease holder ...

There are assets that have different temporal properties than the set
of write-side nodes, which frankly, appear to twinkle.

BTW, my research interest is in spam/http (e.g., comment spam on blogs),
not spam/smtp, and in my problem space the write-side is as intractible,
address list, content signature, authentication exchanges, and so on,
so my interest is on the read-side.

Milage varies, but this is research, so I've no idea what I'm doing.

Eric
Hannigan, Martin
2004-12-22 01:57:57 UTC
Permalink
> -----Original Message-----
> From: william(at)elan.net [mailto:***@elan.net]
> Sent: Tuesday, December 21, 2004 8:38 PM
> To: Hannigan, Martin
> Cc: ASRG list
> Subject: RE: [Asrg] SICS
>
>
>
> On Tue, 21 Dec 2004, Hannigan, Martin wrote:
> >
> > The ISP's cooperate. Going after the zombies is, for the
> > most part, an ineffective approach to the situation.
>
> I'm not talking about reactive approach - I'm talking about
> prevention of
> this in the first place. All that is necessary is that ISPs
> agree to share
> in a standard way a list of host they believe to be
> responsible enough to
> freely participate in SMTP transactions on their own. This
> cuts down list
> of possible zombie targets to very few machines run by users who are
> already likely to have security mechanism that prevents their
> system from
> being taken over.

It'll never happen. What's happening here is that email is
becoming over complicated and the operational expense is increasing
as a result - without (m)any results.

>
> > Search and destroy of the controllers is more effective i.e.
> > 1 controller = 100K downed bots. (example)
> > There's a ton of work going on behind the scenes.
>
> It is certainly good that this is going on, I've been
> involved in couple
> of these "search and destroy" missions myself. But this is
> all work after
> the fact when we should be trying to research ways to prevent
> the occurance

There is work being done on prevention. How about if MS could
bundle AV into the operating system (free)?

> of the problem in the first place. In other words, would you
> prefer to
> face possiblity of being sick with a smallpox rather then the world
> having choosen to immunize everyone against it some time ago which
> effectively got rid of the problem?

Great analogies, but we're talking about bits and nobody cares
about the plague anymore. If you shift the focus onto operational
expense, capital expense, revenue, etc..it might make more sense.

Trying to at least add some on-topic (about spam), the botnets
technically do NOT spam. They sell their zombies and the spammer
usually spams from a host located right here in the USA. The headers
are rewritten so that host is hidden, but it's there. Florida seems
to be the big place to hide-spam from lately.

The solution has to go up high in the network, near the
NSP's. At the exchanges. I have no idea what that solution is.

Best,

-M<
Seth Breidbart
2004-12-22 02:10:57 UTC
Permalink
"Hannigan, Martin" <***@verisign.com> wrote:

> There is work being done on prevention. How about if MS could
> bundle AV into the operating system (free)?

Why would you trust them to get it right? If they could, it wouldn't
be necessary in the first place.

Seth
Hannigan, Martin
2004-12-22 02:27:20 UTC
Permalink
Anyone tried this on the cable swamp?



--
Martin Hannigan (c) 617-388-2663
VeriSign, Inc. (w) 703-948-7018
Network Engineer IV Operations & Infrastructure
***@verisign.com



> -----Original Message-----
> From: asrg-***@ietf.org [mailto:asrg-***@ietf.org]On Behalf Of
> Barry Shein
> Sent: Tuesday, December 21, 2004 8:59 PM
> To: ***@lbreyer.com
> Cc: ASRG list
> Subject: Re: [Asrg] SICS
>
>
>
> I don't know since we block so much at the ingress, such as vast
> swaths of broadband pools which have been a problem, etc.
>
> It's difficult to measure what you block.
>
> I will say that another huge part of the problem is the stuff no one
> ever sees which is the bazillions of user unknowns as the spammers use
> various dictionary type attacks. There's far more user unknown
> attempted deliveries than other deliveries.
>
> So, even if you knew what went (or would go) into peoples' mailboxes,
> opening the floodgates (AKA accepting EFF's point of view) would also
> necessitate sifting through all that spew, increasing costs
> significantly.
>
> Someone would have to pay for that also, and removing those blocks is
> the only way to allow what's called "end-user control" since you don't
> know it's for a non-existant user until you let them begin delivery.
>
>
> On December 22, 2004 at 09:36 ***@lbreyer.com (Laird Breyer) wrote:
> > On Dec 21 2004, Barry Shein wrote:
> > >
> > > At this point there are only two choices, either block
> at the ingress
> > > (e.g., blocking network blocks and similar), or begin
> charging around
> > > $1000/month for e-mail accounts to pay for the bandwidth
> and hardware
> > > necessary to keep up with the unfettered flow (essentially, a
> > > dedicated server with a dedicated T1 per mailbox.)
> > >
> >
> > Since you're an ISP, could you indicate roughly the numbers you are
> > talking about (per user if you like)? How many inbound
> raw mails per
> > day? How much total mail data (megabytes) per day? How much total
> > traffic of all types per day?
> >
> > I don't know if the list already has discussed those kinds
> of numbers,
> > but I'd like to get an idea of the current order of
> magnitude of the
> > problem, from the provider's POV. If others have numbers
> to share for
> > ISPs, let's hear them too.
> >
> > --
> > Laird Breyer.
> >
> > _______________________________________________
> > Asrg mailing list
> > ***@ietf.org
> > https://www1.ietf.org/mailman/listinfo/asrg
>
> --
> -Barry Shein
>
> Software Tool & Die | ***@TheWorld.com |
http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Hannigan, Martin
2004-12-22 03:03:49 UTC
Permalink
If we could make SPAM mitigation as easy as they've made windows........

Making SPAM mitigation harder works against the resolution. As long as
someone will click on f r. E e. V. I. A. G. A. R. A. , there will be SPAM.

We're not the target audience anymore.

They[1] are. We are their stewards and everytime we force the user to take
some additional step, we do them a dis service since they can't remember
what SPAM free is like many of us can.

1. People who want to be end users, not engineers - most people.

-M



---
Martin Hannigan
***@verisign.com
Verisign, Inc.


-----Original Message-----
From: asrg-***@ietf.org <asrg-***@ietf.org>
To: ***@ietf.org <***@ietf.org>
Sent: Tue Dec 21 18:10:57 2004
Subject: Re: [Asrg] SICS

"Hannigan, Martin" <***@verisign.com> wrote:

> There is work being done on prevention. How about if MS could
> bundle AV into the operating system (free)?

Why would you trust them to get it right? If they could, it wouldn't
be necessary in the first place.

Seth
Eric Brunner-Williams in Portland Maine
2004-12-22 08:44:06 UTC
Permalink
> As long as someone will click on <well-known-string="1">
> there will be SPAM.

Actually, in the spam/http problem space even that isn't a necessary
predicate condition. write-side engines insert <well-known-string="1">
where it is unlikely to be seen by people (except for disfunctioning
write-side engines), for the (infered) purpose of harvesting by indexers.

Eric
g***@terabites.com
2004-12-23 21:18:25 UTC
Permalink
>> Rather than a "spam-blocking" program labelling something as spam
> and blocking it, we should have an "unwanted-email-blocking
> program". When the spammer's lawyer screams "It's not spam, it's
> ooga-booga. Stop blocking my client's emails with your
> 'spam-blocking' program", we should respond that it's really an
> 'unwanted-email-blocking-program'. And that our customer has
> indicated that he doesn't want your email.

> That doesn't work: the spammer will get a shill to sign up with your
ISP and claim he _wants_ the spam.

That's ONLY a problem if the decision is made by the ISP and applies to all
recipients.

Once the recipient has the control, then THEY can set the filter as close or as
open as they want, based on which of their recipients they trust and expect to
send them what (type and content of mail).

One could even envision, for example, a scheme whereby an E-mail from a given
whitelisted sender would only be trusted IF it contained one of a small set of
internal "keys" in it, specific to that sender, perhaps in the header or even in
the mail body: the sender's habitual signature file, what E-mail program they
use, their IP address in the header (for users with fixed IP addresses) or some
such.

In any case, the result of doing this RIGHT is that the shill signing up for the
ISP will accomplish absolutely NOTHING, since THEY are the only ones who will
receive the spammer's crap...! The shill's permissions and demands won't affect
ANY other recipient.

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
Seth Breidbart
2005-01-02 00:10:54 UTC
Permalink
***@terabites.com wrote:

>> That doesn't work: the spammer will get a shill to sign up with your
>> ISP and claim he _wants_ the spam.
>
> That's ONLY a problem if the decision is made by the ISP and applies to all
> recipients.

Allowing users to write their own filters leads to annoying problems.
Setting up systems to safely allow users to choose their own filters
tends to be expensive, and is a non-trivial issue. It's not so bad
for a small ISP with clueful users; but I wouldn't want to even think
about AOL's customer support issue if they tried it.

Seth
g***@terabites.com
2004-12-23 21:44:26 UTC
Permalink
[snip]

>> and (3) that spammers will continue to be able
> to recruit zombie spambot armies to do their mailings for them.

> That's certainly likely for the foreseeable future.

It depends entirely on how long it takes people to decide to tackle the problem
in a determined way.

>> First off, I believe that a fine-grained permissions list (with
> permissions based on who messages come from, along with what the
> nature of the contents of those messages are) and which by default
> will not allow either "large", HTML or attachments from untrusted
> senders, together will virtually eliminate E-mail as an effective
> vector for recruiting spambot zombie armies (it will in fact
> virtually eliminate the efficacy of sending worms and viruses in
> E-mail messages).

> In order for it to eliminate zombies, it has to be implemented by
_everybody else_. That's not going to happen.

Well, yes and no.

I agree that you probably won't eliminate 100% of the vulnerability, but at some
point the remainder doesn't much matter. The issue is whether the vulnerability
is widespread enough to ATTEMPT to exploit it.

It also depends on how quickly such a fine-grained permissions list approach is
accepted and installed. Obviously, if Microsoft were to include something like
this in Outlook and Outlook Express by default, it would be much more effective
and much sooner than if Infopoint or some other small software company were to
try to market it as an addon package.

>> Secondly, making HTML-burdened E-mail acceptance CONTINGENT upon the
> sender being whitelisted by the recipient

> You're assuming that you can tell whether or not the _recipient's_ MUA
will attempt to interpret a message as HTML.

It doesn't much matter. Most spammers don't know, either, what specific E-mail
client program a destination address is using. Nor do they care, in fact.
Again, this is the sort of thing that a recipient ought to have some control
over... just how aggressive (or not) such a filter ought to be in their incoming
messages.

>> Third, making spam filtering more effective and harder to defeat or
> evade will dramatically reduce the payback to spammers, and the
> payback is what motivates spamming in the first place.

> For some spammers, maybe. Many others sell their services, and the
worldwide shortage of suckers is not expected soon.

It won't take long for the word to get around that spamming doesn't work
anymore, and that some types work dramatically less well than others.

>>>and a fabuously complex
>>spam filter control panel that almost nobody will use,

>> Oh, that's TRULY rubbish. While obviously it would be CONCEIVABLE
> to implement such a filter in a stupid and clumsy way, a reasonable
> implementation could make this VERY user-friendly (far more
> user-friendly, in fact, than typical "security permissions" for NTFS
> file systems).

> Prove it. Come up with a reasonable implementation that my mother can
handle.

I'll be GLAD to do that, and I'll even bring it to release-ready, if you'll fund
the development. :-)

The point is that this is IMPLEMENTATION-DEPENDENT, and does not need to be part
of a "best practices" advisory. Some companies are hugely better at
"human-engineering" software products than others are; given that, we don't
have to concern ourselves HERE with the fact that some of them might not do a
terrific job of it.

>>>ISPs tell me that when they have crummy filters that leak a lot of
>>spam, people are constantly asking to be able to tune the filters.

>> The fact is that users who are able to simply and easily control
> THEIR OWN spam filtering, using techniques which are understandable
> and logical, are less likely to require as much ISP support.

> As spammers learn to evade those controls, the ISP has to upgrade
them.

I'd be surprised if ANY system didn't continue to evolve, just as the threats
evolve that it's intended to counter.

Again, though, I'll point out that once you default to "no attachments, no HTML"
in E-mails from unlisted senders, you don't leave the spammers much room to
evade much of anything, at least as far as their E-mail content goes.

I will *freely* admit that the battleground will then most likely move to
malicious Web content and browser-based attacks, rather than spam E-mails... but
that IS a different battle, and we don't have to fight that one HERE.

>>> The user has still made a clear decision - one to delegate their right
>> to accept/refuse to the ISP. As long as the ISP's contract makes this
>> clear, then I believe there's no difference in the 2 cases.

>> Sounds good, until the "spammer" can find a case (even just ONE)
> where the intended recipient actually WANTED the E-mail in question,
> and said user didn't feel they had in fact granted their ISP the
> 'right' to MIS-BLOCK mail that the user actually wanted.

> Then the ISP show the contract the user agreed to, and the law
granting it immunity.

There are a LOT of contracts which judges end up NOT holding as enforceable.

But whatever.

>>> The user can exercise choice by switching to an ISP which allows greater
>> control by its users (and as John has pointed out, probably charges a
>> premium to cover the costs of offering that control).

>> I think you're being far, far too presumptuous about the
> practicality of switching to a different ISP. Maybe you haven't
> been paying attention, but the ISP world has been consolidating (and
> especially if we're talking about broadband type services... okay,
> yes, dialup ISPs are more plentiful).

> Mail Service Providers are growing. They don't have to provide
Internet access; some people prefer buying unbundled services.

Many ISPs try (hard) to prevent their customers using other mail service
providers. Agreed that they are not likely to be totally successful at that.

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
Seth Breidbart
2005-01-02 20:04:52 UTC
Permalink
***@terabites.com wrote:

>> In order for it to eliminate zombies, it has to be implemented by
>>_everybody else_. That's not going to happen.
>
> Well, yes and no.
>
> I agree that you probably won't eliminate 100% of the vulnerability, but at some
> point the remainder doesn't much matter. The issue is whether the vulnerability
> is widespread enough to ATTEMPT to exploit it.

And that doesn't take much.

> It also depends on how quickly such a fine-grained permissions list
> approach is accepted and installed. Obviously, if Microsoft were to
> include something like this in Outlook and Outlook Express by
> default, it would be much more effective and much sooner than if
> Infopoint or some other small software company were to try to market
> it as an addon package.

You mean 5 years instead of 20?

>> You're assuming that you can tell whether or not the _recipient's_ MUA
>> will attempt to interpret a message as HTML.
>
> It doesn't much matter. Most spammers don't know, either, what specific E-mail
> client program a destination address is using. Nor do they care, in
> fact.

Precisely. The spammers will attack a vulnerability that affects only
a small percentage. You have to defend against all such
vulnerabilities.

That issue is already known in virus protection. Some email clients
were treating some messages as HTML that the anti-virus software
thought was (safe) plaintext. The spammer didn't care _which_ victim
he got, just _how many_.

>> For some spammers, maybe. Many others sell their services, and the
>> worldwide shortage of suckers is not expected soon.
>
> It won't take long for the word to get around that spamming doesn't work
> anymore, and that some types work dramatically less well than others.

And some spammers will just think it's good that the market is less
crowded, and spam more.

>> Prove it. Come up with a reasonable implementation that my mother can
>> handle.
>
> I'll be GLAD to do that, and I'll even bring it to release-ready, if
> you'll fund the development. :-)

In other words, you're claiming something that doesn't exist and
offering no evidence.

> The point is that this is IMPLEMENTATION-DEPENDENT, and does not
> need to be part of a "best practices" advisory. Some companies are
> hugely better at "human-engineering" software products than others
> are; given that, we don't have to concern ourselves HERE with the
> fact that some of them might not do a terrific job of it.

However, it's close to necessary to demonstrate that it's _possible_
to do an acceptably-good job.

> Again, though, I'll point out that once you default to "no
> attachments, no HTML" in E-mails from unlisted senders, you don't
> leave the spammers much room to evade much of anything, at least as
> far as their E-mail content goes.

As I've already pointed out, "no attachments, no HTML" isn't a
clear-cut decision, and spammers will take advantage of every
implementation difference of opinion.

>>> Sounds good, until the "spammer" can find a case (even just ONE)
>>> where the intended recipient actually WANTED the E-mail in question,
>>> and said user didn't feel they had in fact granted their ISP the
>>> 'right' to MIS-BLOCK mail that the user actually wanted.
>>
>> Then the ISP show the contract the user agreed to, and the law
>> granting it immunity.
>
> There are a LOT of contracts which judges end up NOT holding as
> enforceable.

That's because there are laws against such contracts. In this case,
there is a federal law (in the US) specifically allowing the ISP to
make such decisions, and immunizing it for doing so.

>> Mail Service Providers are growing. They don't have to provide
>> Internet access; some people prefer buying unbundled services.
>
> Many ISPs try (hard) to prevent their customers using other mail
> service providers. Agreed that they are not likely to be totally
> successful at that.

I haven't heard of _any_ doing that. Do you know of any who have
blocked the ports for SUBMIT, POP, etc.?

Seth
g***@terabites.com
2004-12-23 21:54:45 UTC
Permalink
> I've been hit by 1500 zombie pc's simultaneously pumping the same
exact spam at us.

> How cheap does cheap have to be to deal with an attack like that?

At SOME point, this is more a DDOS attack than a spam issue... although if we go
after unwanted/untrusted attachments and HTML, we will go a long way towards
solving the problem (at least for E-mail vectors) of recruiting these zombie
spambots.

There is clearly NO (local) solution for DDOS attacks, because by the time they
arrive at your entrance portal the damage has already been done.


Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
Laird Breyer
2004-12-24 03:53:20 UTC
Permalink
On Dec 23 2004, ***@terabites.com wrote:
> > I've been hit by 1500 zombie pc's simultaneously pumping the same
> exact spam at us.
>
> > How cheap does cheap have to be to deal with an attack like that?
>
> At SOME point, this is more a DDOS attack than a spam
> issue... although if we go after unwanted/untrusted attachments and
> HTML, we will go a long way towards solving the problem (at least
> for E-mail vectors) of recruiting these zombie spambots.

That's the whole point. Spammers are DDOSing the SMTP servers. We're
discussing ways to protect against such DDOS attacks. But looking at
message contents is useless - by the time your SMTP server gets to see
the message, the resources are committed.

> There is clearly NO (local) solution for DDOS attacks, because by
> the time they arrive at your entrance portal the damage has already
> been done.

Available information before that point is limited. You can refuse to
accept the connection based on IP address, or you can accept the
connection tentatively and do some cheap lookups. Before the message
DATA arrives, you have three pieces of information: a HELO (or EHLO)
identification string, the MAIL and RCPT strings (There's also VRFY or
EXPN but keep it simple). Once the DATA command is used, the
connection is no longer cheap, because the data can be arbitrarily
long, and doing any sort of analysis on it will cost unpredictable
resources anyway. So discussing body analysis is irrelevant for this
problem.

--
Laird Breyer.
Barry Shein
2004-12-25 00:17:10 UTC
Permalink
Without belaboring the specific technical suggestions being made,
which seem to me suffer at the very least from new and ever-changing
definitions of heretofore well-understood terms such as
"scaleability", I don't think finding ways to reduce the cost of
handling the ever-growing crush of spam is progress towards any
solution.

Such suggestions, along with announcements of relevant hardware
advancements (ram disks or whatever) are no doubt interesting from a
practical point of view to system administrators but don't strike me
as progress in the context of this group.

But feel free, I'm just explaining why I'm done with this particular
thread.

It's kinda like dealing with muggings by suggesting ways to make money
faster than muggers can steal it.

--
-Barry Shein

Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
The World | Public Access Internet | Since 1989 *oo*
Laird Breyer
2004-12-25 00:21:28 UTC
Permalink
That's a long winded way of saying you'd rather not give out
mail volume statistics on a public forum. A simple refusal would have
saved a lot of unnecessary arguing about unknown unknowns :-)

--
Laird Breyer.

On Dec 24 2004, Barry Shein wrote:
>
> Without belaboring the specific technical suggestions being made,
> which seem to me suffer at the very least from new and ever-changing
> definitions of heretofore well-understood terms such as
> "scaleability", I don't think finding ways to reduce the cost of
> handling the ever-growing crush of spam is progress towards any
> solution.
>
> Such suggestions, along with announcements of relevant hardware
> advancements (ram disks or whatever) are no doubt interesting from a
> practical point of view to system administrators but don't strike me
> as progress in the context of this group.
>
> But feel free, I'm just explaining why I'm done with this particular
> thread.
>
> It's kinda like dealing with muggings by suggesting ways to make money
> faster than muggers can steal it.
>
> --
> -Barry Shein
>
> Software Tool & Die | ***@TheWorld.com | http://www.TheWorld.com
> Purveyors to the Trade | Voice: 617-739-0202 | Login: 617-739-WRLD
> The World | Public Access Internet | Since 1989 *oo*
g***@terabites.com
2004-12-23 22:21:59 UTC
Permalink
> And the invalid user query is a database hit (40 million+ users and
those keep changing very rapidly).

First, relatively few ISPs have valid username lists totalling 40+ million
usernames.

Even if they did, and if they average 12 bytes each, that's only maybe 500Mb of
RAM... about $50 worth.

It would be relatively trivial to put in (as another poster suggested) a proxy
server or other special-purpose machine which had a RAM copy of this list
(probably hashed) which could be queried virtually instantaneously... it could
(given the very low CPU requirement) even be a co-resident task on the mail
server machine.(which would at least eliminate the excess inter-machine
communication overhead, too). So it's not like the mail server has to go off
and do a grunt-slow Oracle search or something to verify a good E-mail
address...!

Even if the ISP used a traditional database for their subscriber list, nothing
would prevent them from also sending a message (even by E-mail!) to update the
mail server when subscriber E-mail address information changes.

So anyhow, I -really- don't see this is a big deal.

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
g***@terabites.com
2004-12-24 01:19:06 UTC
Permalink
>> The biggest, best move would be if we could get Microsoft
>> onboard, and/or some
>> company able to produce something that would piggyback onto
>> Outlook and Outlook
>> Express.

>> That would be the best way to make the biggest impact, the fastest.

>They're already in, at a much higher level.

Well, they are, but only half-heartedly at this point. Their typical approach
is to enable the user to block certain kinds of things, although the
restrictions feature is presently mostly unusable... if you need to be able to
accept HTML or ActiveX or attachments from ANY SINGLE sender, then you end up
having to basically turn it on globally.

If they'd set up finer gradations on what the user is willing to accept, AND
then to make those permissions variable, based upon who the sender is... then
they'd be a lot closer to the technique I believe in.

>Of course, the more
>secure they make their products, the less they will be exploited, but,
>there will *always* be exploits -- there will always be bots, IMHO.

Sure, but by slashing unexpected attachments and HTML, we can SLASH the arrival
of worms and viruses in E-mail, to something very nearly approaching zero. And
we could do that much EASILY. And it doesn't require ANY major global E-mail
infrastructure changes, nor does it require loading new antivirus files every 60
minutes.

(In fact, MY approach would also block most viruses and worms in E-mail EVEN
BEFORE *ANY* ANTIVIRUS PRODUCT was able to detect it...!)

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
g***@terabites.com
2004-12-25 01:56:24 UTC
Permalink
>> BTW, the proxy scenario I gave above isn't intended to be a ready
> solution, rather it's an argument to prop up my claim that handling
> the invalid requests can be handled scalably.

> All I get from it is that an in-memory hash or other fast lookup
algorithm, if it's not being used already, might speed up response
time assuming that your model improves on the resources which are
actually strapped.

> That doesn't address scalability, it just shifts the knee of the curve
much like a faster CPU or more memory might.

How much scaleability do you NEED?

> ...Scalability has to address:

> a) The critical resource(s)

Whatever it might be, it's CERTAINLY not the speed of searching for one of
(even) 50 million valid E-mail addresses in in-memory hash tables.

> b) Permutational effects

I don't consider those to be issues either... or if you're convinced they are,
perhaps you might explain how and why.

> Where (b) is often subtle and significant. For example, if you use N
servers you have to load balance between them. How does that
decision-making scale as N increases. It's often somewhere between
O(N^2) and O(N!).

It's easy (and VERY fast, a mere handful of machine instructions) to split such
a hash table... for example, you can do something as simple as index using the
first character of the E-mail address into an array to determine which of
(possibly) several hash table lookup machines that address would reside in.
Jeez, I can't believe we're even talking about ridiculous low-level, trivial
stuff like this here.

But the whole discussion is silly, since in fact the table would fit EASILY in a
single machine, probably along with the mail server itself, and wherever the
throughput gauntlet is, it wouldn't be THERE.

> At any rate, as fascinating as some might find it I sincerely don't
believe the problem with spam, even at the ingress, is going to be
solved or even ameliorated for very long by some improvement in
recipient validity processing.

Sure, at some hypothetical infinite level of spam ingress volume you end up with
what looks for all the world like a DDOS attack... like trying to drink out of a
firehose.

> It's like someone breaking your windows unchallenged and someone says
I know, let's find cheaper ways to make window panes!

This is part of why I believe the better solution is to consider NOT ONLY JUST
things like sender authorization (which in fact does damned near NOTHING to
prevent infected machines from pouring garbage E-mails, and worms/viruses for
that matter, onto the net using the infected machine's authorizations) but
rather approaches which largely negate the ability to commandeer machines using
E-mail messages (and that means, for the most part, HTML and attachments). As
long as spammers and hackers can readily infect (in a matter of a few hours)
millions of computers belonging to clueless Lusers, there is NOTHING that will
prevent them from using those zombie armies to launch massive waves of spams,
viruses, or even (for that matter) just ordinary simple DDOS attacks.

Corporations, NO MATTER HOW WELL PROTECTED THEIR DATA CENTERS SUPPOSEDLY ARE,
can do **nothing** to prevent such DDOS attacks, since once the garbage gets to
their data center, the damage has pretty much already been done. So THEIR
enterprise-level interests HAVE TO INCLUDE making sure that INDIVIDUAL USERS
have reasonable protection against stuff like that, too.

That's why it's so crucial that the protections be built into the most widely
available packages such as Outlook, Outlook Express, Eudora, Pegasus, etc etc.

Current-level Microsoft stuff allows you to turn off attachments (say, or HTML)
for EVERYBODY, which is ridiculously impractical because nearly EVERYBODY needs
to receive attachments of SOME sort, or HTML, at least occasionally, from
SOMEONE. So the result is that that limitation is turned off, and finally
accomplishes nothing. It needs to be something finer-grained, and way more
selective, that's left at the "protected/armored" condition for the *great*
majority of incoming messages.

Until it's made harder to infect armies of individual clueless Luser-owned
machines, there's truly LITTLE point in debating ANY of these other approaches,
because they can simply be overwhelmed with DDOS attacks (or spam that
effectively becomes that, in volume).

And, happily, while we're at it and putting the hammer down on HTML and
untrusted attachments, we also simultaneously make content-based anti-spam
filters WAY more effective than they typically are today, AND take away (in one
fell swoop) most of the tricks that spammers use to deceive. We also make it
FAR harder for spammers to not just SEND stuff cheaply in large volumes, but
make the probability of it ever being delivered and read vanishingly small...
bringing us far closer to the day when spammers will simply move off to another
type of more profitable and easier [scam] business model.

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
g***@terabites.com
2004-12-25 03:08:25 UTC
Permalink
>> There is clearly NO (local) solution for DDOS attacks, because by
> the time they arrive at your entrance portal the damage has already
> been done.

> Available information before that point is limited. You can refuse to
accept the connection based on IP address, or you can accept the
connection tentatively and do some cheap lookups. Before the message
DATA arrives, you have three pieces of information: a HELO (or EHLO)
identification string, the MAIL and RCPT strings (There's also VRFY or
EXPN but keep it simple). Once the DATA command is used, the
connection is no longer cheap, because the data can be arbitrarily
long, and doing any sort of analysis on it will cost unpredictable
resources anyway. So discussing body analysis is irrelevant for this
problem.

Maybe, but HEADER analysis might be helpful.

After the receiving mail server has gotten the complete full header, then it
knows not only who the intended recipient is but also who the stated sender is,
and that could permit it to establish what that recipient's permissions are for
that sender... and SOME of those, at least, could perhaps be enforced during the
time the incoming mail arrival is being done.

The easiest one to enforce is maximum allowed message size for that sender (or
default 'unknown' sender) since that one already has to be done anyhow if the
user's maximum inbox size is exceeded (and that isn't known, either, until too
much data comes in).

It's a little more complicated (but not HUGELY difficult) to have the receiving
server recognize message divisions and enforce (at least on a preliminary,
tentative basis) recipient permissions for attachments, HTML-burdened
attachments, and such.

But (although there is a potential cost savings from truncating such mails
earlier in the process, before they have been fully transferred) there is a
downside which may override the potential savings... and that is the recipient's
desire to be able to change their mind upon considering the "spam" filtering
decision... it's harder to do that, and go ahead and approve the message for
delivery (and possibly revising the filtering rules for that sender), if in fact
it was blocked or the server connection was dropped during transfer. It may be
that the bandwidth cost (and that's ultimately getting cheaper and cheaper) is
simply less costly than the added complexity of trying to be more clever, "too"
early in the process.

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
Devdas Bhagat
2004-12-25 07:50:51 UTC
Permalink
On 25/12/04 03:08 +0000, ***@terabites.com wrote:
> > > There is clearly NO (local) solution for DDOS attacks, because by
> > > the time they arrive at your entrance portal the damage has already
> > > been done.
>
> > Available information before that point is limited. You can refuse to
> > accept the connection based on IP address, or you can accept the
> > connection tentatively and do some cheap lookups. Before the message
> > DATA arrives, you have three pieces of information: a HELO (or EHLO)
> > identification string, the MAIL and RCPT strings (There's also VRFY or
> > EXPN but keep it simple). Once the DATA command is used, the
> > connection is no longer cheap, because the data can be arbitrarily
> > long, and doing any sort of analysis on it will cost unpredictable
> > resources anyway. So discussing body analysis is irrelevant for this
> > problem.
>
> Maybe, but HEADER analysis might be helpful.

I don't think I quite understand your point here.

A mail transaction typically goes like this:
<tcp connection> from high port to port 25 of the server
The server checks up the reverse DNS of the client.
S: 220 <server.hostname> [ESMTP]
C: HELO <client.hostname> (or EHLO for ESMTP)
S: 250 Ok
C: MAIL FROM:<***@example.com>
S: 250 Ok
C: RCPT TO:<***@example.net>
S: 250 Ok
... (client adds more RCPT TOs here)
C: Data
S: 354 Send CRLF.CRLF to end
Message headers

Message Body
.
S: 250 Ok
quit
S: 221 Ok
<tcp connection teardown>

Now, you know the actual message size only after the data was sent. If
the client sends a ehlo, it may send a size value, but that cannot be
known to be correct. The message headers and body are part of the
message itself, as part as SMTP is concerned.

>
> After the receiving mail server has gotten the complete full header, then it
> knows not only who the intended recipient is but also who the stated sender
> is, and that could permit it to establish what that recipient's permissions
> are for that sender... and SOME of those, at least, could perhaps be
> enforced during the time the incoming mail arrival is being done.

This is more complex, but doable. A restrictions lookup table keys off
the recipient address, and

>
> The easiest one to enforce is maximum allowed message size for that
> sender (or default 'unknown' sender) since that one already has to be
> done anyhow if the user's maximum inbox size is exceeded (and that isn't
> known, either, until too much data comes in).

Actually, this isn't known until delivery is attempted. No one needs to
have the mail storage system on the same server as the receiving MTA.
The complexity and fraility of trying to push quota overflow conditions
in realtime to external MTAs is bad enough that I would just allow for
bounces to be generated later.

>
> It's a little more complicated (but not HUGELY difficult) to have the
> receiving server recognize message divisions and enforce (at least on a
> preliminary, tentative basis) recipient permissions for attachments,
> HTML-burdened attachments, and such.

This requires a large amount of code to be pushed into MTAs. Parsing
text takes up CPU. Gobs of CPU. Just to show how much pre data filtering
can help (I wont quote names here, since I don't have permission to do
that yet):
A canadian ISP had 4 boxes handing 200K mails/day each on average, with
SpamAssassin filtering the content for suspicious mail. Note that this was
post recipient validation, with a few externsions filtered out
(exe/cpl/pif/.., the common viral vectors). SA was running on 9 boxes,
and those were barely able to keep up with the load (SA was daemonised).
By adding a single DNSBL (the cbl-sbl.spamhaus.org list), they reduced
the inflow to 80K mails/day per host on average, and need 6 SA boxes for
filtering comfortably.

I'll let you figure out the bandwidth and server savings, and the
administrative time saved on that (the admins had been trying for three
months at least to tune the SA boxes so that they would keep up).

> But (although there is a potential cost savings from truncating such mails
> earlier in the process, before they have been fully transferred) there is a
> downside which may override the potential savings... and that is the

When do you plan to show actual savings? There are no savings post DATA,
except the cost of putting that message in the temporary mailstore on
the server.

> recipient's desire to be able to change their mind upon considering the
> "spam" filtering decision... it's harder to do that, and go ahead and
> approve the message for delivery (and possibly revising the filtering

Have you considered that message may have more than one recipient, and
that enforcing different message sizes per recipient in the MTA is simply
not feasible (there is exactly one return value after data)?

> rules for that sender), if in fact it was blocked or the server connection
> was dropped during transfer. It may be that the bandwidth cost (and
> that's ultimately getting cheaper and cheaper) is simply less costly than
It is? Not as fast as the spam volume is going up.

> the added complexity of trying to be more clever, "too" early in the
> process.

Devdas Bhagat
Laird Breyer
2004-12-26 01:42:05 UTC
Permalink
On Dec 25 2004, ***@terabites.com wrote:
>
> Maybe, but HEADER analysis might be helpful. (snip)

I fail to see how any of these points address the specific issue being
discussed. A header is part of the DATA, it can be arbitrarily long
and arbitrarily complex to parse. You simply cannot expect to be able to
allocate a fixed amount of resources of all types if you intend to
read a header (or any number of body lines for that matter).

Moreover, being part of the DATA, if you intend to accept a single
header line, chances are you'll receive the whole message anyway due
to buffering, and what are you going to do with the full message? If
you intend to analyse it, it will be simpler overall to accept a connection
directly through a full SMTP server.

The only way to make the rejection of obvious invalid requests truly
scalable is if you can guarantee a small fixed cost in CPU cycles,
RAM, I/O. Then you have to run this on a dedicated system which
doesn't do unpredictable things like swapping memory pages, spawning
processes, etc. It has to work like a self contained appliance basically,
only then will you be sure that the resources used are proportional to
the number of incoming requests, ie it's scalable. That also means no
talking to databases, DNSBLs etc.

A mail system with this kind of front end still has to contend with
too much spam, but I think we can then discuss per user spam volume rather
than incoming SMTP bound volume. Making that point was my only aim in
this thread.

Message analysis has its place somewhere after the message is
accepted, but does nothing at all to reduce the SMTP load.

--
Laird Breyer.
g***@terabites.com
2004-12-26 01:38:54 UTC
Permalink
>> Maybe, but HEADER analysis might be helpful.

> I don't think I quite understand your point here.

> A mail transaction typically goes like this:
<tcp connection> from high port to port 25 of the server
The server checks up the reverse DNS of the client.
S: 220 <server.hostname> [ESMTP]
C: HELO <client.hostname> (or EHLO for ESMTP)
S: 250 Ok
C: MAIL FROM:<***@example.com>
S: 250 Ok
C: RCPT TO:<***@example.net>
S: 250 Ok
... (client adds more RCPT TOs here)
C: Data
S: 354 Send CRLF.CRLF to end
Message headers

> Message Body
.
S: 250 Ok
quit
S: 221 Ok
<tcp connection teardown>

Of course.

> Now, you know the actual message size only after the data was sent.

Well, you know the maximum OFFERED size by then.

IF during the transfer, the recipient's inbox overflows (and, if they are the
ONLY addressee there for the message!) you no longer have to be concerned about
how much more there might be in the wings... once you've decided that the
message is "too big" and is going to be bounced. You don't HAVE to keep
receiving the rest of it, at that point.

> If the client sends a ehlo, it may send a size value, but that cannot be
known to be correct. The message headers and body are part of the
message itself, as part as SMTP is concerned.

Sure, but that doesn't mean that the connection can't be dropped DURING the body
transfer.

>> After the receiving mail server has gotten the complete full header, then it
> knows not only who the intended recipient is but also who the stated sender
> is, and that could permit it to establish what that recipient's permissions
> are for that sender... and SOME of those, at least, could perhaps be
> enforced during the time the incoming mail arrival is being done.

> This is more complex, but doable. A restrictions lookup table keys off
the recipient address, and

Right.

>> The easiest one to enforce is maximum allowed message size for that
> sender (or default 'unknown' sender) since that one already has to be
> done anyhow if the user's maximum inbox size is exceeded (and that isn't
> known, either, until too much data comes in).

> Actually, this isn't known until delivery is attempted.

Right, at the final mail server.

> ...No one needs to have the mail storage system on the same server as the
receiving MTA. The complexity and fraility of trying to push quota overflow
conditions in realtime to external MTAs is bad enough that I would just allow
for bounces to be generated later.

Certainly a reasonable enough position.

>> It's a little more complicated (but not HUGELY difficult) to have the
> receiving server recognize message divisions and enforce (at least on a
> preliminary, tentative basis) recipient permissions for attachments,
> HTML-burdened attachments, and such.

> This requires a large amount of code to be pushed into MTAs.

Oh, not really. I wouldn't try to do 'full' content analysis there, but it
MIGHT make sense to do some levels of stuff (binary attachments and their
extensions, maybe).

> Parsing text takes up CPU. Gobs of CPU.

A lot of that depends on what you're trying to do, and what kind of tools you're
trying to do it with.

> Just to show how much pre data filtering can help (I wont quote names here,
since I don't have permission to do that yet):
A canadian ISP had 4 boxes handing 200K mails/day each on average, with
SpamAssassin filtering the content for suspicious mail. Note that this was
post recipient validation, with a few externsions filtered out
(exe/cpl/pif/.., the common viral vectors). SA was running on 9 boxes,
and those were barely able to keep up with the load (SA was daemonised).
By adding a single DNSBL (the cbl-sbl.spamhaus.org list), they reduced
the inflow to 80K mails/day per host on average, and need 6 SA boxes for
filtering comfortably.

Okay, but I wasn't suggesting anything nearly as complicated as SpamAssassin at
the ISP end (I've always said that this ought to be at the recipient end, where
processing power is cheaper and more plentiful).

SpamAssassin is probably written in Perl or something, and that language is NOT
particularly efficient (especially compared to something that's more powerful
and more efficient for textual analysis and pattern recognition, such as
SPITBOL).

Let's recall that SpamAssassin was DESIGNED to be run at the end-user machine,
and it didn't HAVE to be fast or efficient.

> I'll let you figure out the bandwidth and server savings, and the
administrative time saved on that (the admins had been trying for three
months at least to tune the SA boxes so that they would keep up).

I am not convinced that the effort is worthwhile, since it's so
implementation-dependent in a moving-target situation.

>> But (although there is a potential cost savings from truncating such mails
> earlier in the process, before they have been fully transferred) there is a
> downside which may override the potential savings... and that is the

> When do you plan to show actual savings? There are no savings post DATA,
except the cost of putting that message in the temporary mailstore on
the server.

Depends on how far up the chain you are, and how many forwards will be done
between the sender and the recipient, and how much data transfer can be avoided.
But like I said, any such savings may not be worth the extra complexity.

>> recipient's desire to be able to change their mind upon considering the
> "spam" filtering decision... it's harder to do that, and go ahead and
> approve the message for delivery (and possibly revising the filtering

> Have you considered that message may have more than one recipient, and
that enforcing different message sizes per recipient in the MTA is simply
not feasible (there is exactly one return value after data)?

I hadn't considered that, but it's certainly a valid point... and yet another
reason why (based on complexity cost) it simply might make the most sense to do
the filtering at the recipient end.

>> rules for that sender), if in fact it was blocked or the server connection
> was dropped during transfer. It may be that the bandwidth cost (and
> that's ultimately getting cheaper and cheaper) is simply less costly than

> It is? Not as fast as the spam volume is going up.

Obviously we want to reduce spam volume. My approach is all about that.

>> the added complexity of trying to be more clever, "too" early in the
> process.


Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
Laird Breyer
2004-12-26 02:00:07 UTC
Permalink
On Dec 26 2004, ***@terabites.com wrote:

(snip)

>
> Obviously we want to reduce spam volume. My approach is all about that.
>

From your arguments, I have the strong impression that your approach
is entirely unrelated to the problem being discussed in this
thread.

Your goal is to reduce the number of spam messages visible to end
users. That's a good goal, but has nothing to do with reducing the
amount of data processed by ISPs who operate SMTP server farms for
their customers. Let's not mix up the issues.

--
Laird Breyer.
Devdas Bhagat
2004-12-26 06:10:02 UTC
Permalink
On 26/12/04 01:38 +0000, ***@terabites.com wrote:
> >> Maybe, but HEADER analysis might be helpful.
>
> > I don't think I quite understand your point here.
>
> > A mail transaction typically goes like this:
> > <tcp connection> from high port to port 25 of the server
> > The server checks up the reverse DNS of the client.
> > S: 220 <server.hostname> [ESMTP]
> > C: HELO <client.hostname> (or EHLO for ESMTP)
> > S: 250 Ok
> > C: MAIL FROM:<***@example.com>
> > S: 250 Ok
> > C: RCPT TO:<***@example.net>
> > S: 250 Ok
> > ... (client adds more RCPT TOs here)
> > C: Data
> > S: 354 Send CRLF.CRLF to end
> > Message headers
>
> > Message Body
> > .
> > S: 250 Ok
> > quit
> > S: 221 Ok
> > <tcp connection teardown>
>
> Of course.
>
> > Now, you know the actual message size only after the data was sent.
>
> Well, you know the maximum OFFERED size by then.

Offered? The server shows a limit on the message size offered. We
already know that. The question is whether the client will honour that
limit (real mail servers supporting ESMTP do, and won't send the message
on at all).

>
> IF during the transfer, the recipient's inbox overflows (and, if they are the
> ONLY addressee there for the message!) you no longer have to be concerned
> about how much more there might be in the wings... once you've decided that
> the message is "too big" and is going to be bounced. You don't HAVE to keep
> receiving the rest of it, at that point.

You do have to receive the rest of it.

> > If the client sends a ehlo, it may send a size value, but that cannot be
> > known to be correct. The message headers and body are part of the
> > message itself, as part as SMTP is concerned.
>
> Sure, but that doesn't mean that the connection can't be dropped DURING
> the body transfer.

See the RFCs. It does mean exactly that.

Devdas Bhagat
der Mouse
2004-12-26 07:17:02 UTC
Permalink
>>> The message headers and body are part of the message itself, as
>>> part as SMTP is concerned.
>> Sure, but that doesn't mean that the connection can't be dropped
>> DURING the body transfer.
> See the RFCs. It does mean exactly that.

Well, it doesn't *mean* that, exactly.

I'm not sure which RFC specifies that the server is not permitted to
drop the connection if it feels like it, if indeed there is such a
specification (I can't recall one offhand, but I'm willing to
tentatively believe that there is one). However, the client must be
prepared to handle it if it happens; servers do crash.

It also seems like no less reasonable a thing to do than a lot of the
other things that are done when dealing with spam and associated
issues, including a number that violate the RFCs (such as non-relay
hosts dissecting message headers, or my own "half-open hell" tactic for
dealing with hosts that retry too fast).

/~\ The ASCII der Mouse
\ / Ribbon Campaign
X Against HTML ***@rodents.montreal.qc.ca
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Devdas Bhagat
2004-12-26 08:18:04 UTC
Permalink
On 26/12/04 02:17 -0500, der Mouse wrote:
> >>> The message headers and body are part of the message itself, as
> >>> part as SMTP is concerned.
> >> Sure, but that doesn't mean that the connection can't be dropped
> >> DURING the body transfer.
> > See the RFCs. It does mean exactly that.
>
> Well, it doesn't *mean* that, exactly.
>
> I'm not sure which RFC specifies that the server is not permitted to
> drop the connection if it feels like it, if indeed there is such a
> specification (I can't recall one offhand, but I'm willing to
> tentatively believe that there is one). However, the client must be
> prepared to handle it if it happens; servers do crash.

RFC 0821

Page 25, description of the quit command:
The receiver should not close the transmission channel until
it receives and replies to a QUIT command (even if there was
an error). The sender should not close the transmission
channel until it send a QUIT command and receives the reply
(even if there was an error response to a previous command).
If the connection is closed prematurely the receiver should
act as if a RSET command had been received (canceling any
pending transaction, but not undoing any previously
completed transaction), the sender should act as if the
command or transaction in progress had received a temporary
error (4xx).

RFC 2821 clarifies (section 4.1.1.10):
The receiver MUST NOT intentionally close the transmission channel
until it receives and replies to a QUIT command (even if there was an
error). The sender MUST NOT intentionally close the transmission
channel until it sends a QUIT command and SHOULD wait until it
receives the reply (even if there was an error response to a previous
command). If the connection is closed prematurely due to violations
of the above or system or network failure, the server MUST cancel any
pending transaction, but not undo any previously completed
transaction, and generally MUST act as if the command or transaction
in progress had received a temporary error (i.e., a 4yz response).

>
> It also seems like no less reasonable a thing to do than a lot of the
> other things that are done when dealing with spam and associated
> issues, including a number that violate the RFCs (such as non-relay
> hosts dissecting message headers, or my own "half-open hell" tactic for
> dealing with hosts that retry too fast).

However, this would play hell with hosts that play by the rules (ISP
outbound relay servers, for example, which need not be the same as their
submission servers).

Devdas Bhagat
g***@terabites.com
2004-12-27 02:45:47 UTC
Permalink
>>> Obviously we want to reduce spam volume. My approach is all about that.

>> From your arguments, I have the strong impression that your approach
is entirely unrelated to the problem being discussed in this
thread.

>> Your goal is to reduce the number of spam messages visible to end
users.

That's part of it, but only a part.

Let's not confuse spam QUANTITY (number of messages) with spam VOLUME (number of
messages * average size of a spam message).

> That's a good goal, but has nothing to do with reducing the
amount of data processed by ISPs who operate SMTP server farms for
their customers. Let's not mix up the issues.

I disagree.

First, if we were ever to reach the state where ZERO (or near zero) spam
messages reached end recipients, then spamming would be 100% without profitable
return to the spammers... it wouldn't continue for long under such conditions.

Second, if spammers get the message that spam containing attachments or that is
HTML-burdened has GREATLY REDUCED chances of being delivered and read, then
they're likely to stop using those approaches (and those approaches hugely
inflate the size of most spam messages). Thus, a likely HUGE reduction in spam
volume (including, in particular, that which ISPs receive and process).

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
Laird Breyer
2004-12-27 04:04:47 UTC
Permalink
On Dec 27 2004, ***@terabites.com wrote:

> >> Your goal is to reduce the number of spam messages visible to end
> users.
>
> That's part of it, but only a part. Let's not confuse spam QUANTITY
> (number of messages) with spam VOLUME (number of messages * average
> size of a spam message).

I don't see how you are arguing for any reduction of volume at ISPs,
since you want to accept all messages presented via SMTP, analyze them
and perhaps reject afterwards. What that represents is a reduction in
quantity visible to the end user only.

>
> > That's a good goal, but has nothing to do with reducing the
> amount of data processed by ISPs who operate SMTP server farms for
> their customers. Let's not mix up the issues.
>
> I disagree.
>
> First, if we were ever to reach the state where ZERO (or near zero)
> spam messages reached end recipients, then spamming would be 100%
> without profitable return to the spammers... it wouldn't continue
> for long under such conditions.

That isn't the issue discussed. Long before your client side solution
hopefully blocks 100% of spam for all users, ISPs must address
increasing volume or change their offerings.

Moreover, I am far from convinced that spammers will stop sending mail
just because near zero messages reach the recipients. Given unlimited
resources for free, the correct response to dwindling success rate is
to increase quantity. Whether people actually see spams in their
inboxes is irrelevant for ISPs, as they still have to find a way to
deal with the spams sent to them.

> Second, if spammers get the message that spam containing
> attachments or that is HTML-burdened has GREATLY REDUCED chances of
> being delivered and read, then they're likely to stop using those
> approaches (and those approaches hugely inflate the size of most
> spam messages). Thus, a likely HUGE reduction in spam volume
> (including, in particular, that which ISPs receive and process).

No. Spam volume isn't controlled by HTML, you'd have to outlaw the
MIME standard, otherwise people and spammers can send image
attachments, Word attachments etc. But outlawing MIME won't help
anyway, as people can fall back on uuencode type methods, which are
plain text. The huge reduction in volume you're hoping for based on
message structure will never happen.

--
Laird Breyer.
g***@terabites.com
2004-12-27 20:03:22 UTC
Permalink
>>>> Your goal is to reduce the number of spam messages visible to end
> users.

>> That's part of it, but only a part. Let's not confuse spam QUANTITY
> (number of messages) with spam VOLUME (number of messages * average
> size of a spam message).

> I don't see how you are arguing for any reduction of volume at ISPs,
since you want to accept all messages presented via SMTP, analyze them
and perhaps reject afterwards. What that represents is a reduction in
quantity visible to the end user only.

You missed the (a?) key point,. One spammers realize that (bulky) attachments
and (bulky) HTML (or large E-mails!) are the kiss of death for the spam they're
sending, and will make it FAR less likely that their spam will be read, they'll
be motivated to send smaller E-mails with neither spam nor attachments. Thus,
the reduction in overall, aggregate spam bulk.

>>> That's a good goal, but has nothing to do with reducing the
> amount of data processed by ISPs who operate SMTP server farms for
> their customers. Let's not mix up the issues.

>> I disagree.
>
> First, if we were ever to reach the state where ZERO (or near zero)
> spam messages reached end recipients, then spamming would be 100%
> without profitable return to the spammers... it wouldn't continue
> for long under such conditions.

> That isn't the issue discussed.

It ABSOLUTELY is.. since that is ULTIMATELY the way we want to reduce spam
volume. :-)

> Long before your client side solution hopefully blocks 100% of spam for all
users,

1) It doesn't have to be 100% to have a substantial impact.

2) A client-side solution can be implemented FASTER since it doesn't require a
globsl concurrence, and doesn't require reworking the whole Internet.

> ...ISPs must address increasing volume or change their offerings.

They clearly have a vested interest in helping make sure that their customers
solve this problem (much like the way AOL has started providing free antivirus
software to all of their customers).

By the same token, AOL could similarly make a BIG impact by updating their
E-mail client software to include a fine-grained, by-sender permissions list
approach such as I propose. This would by itself make a huge impact on spamming
profitability, and could be done quickly.

> Moreover, I am far from convinced that spammers will stop sending mail
just because near zero messages reach the recipients.

Some people still aren't convinced we really landed men on the moon in 1969,
either. The fact that some people will always be skeptical isn't a good reason
to stick with the status quo,

> Given unlimited resources for free,

That's why the permissions list idea makes a MAJOR strike against E-mail
transmission of worms and viruses. This will shut the door (at least regarding
E-mail) on zombie recruitment. SPF and other DNS-based approaches do
**NOTHING** to control zombie recruitment, which is a big reason why those
approaches are doomed to failure.

> the correct response to dwindling success rate is
to increase quantity.

"If at first you don't succeed, try, try again." But it goes on to "...but
eventually, quit... there's no point being a damned fool about it." :-)

The cost of spamming (while small) is NOT zero. At some point, as the return to
be earned from spamming shrinks, it is no longer profitable to do it. There are
also increasing dangers of successful lawsuits (as in the recent case of $1B
being levied against the first three spammers in a group of 300...)

> Whether people actually see spams in their
inboxes is irrelevant for ISPs, as they still have to find a way to
deal with the spams sent to them.

Right, but spams have no value if people don't see them. If people don't see
them, eventually spammers will stop sending them, and thus ISPs will benefit
(and greatly) by reduced success rate from spamming.

>> Second, if spammers get the message that spam containing
> attachments or that is HTML-burdened has GREATLY REDUCED chances of
> being delivered and read, then they're likely to stop using those
> approaches (and those approaches hugely inflate the size of most
> spam messages). Thus, a likely HUGE reduction in spam volume
> (including, in particular, that which ISPs receive and process).

> No. Spam volume isn't controlled by HTML, you'd have to outlaw the
MIME standard, otherwise people and spammers can send image
attachments, Word attachments etc.

HTML-burdened e-mails are typically 3-5x bulkier than non-HTML-burdened e-mails
with the same content.

Yes, attachments are also bulky, and the permissions-list approach would (by
default) block or quarantine E-mails containing attachments (images, Word
documents, etc) from unknown senders, too. So those would cease to be a useful
plan for spammers wanting to get their E-mails read.

> But outlawing MIME won't help anyway, as people can fall back on uuencode type
methods, which are plain text.

Note that we're not talking about 'outlawing MIME', but it WOULD involve only
allowing MIME attachments from senders you trust (and even then, only the TYPES
of attachments that you trust them to send you). And I wouldn't have a problem
with considering UUENCODED (or similar) content as an "inline attachment" and
thus not allowing it (by default) from untrusted senders, even if it is
supposedly represented as being "plain text".

> The huge reduction in volume you're hoping for based on
message structure will never happen.

Back in 1977 (December 1st, in fact), when we publicly announced the ARC System,
my friends told me, "Gordon, you're crazy... big business is never going to give
up their mainframes and run their processing on networks of little computers!"

My answer at the time was "...YOU JUST WATCH!!!" And time, I think, has borne
me out.

I've had skeptics before. As the old saying goes, "he who laughs last, laughs
best."

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
Devdas Bhagat
2004-12-27 20:39:53 UTC
Permalink
On 27/12/04 20:03 +0000, ***@terabites.com wrote:
> You missed the (a?) key point,. One spammers realize that (bulky)
> attachments and (bulky) HTML (or large E-mails!) are the kiss of death
> for the spam they're sending, and will make it FAR less likely that
> their spam will be read, they'll be motivated to send smaller E-mails
> with neither spam nor attachments. Thus, the reduction in overall,
> aggregate spam bulk.

I suggest looking up what happened to the spam volume as the end user
filters got more effective.

>
> >>> That's a good goal, but has nothing to do with reducing the
> > amount of data processed by ISPs who operate SMTP server farms for
> > their customers. Let's not mix up the issues.
>
> >> I disagree.
> >
> > First, if we were ever to reach the state where ZERO (or near zero)
> > spam messages reached end recipients, then spamming would be 100%
> > without profitable return to the spammers... it wouldn't continue
> > for long under such conditions.
>
> > That isn't the issue discussed.
>
> It ABSOLUTELY is.. since that is ULTIMATELY the way we want to reduce spam
> volume. :-)

Actually not. The bulk of the spam comes from spammers selling the "get
rich quick by spamming" idea to others. And these people get money per
hit, so they will just increase volume to the point where your filtering
servers die, or you turn off the filters.

"The enemys gate is down".

> > Long before your client side solution hopefully blocks 100% of spam for all
> users,
>
> 1) It doesn't have to be 100% to have a substantial impact.
>
> 2) A client-side solution can be implemented FASTER since it doesn't
> require a global concurrence, and doesn't require reworking the whole
> Internet.
>
> > ...ISPs must address increasing volume or change their offerings.
>
> They clearly have a vested interest in helping make sure that their customers
> solve this problem (much like the way AOL has started providing free
> antivirus software to all of their customers).
>
> By the same token, AOL could similarly make a BIG impact by updating their
> E-mail client software to include a fine-grained, by-sender permissions list
> approach such as I propose. This would by itself make a huge impact on
> spamming profitability, and could be done quickly.

Hmmm, maybe ISPs should STOP offering email at all. If you want email,
run your own server or pay someone else to run it for you. Give people
static IPs, and let them loose.

> > Moreover, I am far from convinced that spammers will stop sending mail
> just because near zero messages reach the recipients.
>
> Some people still aren't convinced we really landed men on the moon in 1969,
> either. The fact that some people will always be skeptical isn't a good
> reason to stick with the status quo,
>
> > Given unlimited resources for free,
>
> That's why the permissions list idea makes a MAJOR strike against E-mail
> transmission of worms and viruses. This will shut the door (at least
> regarding E-mail) on zombie recruitment. SPF and other DNS-based
> approaches do **NOTHING** to control zombie recruitment, which is a big
> reason why those approaches are doomed to failure.

SPF is doomed to failure for other reasons. Let me put it this way:

ANY SOLUTION WHICH TRIES TO WORK AFTER DATA DOES NOT SCALE.

After you have accepted data, your best option is to deliver spam to the
end user and let their Bayesian/other filters handle it. Stop it before
data, or let the end user handle it.

> > the correct response to dwindling success rate is
> to increase quantity.
>
> "If at first you don't succeed, try, try again." But it goes on to "...but
> eventually, quit... there's no point being a damned fool about it." :-)
>
> The cost of spamming (while small) is NOT zero. At some point, as the
> return to be earned from spamming shrinks, it is no longer profitable
> to do it. There are also increasing dangers of successful lawsuits (as
> in the recent case of $1B being levied against the first three spammers
> in a group of 300...)

If you can actually collect...

> > Whether people actually see spams in their inboxes is irrelevant for
> > ISPs, as they still have to find a way to deal with the spams sent to them.
>
> Right, but spams have no value if people don't see them. If people don't see
> them, eventually spammers will stop sending them, and thus ISPs will benefit
> (and greatly) by reduced success rate from spamming.

You assume that spammers will respond to lowered delivery rates by
stopping malicious behaviour. Previous history shows otherwise.
Those who do not learn from history...

<snip>

> HTML-burdened e-mails are typically 3-5x bulkier than non-HTML-burdened
> e-mails with the same content.
>
> Yes, attachments are also bulky, and the permissions-list approach would (by
> default) block or quarantine E-mails containing attachments (images, Word
> documents, etc) from unknown senders, too. So those would cease to be a
> useful plan for spammers wanting to get their E-mails read.

Convenience against security? Security will lose.

<snip>
> I've had skeptics before. As the old saying goes, "he who laughs last,
> laughs best."

So, how soon can we expect the spammers to stop spamming, given that
they have the bigger guns, and the economic advantage (it is cheaper to
send mail via proxies than to receive it)?

The cost advantage of the smaller computer is no longer on the side of
the legitimate servers. See my first quote in this mail.

Devdas Bhagat
Hannigan, Martin
2005-01-03 01:06:12 UTC
Permalink
Did you mean small percentage of vulns "on the list" or vulnerable machines?

They can't easily control the inection rate, but they can control the
utilization rate. One thing they want is non rbl listed zombies. This is a
good tactic re infect many, list them, then silence until needed.

-M

---
Martin Hannigan
***@verisign.com
Verisign, Inc.


-----Original Message-----
From: asrg-***@ietf.org <asrg-***@ietf.org>
To: ***@ietf.org <***@ietf.org>
Sent: Sun Jan 02 12:04:52 2005
Subject: Re: [Asrg] SICS

***@terabites.com wrote:

>> In order for it to eliminate zombies, it has to be implemented by
>>_everybody else_. That's not going to happen.
>
> Well, yes and no.
>
> I agree that you probably won't eliminate 100% of the vulnerability, but
at some
> point the remainder doesn't much matter. The issue is whether the
vulnerability
> is widespread enough to ATTEMPT to exploit it.

And that doesn't take much.

> It also depends on how quickly such a fine-grained permissions list
> approach is accepted and installed. Obviously, if Microsoft were to
> include something like this in Outlook and Outlook Express by
> default, it would be much more effective and much sooner than if
> Infopoint or some other small software company were to try to market
> it as an addon package.

You mean 5 years instead of 20?

>> You're assuming that you can tell whether or not the _recipient's_ MUA
>> will attempt to interpret a message as HTML.
>
> It doesn't much matter. Most spammers don't know, either, what specific
E-mail
> client program a destination address is using. Nor do they care, in
> fact.

Precisely. The spammers will attack a vulnerability that affects only
a small percentage. You have to defend against all such
vulnerabilities.

That issue is already known in virus protection. Some email clients
were treating some messages as HTML that the anti-virus software
thought was (safe) plaintext. The spammer didn't care _which_ victim
he got, just _how many_.

>> For some spammers, maybe. Many others sell their services, and the
>> worldwide shortage of suckers is not expected soon.
>
> It won't take long for the word to get around that spamming doesn't work
> anymore, and that some types work dramatically less well than others.

And some spammers will just think it's good that the market is less
crowded, and spam more.

>> Prove it. Come up with a reasonable implementation that my mother can
>> handle.
>
> I'll be GLAD to do that, and I'll even bring it to release-ready, if
> you'll fund the development. :-)

In other words, you're claiming something that doesn't exist and
offering no evidence.

> The point is that this is IMPLEMENTATION-DEPENDENT, and does not
> need to be part of a "best practices" advisory. Some companies are
> hugely better at "human-engineering" software products than others
> are; given that, we don't have to concern ourselves HERE with the
> fact that some of them might not do a terrific job of it.

However, it's close to necessary to demonstrate that it's _possible_
to do an acceptably-good job.

> Again, though, I'll point out that once you default to "no
> attachments, no HTML" in E-mails from unlisted senders, you don't
> leave the spammers much room to evade much of anything, at least as
> far as their E-mail content goes.

As I've already pointed out, "no attachments, no HTML" isn't a
clear-cut decision, and spammers will take advantage of every
implementation difference of opinion.

>>> Sounds good, until the "spammer" can find a case (even just ONE)
>>> where the intended recipient actually WANTED the E-mail in question,
>>> and said user didn't feel they had in fact granted their ISP the
>>> 'right' to MIS-BLOCK mail that the user actually wanted.
>>
>> Then the ISP show the contract the user agreed to, and the law
>> granting it immunity.
>
> There are a LOT of contracts which judges end up NOT holding as
> enforceable.

That's because there are laws against such contracts. In this case,
there is a federal law (in the US) specifically allowing the ISP to
make such decisions, and immunizing it for doing so.

>> Mail Service Providers are growing. They don't have to provide
>> Internet access; some people prefer buying unbundled services.
>
> Many ISPs try (hard) to prevent their customers using other mail
> service providers. Agreed that they are not likely to be totally
> successful at that.

I haven't heard of _any_ doing that. Do you know of any who have
blocked the ports for SUBMIT, POP, etc.?

Seth
Seth Breidbart
2005-01-03 05:03:24 UTC
Permalink
"Hannigan, Martin" <***@verisign.com> top-posted:

> Did you mean small percentage of vulns "on the list" or vulnerable
> machines?

If even a small percentage of machines are vulnerable, multiply by a
lot of machines and they get a lot of infected machines.

> They can't easily control the inection rate, but they can control
> the utilization rate. One thing they want is non rbl listed
> zombies. This is a good tactic re infect many, list them, then
> silence until needed.

Or set the rent for a non-listed machine higher, and try each victim
from a listed machine, if that doesn't get through, then use a
non-listed machine.

Seth
Hannigan, Martin
2005-01-03 07:12:59 UTC
Permalink
You've got it, for the most part. They also consider how well
connected the machines are i.e. faster link, more spam. They
also know who's watching and read ASRG and others.

-M<

--
Martin Hannigan (c) 617-388-2663
VeriSign, Inc. (w) 703-948-7018
Network Engineer IV Operations & Infrastructure
***@verisign.com



> -----Original Message-----
> From: asrg-***@ietf.org [mailto:asrg-***@ietf.org]On Behalf Of
> Seth Breidbart
> Sent: Monday, January 03, 2005 12:03 AM
> To: ***@ietf.org
> Subject: Re: [Asrg] SICS
>
>
> "Hannigan, Martin" <***@verisign.com> top-posted:
>
> > Did you mean small percentage of vulns "on the list" or vulnerable
> > machines?
>
> If even a small percentage of machines are vulnerable, multiply by a
> lot of machines and they get a lot of infected machines.
>
> > They can't easily control the inection rate, but they can control
> > the utilization rate. One thing they want is non rbl listed
> > zombies. This is a good tactic re infect many, list them, then
> > silence until needed.
>
> Or set the rent for a non-listed machine higher, and try each victim
> from a listed machine, if that doesn't get through, then use a
> non-listed machine.
>
> Seth
>
> _______________________________________________
> Asrg mailing list
> ***@ietf.org
> https://www1.ietf.org/mailman/listinfo/asrg
>
g***@terabites.com
2005-01-04 00:51:55 UTC
Permalink
[zombies]

>>> In order for it to eliminate zombies, it has to be implemented by
>>_everybody else_. That's not going to happen.

>> Well, yes and no.

>> I agree that you probably won't eliminate 100% of the vulnerability, but at
some point the remainder doesn't much matter. The issue is whether the
vulnerability is widespread enough to ATTEMPT to exploit it.

> And that doesn't take much.

I disagree. If the great majority of users are rejecting E-mail-borne zombie
attacks, then I think that hackers and spammers will move to greener pastures
(such as trying to infect via Web sites or whatever).

>> It also depends on how quickly such a fine-grained permissions list
> approach is accepted and installed. Obviously, if Microsoft were to
> include something like this in Outlook and Outlook Express by
> default, it would be much more effective and much sooner than if
> Infopoint or some other small software company were to try to market
> it as an addon package.

> You mean 5 years instead of 20?

I don't agree with your numbers, but the ratio might be about right. A small
software company is going to have a much harder time getting rapid worldwide
distribution.

Such an effort will be implemented worldwide by far more customers, faster, if
it's an upgrade to a program they already have and trust than if it's something
new and different.

>>> You're assuming that you can tell whether or not the _recipient's_ MUA
>> will attempt to interpret a message as HTML.

>> It doesn't much matter. Most spammers don't know, either, what specific
E-mail client program a destination address is using. Nor do they care, in
fact.

> Precisely. The spammers will attack a vulnerability that affects only
a small percentage.

True, but as it becomes a smaller and smaller percentage, their exploit becomes
less and less a matter of concern.

> You have to defend against all such vulnerabilities.

Oh, if you can, I suppose. But if you're waiting to get 100% before getting 80%
that you can get sooner, I think that's a poor implementation choice.

> That issue is already known in virus protection. Some email clients
were treating some messages as HTML that the anti-virus software
thought was (safe) plaintext. The spammer didn't care _which_ victim
he got, just _how many_.

At some point, the answer to "how many" will be "not enough"... just the same
way that spammers don't tend to go after Linux or Mac boxes, even though they
clearly could. The grass is greener elsewhere, so the minority situation can be
pretty safely ignored.

>>> For some spammers, maybe. Many others sell their services, and the
>> worldwide shortage of suckers is not expected soon.

>> It won't take long for the word to get around that spamming doesn't work
> anymore, and that some types work dramatically less well than others.

> And some spammers will just think it's good that the market is less
crowded, and spam more.

Sure, for a while. But the fine-grained permissions list is still the most
likely solution to yield a quick, lasting reduction in spam volume and
virus/worms too, and with a minimum impact on legitimate users.

Ultimately, the way to control spam is to make it less profitable and appealing
for the spammer. The profitability doesn't HAVE to go to zero... it only has to
go low enough that a spammer or virus author will decide that other approaches
are more appealing.

>>> Prove it. Come up with a reasonable implementation that my mother can
>> handle.

>> I'll be GLAD to do that, and I'll even bring it to release-ready, if
> you'll fund the development. :-)

> In other words, you're claiming something that doesn't exist and
offering no evidence.

Anyone who has spent any time working in R&D is quite used to designing products
that don't exist yet. My professional record for conceiving, designing and
successfully implementing practical, usable, innovative products that are the
first-of-kind is pretty good. :-)

>> The point is that this is IMPLEMENTATION-DEPENDENT, and does not
> need to be part of a "best practices" advisory. Some companies are
> hugely better at "human-engineering" software products than others
> are; given that, we don't have to concern ourselves HERE with the
> fact that some of them might not do a terrific job of it.

> However, it's close to necessary to demonstrate that it's _possible_
to do an acceptably-good job.

I think that's pretty evident. Certainly neither you nor anybody else here has
posted convincing evidence proving the inverse.

There are MUCH, MUCH harder problems of software engineering that have been
solved. This one is comparatively quite trivial.

>> Again, though, I'll point out that once you default to "no
> attachments, no HTML" in E-mails from unlisted senders, you don't
> leave the spammers much room to evade much of anything, at least as
> far as their E-mail content goes.

> As I've already pointed out, "no attachments, no HTML" isn't a
clear-cut decision,

It comes awfully close.

And if the implemention of the fine-grained permissions list is built into the
recipient's mail client software (say, Outlook/Outlook
Express/Pegasus/Eudora/etc) then it's possible to make it damned near
airtight... certainly something that a clueless user (the one most likely to be
fooled) isn't likely to be taken in by.

> ...and spammers will take advantage of every implementation difference of
opinion.

Sure. They are awfully determined, and devious. But (again) the *great*
majority of their tricks and deceptions are based on obscured URLs, scripting,
HTML, and attachments. Once you have denied them those, they're left with VERY
little wiggle room.

Likewise, if you don't allow HTML or attachments in mail from
unfamiliar/untrusted sources, and executable content (by HTML or attachment)
from fewer-to-NO sources, then virus/worm/zombie infestation (at least by E-mail
transmission vectors) is essentially going to go away.

[That is SUCH an obvious conclusion that I'm astounded that we have to keep
bringing up the point!! You'd *think* that at least THAT part of my proposal
would receive pretty much universal agreement... especially when you compare its
simple, logical basis with the outlandishly convoluted and complicated
Rube-Goldberg-esque schemes that others here seem to be so fond of!]

Gordon Peterson http://personal.terabites.com/
1977-2002 Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections! http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.
Seth Breidbart
2005-01-04 04:55:01 UTC
Permalink
***@terabites.com wrote:

> [zombies]

> I disagree. If the great majority of users are rejecting
> E-mail-borne zombie attacks, then I think that hackers and spammers
> will move to greener pastures (such as trying to infect via Web
> sites or whatever).

Some will. Some will anyway. Some won't, so long as a sufficient
number (which can be quite a small percentage) of users are
vulnerable. And more users will be vulnerable to new attacks, because
defenses won't be perfected against those.

>>> It also depends on how quickly such a fine-grained permissions list
>>> approach is accepted and installed. Obviously, if Microsoft were to
>>> include something like this in Outlook and Outlook Express by
>>> default, it would be much more effective and much sooner than if
>>> Infopoint or some other small software company were to try to market
>>> it as an addon package.
>
>> You mean 5 years instead of 20?
>
> I don't agree with your numbers, but the ratio might be about right.

How many users do you think are still using Windows 98?

> Such an effort will be implemented worldwide by far more customers,
> faster, if it's an upgrade to a program they already have and trust
> than if it's something new and different.

Maybe; if their hardware and OS support the new version, and it
doesn't break anything they're already doing (including compatibility
with other old programs they're using), and . . .

>> Precisely. The spammers will attack a vulnerability that affects only
>> a small percentage.
>
> True, but as it becomes a smaller and smaller percentage, their
> exploit becomes less and less a matter of concern.

I disagree. So long as their exploit spreads, it's of concern.

>> That issue is already known in virus protection. Some email clients
>> were treating some messages as HTML that the anti-virus software
>> thought was (safe) plaintext. The spammer didn't care _which_ victim
>> he got, just _how many_.
>
> At some point, the answer to "how many" will be "not enough"... just
> the same way that spammers don't tend to go after Linux or Mac
> boxes, even though they clearly could. The grass is greener
> elsewhere, so the minority situation can be pretty safely ignored.

Until there's enough protection in what used to be easiest, then the
spammers go for something else.

> Ultimately, the way to control spam is to make it less profitable
> and appealing for the spammer.

Felony convictions have that effect. Little else seems to.

>> However, it's close to necessary to demonstrate that it's _possible_
>> to do an acceptably-good job.
>
> I think that's pretty evident. Certainly neither you nor anybody
> else here has posted convincing evidence proving the inverse.

Ask anyone who has done customer support.

>> As I've already pointed out, "no attachments, no HTML" isn't a
>> clear-cut decision,
>
> It comes awfully close.

Until there's a new MUA that decides to be "helpful".

> And if the implemention of the fine-grained permissions list is
> built into the recipient's mail client software (say,
> Outlook/Outlook Express/Pegasus/Eudora/etc) then it's possible to
> make it damned near airtight... certainly something that a clueless
> user (the one most likely to be fooled) isn't likely to be taken in
> by.

But by then the email has travelled the last mile.

>> ...and spammers will take advantage of every implementation difference of
>> opinion.
>
> Sure. They are awfully determined, and devious. But (again) the *great*
> majority of their tricks and deceptions

SO FAR

> are based on obscured URLs, scripting, HTML, and attachments. Once
> you have denied them those, they're left with VERY little wiggle
> room.

and will have to come up with something else, which so far they've
always managed to do.

Seth
Continue reading on narkive:
Loading...