Brad Ideas - Tempfailing for spam -- where does it lead - Comments

Main reason

brad — Mon, 26 Feb 2007 11:05:21 -0800

I don't do that is I want to keep the blog posting rate to just a few a week. Many people think a good blog has many postings a day, but I prefer lower volumes.

Ask Brad?!

Anonymous — Mon, 26 Feb 2007 02:41:05 -0800

Why don't you set up an "Ask Brad" feature, either here or elsewhere.
The idea would be that instead of only you initiating threads in your
blog, i.e. blog entry, comments and responses, readers could ask for
your advice, and your answer could be handled like a normal blog entry,
i.e. spawn comments and responses etc.

Of course, you could screen the entries before posting the questions and/or
your responses, require registration for those asking questions etc.

The idea is that many readers of your blog have common interests and often
readers ask questions in the comments to your blog entries. The only difference
is that the readers (with your screening and approval) could start a new blog
thread.

Let me start (in this thread since it has to do with spam): Most people don't
have time to manually screen their spam. What do you recommend: using SpamAssassin
or something similar to mark messages as potential spam, so that the user can then
take whatever action he wants (delete them all, screen all by hand, apply additional
processing (semi)automatically (perhaps dependent on the score) but otherwise ACCEPTING
all spam messages, or, on the other hand, rejecting spam messages above a certain
spam threshold?

With some sort of automatic rejection (which I would like to implement for spams with
a score so high that I can be sure that I am not rejecting any non-spam messages), do
you recommend sending an email to the (perhaps forged) sender, saying that the message
could not be delivered because it is suspected of being spam? The reason I ask is that
my provider can let me have all incoming mail scanned by SpamAssassin before it is
delivered to me. (More precisely, all mail using a particular set of MX records. My port
25 is still open so that other email addresses are possbile which can be delivered bypassing
the spam screening.) Above a certain score set by him (5 SpamAssassin points at the moment),
it is tagged as spam. If I want, I can reject mail above a certain threshold. If this happens,
the (perhaps forged) sender will get an email saying that the mail couldn't be delivered because
it is suspected of being spam. I have no control over this.

I see two potential problems. One is backscatter spam. A spammer uses the addresses he wants
to spam to as the forged senders, sends to an address known to be screened for spam, and thus
has the spam assassin forward the message to his recipients. My provider doesn't seem worried
by this. Should I be worried, since my address might appear in the spam? That is, a spammer
sends me an email with your address as the sender. My provider sends a message to you (perhaps
including the original message) saying it is spam, and you see me as the original recipient and
suspect I might have something to do with the spam.) (If you do similar spam screening, then
one email might start a vicious circle and multiply until the systems are overloaded!) In other
words, is sending an email to the apparent sender a good idea (maybe the message was mistakenly
tagged etc)? (My provider should perhaps be concerned with all these informational emails and/or
with being tagged as a spam relay because of backscatter himself.)

The other problem would occur if, instead of sending information emails, the provider would send
some sort of error code during the SMTP dialog. This is OK for real spammers, but might be a problem
for people which forward email to me---they shouldn't get any error codes which indicate that they
might be sending spam. (For example, I moderate a newsgroup, and posts to the newsgroup go to another
site which emails them to the active moderators. If spam comes in, the relay site should not experience
any problems.)

Spam prize

brad — Fri, 09 Feb 2007 17:49:09 -0800

Right now anti-spam is a big business, not much need for a prize.

In my plan all that’s needed is a large enough network of spare servers to act as these MXs that you can be sure at least one of the ones you use will be up, and the network has the capacity to handle the real mail. The machines are sharing via an internal channel the frequency counts of mail attempts from the billion untrusted IPs, and start to simply refuse connections for a while from IPs sending mail too fast, so their mail load can be managed. But it will still be large.

However, since speed is only a secondary consideration it’s something to easily run on spare servers, older servers, machines sitting around etc. You could also run one of your own if you like but generally you would just subscribe to somebody else’s. (You could also run one of your own on a 2nd IP address used by your own mail server, so no new hardware is needed, and just nice the process down low.)

The servers do delay mail by long enough to send and receive the regular updates on mail volumes over IP blocks. I haven’t designed a protocol for that but I think a fairly efficient one should be possible.

But again the key gain is the rest of the world doesn’t change their SMTP senders at all, not right away. Very little breaks because you install it (only large mailing list hosting on unknown sites.) Most other anti-spams tend to break things right away for large numbers of innocents, which makes them poor choices.

dynmic IPs

Phillip Helbig — Fri, 09 Feb 2007 15:04:37 -0800

I suspect that many people, like myself, who don't send email
directly from dynamic IPs can't do so because someone they
want to send email to has rejected their mail since it comes from
a dynamic IP address. Technically, there is no reason I couldn't
send email directly. Many probably don't host mailing lists
for the same reason. I can avoid the problem by sending through
the SMTP relay server provided by my dynamic-DNS provider, at
negligible cost and with no need for SMTP authentication (since
he knows my current IP anyway). From my point of view, I just
need to change one item in my SMTP configuration.

A static IP address is another option. That would cost more than
using the SMTP relay server at the provider.

Right now, I still receive email directly. I could channel that
through my dynamic-DNS provider as well, but I lose some control
in the process. Still, I might go this route and let him sort out
or rather tag the spam rather than implementing more anti-spam
measures at my end. (I can adjust the anti-spam level rather
finely along a scale, but not decide on whether to use one method
or another, if I let him do it. If I do it myself, I can also
experiment with various methods.)

While my scheme (everyone using a trusted server to send email
through and rejecting email from elsewhere) would cut out spam
from getting through, your scheme would actually reduce the
volume of spam. Dynamic-IP folks could still send through a
trusted server, as I do now.

It seems to me that your scheme does offer many advantages, but
someone needs to run these magic MX servers. Suppose I opt to
have my provider have the main MX record for me (i.e. route all
incoming mail through him) and try to convince him to run one of
these magic servers. If someone tried to send a lot of spam to
my addresses, or other addresses he handles, then they would get
slowed down. However, a typical spammer probably has a more or
less random list of addresses, meaning they would go through
many MX servers. So, to get off the ground, a lot of folks would
have to start cooperating at once; it can't be built up piece by
piece.

Is there an out-of-the-box, turnkey solution for people who would
be interested in running such a server?

I read that Richard Branson is offering 25 million for someone who
finds a method of disposing of excess carbon dioxide. Why not
convince him (or someone similar) to offer a similar prize (perhaps
somewhat smaller) for coming up with such an anti-spam solution.

Some folks do lose money through spam (due to the time and effort
applied to filtering etc); I think it might not be that difficult
to get enough people together to offer quite a substantial reward
for coming up with a foolproof anti-spam system.

Dynamic IP

brad — Fri, 09 Feb 2007 13:18:13 -0800

Dynamic IPs are indeed more difficult to put on trusted lists. It's not impossible, but it's a lot harder. Most people with dynamic IPs don't send directly today, and they don't host mailing lists. In fact, most ISPs these days have taken to not even letting dynamic IP hosts use port 25, you have to manually ask them to remove the block -- if you can even ask. However, their mail will still work, just through the throttles.

The goal of the plan is for most of the main SMTP servers in the world to join the trusted lists, so that 99% of the non-spam flows peer 2 peer as it should. Then 1% of the real mail, and almost all the spam, goes through the throttle servers.

OK, it's clearer now

Phillip Helbig — Fri, 09 Feb 2007 12:00:55 -0800

Thanks for the explanation. OK, some things are clearer
now. However, it seems that I could no longer receive
email directly, i.e. have an MX record pointing to my IP
address or, if I did, I would have to reject stuff from
non-trusted senders. Right? (This is certainly possible.
In fact, I could do it now. But then I have to have someone
not just carry the MX record, but route all of my mail through
his server.) Also, would it be possible for a dynamic IP address
to get on the trusted list? It seems that these folks will
always be on the non-trusted list, unless an ISP wants to vouch
for all of his addresses, in which case he would probably require
customers to send email through his own SMTP relay server.

Alas, once again no.

brad — Thu, 08 Feb 2007 10:30:11 -0800

In the system:

People who want to use it to filter spam set up special low priority MX records.
People who know they are on the trusted list are encouraged to, but not required to, modify their SMTP sender to ignore the special MX records (they have a magic name)
People who are not on the trusted list, or who have a rules-conformant SMTP sender relay mail through the magic MXs. It delivers single mail but not bulk mail.
People who try to bypass the special MX servers who are not on the trusted list are rejected immediately, perhaps with errors the first few times for politeness.
People who are on the trusted list with a modified SMTP sender just send mail directly to the lowest priority MX that does not have the magic name. Ie. it’s just the same as before. Peer to peer SMTP, highly efficient.

If most people join the trusted list (and all list hosts need to) then mail operates just as today. No sender is forced to change a thing which is essential. For trusted senders, it’s as though the throttles are not there (just like today).

I still don't get it

Phillip Helbig — Thu, 08 Feb 2007 09:21:23 -0800

Presumably, you mean that no-one has to change anything because
in your scheme this will be handled transparently by MX records
which belong to this "white hat" MX network. This DOES prevent
me from receiving direct connections to my port 25, though.
(I could still receive them, but most would be spam.) So, someone
has to run the MX server so that I can get email addressed to me.
I don't see any advantage of this as opposed to someone running a
server which I can send email out through.

Your system will work if the MX servers do the throttling. What,
other than the wish to be a good guy, is their motivation to do
this?

How, as someone with a dynamic IP address, can I avoid the
disadvantages of being suspected of spamming if I send several
legitimate emails within a short time?

Again, I think your system will work IF enough MX servers are
willing to operate as you propose. Are you aware of any which
are?

With your scheme, people stop getting spam if they have such an
MX server send them mail. I think this is considerably more work
than finding an MX server to send mail out through.

No, it doesn't

brad — Wed, 07 Feb 2007 17:46:19 -0800

With my system you make no change to your mail system. You don’t change any configuration or any practice (as long as you don’t host mailing lists). For those not hosting mailing lists, nothing breaks, nothing fails, nothing stops working.

Making mail stop working for everybody until they reconfigure their servers is a non-starter.

What am I missing?

Phillip Helbig — Wed, 07 Feb 2007 11:44:50 -0800

What am I missing? With your system as well, people have
to change the way they send mail. They have to send to this
MX network. They can't send directly to port 25 of my (or
any) address. Thus, your system makes people change the how they
send mail as well.

Again, most of my spam is coming from virus-infested PC, spambots
etc, most on volatile IP addresses, who are sending directly to
my port 25 on my IP address. The only way to avoid these emails
is to drop the connections. The only way for real mail to get through
is if I accept connections from trusted relay servers. If other
people do the same, the only way for my mail to get out is to send
it through a trusted server.

All of this applies to both schemes.

I think the only difference is the way the trusted servers
operate. My scheme: non-spammers get someone to relay their
mail (I pay for this, but there are other possibilities). This
server rejects attempts to relay through it from non-customers.
Your scheme: the servers relay everything, but unknown stuff
is throttled. Spammers will realise this, and will send directly.
It will only get stopped if people accept email only from trusted
servers.

With your scheme, there are problems for people who legitimately
send many emails. For normal users, either the server has
to keep track of when the last email from a particular address
arrived, or it slows stuff down coming from unknown addresses.
The former is probably too much work. The latter will slow down
legitimate email from people without a static IP address.

With most people having a volatile address these days, one would
have to use SMTP authentication or the owner of the relay server
would have to know my current IP address. I think your scheme
would only work for someone with a fixed IP address.

The key is

brad — Wed, 07 Feb 2007 11:16:12 -0800

You don't have to break anybody's ordinary email. Your system forces people to change how they send mail. You want a system where the default is to do nothing, and everything still works, unless you are hosting a mailing list.

OK, all clear now

Phillip Helbig — Tue, 06 Feb 2007 12:08:41 -0800

I've read the article several times. :-)

OK, to use your scheme I have to stop accepting
connections directly to my port 25 and accept them
only from "trusted servers". That's essentially
the same thing that I have in mind. With your scheme,
I can send email myself, but only through a trusted
server (at least if I am sending to someone wanting
to avoid spam and who thus only accepts mail from trusted
servers), and I should accept connections also only from
trusted servers (unless I want to get spam again). Under
my scheme, it's essentially the same: I send through a
trusted server (provided by my dynamic-DNS provider) and,
if I drop connections from non-trusted servers, essentially
also accept email only from trusted servers. Not much
difference really.

The only difference is that, under your scheme, an unknown
could send an email into the MX network and it would
eventually get through. In my scheme, he could do so
with no penalty if he sent it through a trusted server.

Do you know of anyone running such a throttling server who
invites everyone to send email through it? If I had the
resources to run such a server, I would rather relay email
from people I know to be trustworthy (and who contribute to
the cost) rather than accepting from anyone and throttling
unknowns (most of which are spammers).

In both schemes, to stop spam computers have to stop accepting
port 25 connections from "just anywhere" and accept them only
from trusted servers. In both schemes, email has to be sent
through trusted servers (one which will relay legitimate email
immediately in my case and not even accept other connections,
one which will relay known senders quickly and unknowns slowly
in your case). In practice, my scheme already works for those
who want to use it. Do you know of anyone running these
throttling servers?

I think your scheme would work well if it existed, but do you
see any way for it to come about? My scheme can be built up
piece by piece. More and more people will stop accepting
connections from arbitrary addresses to stop spam, and accept
them from trusted servers. There is a motivation to send email
through trusted servers, and a market and competition. For example,
dynaccess.com offers a "reasonable" number of emails as part of a
flat fee (and it is possible to register special sender addresses
for sending newsletters etc) with access based on IP address, whereas
dyndns.org offers a micropayment-per-email scheme using
authentication.

Read the article

brad — Mon, 05 Feb 2007 10:39:25 -0800

I suggest you read the article, it goes into all these details I think.

Sites don't accept e-mail connections from unknowns connecting directing. Unknowns can only go through the top priority MX records which point to the MX network because those are the only sites that will even let them open an SMTP connection. They can try random addresses all they want, it won't do any thing for them.

random MX records

Phillip Helbig — Mon, 05 Feb 2007 03:38:20 -0800

I think some spammers have already caught up on this one.
Anyone can do a SHOW MX on an address and get one or more
MX records. Normally, the highest-priority should be taken,
and if that fails, move on to the next etc. Apparently, some
spammers are selecting them randomly, or even going for the
lowest-priority one on the assumption that the most spam checking
will be done on the highest-priority records. (This assumption is
not always true, of course, but as long as it is true more often than
it is false, it is worth it to the spammers.)

The large MX network that you mention, though, can be used by me
only if I use it for my MX records. It won't help me at all if
I continue to receive email directly (i.e. not only receive it
directly on my computers, but have someone sending me email make
a direct connection to my port 25).

If this network comes into existence and I choose to use it, then
in effect I am letting someone else decide what gets through and
what does not. Apparently SOME spam would still get through.
If I am going to let someone else decide anyway, I don't see any
reason not to choose someone who will just reject spam outright
(rather than throttling it), according to criteria I choose (my
dynamic-DNS provider offers such a service). I don't have any
more qualms about dropping connections which are a) from known
spamming IP addresses or b) which are trying to transmit spam
outright than you do setting up spam filtering for comments to
your blog. You don't let spammers and bots into your blog because
it is "free speech"; why the different stance on email? Yes, your
scheme would allow someone to send legitimate mail from any IP
address. How often does one need to do this? I think most people
send from the same computer most of the time (if it has a dynamic
IP address, then this "same computer" might be one other than their
own) and/or use SMTP-authentication.

It's detailed in my essays

brad — Fri, 02 Feb 2007 13:07:31 -0800

But in effect, there is a magic low-score MX record which is a network of throttling relays. Sites that know they are trusted ignore that MX record and mail you directly (and get through because they are trusted.) Sites of unknown status follow the spec and mail that MX and are throttled if not trusted. Non-trusted sites mailing directly, disregarding the spec, don't even open a socket.

This requires a large network of cooperating sites ready to run this large MX network as a central front against spam. However, as the goal of the machines is to be slow and throttling, that is not a hard engineering problem. (At least the being slow part.)

Tempfailing for spam -- where does it lead

brad — Fri, 26 Jan 2007 15:27:04 -0800

One growing technique for use in anti-spam involves finding ways to “fail” on initial contacts for sending mail. Real, standard conformant mail programs try again in various ways, but spammers, in writing their mail blasters, tend to just have them skip that address and go to the next one in their list.

Two common approaches include simply returning a “temporarily unavailable” status on any initial mail attempt that might be spam. Another approach is to have dead MX records both at the “try first” and “try last” end of the MX chain.

Why does this work? Spammers just want to deliver as much mail as possible given time and bandwidth. If one address fails for any reason, it’s really no different whether you spend your resources trying the address again or in a different way, or just move on to the next address. In fact, since many of the failures are real failures, it’s actually more productive to just move on.

And, I admit, some of the spam filtering tools I make use of use these techniques, and they do help. But what exactly are they doing? For spammers, the limiting factor is bandwidth. Dealing with failures, especially timeouts on dead servers, takes very little of their resources.

It doesn’t reduce the amount of spam they send, at least by much, it just redistributes it to those who don’t use the techniques. For a positive spin, you can liken it to putting up a higher fence than your neighbour, so the criminals attack them and not you. For a negative spin, you can imagine it as being like an air filter that filters out the pollution on air coming into your house, and spews it out the back at your neighbours.

So it’s a touch question. Is this approach a good idea? Especially at the start, it was very effective. Over time if it becomes very common spammers will see a reduction in spam they deliver and make fairly simple moves to compensate for it. Is this fair game or antisocial?

There is an old joke about two hikers who meet a bear. The first sits down and starts putting on his running shoes. The other says, “What are you doing, you can’t outrun a bear!” and the first says, “I don’t have to outrun the bear, I just have to outrun you.”

Are we passing the bear onto our neighbours?

(This is part of a larger question of some of the other negative consequences of anti-spam. For example, as text filters got better, spammers moved to sending their spam as embedded images which filters could not easily decode. The result is more and more bandwidth used, both by spammers and victims. Was it a victory or a loss?)