Brad Ideas - Spamcop blacklists autoresponders - Comments

Where the bad links come from

Carol Bush — Tue, 29 Aug 2006 23:42:55 -0700

I was in a forum discussion earlier today and told a guy named Kelly about a bad experience I had with SpamCop. Some spammer was putting MY site URL in their spam (along with many others) to saturate the URLs they wanted in the content of their spam. SpamCop complained to my ISP and my site came down for 12 days while I sorted it out with tech support. Some of the bloggers were calling this shotgun reporting and a menace to the Internet. I agree. SpamCop should not trust URLs in the body of e-mails as always being spam. What! Are they just grabbing all the links and e-mail addresses they can outta the spam? How wrong is that! They should do a little homework.

So you are saying

brad — Tue, 04 Jul 2006 12:43:54 -0700

In order to avoid being unfairly blacklisted, use the unfair blacklist yourself, and not offer delivery to those on it?

This would possibly make sense for a better blacklist, one that follows well-established principles of justice (presumption of innocence, right of to defend yourself, right to see evidence, right of appeal etc.) but not for spamcop.

What if the server your

Anonymous — Mon, 03 Jul 2006 11:54:16 -0700

What if the server your autoresponder is on checks against Spamcop before allowing the mail to come in and BE auto-responded to? Wouldn't that prevent the mail from a spammer from getting responded to in one of Spamcop's anonymous spam traps.

Backscatter spewing - Spamarrest.com

jm — Mon, 26 Jun 2006 19:58:40 -0700

Spamarrest.com, your IP's are bloked at my system due to many, many c/r backscatter attempts. Your Challenge mails go straight to quarantine now. Sometimes I skim over the quarantine, and if I see one, I confirm it.

Such standardization is

brad — Thu, 12 Jan 2006 10:28:23 -0800

Such standardization is already in place (there's an RFC) and in a later blog post you will find more proposals on this very topic.

CR and scaling

WhiteShep — Thu, 12 Jan 2006 05:42:31 -0800

I wrote a CR filter for my OS/2 mail server in 2002. It was increadibly effective but I took it offline to prevent the challanges from being classed as spam.

But I do want to explain some of the reailty of C/R as a spam threat. First off most forged spam e-mails we get are non-existing e-mails. They do not go to a single user. Secondly my (and I assume other) C/R programs will only send one C/R to any email address a week. Since spam addressed are 99% repeated that cuts C/R responces down way farther (even for us getting tens of thousands of spam a day). This does not include a whitelist (auto built from your users SMTP sending)and bayesian filter wich catches most spam before it even hits CR.

The problem is if you have a huge comercial spammer (debatable) who can hit enough C/Rs they can flood an account with a lot of C/R responces. Thus the argument by blacklists is all C/R's should be treated as so better than the original spammer.

Here is the thing. Pulling back from the knee jerk reaction for easy management. If a huge spammer is spamming THAT many addresses with a forged e-mail the forged users account is going to get flooded to beyond belief with flames and angery responces anyways making C/R the least of that account's problems.

Solution: If it was possible for this idea to come forward (without being blacklisted for even suggesting it). I think one solution would be to "standardize" for what is concidered a acceptable C/R responce. Example:

C/R: should start the subject line.
The message must be text and be no larger than 512 bytes. Only one link mail or e-mail in the message and it must go to the same domain as the sending SMTP mail server.

Reasoning: The C/R subject would allow mailers (just as bayesian preforms test emails now) to flag for testing C/R emails (even limit # recieved in X time). This format would work and be too restrictive for most spammers. However it would require the adoption of standards .

Another idea I have been kicking around (which would be easier to implement into existing systems) would be to set up a volunteer C/R server (similar in idea to the blacklists). Only legit registered servers could relay C/R through our server. Our server would then apply the "tests above" to each C/R. No more than 5 C/R's would ever be sent to one e-mail address in an hour. A delay to a ISP's message queue (time to delay queue after so many C/Rs sent at once) based on ISP size to prevent a hacked ISP from spamming (before being caught).

This server model could be distributed via a server farm or participating servers (DNS (round robin) and MX).

I feel this method would give ISP's a "safe source to whitelist" for C/Rs, prevent any abusive volume of mail from C/Rs, and safe guard spammers from abusing the system.

Are you talking about the

brad — Mon, 09 Jan 2006 11:25:46 -0800

Are you talking about the forgery problem? C/R at most doubles the volume of mail currently being handled. However, if your spam filter is already classifying 80% of the mail you get, which it had better, then C/R is only adding 20% to the total volume of mail (with most of the challenges discarded.)

I fail to see anything that doesn't scale about that. The only thing that doesn't scale is the autoresponses to forged addresses, if those victim addresses don't have their own suitable filtering. Beyond that, how can anything that simply does a small linear multiple not scale? "Not scaling" refers to something that gets harder to handle (per unit) the more units you have, not something that remains linear.

CR and scaling

Suresh Ramasubramanian — Mon, 09 Jan 2006 05:06:26 -0800

> This is something I see happening perhaps a few times a week
> if the whole world used C/R.

Please, Brad. Hotmail, Yahoo, AOL, etc add something like several hundred accounts to their service an hour. There's no way CR is going to ever keep pace with it. Even after using CR as a last layer of protection against spam.

http://www.hserus.net/images/minute.png - that's about a million emails a minute rejected, and 100k emails a minute accepted / passed by our filters. Now, even if a fraction of those incoming emails were greeted by CR ... and our mail farm (with 40 million users) is barely a third the size of AOL's.

NO way CR is going to scale in such a situation.

CR doesn't scale?

brad — Fri, 06 Jan 2006 12:10:41 -0800

Some badly implemented C/R schemes may not scacle, but C/R, as implemented according to my best practices, scales fine — leaving aside, for a moment the issue of what to do about challenging mail with fake addresses.

In a world where everybody used C/R, you would only get a challenge when:

You email somebody you have never mailed to or from before, AND
Your message is not clearly identified as non-spam by the recipient’s spam filters, AND
You don’t take any extra steps that become adopted, such as including a cpu-stamp, or signing your mail

This is something I see happening perhaps a few times a week, if the whole world used C/R. That scales fine. There is an issue when you change E-mail addresses. Then you will get a new slew of challenges, temporarily. But it’s not giant burden. In such cases I would recommend the adoption of tools to bypass all anti-spam including C/R because most spam filters do use some level of whitelisting, and you give up your whitelist privs when you change E-mails.

Non delivery is a failure

Suresh Ramasubramanian — Fri, 06 Jan 2006 03:28:29 -0800

Non-delivery is a serious failure of the mail system. It must not go unreported. Some would argue it should be delivered to both parties. It could make sense for the sender to decide who to deliver it to, though you canâ€™t easily stop the recipient from superseding that.

I agree. That's what smtp has DSNs for. Any 550 error message (which turns into a mailer-daemon notification in the sender's mailbox) we issue has something like "Mail refused, please see http://spamblock.outblaze.com/202.54.30.2" (or whatever the blocked IP is).

Similarly for blocked domains, or for other filter rules. And that's accompanied by a fairly easy auto removal mechanism that can be used just once - and a link to contact the site postmaster (me and my staff) - and we respond to tickets reporting false positive blocks within a business day, or even sooner.

Blocking is going to keep happening, and C/R bots just dont scale .. but blocking should be done responsibly and should be blocked by people who are willing to listen and respond fast to false positive reports.

srs

I know people disagree with c/r

brad — Thu, 05 Jan 2006 11:04:01 -0800

But autoresponding isn’t going away, it’s too useful in bounces after mx, for mailing list confirmation, confirmation of email addresses entered on web pages etc.

The answer is not blacklisting autoresponders, but working to fix the autoresponse to forgery problem. Autoresponses normally don’t multiply spam like open relays, they just reflect it. That’s not good for the person it is reflected at, of course. But the autoresponse does not advertise the product so the spammer is not interested in it. He’s just putting in a forged From to get past whitelists and detectors of invalid From lines.

For autoresponses to emails, we may need to move to a regime where those who want autoresponses sign their mail. However, long before that we could move to some standization in automated responses, so that it’s easy to detect autoresponses to messages you never sent out, and be rid of them.

C/R is worth protecting because it is the only system that can turn an anti-spam filter into a no-false-positive filter. The correct approach is to discard spam you are sure is spam, pass through what you are 99.9% sure is ham, and challenge the small quantity of stuff you can’t figure out.

Some people say, “I don’t like challenges” but if you pose the question, “would you rather have a challenge, or would you rather your mail was discarded or put into spam folder that may or may not get looked at?” — the answer is different. I sent the mail for a reason, and I want a chance to override the spam filter if it decides not to deliver it. The message might be important.

One could also consider a flag to say “Don’t bother challenging me” for those who don’t want a challenge if their mail is about to be not delivered.

Non-delivery is a serious failure of the mail system. It must not go unreported. Some would argue it should be delivered to both parties. It could make sense for the sender to decide who to deliver it to, though you can’t easily stop the recipient from superseding that.

If the algorithms can’t figure out whether to deliver, only a person can.

I dont use the spamcop blocklist, but ..

Suresh Ramasubramanian — Thu, 05 Jan 2006 05:18:03 -0800

C/R systems are more or less the smtp equivalent of what a network admin would describe as a smurf amp. As others posting here have pointed out, while they may reduce your spam levels to some extent, one thing they serve to do is to generate double the unwanted traffic when someone's being forged into spam.

That, plus lots of mailing list admins just dont like being at the receiving end of C/R systems .. Dave Farber was saying he'd unsub anybody who sent a C/R challenge back in response to IP email, for example.

Even with all the steps that are being suggested (first running spam through a bunch of filters before you run a CR bot on it as a last resort) it is not a good idea. And there's even more fun when it conflicts with another technique that I dont like too much - graylisting. Watch a c/r bot end up sending challenges back to a mailserver that uses graylisting, its a highly entertaining (!) experience, I assure you.

On a tangent from your original post, responding to Daryin. I'm too much of a fan of SPF either. Two interesting posts on circleid -

http://www.circleid.com/posts/spf_loses_mindshare/ (by John Levine)
http://www.circleid.com/posts/port_25_blocking_or_fix_smtp_and_leave_por... (which I wrote sometime before John wrote his article)

We were the first email provider to publicly stop publishing spf records, even conservative ones. This was back in February 2005. Earthlink followed suit a few months after we did - in July or August 2005.

-srs

Nice blog Mr Templeton

Ikhyun — Wed, 04 Jan 2006 06:02:55 -0800

I visited your photography page and saw the gallery of photos taken from around the world. Nice work especially on the Australia photos - keep up the good work!

Use A Filter on Top

Roy Schestowitz — Tue, 03 Jan 2006 23:25:50 -0800

Reducing the amount of challenges is probably the right move to take. You can reduce the likelihood of banishment in this way, as well as become less of a nuisance to the Net. In other words, try ruling out cases when messsages are rather obvious spam. It leads to lower volume of messages being dispatched, which in turn can avoid blacklisting.

I use SpamAssasin, which is active at a layer higher than challenge/response (in this case Apache with BoxTrapper). Whatever gets scored as spam will be put aside in a mail folder which is reserved for spam. Only messages not marked as spam (and not in the whitelist either) will have a challenge delivered. This cuts down the number challenges by about 70% in my case. It never entails any false positive because I set the thresholds high.

Join the club

Brent Chapman — Tue, 03 Jan 2006 21:33:23 -0800

As near as I can tell, GreatCircle.com is more-or-less permanently stuck on the SpamCop blacklist unless I want to stop running Majordomo. Yeah, like that's gonna happen...

-Brent

Spamcop blacklists autoresponders

brad — Mon, 02 Jan 2006 23:02:33 -0800

I learned a couple of days ago my mail server got blacklisted by spamcop.net. They don’t reveal the reason for it, but it’s likely that I was blacklisted for running an autoresponder, in this case my own custom challenge/response spam filter which is the oldest operating one I know of.

I understand the debate about the merit of C/R spam filters. Like all autoresponders, they can generate unwanted mail when spammers and viruses send mail with a forged From address, and the responder annoys the innocent victim. However, this is a problem common to all autresponders, and unlike the even-more-hated open-relay, it doesn’t magnify the spam problem — there is one possibly annoying response per spam, not hundreds.

I am bothered because I don’t want to see anti-spam advocates fighting other anti-spam methods because they don’t agree with them, or blacklists in general used to punish people you don’t agree with. Spamcop should be fighting spammers, not anti-spammers.

In addition, e-mail autoresponse is an important mail tool. In fact, anti-spammers insist that mailing lists do a confirmed opt-in (also known as double opt-in), generally by autoresponse, before adding a person to a mailing list. When a mail server bounces directly delivered mail it can avoid doing an autoresponse, but if mail comes in through an MX — a vital feature of mail — it requires an autoresponse to bounce it. Vacation programs and many other tools use this ability.

Check to see if your mail system uses spamcop.net as a blacklist. If it does, disable it or switch to something else until they change this policy. Otherwise you won’t receive mail from me, and many others.

Update: My server is no longer blacklisted. I didn’t do anything (other than this blog post and a few complaints to people using the spamcop BL) so perhaps they auto remove. But it could happen again at any time until they change their policy. This is also a nasty DOS attack. Find anybody with any autoresponder, including a bounce of MX’d mail. Send forged mail to it with a From set to a spamtrap address — and they’re blacklisted. Also can be used against any sites that have you enter an E-mail address on a web page and then email that address to confirm you own it — you can get these sites blacklisted trivially. Every web form that can enter an E-mail address is at risk.