New Essay on Autoresponder practices

Topic: 

I wrote earlier this week on the discovery that people were blacklisting sites with email autoresponders. More thought and debate on the issue has led to a number of thoughts over how to solve the issues around autoresponders, in particular the concern that they will respond to messages with forged From addresses.

These thoughts have been laid out in this essay on practices for autoresponders which starts off by pointing to RFC3834, and goes further in a world where people might want to blacklist sites just for autoresponding.

The RFC specfies a way for an autoreponse to be reliabily identified as such. Those who are blacklisting or filtering autoresponders can use this so that if they are going to go about blacklisting a site for running an autoresponder (as is required in the SMTP spec) that they only blacklist further autoresponses, and not ordinary mail from the same server. While some blacklisters, unfortunately, have a capricious disregard for the consequences of their actions, most of them agree that they should wish to block as little legitimate, desired mail as possible, ideally zero, so techniques which can make this happen deserve their attention.

There are many other techniques outlined in my essay on challenge-response best practices which are still not followed (admittedly in a few cases even by my own code, since I never put it into public distribution.) These techniques make C/R not only workable, but I believe a must in any good anti-spam system. If somebody's anti-spam system is going to block my mail, I want the ability to know about it and reverse that decision by proving I'm not a robot. While it is annoying to have to respond to a challenge, if the alternative is not having your mail read, most people would take the challenge -- if it was really necessary. C/R systems allow systems to have no false positives, at least for non-anonymous mailers, and that should be the goal for everybody.

Comments

I honestly believe the challenge-response system is going to be the only spam-free way of doing things going into the future. How can we combat the millions of spammers with software solutions when they'll always find a way into your inbox?

Only spam free for you, perhaps. However that is as I and others have pointed out, an idea that thrives on inconveniencing other, innocent parties in an effort to make your mailbox spamfree.

As for millions of spammers, there's no way CR is going to scale to millions of mailboxes.

If you have the patience to plow through about 35 pages of text, please try http://www.oecd.org/dataoecd/5/47/34935342.pdf - it might help. And saves me trying to summarize that lot in a followup to a blog post.

Brad, that last comment on your blog post - an ad for a DVD of Magnum PI - seems to have been posted AFTER the guy got through your C/R.

And there's an entire cottage industry in cheap labor places like the Phillipines, that do nothing except respond to C/R challenges and type in fuzzy words from captcha protected signups, at a dollar for a hundred captchas, or something similar.

C/R isn't too much of a defense against other labor intensive spam operations, such as Nigerian spam. It is just that the volume of C/R right now is just too damned low for you to notice any steps at all that spammers are taking to do an end run around them. But if (and that's a big if) C/R does get more popular, you'll notice that its trivially easy to game it.

That kind of gaming of C/R and captchas is entirely besides the elementary scaling problems that captcha faces.

I tend to call it "Challenge Response Authentication Procedures" btw .. makes for a nice, catchy acronym, that.

Of course humans can get past C/R. Truth is humans can get past anything. If we build a spam system so good a human being can't get past it, we're surely blocking tons of real mail.

People forget that our paper mailboxes get half a dozen pieces of junk mail every day, even though in that world it costs 50 cents to a buck to print and mail each piece. But 6 pieces of junk mail, once a day, while annoying, is only that -- annoying. It doesn't overwhelm the mail system.

As noted, C/R is the system to use when you can't identify an incoming mail as spam or not spam. If you make a solid identification either way (for example it is talking about millions of dollars held in a nigerian bank account) you don't challenge it.

My simple prompt (not really a challenge) on this blog has a human spammer get "past" it once or twice a week. A low enough rate that I manually delete them fairly quickly. Frankly, I don't know why they do it, since I put "nofollow" on all the links in posts so the search engines ignore the links they enter.

However, the 1 to 2 is not unexpected. You don't want a perfect anti-spam system. What you want is a perfect delivery system, that delivers real mail as reliably as possible, while getting rid of the most spam that it can. C/R as the final stage of a spam filtering tool is the best currently available approach, if the forgery problem can be handled.

C/R is going to retain that level of utility only as long as it doesnt get popular enough for spammers to actively devoting their energies to try game it. They've learnt enough about captchas to game them .. and a whole lot of CR bots are heavily tied to captcha technology.

But this is something where I guess we'll have to agree to disagree.

Actually, captchas are overkill for challenge/response. Better is just a simple human question like the one I use here -- as long as the question is made up by each person, and there are no patterns. Spammers could start maintaining databases but I doubt it's worth it for them. I know spammers have taken the challenge of captchas up when it comes to automating creation of webmail accounts and so on, which have long term value for them. Are there reports of them doing it for single mail challenges? Hardly seems worth it.

Right now the normal spammer philosophy is that if you fail to get a mail through it's cheaper to just go on to the next one. Once anti-spammers get good enough to make this no longer viable the spammers will find it useful to defeat turing tests etc.

My hope is that the spamemrs won't drive us into an Orwellian world where you can't do anything on the net without providing your ID. "Digital papers, please!"

Current volumes of CR arent worth it for the spammers. Except in some rare cases here and there where an ISP offers CR as an option.

Nowhere in current CR or captcha is anybody at all asked to provide valid ID though, so 1984 analogies are wasted on these things :)

Not really. There are quite a few anti-spam proposals out there which break down to requiring everybody sign their mail and have an ID certificate. At first it is usually suggested as a way to get through anti-spam filters but it's clear that the long term goal of many is to simply refuse mail that comes without proper signatures and ID certs.

Not everybody is proposing this, but some are, and their proposals are not immediately discarded.

Add new comment