The Unintended Consequence of the Spam
Rocky Mountain Internet Users Group
Minutes of the 2 July 2008 meeting, "The Unintended Consequence of the Spam
Wars: Why Your Email Isn't Getting Delivered"
About 20 people attended tonight's pre-holiday meeting. Josh Zapin facilitated and Jeremy Kohler recorded the minutes.
Microstaff (www.microstaff.com) provides refreshments, Copy Diva
(www.copydiva.com) provides the audio-visual equipment, NCAR
(www.ncar.ucar.edu) provides the facility, and ONEWARE
(www.oneware.com) sponsors these minutes.
INTRODUCTION (Josh Zapin)
We all know spam. We all hate spam. It clutters our inbox, offends us (do I really need to see another Viagra ad?), and is just a pain in the neck. Some researchers have estimated that every 24 hours, 100 billion spam messages are sent. That's 100 billion useless emails every day. Ferris Research estimates that the lost productivity costs businesses $100 billion worldwide, of which $35 billion is in the USA alone. I think we can all agree that if we could obliterate it completely we would.
While obliterating is probably impossible we sure are more or less succeeding. With a litany of cool-named products like Spam Assassin, Spam Eater, and Spam Agent, we are starting to see a decrease in spam's growth rate. 2007 saw an increase of spam of about 10% down from a 53% the year before that and over 100% the year before that.
Some people are saying this is the case because antispam products are working, making spam a less attractive avenue for marketing.
Using fancy algorithms and other methods, these products "read" your email and determine whether the email is truly worthy of your attention. While certainly not perfect, they are helping to reduce the clutter.
But are they doing their jobs "too" well? Increasingly people are finding that critical emails are lost in their "spam" folders because some attributes of these legitimate communications fail the algorithm.
So we may win the battle but not the war because email is such an important communication device.
ABOUT THE SPEAKER
Anne P. Mitchell, Esq. (firstname.lastname@example.org) Anne is the CEO and President of the Institute of Spam and Internet Public Policy.
Mitchell brings with her nearly 10 years of experience in the Internet and email industries, both from the legal and technical side. Mitchell was the Director of Legal and Public Affairs for Mail Abuse Prevention Systems (MAPS), the original antispam blacklist. Following her time at MAPS, Mitchell was cofounder and CEO of Habeas, the first of the email reputation services.
Institute for Spam and Internet Public Policy: http://www.isipp.com
The Email Deliverability Blog: http://www.GettingEmailDelivered.com
A question for the audience: What's your biggest interest in this topic? Why are you here?
At work we use email in place of talking. Why don't my emails get delivered?
We're extremely dependent on email with multiple email systems, so spam is a big problem for us.
I work with email marketing systems, so I want to catch up on trends in delivery and permissioning.
A family friend sent some messages I wanted and some I didn't, so I tagged his mail as spam, not wanting to offend him. I'm not sure if that's the best way to deal with it.
I run an email hosting company, so I try to make sure my servers don't get blacklisted. We also have some trouble with backscatter.
I want to know what is "responsible" email marketing?
My web clients are getting hammered by spam from other countries and I'm looking for solutions.
ANNE MITCHELL: A lot of admins actually just block entire countries.
We use an in-house email deployment system, and we want to keep our servers whitelisted.
I just want to know how to block spam.
I can get 50 to 500 spams per hour because our email addresses are posted on a web page.
ANNE MITCHELL: You can disguise your email addresses in code so that machines harvesting email addresses for spammers can't read it.
I'm just trying to get our legitimate bulk emails delivered.
I work in IT and I need to keep up with this stuff.
I'm interested in working in this space.
I don't know much about spam.
I want to know what can we do as responsible citizens to help in the fight against spam. How should we report it? How should we deal with phishing emails, and what about these abuse addresses that are set up for reporting?
I work in IT and I'm tired of hearing my clients complain about spam.
Is spam is slowing down? End users say yes, there's less spam, we're doing a better job. But IT and admins say just the opposite, because incoming spam has not gone down at all. ISPs for the most part are absorbing this problem.
So the filtering is getting better but the spam isn't.
Remember when spam started including images of text? That was to get around filters looking for "viagra" and other words. Now spammers are starting to send PDFs. In response, spam filters have started blocking emails with attachments.
So with all this filtering going on, is your email getting to where you want it to go?
People do have problems getting their email delivered to customers, and it costs them money. This affects people on all levels.
The problem is that your good mail is getting caught up in the spam filters as the filters try to keep up with the spammers.
Sometimes email doesn't even reach the recipient's spam folder because the ISP didn't even send it along.
THE EMAIL PATH
Your email server sends a message to the recipient's ISP, which looks up your IP address in a database via a DNS query. The ISP queries a whole bunch of databases all over the world to see if the sender's IP is blacklisted somewhere. It's pretty easy to set up a blacklist--which can cause problems--but fortunately now the industry does a little due diligence to look for "genuine" blacklists. Some blacklists include IPs simply because someone doesn't like them.
Fortunately most ISPs don't pay attention to those lists. To be on a genuine list, you have to truly be spamming or you haven't taken steps to fix a clear problem--like a hosting company not dealing with a customer who sends spam.
Your email has to run the gamut of dozens of spam filters, and they all filter differently.
Some ISPs use their own filters, others use off-the-shelf stuff. Spam filtering is all over the map. ISPs can use various combinations of blacklists and filters, so it's hard to deal with the nonstandardization, plus ISPs don't always reveal what they are doing.
Spam Assassin is one of the most widely deployed filters out there. A lot of ISPs use it because it's open source, easy to use, and customizable. Our service is listed with Spam Assassin.
It looks for different traits and assigns "points"--too many points and it's tagged as spam. Kind of like failing a driving test--you can make a few minor mistakes and still pass, but if you make too many you fail.
Spam Assassin also assigns credit for unspamlike traits.
With Spam Assassin, the recipient email server checks the sender's IP address for blacklistings. For example, if your IP address is using open relays, it might be blacklisted. It also checks if your IP address matches your domain--spammers often spoof a domain name, which creates a mismatch.
Then it analyzes the mail headers. For example, it analyzes the the subject lines--searches for gappy versions of commercial products as well as specific words and phrases.
HTML vs plain text mail: Text has better deliverability because HTML is preferred by spammers. So lots of HTML raises your spam point score. The software also looks for certain HTML tags, such as really tiny or really large font sizes--these things have to be "just right"
Spam Assassin also looks at the body of the email. Don't say that you comply with spam regulations, for example, because it will cost you points.
"Unsubscribe" links also cause spam demerits. Even though you're supposed to include that stuff as a good citizen, don't mention anti spam laws because spammers are doing it too (but you MUST include an unsubscribe link, and honor it!)
Filters might catch even common terms that you might use in normal writing.
Lots of regular text is now being identified as spam indicators.
So given all this testing, hitting "send" is like sending your baby out into the world, wondering if it will make it to the other side.
What can you do to ensure delivery?
One method is to outsource your email to a service provider. This is recommended for people sending lots of email. Check their reputation.
They might be being blocked too. See if they are participating in our program or one of the others. See if their IP address is blacklisted at rbls.org. Check your own IP address too.
If you must use your own server, beware that if your ISP is hosting spammers and won't deal with them, the entire ISP might get blacklisted through no fault of your own. So then you may have to switch your ISP--there's no other solution if your ISP doesn't clean their act up.
Get your header information accurate and complete. Make sure your IP address matches your domain. Don't use a nonexistent "From:"
address--that's spoofing, and it makes you look like a spammer.
Set up reverse DNS so your IP will resolve your domain name. DNS is like directory assistance. Forward DNS takes a domain and finds the IP address.
Reverse goes the other way. Your ISP has to set that up for you. And now ISPs do reverse lookup because spammers do a lot of spoofing.
Publish authentication records like SPF and Domain Keys.
Doing these things makes it look like you're doing the right thing.
Kind of like displaying good manners, even if it doesn't always work.
If you're a big emailer, develop a personal relationship with all of the ISPs to which you send an appreciable amount of email. It's almost impossible to get good responses from you're the ISPs without a personal relationship. Many volume senders have a full-time ISP relations person.
You really need to be aware of things that can trip a spam filter, and they are legion.
Audience Comment: Always include a return path header for bounce handling.
You can also test your emails. Send a draft through a content checker, or send it to yourself.
A lot of ISPs run Spam Assassin on the outbound mail server, so your email might never even get past the gate--it won't even reach the recipient's spam folder.
Text or HTML? Going to text sacrifices your data on click-through rates that come with HTML, so it's scary for some.
Tell your recipients to whitelist your address! That's important because some people never check their spam folders.
For commercial email: How you build your mailing list affects your deliverability. You need to prove that recipients gave you permission to send them email. The gold standard for this is double opt-in, where you don't put someone on your mailing list unless they specifically responded "yes" to your request to opt them in.
Just providing opt out isn't good enough. Some businesses automatically opt you in, and then offer opt-out. It's perfectly legal and some big companies do it that way.
You need to be able to show opt-in information; if you don't, and just go with opt-out, it'll get you into trouble. It's not worth it.
Antispammers have taught users not to unsubscribe from spam. So instead everyone clicks the "Spam" button instead to report it. Now we're retraining people to unsubscribe. Fortunately, ISPs set up feedback loops so that when someone clicks you as spam you get notified. Some ISPs even click your unsubscribe link for you.
When someone signs up with us for accreditation, we require feedback loops.
Important to remove email addresses from your lists that bounce.
A spammer spoofs your email address so you get all the bounce messages. This is because spammers don't clean their lists.
Backscatter can be enough to crash smaller servers. It's a pain, but it won't get you in trouble. You have to have your coder find whatever is unique about the backscatter and set up a filter based on that. Of course you don't want to block the whole ISP.
A year ago I could never say this: We outsourced our own spam filtering to Postini. They have really turned around from their earlier reputation of being unresponsive to senders, and I can highly recommend them now. Their service is well worth it. This way you outsource your spam filtering: all your mail goes to directly to Postini first (you set up your MX record to handle that).
People actually sign up antispammers for mailing lists and that gets them blacklisted.
Audience Comment: Disgruntled employees might opt in their old boss to a mailing list.
Closed-loop (double) opt-in will prevent that by sending a confirmation email before really signing you up. That verifies that the email address is owned by someone who wants to opt in. This is what ISPs are looking for.
They want to see the confirmation emails or logs showing click-throughs on subscribe links.
If you send out a newsletter that requires a fee, that's another way to verify confirmation--someone paid to be on your list. Always try to send a confirmation message.
Some older lists are legitimate but were built before confirmed opt-in existed. So you might have to reconfirm your legacy lists. The trick is to make your message really compelling and split your list. Offer an incentive for reconfirming; those that don't reconfirm you can eventually drop. An incentive might be something like a free subscription.
Be CAN-SPAM compliant, but don't say it in your emails.
Don't try to "game" the spam filters by fussing with headers and servers--it just makes you look like a spammer.
So email deliverability problems is a big issue and everyone has it.
Watch your dos and don'ts, and consider using an email service provider--that can take a great weight off your shoulders.
And remember: Filters don't know the difference between "looks like spam"
and "is spam."
QUESTIONS and ANSWERS
Q: Is there anything wrong with using a complex email address with a number at the end?
A: A lot of spammers use those complex addresses with numbers and stuff.
There's no rule per se, but you might trigger it somewhere.
Monitor your own email deliverability. Service providers can do this for you. Or open up a bunch of free email accounts - like at Yahoo and Hotmail and AOL, and see if your mail gets there.
Q: SpamCop blacklisted me because we got some backscatter and bouncing. I want to keep track of bouncing and inform my customers that their emails didn't make it someplace.
A: Not a big piece of the problem. ISPs know that spammers will spoof your email address and cause backscatter. Most of them won't hold it against you.
That said, it's important to bear in mind that ISPs do not have to accept your email, so if something you are doing is causing a problem then you just gotta do what they want.
Q: How do I avoid people mistaking my emails as phishing attacks?
A: Don't send IP addresses and make sure your links are normal. This has been a big problem for financial institutions. The average end user can't tell phish from legit. So just make sure you don't look like those phishing emails you receive. You choose not to provide click-through links--instead provide text instructions that tell customers to go and log in to their accounts.
Q: How about return-path certification logos?
A: No one thing will kill you. Even we have to be careful with our own accredited mailings because local spam filters can catch them after they pass the ISP. Just be careful.
Q; Email postage? This could save ISPs lots of money.
A: That concept has met lots of resistance. Email is supposed to be free.
But how would an ISP know that postage was paid anyway? Spammers could spoof that too. Of course, senders pay our company to ensure that their mail gets through. So that's not terribly different from the postage idea.
Q: Are there any ways of fixing this really screwed up email system?
A: We are testing a system that does an end run around ISPs and all spam filters, and delivers email directly to a user's in-box. Check it out at mailflipz.com. It's RSS based: it pulls email instead of pushing it. Works great, but requires the user and sender to sign up.
Q: How about mass adoption of certification (publishing authentication)?
A: SPF (Sender Policy Framework) is happening. But it's just one indicator that you're doing the right thing. You can't satisfy every filter out there.
Q: Do filters monitor your email volume?
A: Some ISPs will look at weight limits. It hasn't been a big issue.
But if you bring a new IP address on line, be careful to use it slowly at first to build its reputation--mass emails right off the bat won't help , in fact they'll likely get the mail coming from that IP address blocked.
Q: Can we have better laws?
A: CAN-SPAM is an opt-out law. Even places with tighter laws can't do much.
Spam gets routed all over. We have always said that it takes a 3-prong
approach: LAWS, TECHNOLOGY (filters), and USER EDUCATION.
That last one we've really fallen down on. That's the problem. People buy stuff through spam. Until we educate the masses to not click through, it's going to remain a big problem.
Q: What is some good email software?
A: Most of the Mac OS apps are great. I use Mail.app. For the greatest security we recommend a Mac because it's a whole lot safer. If you're running a PC you're just asking for it right now.
Audience Comment: Cloudmark service is based on how many people are reporting back--it's good if you've got a PC.