Some of the false positives need to stand up and start shouting.

At the end of this post, and admittedly this got longer than expected, I’m asking for your help. In fact, this post got so long that it’s the longest post published here to date. It’s an important topic and one that anybody who spends any time online should appreciate and in subsequent editing, I found my thoughts on the subject only grew. From an editor’s perspective this isn’t always a positive trait, so I’ve added the following header section.

If you are in a hurry and don’t have time for the details at the moment but would like to help me, then you can safely skip to the last bolded section of this post. The title above and the final paragraph of this post will give you the necessary details. The guts of this post give background and substance into my feelings on spam/spammers, anti-spam tools, false positives and how it impacts communication online. It also delves into why I’m concerned that we don’t go down roads we’ve already travelled with comment spam solutions. If that’s a topic of great interest for you, then enjoy and please consider getting involved at the end of this post.

It began with a conversation
This morning a friend of mine, darkmoon, told me that as recent as a month ago he was getting trackbacks sent from this blog flagged as ‘SPAM’ and that he had no easy way of whitelisting Hmm. I suspect that there is a code workaround to stop this from happening — there always is — the ability go into the code and whitelist a domain or list of domains. However for the purposes of this post (and reality) let’s not assume that everybody with a blog who uses these anti comment spam tools is a programmer. I’ll save the technical dialog for another post perhaps another day. I am interested in the technical side of things too, but I’m trying instead to cover the social implications in this post.

It appears from the Movable Type screenshot of the Akismet plugin that darkmoon sent me (pictured above) there is a way to credit false positives as “ham” which hopefully makes the score for the domain less likely to occur in the future. Thank you darkmoon for the screenshots.

The problem with which everybody reading is familiar is that unless people look through the filtered posts they may not see the trackback or comment at all. It’s the same as being filtered on an email list and what has caused the downfall of email as we know it today.

With email, the perception is more and more that it isn’t reliable. On a local talk show the other day the host was railing against the fact that over 9 in 10 emails are spam. It’s a broken medium and that’s where blog comments are headed if something isn’t done. Trying to use the same techniques that are being used — and broken — in email won’t get the job done. Blacklisting and current anti-spam filtering code does. Not. Work.

Sorry to every anti-spam program out there but you all suck. Some more badly than others, yes, but you are trying to take a 1990s solution for a 2000 evolved spammer crowd. It’s like trying to fight a new strain of virus with something that worked 10 years ago.

It isn’t working.

How do the blog commenters feel?
From the commenter’s perspective when a comment they make doesn’t appear, it’s not exactly the same way as sending email that never receives a response. The reason being is the person sending the comment doesn’t always know you are using a comment anti-spam plugin or, as was the case with darkmoon, why you were being flagged as a spammer by the service?

It’s like somebody saying hello and the other party walking the other way, unaware, unhearing. It’s a good assumption that most everybody sane using email is using some sort of anti-spam plugin but can the same assumption be made about every blogger? What about the ones who don’t have any choice what anti-spam plugin their blog uses? Services like TypePad or MSN Spaces don’t allow those types of plugins.

Socially, most people realize this possibility can be perceived as being antisocial and downright rude. The solution? Search through the filtered sites when they get time.

See, we’re back at 1990s problems again.

I know most of us do this and it means we are trying to find the false positives and still seeing the spam to some degree. That plays right into the real spammer’s hands and is the same problem with email all over again.

Solutions for dealing with comment spam
1. if you don’t want to hack the code or keep marking this domain as Ham, stop using Akismet or whatever anti comment spam tool your using and try something else. I didn’t ask darkmoon whether or not he was interested in trying a different solution, but he can respond to that on his own blog or in the comments below. This could mean more or less comment spam being received depending on how good or bad the new tool works. It’s possible to drop Akismet for something much worse and vice versa. You have to look around, but you know what? I’ve been doing that for years now.

I’m still looking for the killer anti-comment spam tool that has an extremely low false positive rate. All of them use the same type of filtering techniques and RBL blacklists, plus or minus a few hooks that spammers end up figuring out and exploit. Some of them say they are better because they employ large amounts of people helping them socially filter sites. That’s BS. That just means more people can contaminate the pool by intentionally marking false positives. Don’t like me? Just mark any trackback received from me — even when it’s not spam — as spam.

2. stop accepting comments/trackbacks altogether. It’s kind of difficult to argue against heavily trafficked blogs throwing in the towel, although one could argue (and I still do) that the more traffic a sites receives, the more money they have to pay one or more people to help moderate the comment flow.

3. create your own custom filtering solution. This seems to be what more and more sites are doing. It’s the option we use here at Hmm.

Before continuing, the title of this post aside, I’m not saying that tools like Akismet are useless. Right now, they are among your only defense against the ultimate comment spammer’s dream: having no comment spam plan at all.

They all are using flawed sources and filtering algorithms. And if it is flawed with this domain and me, it might be flawed with your site and you too. Any software meant to protect against the bad people that is filtering out good people is flawed.

I’m Ham, Sam, I am.

Spamlookup.net
During this past summer darkmoon alerted me via email that there was a spam entry for makeyougohmm.com at bsb.spamlookup.net. I contacted the owner of our hosting company used for the dedicated server that runs makeyougohmm.com and asked what the deal was with these spam entries for a domain that has never had any kind of mailing list and doesn’t spam anybody?

The hosting company owner, Chris, replied that they contact these anti-spam organizations when customers leave that were found to be spamming and ask that the IP addresses be released when they’re reassigned, as was the case with makeyougohmm.com. He seemed very disturbed that many of these organizations are terrible about releasing these listings, even when notified. As if it’s the hosting company’s problem and all their clients get to suffer the consequences or leave for another hosting company and different block of IPs. I don’t want to change IP addresses or hosting companies, although that might be the only solution to clear this site’s name long term in some of these lists.

This situation reminds me of my direct experience with numerous security vulnerability sites. These sites are great about listing vulnerabilities but I’ve faced challenges trying to get them to update erroneous information in their databases, as well as providing details on patches for software where there were vulnerabilities that have since been fixed.

At some point a one developer run shop like ours has to decide whether to spend his time fixing code, updating and upgrading or trying to tell a dozen or more different security vulnerability sites that a patch exists. I don’t think it’s my job, nor do I want, to jump through hoops at third party sites to correct public misinformation, even if it’s negative toward my business. If their site is one to report vulnerabilities shouldn’t they be equally as concerned about reporting fixes, when available?

Maybe because I didn’t call in legal eagles to help me with these problems was the reason I haven’t had better success, but that’s my experience.

Back to the spam situation.

So it seems that makeyougohmm.com has inherited a spammer IP address and there are some lists out there which refuse to refresh/update their lists to acknowledge that the current owner isn’t spamming.

Unfortunately, this isn’t the first time I’ve had this problem. Enter Six Apart. When signed up and tested, I couldn’t leave comments on my own brand new TypePad blog (tdavid.typepad.com). This was around Christmas time two years ago and it took a week or so to find out that the reason my comments were being blocked was because makeyougohmm.com was listed on an old spam service.

Although the comments started working after the engineer assured me makeyogohmm was removed (thank you), I didn’t keep that test blog and have still experienced problems seeing comments left on some TypePad blogs in the years since (more proof that Akismet isn’t the only service with these problems).

Some of this I figured might have been owner moderation although I do not feel like I leave spammy comments. I rarely leave comments with links back here nor do I wear some anonymous troll hat. I definitely have left some spirited and sometimes downright testy comments on other blogs, but spam? No. Disagree? Have proof otherwise? Bring it in the comments below, I’m all eyes and ears.

Even so, I won’t blame every lost comment or trackback as owner moderated and it doesn’t bother me if owner’s manual moderation prevented a few comments from appearing. Heck, we moderate comments here too.

A few weeks back I made a comment on Sterling’s blog that wasn’t even remotely spammy and it too didn’t appear. He commented back here saying he never saw the comment.

He uses Akismet.

Got to admit this has reduced my interest in leaving comments on Sterling’s blog, albeit only a little bit. And this is a guy I happen to like and respect, but then I’m at his blog and the thought crosses my mind: will this be blocked?

Why should one go through the exercise of leaving something that might never be seen? Why bother? See the problem?

I’m better off leaving my comments and hoping he sees/reads it here.

Another friend of mine here locally runs this great blog on videogames. I’ve tried leaving comments over a dozen times, including comments at a local blog meetup with him in the same room and he still doesn’t get them. Blocked. He uses Movable Type and some anti-spam plugin (not sure which).

Eventually it’s give up and move on.

Not a very good solution, but that’s where I’m at with some bloggers and blogs out there that are using tools which prevent me from commenting. I’m not some dolphin that will jump for shrimp at command. Make it easy for me to leave comments and see them show up (so I know they were actually read) and sooner or later I will. Make it more challenging or downright impossible and I’ll probably move on. I like to have two-way conversations with people, not one way conversations. That’s a lecture, not a conversation.

How many readers do you have with the same commenting problems I’m facing? Don’t ask me. Ask them.

I value relevant, thought-provoking comments and consider them to be an important part of the overall post. Sometimes I’ll riff off these comments and make entirely new posts. Some posts here the comments are more important to me than the post. Some posts have 50+ comments, really, really good ones that said more intelligent and thought provoking things than I’d written. People who think of commenters as some kind of negative lifeforce, miss out on those commenters who do add value, substance and balance to the topic. Yeah, the pool might be small — really small when you first start out, but it does grow over time — it’s there and these people are important and well worth encouraging.

One of the things I like about blogs, generally speaking, is they provide better communication than sending email. More and more I feel like I’m having better success communicating with relative strangers by writing posts here instead of sending emails. Particularly when it comes to writing about people some might or do consider to be celebrities. I’m talking authors, owners, reporters, CEOs, people with whom I would contact and be lost in their sea of correspondences. Yes, even some of those so-called A-list bloggers (though I don’t care for that verbiage).

Being filtered completely by anti-spam tools sucks.

What email does makeyougohmm.com send?
The only email this site sends out as of this writing are replies to comments in threads where people are subscribed. There are currently and never have been any advertisements in these replies and each one carries an unsubscribe link at the bottom. When you leave a comment here you can uncheck the box and never receive those replies. You could also dummy up an email address and never get them if you are that paranoid. Want to test this? Leave a comment at any thread here that allows comments (99% of posts allow comments but a few have comments turned off).

The only other email this site generates are daily notification of the posts via RSS subscription through the third party service FeedBlitz. Signing up for that email requires confirmation, so it’s double opt-in.

I don’t email people directly or indirectly with bogus, affiliate-laden offers or “hey, you should buy this or that” using makeyougohmm.com. I rarely — and I mean maybe a half dozen times in 3.5 years and 4,000+ posts I’ve made — drop an email to somebody else unsolicited that they might be interested in checking out something available on this domain. If you are reading and have ever received or had any unsolicited email correspondence with me period, that’s very rare. I’ve had lots of situations replying to feedback from readers, but I’m talking about me out of the blue composing an email and encouraging anybody I know to look at something here at Hmm. Just doesn’t happen very often.

And when it does happen, in all these cases they were very personalized emails, not anything mass mail. Definitely not intended to look like or be spam. They were something I legiitmately felt would be of interest to the person I contacted.

Does this sound like the work of a spammer?

I hate spam and hate spammers. Talk is cheap though, so instead of listening to what I’m writing, spend some time looking around and judge this domain for yourself. Compare it to other domains with similar amounts of traffic. This site receives over 4,000 visitors a day according to server logs and about 2,600 according to the third party site meter service which is and has been publically available since 2/15/2004. This is a healthy increase in visitors since the summer (thank you and welcome to new readers and subscribers). I think these people keep reading because they sense that I’m one of the good guys and trust me.

Trying not to violate trust
Look, one of the major things we own is our integrity and the only person who can destroy that is yourself. I’m not going to screw up my reputation and integrity that I’ve worked my ass off to build by sending out spam to anybody and everybody and that includes trackback spam. In fact, every trackback sent from here, perhaps 99.9% have all been from posts that have a link back to the place I’m sending. I don’t like one way trackbacks and try never to send them. I don’t say 100% because there is probably a couple exceptions but they are so rare it would be like finding a needle in a haystack. Don’t send them.

Two-way trackbacks are encouraged and welcome of course. In fact, let me go a step further here: don’t send this blog a trackback unless you honestly have something to say about the post or subject here. I’m not a recip link whore and could care less what my Technorati rating is (it’s declining over the last six months). I’d be happy being C or D-list with the popular crowd and an A with readers/subscribers. I care about the people who visit this site and spend time here more than what any third party blogger/site thinks. Doesn’t mean I won’t argue and disagree with readers, but I do care about them — you.

Contrast that to thousands that have sent MakeYouGoHmm one way trackbacks. Sploggers, spammers and even some respected people like Mashable’s curator Pete Cashmore. Pete’s site sent at least a half-dozen one way trackbacks to this blog despite me writing here several times that I don’t like them nor do I want them. I even emailed Pete personally to tell him please don’t send any more one way trackbacks to this site.

He never replied to my email — and it wouldn’t surprise me if he shows up after I’ve published this and says he never saw any email from me about this and would be happy to comply — but that doesn’t change what happened and when. He and I both know he sent trackbacks this way without links in his posts. In Pete’s defense, the posts were always related, only he chose to tell me about his post and link to other people and not me. I take a dim view of somebody one way trackbacking their related post and linking to other people and not me. It’s like saying: hey, check out what I have to say about this and here’s my post with links to my friends. Your site and commentary wasn’t worth linking to, but I think you should tell your readers about my post.

Rude.

I haven’t seen any one way trackbacks from Mashable in quite awhile so perhaps I made it off Pete’s “must send one way trackback” list. Always happy to receive trackbacks of the two-way variety.

Practicing good netiquette
I do my best to operate above the spam fold on all levels here and it bothers me when friends of mine are trying to receive communication from me — want to receive communication — and some filtering tool is blocking this communication. Darkmoon seemed a bit annoyed that he has limited ways to whitelist me/this domain and I’d like to help.

Why aren’t there better whitelisting features in Akismet, Matt Mullenweg? Or, if there are, how should people like darkmoon go about employing them? Sure hope you aren’t harboring any grudges over the negative comments I made about your -9000 pixel Wordpress tricks, Matt.

Just FYI: I did not email Matt to point to this post. It happens to be the kind of rare post I would like to contact the other party about to get their POV before publishing, but I’m almost 100% certain the Akismet team will tell me they have nothing against makeyougohmm.com and don’t see this site as spam. Too bad their software does. As for Matt? He’s a smart guy and will see this post. I’m fairly confident the message will be sent and received which if instead I sent via email, who knows if it would end up being a false positive? (Update 7:02pm PST: Matt was the second person to comment on this post, see comments below).

Use your eyes, what is spam to you?
Please take a look around this website for a minute. I mean, really, really look around. We don’t use popups, popunders, hidden links, interstitials — all types of advertising we could employ that many people feel are spammy and need to be blocked by Greasemonkey but would be well within our capability to use — but don’t. I could quadruple the income of this site in one day by making some of these aggressive advertising changes — but I happen to like the reader experience the way it is now from both the website and in the full text RSS feeds. And I think readers who do come to this site do too, but they are welcome and encouraged to echo their feelings as always below. I think while we could increase income, we’d negatively impact readership just to make more money as the primary goal. The site is still very profitable, there is no need to get greedy.

There are some places here that contain zero advertising like the search results pages. That’s a prime spot to place advertising that we will be putting advertising someday but after some 3.5 years that space isn’t being used for any advertising, not even a banner, button or text link. How many truly clean search results pages do you see out there anywhere? A lot if you are forced to block them with Greasemonkey and other ad blocking tools. You don’t need to worry about that here.

The problems false positives create
I worry about false positives. Really.

I mean, if somebody like me who tries to create good content, respect readers with a minimal amount of advertising and doesn’t mass mail people is being flagged as spam what does that say about the reliability of these services? How many people are using tools like Akismet and not seeing ham that should be credited?

Why is any place flagging sites like MakeYouGoHmm.com as spam? I’m not going to spend every day tracking down services which are reporting back inaccurate information. But will this be an important job for busy sites to hire somebody to do that for us? Be the ‘we’re not spammer’ cop?

What tools are we using here to combat comment spam?
As mentioned earlier in this post, we aren’t using any third party filtering tools. In the past I’ve tried most all of them including CAPTCHAs and all solutions have been flawed and carried consequences that I felt reduced the amount of good communication between authors and readers.

I don’t think our current solution — which I coded myself — is any better BTW, but it does keep the numbers down to a reasonably acceptable level. To give you an idea of the amount of spam we’re seeing in the comment moderation queue, we deleted over 265,000 comment spams yesterday. That’s not a misprint or exaggeration. That’s the number of comment spams received this year to date to this site. I’ll say it again, just for emphasis and the readers skimming:

Makeyougohmm.com has received over a quarter million comment spams in less than a year

I shudder to think how many comment spams sites with a lot more traffic than ours are receiving. It has to be in the millions yearly. Yes, millions.

Your help is needed if you own a blog and can receive trackbacks
I’d like your help with a test if you use the Akismet system or any anti-spam system with your blog and accept trackbacks.

Please make a test post on your blog that I can send a trackback from this post to yours. You do not need to link to this blog in your post, although it might confuse your readers if you don’t. That part is up to you.

Before anybody asks, this isn’t some clever way of me trying to get links from other sites. In fact, you are welcome to remove the test post at your site when we’re done testing. I don’t want either of our sites to get in trouble with the search engines for an experiment meant to help discover how wide ranging and what tools specifically are blocking trackbacks from makeyougohmm.com. I’m simply wanting to test with others across different IPs and servers trackbacks sent from this blog. Any information including screenshots that you can email me (tdavid + spam at gmail dot com) in relation to the trackback I send you would be helpful in tracking down what services are blocking this site.

I will be sharing this information here so that these services can see these problems and hopefully remove MakeYouGoHmm.com from their list. At the same time, if anybody reading this from any of these services has specific first party proof that this domain is responsible for any spam-related activity then right here, right now is your chance to speak up. This is your chance to publically air out the evidence you have on this site so we can investigate and resolve.

If you would be willing to help me I’ll be grateful and won’t remove the link from the comment area here as long as it’s relevant and not spam (ironic, isn’t it?). I want other blog authors to think one thing when they see a link from MakeYouGoHmm: that is a quality trackback. It’s from somebody who actually looked at and read what you had to say and it was interesting enough to reply and link back to you.

Testing links (updated, trackback pings sent where possible, see comments for details)
LUX ET. UMBRA: When anti-comment spam doesn’t work
Chips Quips: Chipping the web - calendars met on

Some of the false positives need to stand up and start shouting. Did you hear my gentle roar?

Related Posts

• Akismet first catch a false positive
• Mullenweg on Comment spam
• Over 90% of blog comments are spam, reports Akismet
• TypePad support recommends using proxy to get around their own anti-spam filters
• Blog comment spam and SE impact: is it that significant?
• Proof that Google spam fight is not a “sham”