Learned about this one awhile back and meant to blog about it, but searching through the archives it appears I never did. I was reminded about the Techmeme bot being blocked by the Wordpress plugin Bad Behavior by default when I read thatedeguy’s recent complaint:

I am on the Techmeme Blacklist. Yes, it does exist. I’m quite certain that Gabe, the owner of Techmeme will deny that I am on the list and perhaps he’ll even show the list to prove that I’m not on it.

I believe Gabe that there’s no blacklist, thatedeguy. Yes, this blog gets listed on Techmeme so there’s my disclaimer, but there was a time when this blog stopped showing up and I noticed it and contacted Gabe. The only thing we’d changed was adding the Bad Behavior plugin so my immediate concern was if somehow that plugin was blocking his techmeme bot.

If thatedeguy isn’t using the Bad Behavior plugin maybe he’s using some other plugin or program that is blocking the Techmeme bot? This isssue isn’t unique to the Bad Behavior plugin or Wordpress, it could occur with any filtering program that is using a similar methodology as the Bad Behavior plugin.

Maybe it’s because I’m a developer but I tend to look at problems internally before looking externally on issues like this. I don’t think it would be very logical for a site like Techmeme which exists to push current news that sites are talking about to have any kind of blacklist, as Gabe indicates in his comment on Thatedeguys’s blog.

Tom Raftery had the same problem as I and wrote about it in August 2006 prompting a somewhat heated discussion in the comments between the Techmeme and the Bad Behavior plugin developers. I don’t know if this is the exact issue for Thatedeguy but it was for Tom and I. Chris Messina also had the same problem.

As for the possibility of whitelisting Techmeme in Bad Behavior, I visited Michael Hampton’s site to see if his plugin has any specific information related to the Techmeme bot conflict and couldn’t find any with a quick perusal.

Next I sent Mr. Hampton an email asking about the situation and letting him know I was going to write a blog post about this situation. I was curious what he suggested for those who want to use his plugin and also want to be listed at Techmeme. His response, which he gave me permission to quote verbatim:

When Gabe first e-mailed me about his bot’s problem, about a year ago, I
told him exactly why it was blocked, and gave him two alternative
methods he could use to fix his bot. At first he did neither. Instead,
he chose to attack me, saying I was being unfair to him by not giving
his bot a free pass to violate protocol, (I don’t give Googlebot a free
pass, why would I give him one?) and calling Bad Behavior a stupid way
to go about screening out spambots.

A few months ago, he chose to take his attacks public.

He did eventually take one of the two approaches I outlined for him so
long ago, at about the same time he took his attacks public. Later I
learned that his bot was still being blocked, but for some other reason.
I don’t yet know what that reason is, except that the error returned is
400, indicating he’s likely violating the HTTP protocol (as Bad Behavior
uses 400 for protocol violations). But the reporter didn’t send any
bad_behavior log entries, so I don’t know exactly what error still lies
in Gabe’s bot.

It’s too bad. I like Memeorandum and Techmeme. I’d even like to be
listed in them. But that’s not going to happen until Gabe fixes his bot.

I also contacted Gabe by email this evening to ask about this situation and his side of the story. Remember, this is the weekend and both Michael and Gabe responded rather quickly. Gabe responded to the situation by saying that he was working on a solution and might have it posted by “midnight tonight.” He indicated that he had discovered some sort of workaround for the issue. As I publish this it’s a few minutes before midnight, so maybe I’ll be updating this shortly.

Update 10/30/2006 7:30pm PST: Gabe responded below in the comments and also privately to me since that he strongly refutes the notion that he ever attacked Michael Hampton. I asked Hampton to stop by here and address specifically when and where Gabe attacked him and he hasn’t shown up yet. Hampton did send me an email notifying me that he now saw one of his sites on Techmeme so the changes Gabe made seemed to have worked. Also, Thatedeguy responded with additional information. As always, readers are encouraged to visit the comments section for details and response. I try to in cases like this to update when it’s important to the parties the post body or write a follow-up post, but the comments usually come from the horse’s mouth (so to speak, no offense).

Resolution on the way
When I see Gabe’s resolution I’ll post a link to it or maybe Gabe or somebody else can do so in the comments below if you see it first. The reason for this post is to further help make aware those who are using the Wordpress Bad Behavior plugin — or another plugin or program that uses similar methodology to block bots — and want to be listed in Techmeme that there is a confirmed conflict. The solution of not using Bad Behavior isn’t a good solution and it seems by trying to whitelist the IP of the Techmeme bot wasn’t working as planned because the IP addresses change.

Gabe has said several times there is no blacklist that keeps sites out of Techmeme, but if his bot is denied access by the plugin, that means there is zero opportunity for your blog to show up on Techmeme. No matter who links to you.

If you see Wordpress bloggers complaining about not being listed on Techmeme ask them in the comments or privately if they are using Bad Behavior plugin and point them to this post or to Gabe’s once he has published the resolution (my guess is nothing different will be required on the our end with Gabe’s solution, but we’ll have to wait and see). Hopefully Mr. Hampton will point to the resolution for Bad Behavior users, or point to this post if he feels more comfortable.

Sometimes emotions can get in the way of solutions and as someone who likes both the Bad Behavior plugin and Techmeme, I’d like to see a resolution published for a not very well known problem that clearly has created some developer to developer friction. This post will be updated when a resolution is made available.

Did this post make you go hmm?

 Loading ...

• “Keep in mind this isn’t a tech audience,” said Jennifer Jones at PodTech.net
• Again Techmeme Gabe shows he is smarter than the ad-verage bear
• Being off by hours, Wordpress timezone change impact on archived posts and comments
• Tim O’Reilly responds to Web 2.0 controversy, asks for apology over blog mob being stirred
• Developer contest updates: Wordpress and OneNote
• Hmmcast #26: getting in shape