The company that truly cures the virus, spyware, phishing and spam plague that runs rampant on the web could become the next Google. I know this hyperbole is used all the time to describe companies and I don’t want to join in the chorus, but if we look at the most serious current problems on the net that need solving then some sort of incredibly effective anti-malware solution is needed for regular people. And I’m not talking about the security suites from Symantec and other competitors. They start the process, but are about as effective for Johnny Lunchbucket at stopping the slime as an umbrella in the face of a meteor shower.

The State of the Net survey by Consumer Reports projects that American consumers lost more than $8 billion over the last two years to viruses, spyware and various schemes.

Another study I read before vacation showed 96% of email is spam. 96% of mail resources wasted on spammers and scammers. Bill Gates dream of wiping out spam in the next two years could not have been any less prophetic.

I think something with some very sophistocated artificial intelligence is going to need to be deployed to help out people who aren’t yet wise to the scum peddlars on the web. We just got back from Nevada and maybe there is something down there lurking in the warehouses of Area 51. Get a programmer open house running at Area 51. Let’s get E.T on the case!

Might seem absurd, but perhaps some crazy thinking is needed to move toward a real solution? $8 billion in America alone is too much.

These beta invite schemes are really becoming sleazy. They had already encouraged and promoted elitism, but now they are sinking to an even deeper low: marketing without advance disclosure to people who signed up for beta invites.

Some days I have to remind myself why I bother signing up for them. And while it isn’t really spam since you signed up approving the email (see ink above) it feels a lot like spam. Am I masochist or what? Why?

1) because not every beta invite programs treats your email address like a crack whore
2) because we can still hit delete and ignore
3) because I look forward to finding and sharing new/different/unique/updated products and services that save time and increase productivity or just make life more enjoyable

Getting in early on something is the golden time to be heard and have the greatest chance of making a difference. As products/services grow, they, generally speaking, become harder to penetrate to the people who matter. This is one area that I think blogs have helped improve, but if you want to get close to the metal and talk to the people behind a company and help shape a product/service, the best time is usually in the early days.

A trend that Google helped popularize with Gmail invites spawned a whole bunch of marketing copycats. When you signed up for the Gmail invite they didn’t market you during the process, they just send the freaking invite. Google claimed this wasn’t a marketing practice, it was because they didn’t have enough servers. Some people believed this, but I didn’t, especially when at around the same time they were offering the Wikipedia webspace.

Unfortunately, some others are using this beta invite period as a recruiting and marketing campaign. Those who are doing this with their web pooh point oh business, please stop. Think about what people are actually committing to when they give you an email address: an invite. That’s all they want. An invite. Not spam.

An invite request is not an invitation for bragging about what we can’t see
My least favorite private beta invite request scenario is getting email telling us when we will receive the invite, and then using the time to pitch us about how great the product is that we can’t see yet. Even worse is when the email goes on to tell us what others have thought about the site/service we can’t see. People I know and publications I read. The infamous they got to look, but you can’t, neener, neener pitch.

It’s like saying: hey, you aren’t good enough to check this out, you aren’t part of the club elite, but here is what the club elite is saying. And hey, here are a few top secret screenshots!

If you do that to me on a beta invite list, my interest in your product/service just went from interested to annoyed. Use your blog to tease, not the beta invite list.

The most recent to do this was Craig Fitzpatrick CEO of Devshop.com. I emailed [Fitzpatrick] him privately before writing this blog entry complaining about this very practice. I’m not going to quote from private email conversations without permission, but anybody else on their 2,700+ invite list has already seen the same message I complained about. I am not accusing devshop of spamming but I find it very ironic that the signup text reads: “Your address will only be used to send you the occasional notification. We’re no spammers.”

Just as I told Craig directly, I’m probably the only one of the 2,700+ who complained, but because people don’t speak up doesn’t mean they don’t have an opinion on the subject. In Craig’s defense I will say he responded promptly to my complaint and seemed legitimately concerned.

Now, it’s your turn to declare my general complaint about this beta invite marketing strategy as baseless, petty or warranted. Despite giving a recent example above I’m not interested in talking about any specific company or invite list but the practice itself. When you sign up for a beta invite list are you expecting to get anything besides the invite and/or possibly information about when the invitation will arrive?

Lastly, good manners suggest that an invitation is just that and it seems very boorish to complain about an invite to anything. I fully realize that and some will probably flame me for it, but please remember if nobody questions these practices then they continue and can worsen. Let’s get the conversation started and let businesses know whether or not this is an acceptable marketing path. Fire up your keyboard and comment or trackback in with your 2 1/2 rusted pennies.

My opinion is clear: if we sign up for something specific like an invite to check out your beta, just send us what you promised when it is ready. Send us only what you promised and don’t try to market us or add to some other list or squeeze in some other marketing messages unless you clearly disclose this will be happening before we subscribe. And I don’t understand marketing the same thing to someone who is already signed up to check it out. Is the company marketing without our permission to remind us why we signed up?

That’s inviting them to change their mind.

I’m sure many of you remember the Million Dollar Homepage [Hmm review], the idea being to sell pixel space at $1 per pixel with a minimum of $100 blocks, so why not try that on a building? Holy zoning laws, Batman! Digg commenter TDot1980 sees why not:

The point was to have a backlink from a very very popular page with tons of visitors every day, thereby increasing your chances of being ranked higher in search engines. A building can’t do that, and most ads will get lost in the eyesore that is that building.

They are calling this Artvertising. This building would have a hard time going up anywhere in the states, but in Amsterdam? Heck, they’ve got everything in Amsterdam, why not an artvertising building. Maybe Vegas would do something like this, just think they could wager on which graphic got the most clicks, er, wait, that’s right … no clicks. Bummer.

In 2004 I decided to create my own URL shortening service. The plan was rather than use somebody else’s short URL service — there are plenty of them — I’d use my own in email correspondences, forum posts, anywhere that I needed to shorten a long URL. I registered the domains tdurl.com and the alternate tduri.com, created the database and small script that powers the program and began using it with little fanfare. Never even posted about the launch here, although I have mentioned or used it in a half dozen posts according to the search.

Clearly this hasn’t been something I’ve promoted for others to use, although you are more than welcome to use it for legitimate purposes. Really something I intended mostly for myself, friends and anybody that came along after following the shortened URLS. I’ve even thought about making the code behind it completely open source so that others could create their own short URL services. Unfortunately, lately it’s been getting pounded by spammers.

At first I thought it was just a couple spam URLs but it was worse than that.

Recently I’ve been getting email notifying me of the severity of the problem in some other places and have been having to put on my cop’s hat and spend about half the amount of time I wrote the program cleaning it up and further securing it against use by spammers. Fortunately, I setup click count tracking from the beginning so I could see what redirects are getting the most activity and follow them. A couple of database queries and I had all the information needed to jump into anti-spam mode.

Redirecting the redirect
During my investigation I noticed spammers taking affiliate links and running them through my short URL service and then spamming the shortend URL everywhere. Several of these URLs were posted on craigslist with just one redirect being clicked well over 140,000 times (heaven knows how many people saw that link, that’s how many who clicked it). I also found some 127.0.0.1 links which for those not familiar will open a local host. Clearly, some were at least attempting to use my service for nefarious purposes.

My first plan of defense was to cut off the money source by redirecting these spammy redirects. When/if the spammers realized that their links would be changed then using my service in the future would be futile because I’d just redirect their redirects back home. The hope being that they would go off and find some other short URL service that would be less dilligent. The problem almost overnight cleaned itself up, literally and figuratively but I’ve had a couple stragglers still banging away trying to figure out how to poke holes in the program so they can use it to spam.

Where to send the clickers?
I thought about creating a blackhole directory like the one that exists here at Hmm (not linked) and is intended for malicious bots, but then I decided instead just redirect these people clicking to the service homepage which has in bold red text “please do not spam” as one of the instructions for use. My gut reaction isn’t to be so nice with spammers and give them more of a reaming, but it’s possible that people don’t realize that masking an affiliate URL and then putting it places where other sites have strict rules against spam is still spamming.

There is no advertising or any money being made whatsoever from my free URL shortening service. Never was and currently isn’t as of this writing. It wasn’t done to make money and now in some ways I’m starting to regret sharing it with some others for free. I have to wonder why there are any free URL shortening services out there if others are experiencing what I have without promotion and advertising.

The ironic hidden value experience
The flip side is that the spammers have made the service better. They’ve forced me to turn the code into being more anti-spam than it was when it was first put out there. I can see how creating, executing and maintaining a free URL shortening service could be a good learning lesson for somebody researching and learning about anti-spam techniques and technology. That wasn’t one of my initial goals, but lately anyway, that’s become one of the realities.

While my service isn’t as full featured as some other URL shortening services, folks can subscribe to an RSS feed of the most recent clicked URLs, which at least at the time I created the service was somewhat progressive. If I really had wanted to get serious about it, I would add even more features, including some way to group and remember short URLs users have created. Starting to ponder that a little more seriously now that I’ve been spending more time back in the database and code.

What you gonna do when they come for you? Bad boys, bad boys …
When I started removing some of these spammed URLs and techniques as well as blocking the ability to sign up anew, I started noticing the spammers using multiple layers of redirects. Sneaky. They would use a service like tinyurl to mask their URL to mine, so next I had to block all the other short URL services out there too. What legitimate use would somebody want to use one short URL service for another short URL service anyway?

Next I had to block a couple foreign country URLs as I noticed a high percentage of these spam URLs coming from certain countries. I also decided to block IP address URLs. I realize this eliminated some legitimate long URLs that don’t have domain names and only use their IP address, but the investigation revealed that the vast majority of the long URLs that didn’t have domain names were spam.

Free web services have hidden costs
If you are developing a free, web-based service, expect to spend more time policing it than if it were a commercial service. If I’d have made the service free for me and $1/month for everybody else, I’m curious what the difference in spammer usage would be?

Something to add to the cost of all these new web-only startups because you just know some of them are spam magnets too. It might be free to netizens, but the labor and resources in keeping these services running is a whole other story.

From the what will the spammers (try to) ruin next department comes a series of del.icio.us popular links leading to splog pages.

Solutions
Yahoo could help out by filtering out exess results so no one user account ever fills up the popular list with more than 1 “most popular” thing? Crap from the spammer who loves internet today (pictured to right and a second screenshot from delicious/popular as of this writing) wouldn’t invade our RSS boxes, if there was a maximum of their most popular of the most popular hit the page. And if there were more than say three at one time, that should set of red flashing lights and spam alarms at Yahoo HQ.

Using Feed Rinse to filter RSS spam
Fortunately other tools exist to combat these situations like Feed Rinse. In less than a minute, I generated the following rinsed feed to filter out anything from this miscreant. You can subscribe here

Some publishers are against services like Feed Rinse but cases like this one make a strong case for protecting these tools. Last month Cory Doctorow wrote on Boing Boing:

This is a service that’s both so vital and so obvious that it’s practically an indictment of RSS feedreaders that they don’t all include this already.

I’d like to see a built-in plugin for Reblog that adds filtering functionality. In a world where there was unlimited time, I’d create one myself. If any other developer reader creates or locates one, please let me know.

Update 8:29am PST: Om noticed the problem this morning too.

Matt notes that blog comment antispam tool Akismet is now sharing its ham and spam stats on a special page:

These numbers seem more realistic to me than other reports on comment spam. I’d say the numbers for targeted PR blogs are more like 95%+ are spam, at least according to our own filters. Akismet is free to blogs that do less than $500 USD/month in revenue and was first reviewed at Hmm in December and then mentioned the false positive concerns to Matt directly at Northern Voice in February.

I’m not surprised that Apple closed their sacred Mac OS X for the newer Intel machines. Macworld.uk:

Thanks to pirates, or rather the fear of them, the Intel edition of Apple’s OS X is now a proprietary operating system. Mac developers and power users no longer have the freedom to alter, rebuild, and replace the OS X kernel from source code. Stripped of openness, it no longer possesses the quality that elevated Linux to its status as the second most popular commercial OS.

People criticize Microsoft for being closed — and rightly so historically on a number of fronts — but Apple has their share of proprietary products and offerings too.

Perhaps the bigger story here isn’t proprietary hardware and software — both of which are neccessary, despite the hardcore open source cries (hey, I like open source too) — but the issue of the bad guys continuing to be the thugs of the web. Blue Security threw in the towel with their fight fire with fire campaign against spam.

Open source has its pros and cons. Pure cons to me are the bad guys of which can study and scrutinize every line of code finding the exploits. If they would only report them to the proper parties — and these companies would do something (cough, cough, Sony) before going to these security sites to make names for themselves or worse going underground and exploiting for personal gain.

I don’t blame Apple here. Yeah, this might mean less cool OS hacks, but then there is nothing stopping somebody from making their own brand of Linux and keeping that open source like Ubuntu. The article says they admit to things being in “flux” so nothing is permanent here.

Last week I got two spams that seaped through comments on problogger. I sent Darren a short reply sharing my disappointment that these “undesirables” slipped through his filters. Today, I see Darren has written about hitting comment spammers and plagiarists where it hurts by reporting the ones that use Adsense:

Every day I find people wasting my time and trying to make a quick buck from my blogs in ways that are either illegal, malicious, morally corrupt or deceptive. Most of them do so using some sort of mainstream Ad network (usually AdSense) and I’ve decided to report each one of them to AdSense using the built in mechanism on each AdSense ad.

Tear ‘em up, Darren. I’m with you 1000% on this one.

Yesterday during Mother’s Day this blog received six separate 150+ comment blasts from comment spammers targeting keywords related to online gambling and drugs. My filters trapped and dispatched all these comments before ever making it live on the site, but I think it’s especially telling how lowlife some of these people have become. They will do anything on any day, including holidays and weekends when they tend to accelerate their scams, to get their crap in front of us.

Robert Scoble who is currently going through some personal hardship with his mother who recently had a stroke and has received overwhelming support in the comments area of his blog also sounded off about how clueless some marketing people are. These insensitive fools couldn’t take enough of their time to look at his blog and see what was going on, but had plenty of time to email stuff to him to be reviewed.

If there are any spammers reading this — extremely doubtful, I realize — you better be aware that our servers are known hostile area for your BS and you will not get through. Spam us and risk having your ass being abuse reported to your hosting company and the affiiliate program(s) being notified. If the activity persists and the affiliate owner does nothing, we will notify the affiliate program’s hosting company complaining about the spam too. Hosting companies do not want to get onto spammer blacklists because that can impact all their customers, including the legitimate ones.

The only way we’ll rid the spam scourge from the internet is by taking away the incentives, cutting off the money at the source. While it will never be completely eradicated, I believe if enough webmasters bind together, the risk of spamming will be so high that only the insane will partake.

Brook Schaaf crunches the numbers from an extensive siteadvisor study evaluating the safety of search engine results. MSN fared the best in returning safe results while ask.com was the worse (disclaimer: I own stock in Ask, Google and Yahoo) . Google? In the middle of the pack.

Since the end of February I’ve been using the Firefox extension for siteadvisor [hmm siteadvisor review] and like its easy red, gray, yellow and green color scheme for the safety of sites. Whenever I see red (unsafe), yellow (alert) or gray (no results found) I will rethink whether I want to visit the website or not. With the extension you can see the siteadvisor results proactively at the end of each search result. Very handy.

Another interesting part of the siteadvisor unsafe search results study that Brook also highlights: “Sponsored results were two to four times more likely than natural results to return something unsafe.” How’s that for instilling advertiser confidence?

Sterling wisely pointed out that the data is only as good as the submitters. From what I’ve seen after using this service daily for a couple months, the submitters and siteadvisor’s algos are pretty good at sniffing the trash. Siteadvisor is becoming my favorite Firefox browser condom.

Didn’t I just talk about not trusting most third party stats vendors (IE. Alexa) a couple posts ago? Check this screenshot out:

For those who don’t use Wordpress this is one of the sidebars in the admin area (default) which shows recent link activity reported by Technorati. You can click thru on the “more” part and be taken to Technorati most recent links to your blog.

Spammers and sploggers eating up their index? Momentary glitch? Thing is I’ve seen this happen several times before. I wrote about how abysmal Pubsub’s stats were here. Don’t get me wrong I think Technorati is atop the pile for what they do but this is simply more evidence that you can’t put too much stock in the numbers they — or any other third party tracking service — provide. And I certainly wouldn’t base any strong opinions or doom prophecies based upon them. They do provide a somewhat useful service and can be helpful sometimes, but their accuracy is seriously in question. I feel for them trying to police against the sploggers and spammers and it’s that a main reason nobody has bought their company yet.

I’ll take the Apache logs over any third party information when accuracy truly matters.