In 2004 I decided to create my own URL shortening service. The plan was rather than use somebody else’s short URL service — there are plenty of them — I’d use my own in email correspondences, forum posts, anywhere that I needed to shorten a long URL. I registered the domains tdurl.com and the alternate tduri.com, created the database and small script that powers the program and began using it with little fanfare. Never even posted about the launch here, although I have mentioned or used it in a half dozen posts according to the search.

Clearly this hasn’t been something I’ve promoted for others to use, although you are more than welcome to use it for legitimate purposes. Really something I intended mostly for myself, friends and anybody that came along after following the shortened URLS. I’ve even thought about making the code behind it completely open source so that others could create their own short URL services. Unfortunately, lately it’s been getting pounded by spammers.

At first I thought it was just a couple spam URLs but it was worse than that.

Recently I’ve been getting email notifying me of the severity of the problem in some other places and have been having to put on my cop’s hat and spend about half the amount of time I wrote the program cleaning it up and further securing it against use by spammers. Fortunately, I setup click count tracking from the beginning so I could see what redirects are getting the most activity and follow them. A couple of database queries and I had all the information needed to jump into anti-spam mode.

Redirecting the redirect
During my investigation I noticed spammers taking affiliate links and running them through my short URL service and then spamming the shortend URL everywhere. Several of these URLs were posted on craigslist with just one redirect being clicked well over 140,000 times (heaven knows how many people saw that link, that’s how many who clicked it). I also found some 127.0.0.1 links which for those not familiar will open a local host. Clearly, some were at least attempting to use my service for nefarious purposes.

My first plan of defense was to cut off the money source by redirecting these spammy redirects. When/if the spammers realized that their links would be changed then using my service in the future would be futile because I’d just redirect their redirects back home. The hope being that they would go off and find some other short URL service that would be less dilligent. The problem almost overnight cleaned itself up, literally and figuratively but I’ve had a couple stragglers still banging away trying to figure out how to poke holes in the program so they can use it to spam.

Where to send the clickers?
I thought about creating a blackhole directory like the one that exists here at Hmm (not linked) and is intended for malicious bots, but then I decided instead just redirect these people clicking to the service homepage which has in bold red text “please do not spam” as one of the instructions for use. My gut reaction isn’t to be so nice with spammers and give them more of a reaming, but it’s possible that people don’t realize that masking an affiliate URL and then putting it places where other sites have strict rules against spam is still spamming.

There is no advertising or any money being made whatsoever from my free URL shortening service. Never was and currently isn’t as of this writing. It wasn’t done to make money and now in some ways I’m starting to regret sharing it with some others for free. I have to wonder why there are any free URL shortening services out there if others are experiencing what I have without promotion and advertising.

The ironic hidden value experience
The flip side is that the spammers have made the service better. They’ve forced me to turn the code into being more anti-spam than it was when it was first put out there. I can see how creating, executing and maintaining a free URL shortening service could be a good learning lesson for somebody researching and learning about anti-spam techniques and technology. That wasn’t one of my initial goals, but lately anyway, that’s become one of the realities.

While my service isn’t as full featured as some other URL shortening services, folks can subscribe to an RSS feed of the most recent clicked URLs, which at least at the time I created the service was somewhat progressive. If I really had wanted to get serious about it, I would add even more features, including some way to group and remember short URLs users have created. Starting to ponder that a little more seriously now that I’ve been spending more time back in the database and code.

What you gonna do when they come for you? Bad boys, bad boys …
When I started removing some of these spammed URLs and techniques as well as blocking the ability to sign up anew, I started noticing the spammers using multiple layers of redirects. Sneaky. They would use a service like tinyurl to mask their URL to mine, so next I had to block all the other short URL services out there too. What legitimate use would somebody want to use one short URL service for another short URL service anyway?

Next I had to block a couple foreign country URLs as I noticed a high percentage of these spam URLs coming from certain countries. I also decided to block IP address URLs. I realize this eliminated some legitimate long URLs that don’t have domain names and only use their IP address, but the investigation revealed that the vast majority of the long URLs that didn’t have domain names were spam.

Free web services have hidden costs
If you are developing a free, web-based service, expect to spend more time policing it than if it were a commercial service. If I’d have made the service free for me and $1/month for everybody else, I’m curious what the difference in spammer usage would be?

Something to add to the cost of all these new web-only startups because you just know some of them are spam magnets too. It might be free to netizens, but the labor and resources in keeping these services running is a whole other story.

- Firefox screenshot extension that grabs either part of current page or entire website and saves as screenshot. Also see Webshot (Windows) and Paparazzi (Mac).
- 6 of the last 7 months the PS2 has outsold the Xbox 360
- Bloggerkit provides an easy way to insert Amazon ads into blog posts (already not that hard with various plugins out there, and I’m not sure how many more third party javascript copy/paste programs bloggers will be interested in). The price? Bloggerkit shaves 15% of the code displaying their Amazon affiliate ID, with 85% for your ID. After adding the code to your blog, just insert the keywords. An example is shown below for blog-related terms.

bk_keywords: blog, blogging
They suggest coloring the keywords white so they won’t be seen, but that’s frowned upon by the search engines. Better off just leaving it show, if you want, or perhaps better yet just use one of the many Amazon plugins out there where you will receive 100% of the Amazon ID showing.

Note: this isn’t the same thing as an 85/15% revenue split. It’s purely based on script views, so that means you could have a really popular page where 100% of the revenue went to bloggerkit’s Amazon ID (people bought during the 15% time displayed) and 0% went to your code if the purchases were made during the Bloggerkit displays. The converse is true as well. Something worth keeping in mind, especially because it’s difficult to audit this type of affiliate arrangement unless you periodically run tests checking the generated code. It would have been nice if they would have provided a layer of stats to backup 85/15 display ratio rather than say: “Amazon provides all the tracking record you need.”

- one of the most comprehensive pages I’ve seen of Cascading Style Sheets (CSS) resources.
- At the eBay Live conference in Las Vegas Digg President and Tech TV alumni Kevin Rose announced that in the “next month or so … will add coverage of world news, entertainment, politics, and more. Major redesign is in the works.” [Business Week]
- AOL has beefed up their open AIM developer program “with new tools for providing PC-to-PC voice communications, creating IM robots and detecting users’ geographic location.” [see developer.aol.com]

I thought I’d blogged this before but a quick search didn’t reveal the answer here, so I’m blogging it again. In the summer make sure the blades are spinning clockwise and in the winter counter-clockwise.

Fan change reminder script
When it gets too hot you’ll be reminded to search for the cooler setting and vice versa in the winter, but I decided I wanted to whip up some quick PHP code to automatically remind me on one of the admin pages I use. Something along these lines could be done in just about any language, perhaps as a widget/module/gadget, whichever you prefer.

Web2.0Central has a list of 13 reasons developers should CONSIDER (their caps, not mine) using Microsoft for web apps. Among the list that I disagree with, bolded is theirs, the comments are mine:

2. Free Database Which I would add the word “crippled” between those two words like many of Microsoft’s free developer tools. It’s curious that the author goes on to list reasons I would be against using SQL Server Express for most, if not all projects. If you want to do custom deployment for your application, although web apps are not a good example of course, Visual Studio Express tools are not for you. It’s a good thing they offer some kind of database, but this developer would rather use PostgreSQL or MySQL than plan for the day the app can no longer run on SQL Server and the Microsoft cash registers will be ready to hold the application hostage.

3. Microsoft Atlas makes AJAX easier. I’m not sure who decided that javascript and XML were so difficult that expensive (in terms of weight and size, not dollars) libraries needed to be created and deployed and now used as potential selling points to developers. Rasmus Lerdorf, creator of PHP, debunked this myth over a year six months ago writing:

Web 2.0 and the programmable web that I and others have been talking about for a while has mostly been vapourware so far.

Interestingly enough, Rasmus works for Yahoo which created a set of AJAX-powered UI tools and he added an addedum to his original post to include the Yahoo UI tools:

What I was hoping to get across here is a simple example of how you can use PHP as-is, without additional complex external layers, to apply an MVC approach with clean and simple views and still have all the goodness of fancy Web 2.0 features. If you think I am out to personally offend you and your favourite framework, then you have the wrong idea. I just happen find most of them too complex for my needs and this is a proposed alternative. If you have found a framework that works for you, great.

My experience using these type tools has not been as favorable as just writing the raw code which is usually much, much smaller and less bulky as Rasmus had originally pointed out. That doesn’t mean I don’t think it’s a good idea that Microsoft and Yahoo offer these tools to developers, however, I don’t see it as a compelling reason to switch.

5. Microsoft solutions can scale..

I have heard and talked to people who swear this is true but I’ve never experienced this firsthand because all of our servers are UNIX-based and 99% of the customers/clients I work with also use servers running some form of Linux. Clearly, I’m inexperienced with Windows servers, so I’m just throwing out user experience as a guide, not developer experience. Also, I’m reminded of recent events like Microsoft releasing their betas for download and experiencing — see screenshot above — of errors from programs because of server load. Scaling? Sorry, although their MySpace example is definitely compelling, not sure I buy this one.

7. Ray Ozzie. I’ve never used Groove so Ozzie bringing these features to Microsoft products is supposed to impact me as a developer how? The post points out two examples that were good for making Microsoft more open:

innovations like Live Clipboard & Simple Sharing Extension are important plumbing in the Web 2.0 world and by contributing these ideas in an open manner is a sign of Microsoft being a better corporate citizen.

Hooray for adding an additional RSS extenstion and making it open. Live clipboard is cool, though.

8. Robert Scoble. Come on. I like Robert, read Robert’s blog regularly, have met Robert a few times, think he’s a good blogger and think he’s underpaid as a Microsoft PR-type guy but his blog hasn’t motivated me one iota to develop using more Microsoft products/services. He has taken some interesting Channel 9 videos but I believe Robert’s value is less to developers, even though he works in the C9 area, than it is to the overall face of Microsoft the company. I’ve said this before, give him a raise, but don’t put him in a list of reasons developers should CONSIDER using Microsoft.

9. Being based on the Microsoft platform doesn’t limit your acquisition options. I think Writely is a bad example, not a good one. This says to me that Google made a bad investment, not that an ASP-powered web service in any way betters the chance of being acquired. This isn’t saying Writely is a bad product and would be better if it didn’t use ASP, but I disagree that any online wordprocessor that I’ve seen/used/tested is superior to the traditional desktop wordprocessors. I’d much rather use any version of Word than Writely.

12. Microsoft has good development resources. I would agree with this one except to add that the resources they provide are far too scattered. I often use Google to find the developer answers I need. On the bright side, search has been improving.

13. Microsoft speeds web application development. Even the author doubts this one and so do I. Let’s take making a ’simple’ Microsoft Gadget vs. creating a Google Module. I was able with zero learning time to create a full blown keno game as a Google module. I’m only able to create slightly more advanced versions of Hello World in the Microsoft Gadget environment with the same amount of zero learning to build time. That’s not speedy web application.

To add to this frustration, Microsoft still to this day says the gadget SDK is experimental and subject to change, so why should I get involved, learn and build, and then have to redo/change my app to comply with the version they decide upon? I would like to build Live gadgets, but at what price in future development time? Make up your mind, Microsoft, find a format, lock it down, and then we can compare if it actually speeds web application development.

Hat tip to Pete from RasterWeb who has an item by item counter for the CONSIDER list.

Not all developers hate Microsoft
I don’t hate Microsoft, although some of my writing might make it seem that way. My whole family, save for our youngest, travels up to Redmond to help beta test products and we enjoy doing so (also like getting ‘paid’ in software). I wanted to go to Search Champs, was excited and delighted to be chosen, and I continue to be involved there as much as the opportunities arise. In fact, just last night I signed up for something in beta that was offered via the private search champs mailing list. Within an hour of seeing that invite, I went in and spent time using the product and offering feedback. I like the employees, although I strongly believe some of them deserve to be shown the door instead of given towels. I really would like to use Microsoft more in my web development but I’m just not there yet. This doesn’t mean I’m close-minded and will never be there, just not there yet. I do like using Microsoft for desktop development and I think Visual Studio 2005 is a very good product. The consider list doesn’t do it for me despite having a few favorable points there, which are the points I didn’t list, BTW.

What am I missing here? Why should small developers like me move web development time in Microsoft’s direction?

If you want to do any sort of local Microsoft gadget development and testing with Visual Studio 2003 or Visual Studio 2005 .NET on a Windows XP Machine you’ll need to set up a local webserver. The following instructions were written by Forser and myself and intended to be published at our script school blog, but is being published here first.

1. First take a look if you have IIS already installed or not.
Go to control panel / Administrative Control and look if you have an icon saying Internet Information Services.

If you don’t then you have to install IIS by following these few steps.

(Note: if you are using Windows XP Home, then you will need to see these instructions)

2. Go to Control Panel and click Add or Remove Programs and click Add/Remove Windows Components.

3. Click the Internet Information Services (IIS) checkbox and press Next to install, You need the CD during the install.

Afterwards you check if you have the Internet Information Service icon under Control Panel / Administrative Tools.

If you have the Icon then next step is to install the ASP.net.

5. Start a command by selecting Start / Run and type cmd

Now you should have a CMD window open.

6. Now go to your dotnet folder in windows

cd \windows\Microsoft.NET\Framework\Version\

Replace Version with the dotNet version you have installed, like v1.1.4322 is dotNET v1.1 and v2.0.50727 is dotNET v2.0

7. in that folder you type aspnet_regiis.exe -i and it will say Finished installing ASP.NET (1.1.4322.0) if you install v1.1.4322 that is.

8. Afterwards you close the cmd by typing exit and it will close the window.
9. Go to Control Panel / Administrative Tools and click the Internet Information Services icon so the internet information service manager starts.

10. Now press the + next to the name and right click Web Sites and select properties and look under the ISAPI Filters after ASP.NET name, if it shows up then you have ASP.net installed.

11. Now to install the Microsoft Gadgets, Firstly you need to unpack the gadgets file you downloaded from Microsoftgadgets.com into a folder like c:\gadgets\

12. Now, press the + next to Web Sites in Internet Information Service and right-click Default Web Sites and select New > Virtual Directory

13. Press Next on the first screen, Enter a alias on the next screen like Gadgets and press Next.
Now select Browse and point it to your directory where you installed the Gadgets like c:\gadgets\ and press Next and on next screen let it be the default Access Permissions and just press Next and then Finish.

14. Now you need to add Live.com to your Internet Explorer Trusted Sites by going to Control Panel, select Internet Options and there select the Security Page, click the Trusted Sites Icon and press Sites so it looks like the picture below.

15. Now enter http://www.live.com and press Add, Make sure that you don’t have the Require server verification (https:) for all sites in this zone checked.

To see if the samples work type http://localhost/gadgets/hello/helloworldgadget.xml (or the name of the xml file if it’s changed, see the XML file name inside the Hello folder) and it should show you the XML for the hello world sample.

16. Next is to add your sample/work to the http://www.live.com site.
First go to it with Internet Explorer and press Add Content and you will see a Search box under Add Content., Press Advanced Options and enter http://localhost/gadgets/hello/hello.xml into the Add a Gadget by URL box and press Add.

Now it should show the Hello World gadget under My Web if working, and if you press the gadget name Hello World you will get a question if you want to install it.

17. Press Install Gadget and it will run the Gadget on the website and you will see it saying Hello World.

Now you are done and can continue to play around and develop your own gadgets, read more on Microsoft Gadgets which is a gadget gallery, development forum and blog. Also, coming soon is dev.live.com which will be a primary development area for live.com-related projects.

Kudos to the Microsoft Office team (or whomever is responsible) for coming up with the send a smile tie-in for Office 2007 testers [download here Update 10/19/2006 4:15pm PST: Reader Raven reports and we confirmed the download is no longer available.] :

Send a Smile is the best way to send feedback about Office 2007. Believe it or not, your feedback goes directly to us on the product team—directly to the people responsible for making decisions about the product in every area. In some cases, if you choose to give us your e-mail address (which is optional) we might contact you to follow-up and get more information.

Although normally I’m against programs running in my systray — too many try to take that liberty — utilities like this are smart.

A few things I’ve sent a smile or frown already about with Word 2007:

- word count displayed by default
- integrated blogging (smile)
- blog image HTTP / FTP instructions (frown)

- send a smile/frown keeps track of all the feedback you’ve sent, along with screenshot so you can refer to what you’ve sent in case the team wants more information.

Hat tip to Download Squad for the head’s up on this one.

Yahoo offers some neat tools for developers, but do they scale? Matt from Wordpress writes of troubles they are having:

I loooove the way Yahoo provides results as serialized PHP in addition to XML (every API provider should do that!) but the underlying search product seems to be built on a shaky foundation.

I haven’t experienced the Yahoo search API tools falling over in our own projects, but clearly something is amiss in Matt’s camp with using them. For developers wondering about how to use the Yahoo API, I wrote an article last year published by Informit that should get you up to speed fairly quickly.

Meanwhile eBay and Yahoo have cut a deal for YPN search results to show up in eBay search results which some are referring to as late and “reactionary” to Google:

“There’s no question that this is a significant partnership, but it seems reactive,” says Allen Weiner, a Gartner Inc. analyst. “It’s anti-climactic, because we’ve been waiting for years. If they had done this two or three years ago, they would have done it from a position of strength.”

Acquisition rumours continue with Microsoft allegedly taking a serious look at buying eBay, which again some believe would be a counter to Google, but that is staunchly rebuked by Microsoft CEO, Steve Ballmer:

Ballmer said: “A tie-up with the sole aim of shutting out a competitor makes no sense. The partnership must produce something that is useful for users and advertisers.” He added, however, that he did regularly talk to eBay’s CEO Meg Whitman, who he said was a close personal friend.

Logically, there are executive concerns about anti-trust violations in such a deal. I don’t see an eBay buyout being a very likely reality, but it’s interesting to ponder what PayPal and Skype would be like under the Microsoft umbrella. Just think of the possible names:

Microsoft Windows Live Skype
Windows Live Skype Messenger
Windows Live PayPal

And don’t forget the developers:

Windows Live .NET PayPal API
Windows Live .NET Skype API

Seriously, I think an eBay buyout, especially with the Skype debt, would be too expensive an undertaking, even for Microsoft. If these rumors still have any traction after the holiday weekend Microsoft and eBay might see a tiny punch.

Techcrunch points out that Time has started using the Sphere It function on its articles. I took a look at the functionality and liked the results it returned. Note: Sphere beta was reviewed at Hmm in November 2005.

The Sphere It function analyzes the context of a page and offers related post results from other blogs. I checked out a few posts at Hmm to see if Sphere It was returning relevant results and decided this could be useful in the share/bookmark link section at the end of each post here for readers. I wanted to implement this.

However, I did not want this Sphere It link to show up on posts without a certain (configurable) amount of words, so I create a small hack for the WP single.php template to check the number of words in the post and if the criteria was met (more than 75 words) return the Sphere It link with the proper permalink for that post. The code would also strip out the HTML tags so those weren’t counted in the total number of words for each post.

Working examples
See this post which contains more than 75 words. Note the “save/share/explore” link with Sphere it afterwards. All other posts show “save/share” only after the post and the Sphere it link is not there. See example with less than 75 words here.

Instructions
Though the following how to instructions are intended for Wordpress server-side powered blogs (not wordpress.com) it is possible to use a similar technique in other blogs that allow PHP scripting.

Edit the single.php template (found in admin area under Presentation->Theme Editor->Single Post) by replacing the following code:

with:

STEP 2. Save the changes by clicking on “update” and then visit a post with more than 75 words and one with less to test. All well, no errors? You’re done.

Notes
This code would be neater wrapped in a plugin, but I wanted something quick and dirty so I could update from within the template. Feel free to make a plugin out of this if you want.

If you need to collect money online for some purpose from a group Chipin might come in handy.

During beta testing there are no fees, but after they leave beta they promise: “we will propose a tiered fee structure that will keep costs absolutely minimal - and in many cases still 100% free to organizers, participants, and recipients.”

Our VTOR group inside Second Life (SL) uses a voluntary donation script linked to an object which looks like our group mascot. Recently, Brady from O’reilly talking about how Second Life for micropayments:

The amazing thing about these transactions is that over 85% (just under 4.5 million) of them are conducted for amounts under a dollar; 57% of them are conducted in amounts under $0.07.

SL might be the most successful example I’ve seen yet on the web of collecting small amounts of money online. Next week they are releasing version 1.10 of the Second Life software which (finally) will include HTTP functionality. This could lead to some exciting new development.

One of the things on my to-do list today was building some subscribed links for certain Google queries to generate relevant content on Hmm. I would collect these query results under a single MakeYouGoHmm subscribed link XML file and then subscribe to it under my name. I’m recording my notes and code from the process here.

My first test was altering the subscribed links to point to my bio whenever someone subscribed and logged in typed a query for my name. Since many sites/people have a profile or bio page somewhere, this seems like a more real world useful example than Hello World which is typically offered as a “hey this works” example.

Follow along if you’d like to build one for linking to your about/bio page, here’s how to do that:

STEP 1. fire up your favorite text editor and add “hello world” example to start with. Save this file as “domain-subscribedlink.xml” — in my case I used “makeyougohmm-subscribedlink.xml.” You can name this file anything you want, of course, but this is the naming convention I used.

STEP 2. In the Hello World example code, change the title from:

Replace YOURNAME with your name:

STEP 3. Change the more_url code:

Replace with YOURURL without the http:// part, which is the location to your about/bio page. Here is how mine looks:

STEP 4. Add from 1-3 lines of short text about you by changing:

Replace with at least one line of description text like this (change YOURNAME to your name):

STEP 5. Verify and save the XML file. If all the above steps are done correctly you should have a file which looks something like this:

Everything look correct? Saved? Move on.

STEP 6. FTP the domain-subscribedlink.xml file to your website space. I chose to put in an ext subdirectory for Hmm, you can put wherever you want on your site. Tip: might want to make it a somewhat uncluttered place, so you don’t forget about it.

STEP 7. Per the Google instructions:

Go to http://www.google.com/coop/manage/subscribedlinks. You’ll see a page with a space to enter in a URL for the XML files defining your Subscribed Links. Enter the URL of your modified file and click the “Add feed” button.

If you want to try my test URL out above use the following URL:
http://www.makeyougohmm.com/ext/makeyougohmm-subscribedlink1.xml

OR you can also subscribe by going to step #8

STEP 8. Now wait for a few minutes. Your subscribed link(s) should show up on your profile page [here’s my profile page]. Be sure to keep track of your profile URL so you can share on your website. Others can subscribe to your subscribed links from your profile. Have a few minutes gone by? Now you can run a test query for “YOURNAME” — in my case try “tdavid”

My example looks like the picture at the top of this post. How does yours look?

Multiple queries for same search results
Sometimes people call me “TD”, sometimes “TDavid” and I also use “TDavids” plural when the singular isn’t available, so I wanted to modify the file to be able to show the same bio page for all three of those specific queries. In my next How To for subscribed links, I’ll go through step by step how to alter this file so multiple queries can lead to the same result page as well as using links inside the description text.

Expect that link a little later (will update the link on this post), as I’m still working on that post.

Update 12:58pm PST: Seems like their system isn’t working at the moment for any changed/new XML files.

Update 1:24pm PST: Ahh, my bad. I had wrongly used the same ResultSpec ID for my first and second XML file, invalidating both of them. Fixed that by removing the file, fixing the error and updating the XML file.