Tailrank 2.5 Now Available

Not only are we announcing Spinn3r 2.0 today but we're announcing that a new version of Tailrank is being released as well.

If you've been a regular reader of Tailrank over the last few months you might have noticed a number of incremental improvements. Tailrank 2.5 is far more evolutionary than revolutionary.

We've spent a lot of time focusing accuracy of Tailrank's core internal algorithms. What works for one blog or even 1M of blogs in our index tends to fail from time to time when working on 12M blogs.

In this release you'll also see:

MUCH Larger index

Tailrank uses Spinn3r 2.0 as it's core crawling platform. Spinn3r 2.0 is now indexing 12M weblogs which means that Tailrank has a much larger index as well.

This makes Tailrank a lot more democratic (as well as the largest memetracker). You don't have to be an A-list blogger anymore. Now you just have to say something insightful and intelligent and you'll be included in the discussion.

Improved Document Clustering

We've improved upon the design of our clustering algorithms so they can process much more data than before. In fact, we're now clustering within a sixty day window so that if newly published stories cover a meme which happens in the past it will be published within the correct meme.

Spam Prevention

As usual we've spent a lot of time on spam prevention.

A great deal of this work has been done within Spinn3r but there's another layer on top of Tailrank which handles other types of spam which are fatal for memetrackers but not for regular web crawlers.

Improved Accuracy

A lot of time and effort has been spent improving our summary extraction, post categorization (tech, politics, etc), and title extraction.

This seems like a smaller issue but it really helps Tailrank seem professional and seriously improves the readability of the product.

What's Next for Tailrank?

We're hard at work on Tailrank 3.0. This will unify technology present in Tailrank with the Spinn3r backend which is now more advanced in a number of areas.

Tailrank 3.0 will be a big release for us. Just as big as Tailrank 2.0 which we launched nearly a year ago.

Tailrank Blocks PayPerPost Bloggers

Today Tailrank had the unfortunate task of blocking a few weblogs from Tailrank's index due to link spamming.

These were PayPerPost bloggers linking to Sproose which purchased a PayPerPost campaign to astroturf their release.

We have no problem with bloggers selling ads and trying to make money but this type of linking behavior is essentially spam.

We're willing to reinstate these blogs if they'll migrate to using rel="nofollow" for future PayPerPost sponsored posts. For example, nearly all links in this post use rel="nofollow" to avoid confusing memetrackers and search engines.

Sproose is also blocked and we're willing to reinstate them as well if they'll stop running PayPerPost campaigns without insisting on a nofollow link. Spam is a top priority for a search engine and for them to resort to link spam to advertise their product is a bit hypocritical.

The spammed post will remain in our index for historical purposes but the ranking is reset and won't show up on any of our archive pages.

To date, we've been very trusting when adding weblogs to our index. This has paid off because we haven't attracted the spam that is problematic with other services. In fact, this is only one of a handful of spam posts we've had to deal with in the last year since our launch.

PayPerPost has received a great deal of criticism in the press for their lack of ethics. What I find most disturbing is the fact that PayPerPost is willing to hurt the search rankings of bloggers by not communicating the problems (spam) with selling links.

Tailrank isn't the only large site with this policy. MSN search has also promised to block websites that sell links. They're sending off email messages notifying them that they've been dropped from their index:

Your site is acquiring links through posting to or exchanging links with sites unrelated to your site content. Techniques which attempt to acquire unrelated spam links in order to increase ranking are considered spam and your site has been excluded from our index as results. Please contact us once you've removed these links and we will reevaluate.

The only party I find at fault here is PayPerPost. We'd really love to reinstate these bloggers and add them back to our index and welcome them home with open arms. I assume they simply weren't aware of the problems with selling this type of link spam.

Tailrank on the Scoble Show

I saw down with Robert Scoble about a week ago to talk about Tailrank for the Scoble Show:

Kevin Burton is a talented developer who has worked on a variety of startups already including Rojo, and now TailRank which he started to be able to see what bloggers were talking about. Here I sit down with him for an interesting conversation in the lobby of San Francisco's Palace Hotel.

I think the interview turned out pretty well. The only mistake I made was that I left my cell phone on which is a slight problem. Luckily no one else called during the interview (sorry Robert).

I also gave a demo of Tailrank. Unfortunately, the realtime IM delivery feature actually worked right after they shutoff the camera. It was pretty amazing actually. Our crawler found a post on that topic right after I subscribed to the meme.

Tailrank Interview on Folksonomy.org

The guys over at Folksonomy.org have published an interview with me that they conducted last week. Short but sweet.

What is TailRank and what are its major advantages over similar services?

Tailrank is a service that allows you to track the hottest news stories across the blogosphere.

There are a few similar services but we track more blogs, allow the user to create their own version of Tailrank, support full-text search, and allow delivery via Instant Messenger.

Please Take the Tailrank Reader Survey.

If you guys have a few seconds I'd really appreciate you taking the Tailrank reader survey.

This will help FM Publishing sell advertising on our site which means we can use the profits to invest in our infrastructure.

It also means we'll start to see more cool ads like Apple Computer and Dice.com which are much more tasteful than the bouncing heads or "punch the monkey" alternatives.

Tailrank in Top Ten Fastest Sites

We just rolled out some performance updates the other night and it looks like they've worked out very well! Tailrank is now in the top ten fastest sites over on grabperf. We're even faster than Google News!

What's really interesting is that I think I can get another significant performance boost out of the system which might allow us to take on the #1 position. I've got my eyes on you Technorati Mobile! ;)

Actually, I should get Tailrank Mobile on there. I'm sure it would take #1 slot right away.

Tailrank Indexes More Weblogs

If you've noticed more traffic on your blog recently it might be because Tailrank is now indexing more weblogs. We've deployed a new crawler and this week alone we've added 15k new blogs!

What's really amazing about this system is that we're adding new weblogs according to our proprietary ranking algorithms. This means that we should totally skip spam blogs altogether and weblogs will be added based on their ranking. This way Tailrank is always indexing the top weblogs (which have the most influence).

The goal of course is to add even more of the long tail into our index until we're indexing the whole blogosphere. We're in the process of deploying new hardware and working towards Taillrank 2.0 which should allow this to happen.

We'll be talking about this more and more in the coming weeks and we have a few surprises in the pipeline which should be pretty interesting.

Update:

Actually it wasn't a week. It was four days. We're also planning on cranking up the volume here in a few days so we'll be adding weblogs at an even faster rate!

Tailrank wins Time.com's Top 50 Coolest Sites

Time.com (in their infinite wisdom) seems to think Tailrank is one of the top 50 coolest websites on the Internet! I couldn't agree more!

Tailrank culls the day's top stories from thousands of blogs (both liberal and conservative) and news sites; the posts that are linked to the most and discussed the most bubble to the top. (Technology and General News are covered under separate tabs.) Registered users can create their own customized filter; there's a mobile version too and an RSS feed.

Pretty sweet I must say. We placed in the News and Information section along with Digg.

I also note that my friends at Phonescoop and Pandora were anointed with coolness!

Tailrank's River of News Memetracker View

This post started by leaving a long comment over on Scobleizer's blog but I figured I'd just transcribe it here.

The "river of news" style of reading is a big one among RSS aggregator fans. I'm a personal fan as well and try to use river of news often (really depends on my current goals really).

This thread restarted after Earthlink released its RSS reader (you can follow the full thread on Tailrank and Techcrunch has a great writeup) which supports a river of news view (which is Dave's favorite).

It's not an "either or" proposition though. There's no reason an RSS aggregator can't have both river of news as well as a more conventional folder view.

Tailrank of course has a river of news view for breaking news which is kind of like a river of memes.

New Tailrank Release (Session Fixes, and Links to Yesterday)

We pushed out a new version of Tailrank tonight which I wanted to talk about.

A few people have emailed me over the last few days to note that they couldn't consistently login to the site. This was a bug with our load balancer which should be fixed now. We're actually using a new distributed session feature which should make the site easier to scale moving forward.

We also shipped a new feature I want to mention. We now have links to 'yesterdays' top posts directly in the user interface. Each day has a permalink for the top stories found within Tailrank. For example, today news is located here. This wasn't immediately obvious to some people that you could navigate back in time so we added a sidebar option highlighting this feature. You can click on the title bar to load the full of page of yesterday's stories in the browser.

Technically Tailrank supports a flexible backend that can allow you to change the timeframe for the search query. You can specify a start, end, and time duration and Tailrank can compute rankings for this range. We've struggled how to express this in the site but now this should help out a good deal. Some day we'll end up shipping an API for this functionality.

Here's a screenshot of the bottom right portion of the page showing the new feature in action.

Marketing Monger Interview on Tailrank

Early this week I was lucky enough to participate in a podcast interview with Eric Mattson of the Marketing Monger

In my last post I said I wasn't going to do any more podcasts until next week but, thanks to a truly bizarre series of events that kept me in Stockholm on Saturday, I managed to connect with Kevin Burton of Tailrank for my 40th podcast.

We talked about RSS, Tailrank, blog marketing and a bunch of other fun topics.

I'm spending this month working from Thailand on a new version of Tailrank and I talk a bit about what's going into the next version.

We conducted podcast using Skype over a 4Mbit satellite link and I was a bit nervous that the call would drop but everything worked great.

A podcast over Skype via satellite between Thailand and Sweden about a virtual company based in San Francisco. The World is Flat.

Link to mp3