Tailrank 2.5 Now Available

Not only are we announcing Spinn3r 2.0 today but we're announcing that a new version of Tailrank is being released as well.

If you've been a regular reader of Tailrank over the last few months you might have noticed a number of incremental improvements. Tailrank 2.5 is far more evolutionary than revolutionary.

We've spent a lot of time focusing accuracy of Tailrank's core internal algorithms. What works for one blog or even 1M of blogs in our index tends to fail from time to time when working on 12M blogs.

In this release you'll also see:

MUCH Larger index

Tailrank uses Spinn3r 2.0 as it's core crawling platform. Spinn3r 2.0 is now indexing 12M weblogs which means that Tailrank has a much larger index as well.

This makes Tailrank a lot more democratic (as well as the largest memetracker). You don't have to be an A-list blogger anymore. Now you just have to say something insightful and intelligent and you'll be included in the discussion.

Improved Document Clustering

We've improved upon the design of our clustering algorithms so they can process much more data than before. In fact, we're now clustering within a sixty day window so that if newly published stories cover a meme which happens in the past it will be published within the correct meme.

Spam Prevention

As usual we've spent a lot of time on spam prevention.

A great deal of this work has been done within Spinn3r but there's another layer on top of Tailrank which handles other types of spam which are fatal for memetrackers but not for regular web crawlers.

Improved Accuracy

A lot of time and effort has been spent improving our summary extraction, post categorization (tech, politics, etc), and title extraction.

This seems like a smaller issue but it really helps Tailrank seem professional and seriously improves the readability of the product.

What's Next for Tailrank?

We're hard at work on Tailrank 3.0. This will unify technology present in Tailrank with the Spinn3r backend which is now more advanced in a number of areas.

Tailrank 3.0 will be a big release for us. Just as big as Tailrank 2.0 which we launched nearly a year ago.

Tailrank on LunchMeet

Jonathan and I had the opportunity to hang out with Eddie last week for LunchMeet:

The blogosphere is a noisy place filled with many interconnected conversations on all sorts of disperate topics. Tailrank, which calls itself a memetracker, is a service that tracks the zeitgeist of conversations in the blogosphere. I sat down with Kevin Burton, Tailrank's CEO and founder, and Jonathan Moore, brilliant engineer and hacker, at their home office to learn a bit about their services. Tailrank is built upon Spinn3r, another service that Burton and Moore built, that spiders and indexes the blogosphere.

Tailrank Now Indexing over 1M Weblogs

Today is a big day for Tailrank.

We have two announcements. The first is that Tailrank is now indexing over 1M weblogs.

This should really help prevent the echo chamber problem which seems all too common with the blogosphere.

The second is that we're launching Spinn3r.

Tailrank is the 'Cream of the Crop'

Frantic Industries did a review of eight memetrackers. Long story short. Tailrank is the 'Cream of the Crop' (which I won't argue with).

When it comes to news aggregation, Tailrank clusters the articles revolving around one main story, but it doesn’t show as many stories as Techmeme or Megite right on the front page. To some, this will be a disadvantage, while others will appreciate less clutter and more clarity that this approach offers. It also gives Tailrank a very “neat” design, contrasted to sometimes chaotic look of the competitors. Tailrank also shows the top news items for today and yesterday in a handy box on the right side of the page.

Overall, Tailrank is the cream of the crop. Its smooth design, solid number of additional features and tools and timely deliverance of news will make it a favorite choice for many.

Tailrank and the San Francisco Wireless Office in the Chronicle

Dan Fost wrote a great piece in the Chronicle about the San Francisco wireless beduin flashmob:

A new breed of worker, fueled by caffeine and using the tools of modern technology, is flourishing in the coffeehouses of San Francisco. Roaming from cafe to cafe and borrowing a name from the nomadic Arabs who wandered freely in the desert, they've come to be known as "bedouins."

...

Burton arrives at 11 a.m., and his lead engineer, Jonathan Moore, 30, arrives a few minutes later. Burton has a double latte and a cupcake, and starts explaining how his site uses "wisdom of the crowds" algorithms to scrape the hottest news from the blogosphere and upend the mainstream media.

As he talks, Gallagher joins in, and advises, "Lower your voice. I already know the ins and outs of your business plan from the last time I was here."

Check out the flash coverage as well. Pretty slick and innovative for mainstream media!

Jonathan (foreground) joined the company about a month ago. We've both been really heads down working on some cool stuff (stay tuned).

Tailrank Blocks PayPerPost Bloggers

Today Tailrank had the unfortunate task of blocking a few weblogs from Tailrank's index due to link spamming.

These were PayPerPost bloggers linking to Sproose which purchased a PayPerPost campaign to astroturf their release.

We have no problem with bloggers selling ads and trying to make money but this type of linking behavior is essentially spam.

We're willing to reinstate these blogs if they'll migrate to using rel="nofollow" for future PayPerPost sponsored posts. For example, nearly all links in this post use rel="nofollow" to avoid confusing memetrackers and search engines.

Sproose is also blocked and we're willing to reinstate them as well if they'll stop running PayPerPost campaigns without insisting on a nofollow link. Spam is a top priority for a search engine and for them to resort to link spam to advertise their product is a bit hypocritical.

The spammed post will remain in our index for historical purposes but the ranking is reset and won't show up on any of our archive pages.

To date, we've been very trusting when adding weblogs to our index. This has paid off because we haven't attracted the spam that is problematic with other services. In fact, this is only one of a handful of spam posts we've had to deal with in the last year since our launch.

PayPerPost has received a great deal of criticism in the press for their lack of ethics. What I find most disturbing is the fact that PayPerPost is willing to hurt the search rankings of bloggers by not communicating the problems (spam) with selling links.

Tailrank isn't the only large site with this policy. MSN search has also promised to block websites that sell links. They're sending off email messages notifying them that they've been dropped from their index:

Your site is acquiring links through posting to or exchanging links with sites unrelated to your site content. Techniques which attempt to acquire unrelated spam links in order to increase ranking are considered spam and your site has been excluded from our index as results. Please contact us once you've removed these links and we will reevaluate.

The only party I find at fault here is PayPerPost. We'd really love to reinstate these bloggers and add them back to our index and welcome them home with open arms. I assume they simply weren't aware of the problems with selling this type of link spam.

Tailrank 2.1 - Now with Video Memetracker Support

Over the xmas break we pushed out a new release of Tailrank which includes a super cool and enhanced video memetracker.

You can see the new Tailrank video feature in the screenshot below:

There are a bunch of additional small changes here including the ability to view river of news and breaking posts per category. Here's the tech breaking news page. All of these have RSS feeds of course.

We're also working on some new super cool features which will be coming out in the following weeks so stay tuned.

Update:

More from NewTeeVee:

Megite and Tailrank are constantly nipping at each other’s heels, especially as far as press coverage goes. When Matthew Chen of Megite emailed us about their upcoming video tab, we asked Kevin Burton of Tailrank if he had anything in that category up his sleeve. Sure, he replied, video was near release. In fact, he’d push out a version tonight!

This was all really odd timing. I might add that our version of video memetracking is public while theirs is still in private beta.

Tailrank Follows that Meme

Great post about Tailrank from the perspective of a new user:

I'm intriqued by a service called Tailrank - it's a memetracker that spiders more than one hundred thousand blogs (presumably chosen based on Technorati rank, but I don't see any details about this at their site) and publishes well-organized, comprehensive threads that track the spread of the hottest blogosphere topics for any given day. There are other, better established memetrackers ... but I find Tailrank's organizing principles to be really intuitive and, to me, it just looks a whole lot prettier.

...

Just using the basic memetracking tool, I've found it easier to follow widely distributed conversations (taking place across many many blogs) and have discovered some great marketing blogs I hadn't previously known about.

Tailrank on the Scoble Show

I saw down with Robert Scoble about a week ago to talk about Tailrank for the Scoble Show:

Kevin Burton is a talented developer who has worked on a variety of startups already including Rojo, and now TailRank which he started to be able to see what bloggers were talking about. Here I sit down with him for an interesting conversation in the lobby of San Francisco's Palace Hotel.

I think the interview turned out pretty well. The only mistake I made was that I left my cell phone on which is a slight problem. Luckily no one else called during the interview (sorry Robert).

I also gave a demo of Tailrank. Unfortunately, the realtime IM delivery feature actually worked right after they shutoff the camera. It was pretty amazing actually. Our crawler found a post on that topic right after I subscribed to the meme.

Tailrank Mobile Now Supports Verticals

Tailrank mobile now supports tech, politics, and entertainment. In the past we only supported the global view of all stories but now you can narrow your focus.

If you haven't yet used Tailrank mobile and have a modern cell phone with a browser you should definitely check it out. A lot of people really like Tailrank mobile and I'm pretty impressed with the feedback we've received so far.

I bumped into Eric Lin of Phonescoop today and he commented how much he liked our mobile version which reminded me that we've never implemented verticals.