Tailrank 2.5 Now Available

Not only are we announcing Spinn3r 2.0 today but we're announcing that a new version of Tailrank is being released as well.

If you've been a regular reader of Tailrank over the last few months you might have noticed a number of incremental improvements. Tailrank 2.5 is far more evolutionary than revolutionary.

We've spent a lot of time focusing accuracy of Tailrank's core internal algorithms. What works for one blog or even 1M of blogs in our index tends to fail from time to time when working on 12M blogs.

In this release you'll also see:

MUCH Larger index

Tailrank uses Spinn3r 2.0 as it's core crawling platform. Spinn3r 2.0 is now indexing 12M weblogs which means that Tailrank has a much larger index as well.

This makes Tailrank a lot more democratic (as well as the largest memetracker). You don't have to be an A-list blogger anymore. Now you just have to say something insightful and intelligent and you'll be included in the discussion.

Improved Document Clustering

We've improved upon the design of our clustering algorithms so they can process much more data than before. In fact, we're now clustering within a sixty day window so that if newly published stories cover a meme which happens in the past it will be published within the correct meme.

Spam Prevention

As usual we've spent a lot of time on spam prevention.

A great deal of this work has been done within Spinn3r but there's another layer on top of Tailrank which handles other types of spam which are fatal for memetrackers but not for regular web crawlers.

Improved Accuracy

A lot of time and effort has been spent improving our summary extraction, post categorization (tech, politics, etc), and title extraction.

This seems like a smaller issue but it really helps Tailrank seem professional and seriously improves the readability of the product.

What's Next for Tailrank?

We're hard at work on Tailrank 3.0. This will unify technology present in Tailrank with the Spinn3r backend which is now more advanced in a number of areas.

Tailrank 3.0 will be a big release for us. Just as big as Tailrank 2.0 which we launched nearly a year ago.

Tailrank is the 'Cream of the Crop'

Frantic Industries did a review of eight memetrackers. Long story short. Tailrank is the 'Cream of the Crop' (which I won't argue with).

When it comes to news aggregation, Tailrank clusters the articles revolving around one main story, but it doesn’t show as many stories as Techmeme or Megite right on the front page. To some, this will be a disadvantage, while others will appreciate less clutter and more clarity that this approach offers. It also gives Tailrank a very “neat” design, contrasted to sometimes chaotic look of the competitors. Tailrank also shows the top news items for today and yesterday in a handy box on the right side of the page.

Overall, Tailrank is the cream of the crop. Its smooth design, solid number of additional features and tools and timely deliverance of news will make it a favorite choice for many.

Tailrank Blocks PayPerPost Bloggers

Today Tailrank had the unfortunate task of blocking a few weblogs from Tailrank's index due to link spamming.

These were PayPerPost bloggers linking to Sproose which purchased a PayPerPost campaign to astroturf their release.

We have no problem with bloggers selling ads and trying to make money but this type of linking behavior is essentially spam.

We're willing to reinstate these blogs if they'll migrate to using rel="nofollow" for future PayPerPost sponsored posts. For example, nearly all links in this post use rel="nofollow" to avoid confusing memetrackers and search engines.

Sproose is also blocked and we're willing to reinstate them as well if they'll stop running PayPerPost campaigns without insisting on a nofollow link. Spam is a top priority for a search engine and for them to resort to link spam to advertise their product is a bit hypocritical.

The spammed post will remain in our index for historical purposes but the ranking is reset and won't show up on any of our archive pages.

To date, we've been very trusting when adding weblogs to our index. This has paid off because we haven't attracted the spam that is problematic with other services. In fact, this is only one of a handful of spam posts we've had to deal with in the last year since our launch.

PayPerPost has received a great deal of criticism in the press for their lack of ethics. What I find most disturbing is the fact that PayPerPost is willing to hurt the search rankings of bloggers by not communicating the problems (spam) with selling links.

Tailrank isn't the only large site with this policy. MSN search has also promised to block websites that sell links. They're sending off email messages notifying them that they've been dropped from their index:

Your site is acquiring links through posting to or exchanging links with sites unrelated to your site content. Techniques which attempt to acquire unrelated spam links in order to increase ranking are considered spam and your site has been excluded from our index as results. Please contact us once you've removed these links and we will reevaluate.

The only party I find at fault here is PayPerPost. We'd really love to reinstate these bloggers and add them back to our index and welcome them home with open arms. I assume they simply weren't aware of the problems with selling this type of link spam.

Tailrank 2.1 - Now with Video Memetracker Support

Over the xmas break we pushed out a new release of Tailrank which includes a super cool and enhanced video memetracker.

You can see the new Tailrank video feature in the screenshot below:

There are a bunch of additional small changes here including the ability to view river of news and breaking posts per category. Here's the tech breaking news page. All of these have RSS feeds of course.

We're also working on some new super cool features which will be coming out in the following weeks so stay tuned.

Update:

More from NewTeeVee:

Megite and Tailrank are constantly nipping at each other’s heels, especially as far as press coverage goes. When Matthew Chen of Megite emailed us about their upcoming video tab, we asked Kevin Burton of Tailrank if he had anything in that category up his sleeve. Sure, he replied, video was near release. In fact, he’d push out a version tonight!

This was all really odd timing. I might add that our version of video memetracking is public while theirs is still in private beta.

Tailrank Follows that Meme

Great post about Tailrank from the perspective of a new user:

I'm intriqued by a service called Tailrank - it's a memetracker that spiders more than one hundred thousand blogs (presumably chosen based on Technorati rank, but I don't see any details about this at their site) and publishes well-organized, comprehensive threads that track the spread of the hottest blogosphere topics for any given day. There are other, better established memetrackers ... but I find Tailrank's organizing principles to be really intuitive and, to me, it just looks a whole lot prettier.

...

Just using the basic memetracking tool, I've found it easier to follow widely distributed conversations (taking place across many many blogs) and have discovered some great marketing blogs I hadn't previously known about.

Tailrank on the Scoble Show

I saw down with Robert Scoble about a week ago to talk about Tailrank for the Scoble Show:

Kevin Burton is a talented developer who has worked on a variety of startups already including Rojo, and now TailRank which he started to be able to see what bloggers were talking about. Here I sit down with him for an interesting conversation in the lobby of San Francisco's Palace Hotel.

I think the interview turned out pretty well. The only mistake I made was that I left my cell phone on which is a slight problem. Luckily no one else called during the interview (sorry Robert).

I also gave a demo of Tailrank. Unfortunately, the realtime IM delivery feature actually worked right after they shutoff the camera. It was pretty amazing actually. Our crawler found a post on that topic right after I subscribed to the meme.

Tailrank Mobile Now Supports Verticals

Tailrank mobile now supports tech, politics, and entertainment. In the past we only supported the global view of all stories but now you can narrow your focus.

If you haven't yet used Tailrank mobile and have a modern cell phone with a browser you should definitely check it out. A lot of people really like Tailrank mobile and I'm pretty impressed with the feedback we've received so far.

I bumped into Eric Lin of Phonescoop today and he commented how much he liked our mobile version which reminded me that we've never implemented verticals.

Tailrank Interview on the Web 2.0 Show

A while back the Web 2.0 show interviewed me about Tailrank.

The quality of the Skype call was really amazing. Apparently the trick is port forwarding so that it doesn't have to do P2P tunneling.

Tailrank Interview on Folksonomy.org

The guys over at Folksonomy.org have published an interview with me that they conducted last week. Short but sweet.

What is TailRank and what are its major advantages over similar services?

Tailrank is a service that allows you to track the hottest news stories across the blogosphere.

There are a few similar services but we track more blogs, allow the user to create their own version of Tailrank, support full-text search, and allow delivery via Instant Messenger.

More Buzz around Firefox 2.0 than Internet Explorer 7.0

Two major browser releases nearly a week apart signal the start of Browser Wars 2.0 (and maybe Web 2.1).

Blogger reaction clearly favors Firefox 2.0 over Internet Explorer 7.0

Nearly three times as many people are talking about the release of Firefox 2.0 vs IE 7.

Please Take the Tailrank Reader Survey.

If you guys have a few seconds I'd really appreciate you taking the Tailrank reader survey.

This will help FM Publishing sell advertising on our site which means we can use the profits to invest in our infrastructure.

It also means we'll start to see more cool ads like Apple Computer and Dice.com which are much more tasteful than the bouncing heads or "punch the monkey" alternatives.

Updated Tailrank Release - No new windows and better splash message

Alright guys. We have a new Tailrank release out the door. I usually don't blog about all the changes but I wanted to make a few notes.

When you click on an external link we now open the link in the same window. We get mixed reactions here but it seems that a lot of people want it to open up in the same window. I think this might be the default moving forward and we'll add a pref to allow people to change it so the page opens in a new window.

We also have a new splash message system which you can see in blue (almost cerulean btw) at the top right. The first time you visit Tailrank you're given a "What is Tailrank?" message but after that we give you random tips and tricks.

Tailrank in Top Ten Fastest Sites

We just rolled out some performance updates the other night and it looks like they've worked out very well! Tailrank is now in the top ten fastest sites over on grabperf. We're even faster than Google News!

What's really interesting is that I think I can get another significant performance boost out of the system which might allow us to take on the #1 position. I've got my eyes on you Technorati Mobile! ;)

Actually, I should get Tailrank Mobile on there. I'm sure it would take #1 slot right away.

Tailrank Indexes More Weblogs

If you've noticed more traffic on your blog recently it might be because Tailrank is now indexing more weblogs. We've deployed a new crawler and this week alone we've added 15k new blogs!

What's really amazing about this system is that we're adding new weblogs according to our proprietary ranking algorithms. This means that we should totally skip spam blogs altogether and weblogs will be added based on their ranking. This way Tailrank is always indexing the top weblogs (which have the most influence).

The goal of course is to add even more of the long tail into our index until we're indexing the whole blogosphere. We're in the process of deploying new hardware and working towards Taillrank 2.0 which should allow this to happen.

We'll be talking about this more and more in the coming weeks and we have a few surprises in the pipeline which should be pretty interesting.

Update:

Actually it wasn't a week. It was four days. We're also planning on cranking up the volume here in a few days so we'll be adding weblogs at an even faster rate!

Tailrank wins Time.com's Top 50 Coolest Sites

Time.com (in their infinite wisdom) seems to think Tailrank is one of the top 50 coolest websites on the Internet! I couldn't agree more!

Tailrank culls the day's top stories from thousands of blogs (both liberal and conservative) and news sites; the posts that are linked to the most and discussed the most bubble to the top. (Technology and General News are covered under separate tabs.) Registered users can create their own customized filter; there's a mobile version too and an RSS feed.

Pretty sweet I must say. We placed in the News and Information section along with Digg.

I also note that my friends at Phonescoop and Pandora were anointed with coolness!

Dave Needs Tailrank Mobile

Dave Winer just got a new Blackberry cell phone and now realizes he wants a mobile RSS aggregator:

Now that I have a BlackBerry, I want to take a PDA version of everything I do on my desktop with me into the PDA-land (otherwise known as The Real World). Immediately, every time I pick it up, I want to know what's new. I've got SMS to connect with other PDA users, and email for everyone else. But what about news? At first I thought a "mini" version of my news aggregator would be the right thing, but that's too much news. I want just-enough, which is less than the firehose

Sounds exactly like Tailrank mobile.

Dave. Just create a Tailrank filter and you can read it with Tailrank mobile. Just create an account, import your OPML, and then you can read it on your Blackberry.

Here's my filter for example. We don't have a UI to create this on the mobile device but once your account is setup you can view it just fine.

The great thing about this is that its exactly the opposite of the firehose. You can control how much water you want to drink at once. If you want to be more picky you can increase the value of min_ranking and Tailrank will show you fewer stories and with higher relevance.

Tailrank Site Specific Memetracker Support

GigaOm shipped a new look and feel today (which is pretty sweet if you ask me) and they created a new tool near the top of the page named "Most Popular Stories" which shows the hottest posts on the site at any given time.

Here's a preview.

Did you know you could do this with Tailrank for your site? You can and its fairly easy. Just search for the URL of your blog (in this case http://gigaom.com) and we'll show you the most popular posts sorted by time. Here's what gigaom.com would look like.

There''s an associated RSS feed for this guy and you can just use one of the widgets that hosts RSS. Either that or you can click the " Add this page to your blog!" link in the left sidebar to include the content via javascript.

Apple Advertises on Tailrank

Apple is running a cool ad on the new Mac Pro on Tailrank (thanks for my friends at FM)

I soooooo want one!

Tailrank's River of News Memetracker View

This post started by leaving a long comment over on Scobleizer's blog but I figured I'd just transcribe it here.

The "river of news" style of reading is a big one among RSS aggregator fans. I'm a personal fan as well and try to use river of news often (really depends on my current goals really).

This thread restarted after Earthlink released its RSS reader (you can follow the full thread on Tailrank and Techcrunch has a great writeup) which supports a river of news view (which is Dave's favorite).

It's not an "either or" proposition though. There's no reason an RSS aggregator can't have both river of news as well as a more conventional folder view.

Tailrank of course has a river of news view for breaking news which is kind of like a river of memes.

New Tailrank Release (Session Fixes, and Links to Yesterday)

We pushed out a new version of Tailrank tonight which I wanted to talk about.

A few people have emailed me over the last few days to note that they couldn't consistently login to the site. This was a bug with our load balancer which should be fixed now. We're actually using a new distributed session feature which should make the site easier to scale moving forward.

We also shipped a new feature I want to mention. We now have links to 'yesterdays' top posts directly in the user interface. Each day has a permalink for the top stories found within Tailrank. For example, today news is located here. This wasn't immediately obvious to some people that you could navigate back in time so we added a sidebar option highlighting this feature. You can click on the title bar to load the full of page of yesterday's stories in the browser.

Technically Tailrank supports a flexible backend that can allow you to change the timeframe for the search query. You can specify a start, end, and time duration and Tailrank can compute rankings for this range. We've struggled how to express this in the site but now this should help out a good deal. Some day we'll end up shipping an API for this functionality.

Here's a screenshot of the bottom right portion of the page showing the new feature in action.

Tailrank in FeedDemon?

Nick Bradbury writes about the coming personal memetracking in FeedDemon.

But I'm in the middle of another coding frenzy right now, and this time I'm going to open the kimono by talking about a "memetracker" feature I'm considering for a future version of FeedDemon (fdmeme.png). I enjoy memetrackers like Tailrank and Techmeme because they let me know what people are talking about, but I'd like to have a memetracker that looks only in feeds I'm subscribed to in FeedDemon. Not only would this inform me about popular topics, but it would also let me read multiple posts about the same topic at once so I don't have to read them one-at-a-time as I come across them.

Nick, I'd love to talk about some way Tailrank could help out FeedDemon here. We have a set of APIs which you could use (and they're all RESTian). You could use our memetracking technology directly or use it to augment your own within FeedDemon.

Ping me if you want to talk!

Reading the Full Thread within Tailrank

We shipped a new feature within Tailrank last week which I want to talk about.

It is now possible to view the full thread around a meme right within Tailrank. This can save you a lot of time and you can read stories just like you were in your favorite aggregator.

All you have to do is click the "X links in thread (read now)" permalink within Tailrank which will take you to a dedicated page.

You can see the link on every story on the main page.

This will then take you to a permalink page with every main post for the story.

Here's an example. Note the full post for the "stop the ACLU" entry.

I also wanted to note that whether or not we include the full-text of a post is controlled by the publisher. We only include the full-text of a post if the publisher uses a full-text RSS feed. If not we still use a summary.

Netscape Is a Memedigger

More Memediggers enter the market. Now Netscape wants to throw its hat in the ring with their new public beta.

Behold the new Netscape:

Update: More from Techcrunch and the Read/Write web

Who Filters the Filters?

Webmonkey has a nice piece on memetrackers (which they call filters in this article):

... the new utopian open collaborative Web, typified by del.icio.us brought with it a parade of new sites that use the power of the people to aggregate links and determine their relative coolness. In the new paradigm, editors (damn them) have been deposed and users have the power. Anyone can post links on the site, as often as they please, and the links appear immediately and uncensored. The result is a database of trillions of links to whatever any random person thinks is cool or memorable or otherwise linkworthy.

They also have a nice description of Tailrank.

Tailrank hides all but the most-linked-to stories, and favors news from blogs; its algorithm takes into account external links to a story as well as the number of Tailrank users who linked it. The result is a slick news and politics filter, with a reasonable dose of material you won't see elsewhere, along with plenty you will.

The feature breakdown is really interesting as well.

All and all a very good article and they were very nice towards Tailrank!

New Tailrank Architecture Now Live

For the last month we've been working hard on a new architecture for Tailrank's backend. I'm happy to say that yesterday we deployed the first wave of updates and the new system is online and firing on all cylinders.

What I'm really excited about is that this will provide us with a great deal of flexibility moving forward in terms of ranking and spidering of the blogosphere.

Most of these upgrades are under the surface and not visible as user features. The new infrastructure should now allow us to innovate more on features in the main product so stay tuned for new and cool functionality!

Marketing Monger Interview on Tailrank

Early this week I was lucky enough to participate in a podcast interview with Eric Mattson of the Marketing Monger

In my last post I said I wasn't going to do any more podcasts until next week but, thanks to a truly bizarre series of events that kept me in Stockholm on Saturday, I managed to connect with Kevin Burton of Tailrank for my 40th podcast.

We talked about RSS, Tailrank, blog marketing and a bunch of other fun topics.

I'm spending this month working from Thailand on a new version of Tailrank and I talk a bit about what's going into the next version.

We conducted podcast using Skype over a 4Mbit satellite link and I was a bit nervous that the call would drop but everything worked great.

A podcast over Skype via satellite between Thailand and Sweden about a virtual company based in San Francisco. The World is Flat.

Link to mp3

Memetrackers Need a good Title

This is a really good story by the NYTimes on how some publishers select titles which look bad in Google:

So news organizations large and small have begun experimenting with tweaking their Web sites for better search engine results. But software bots are not your ordinary readers: They are blazingly fast yet numbingly literal-minded. There are no algorithms for wit, irony, humor or stylish writing. The software is a logical, sequential, left-brain reader, while humans are often right brain.

...

In newspapers and magazines, for example, section titles and headlines are distilled nuggets of human brainwork, tapping context and culture. "Part of the craft of journalism for more than a century has been to think up clever titles and headlines, and Google comes along and says, 'The heck with that,' " observed Ed Canale, vice president for strategy and new media at The Sacramento Bee.

This is a real problem. I often see stories in Tailrank that are very out of context. Your witty title on your blog might not be that smart when displayed in a Memetracker.

eHub Interviews Tailrank

The Web2.0 blog eHub interviews me about Tailrank and memetrackers which I think you guys might like.

TailRank: Release early release often. I’m a poor designer but I find that the feedback from our user community has been invaluable. We try to listen to our customers as much as possible and push about two releases of TailRank a week.

So far this has worked amazingly well and TailRank just keeps growing and growing.

eHub: What technologies are you currently using?

TailRank: We’re a Linux, Java, Apache, and MySQL shop. The software stack we’re using is designed to scale so we should be able handle a lot of traffic here moving forward.

I forgot to mention PHP here. I'll never again use a Java based web infrastructure.

New Tailrank Release - Categories, New Blog, and FM Publishing

We have a new version of Tailrank available. Part of what you'll notice right away is our new blog (which you're reading right now). Feel free to go ahead and subscribe because in the future all Tailrank specific news will be published here instead of my personal blog.

What's new you ask? Good question!

Categories

A popular request has been to ship categories including technology and politics and we now have them at tech.tailrank.com and politics.tailrank.com

We're also going to be pushing a few more categories here shortly. If you'd like to help or want a specific category please feel free to contact us as we'd appreciate the help.

Now Part of FM Publishing

We're now part of John Battelle's FM Publishing media empire. These guys have been great to work with so far and are doing an awesome job.

I'd also appreciate it if Tailrank members would take the FM publishing survey and help explain who you are and why you use Tailrank. It will help advertisers decide if they want to sponsor us and keep us innovating in the memetracker space.

One of the things I like about FM and CPM ads in general is the fact that advertisers become part of the site experience. With Adsense I'm really just part of a larger network of sites that match on some proprietary Google algorithm.

With FM publishing my advertisers are aware of Tailrank and will generally run ads similar to the content of the site. So far they've done a great job with BoingBoing.

Feel free to contact FM if you'd like to advertise on Tailrank.

Polish and Attention to Detail

You'll also notice a lot more polish and attention to detail including performance and UI updates. Many operations are now Ajaxian (including weblog subscription) and we have a bunch of relevance changes under the surface.

Feel free to add a comment if you have any feedback or additional suggestions.

Welcome to the New Tailrank Blog!

This has taken a while but we finally have a dedicated company weblog! In the past I've blogged about Tailrank over on my personal blog at feedblog.org so if you've been following Tailrank's progress over there you should subscribe to this blog in your RSS reader or just check back every now and then.

Wired on Tailrank, Memetrackers, and the Man vs Machine War

Wired just released a piece by Ryan Singel entitled Man vs Machine in Newsreader War which is pretty hot (and pushing some serious pages on Tailrank right now). I really like the man vs machine analogy (though I think in the end a hybrid approach will be better).

Great piece and Ryan covers all the bases.

... Tailrank, a San Francisco-based startup founded by Kevin Burton, also relies heavily on smart code to find cool stories -- not just from news outlets, but also from tens of thousands of blogs.

Burton, who wrote a news-reading application called NewsMonster and co-founded Rojo, an online news-reading tool, started Tailrank as a way to handle the information overload created by easy access to blogs and media outlets via RSS.