Subba’s Serendipitous moments

September 23, 2009

Netflix’s “crowdsourcing” approach is a success

I have been following Netflix unique experiment to improve its Web site’s movie recommendation system. This week Netflix announced the winner of a three year contest with the winner BellKore comprising of statisticians, computer scientists, data mining experts netting a cool million dollars.

The rules of the competition was fairly straightforward. The qualification for the prize was that the winning team has to improve by at least 10% the prediction of what movies customers would like as measured against the actual ratings. The teams were grappling with a huge data set of more than 100 million movie ratings.

Over the past three years there have been 44,014 entries from 5,169 teams in 186 countries vying for the top prize

I think with this experiment and with Google’s experiment with crowdsourcing described here, there will be a significant shift towards innovation management. The fact that there exists more intelligence and wisdom and the collective effort outside the company’s eco-system has gained credibility. I expect many such organizations embarking on the contest mode to solve intractable problems.

There are a number of lessons that this contest brings about.

First, it indicates that there can be a marketplace for innovation where companies could post their product development challenges and for an interesting contest, the best brains are willing to compete. It sharpens their own abilities.

Second as the BellKore team and other teams demonstrated there is a willingness for disparate people to actively collaborate. While cooperation and collaboration within many organizations has been challenging, I wonder how such disparate people could come together and collaborate easily for a bigger goal.

Third, for people who believed in having an inhouse R&D and saw that as a competitive advantage, this experiment seeks to blow that myth away.

Note: Netflix Prize 2 would challenge competitors to recommend movies based on demographic and behavioral data.

July 27, 2009

When intuition outsmarts rationality

In October 2001, a fire crew was fighting a fire in a disused bingo hall in Leicester in the UK. Even though it was big, the fire chief decided it was safe enough to send the crew into the building.

They were starting to make progress in knocking the fire down when the fire chief decided something was wrong, and ordered his team out of the building. The team protested, unwilling to give up the progress they had made. But the fire chief insisted and as they exited the building it exploded in a massive fireball. If the decision to evacuate hadn’t been made the entire team would have been killed.

It turns out that the fire was one of the rarest and most dangerous phenomenon in firefighting – a backdraft. The fire chief had never experienced a backdraft before, he just knew that something was wrong and they needed to get out. In the ensuing investigation it turns out there were three things that were unusual: the smoke was more orange than usual, air was rushing into the building rather than out of it, and the fire was unusually quiet. The fire chief was right in his decision, he just didn’t know why at the time.

Well, all is well, that ends well.

But let’s take a moment and reflect what could have happened to the same event in a different set of circumstances. Assume that the fire chief was not the decision maker but he had to refer the decision to his boss.

There was clearly no evidence that something unusual was underway and that the teams were in disagreement with the fire chief. The teams were actually making progress and were engaged in a great endeavor to put out the fires. Normal rational thinking would have demanded that the boss would overrule the fire chief. The firemen would continue to fight the fire and the entire team would have been killed.

An investigation would have ensued and the decision would have been termed as rational and the whole thing written off as a terrible tragic accident.

March 3, 2009

Experience can be a big liability

The Wall Street Journal has an interesting and a provocative article on the relevance of experience. Though the article by and large focuses on the project management area (an area where I have significant experience both as a practitioner and as a consultant and hence competent to comment), the broad implications of the article are relevant across all disciplines.

In times of profound change, the learners inherit the earth, while the learned find themselves beautifully equipped to deal with a world that no longer exists.

In my own personal endeavor, I have often felt that to remain current, one needs to constantly have a curious and questioning mind. It is too tempting to impose a solution that has worked in the past, or a solution that we are personally biased towards and force-fit the problem to the solution. As a consultant and an analyst, the temptation is all the more greater. And I have seen some unpleasant consequences of using an obsolete approach in the name of best practices.

This is another situation where an ounce of perspective is worth more than a pound of analytics and regurgitation.

I have blogged about the how being trained to see often masks our ability to see and why despite being wise, not being able to have the much needed perspective often leads to us going astray here.

Tags: , , ,

Powered by Qumana