Netflix’s “crowdsourcing” approach is a success

I have been following Netflix unique experiment to improve its Web site’s movie recommendation system. This week Netflix announced the winner of a three year contest with the winner BellKore comprising of statisticians, computer scientists, data mining experts netting a cool million dollars.

The rules of the competition was fairly straightforward. The qualification for the prize was that the winning team has to improve by at least 10% the prediction of what movies customers would like as measured against the actual ratings. The teams were grappling with a huge data set of more than 100 million movie ratings.

Over the past three years there have been 44,014 entries from 5,169 teams in 186 countries vying for the top prize

I think with this experiment and with Google’s experiment with crowdsourcing described here, there will be a significant shift towards innovation management. The fact that there exists more intelligence and wisdom and the collective effort outside the company’s eco-system has gained credibility. I expect many such organizations embarking on the contest mode to solve intractable problems.

There are a number of lessons that this contest brings about.

First, it indicates that there can be a marketplace for innovation where companies could post their product development challenges and for an interesting contest, the best brains are willing to compete. It sharpens their own abilities.

Second as the BellKore team and other teams demonstrated there is a willingness for disparate people to actively collaborate. While cooperation and collaboration within many organizations has been challenging, I wonder how such disparate people could come together and collaborate easily for a bigger goal.

Third, for people who believed in having an inhouse R&D and saw that as a competitive advantage, this experiment seeks to blow that myth away.

Note: Netflix Prize 2 would challenge competitors to recommend movies based on demographic and behavioral data.

Social network for Government 2.0

Government 2.0 is clearly gaining momentum. I just stumbled on a social network platform to discuss government 2.0 initatives. GovLoop is the Premier Social Network for Government 2.0 connecting over 12,500 Federal, State, Local, Academics, and Good Contractors.

This is what I call tapping the wisdom of the crowd something that I have been strongly advocating. Some prefer to call it crowdsourcing.

Will Singapore usher in Government 2.0?

President Obama will surely go down in history for a number of things. Amongst many things, he was the first one to use the power of social networking so effectively which led him to win the Presidential elections decisively. He appointed Vivek Kundra as CIO in his administration and Aneesh Chopra as a CTO. Surprisingly they are not marquee names as one would have expected, but people who have blazed a new trail defying conventional practices. Some prefer to call them the iconoclasts.

Vivek Kundra, the CIO for the Obama Administration launched a new website called Data.gov which for all its radical breakthroughs was announced quietly. I hope it gets the publicity it so rightfully deserves. The intent of the website is to release vast amounts of raw data so that tax payers can see what’s happening in the government and buraucracy.

The new site has 50 feeds and is intended to grow to about 240,000 feeds next month itself. It will be a one-stop shop for free access to data that will be generated across all federal government agencies.

This is a paradigm shift and in some sense unparalleled in the history of Government IT. First, it establishes beyond doubt the credentials of President Obama to be as transparent as possible. Second, ingenious entreprenuers can quickly develop Web applications more easily (with mash-ups becoming so common) using government data and take it to market. Finally, interested citizens can provide ideas to the government’s problems, now that they have access to better, reliable and immediate data. Finally, it also goes to show that the government is prepared to accept that it may not have all the answers to its problems and that crowdsourcing must be encouraged.

The Singapore government has been not just an early but a staunch user of IT. It has in the past, pioneered effective applications and can rightfully claim credit for the high level of IT penetration in Singapore. Yet, in recent years it seems to have lost both the momentum and direction. The Government IT directions are managed in a hybrid model with the Government CIO being part of the IDA. One look at the Government CIO mandate here shows it is inwardly focused, tactically driven and continues to o continues to operate from a traditional mindset.

At this stage of IT maturity merely notching up some incremental percentage points on the efficiency scale is not going to help either the Government or the citizens. It needs a more forward looking radical approach if it has to remain relevant and regain the respect that it once had. It needs a President Obama philosophy and a Kundra’s impetus for action.

If what is stopping this leap is imagination, it needs new blood; a set of iconoclasts. If however they believe that the government knows best and that there is no wisdom in the crowd, then sadly, only a serious failure will force a rethink. If the Government does not want to be more transparent, it is only inhibiting the natural empowerment and evolution of the citizens. If the government needs a role model, President Obama has accepted to be one.

Let’s not forget that one constant dimension of the various developments in the IT world is empowerment. This has happened not just within the firewall but as part of the extended enterprise. Choosing to ignore the philosophical underpinnings of empowerment is choosing to ignore the true potential of IT and in a way also choosing to ignore true progress.

I will have the opportunity to speak on Government 2.0 at a National IT conference very soon and this gives me a lot to conceptualize things better.

Collaboration amidst competition

Much has been said about “coopetition” — the phenomenon of collaborating with one’s competitors in business literature. It undoubtedly has its merits though it is a huge challenge to pull it off.

The entrance test to the Indian Institute of Technology (IIT) is the most difficult exam on the planet. The acceptance ratio is somewhat closer to 250: 1 and hence competition is intense.

Under such circumstances, it is unusual to see student collaborate — not for a common cause but to help each other.

The piece here describes an instance of how a group of underprivileged students (some of them didn’t know what a carpet was) collaborated with each other and ensured mutual success.

What was also heartening for me personally, (as it echoes my own definition of a good teacher) is one student’s description of a good teacher. He said “My definition of a good teacher is one who doesn’t give answers but keeps asking the right questions, thereby helping the students to get to their own answers.”

Tags: IIT, student, learning, teacher, collaboration, competition

Powered by Qumana

Eric Schmidt unveils an exciting future.

Eric Schmidt in a very short speech at the recent IBM Partners leadership conference shows to all of us what the future could be in a very interesting compelling way.

Some of the facts:

1. Currently the Internet has 1.3 billion users, with 200 million getting added each year.

2. In Japan, 3 of the most popular books were delivered first on the mobile readers and subsequently done on print.

3. There were just 400 servers in 1983, and now there are more than 500 million servers worldwide.

4. There are 70 million blogs, with over 120,000 blogs being created every day

5. 7 million photos are uploaded to Picasa (Google’s photo sharing site) each day

6. 10 hours of video uploaded on YouTube each minute.

7. 500 million Wifi chip sets will be sold next year.

His most provocative statement of the future: By 2019, there could be a device that could sit on the belt or kept in the wallet that could have 85 years of video on it. You will be dead before you can see all of it. One of the ultimate frustrations in life.

But what he said about Convergence was interesting. Convergence is not everything (services) going into one device. It is entering (all the services) into one server or services in the cloud and hence even if the devices are different, the content in all that will remain the same.

His quote on Breakthroughs was equally profound: Great breakthroughs are closer to what happens in a flood pane. It is not one idea. A dozen tributaries converge and the rising waters lift the genius high enough so that he or she can see the conceptual obstruction of the age.

His entire address and the subsequent panel discussion can be viewed here. Each time I hear Eric speak, I come back with more knowledge and insight. An earlier interview of Eric is also available on my blog here.

As you see this, maybe you should also see some of the other great CEO interviews and discussions. A few are listed here

Tags: Convergence, cloud computing, our future, breakthroughs, collaboration, innovation, Google, IBM, Eric Schmidt, consumer expectations

Powered by Qumana

Enterprises using Wikis

Based on a IDC report, Wikis are slowly making their way into the Enterprise.

It is interesting because this solves some key issues in the collaboration mechanisms that’s been detrimental for effective knowledge sharing. However, it requires more than a technology platform to be effective. Other organizational interventions are also needed.

Though a few enterprises in Asia Pacific express a keen interest in Wikis, I have yet to come across of a serious deployment of Wikis in the Enterprise.

Tags: IDC study, Wikis, Web 2.0, Enterprise 2.0, collaboration

Powered by Qumana