By STEVE LOHR
Published: September 21, 2009
Even the near-miss losers in the Netflix million-dollar-prize competition seemed to have few regrets.
Netflix, the movie rental company, announced on Monday that a seven-man team was the winner of its closely watched three-year contest to improve its Web site’s movie recommendation system. That was expected, but the surprise was in the nail-biter finish.
The losing team, as it turned out, precisely matched the performance of the winner, but submitted its entry 20 minutes later, just before the final deadline expired.
Under contest rules, in the event of a tie, the first team past the post was the winner. “That 20 minutes was worth a million dollars,” Reed Hastings, chief executive of Netflix, said at a news conference in New York.
Yet the scientists and engineers on the second-place team, and the employers who gave many of them the time and freedom to compete in the contest, were hardly despairing.
Arnab Gupta, chief executive of Opera Solutions, a data analytics company based in New York, took a small group of his leading researchers off other work for two years. “We’ve already had a $10 million payoff internally from what we’ve learned,” Mr. Gupta said.
Working on the contest helped the researchers come up with improved statistical analysis and predictive modeling techniques that his firm has used with clients in fields like marketing, retailing and finance, he said. “So for us, the $1 million prize was secondary, almost trivial.”
Indeed, since it began in October 2006, the Netflix contest was significant less for the prize money than as a test case for new ideas about how to efficiently foster innovation in the Internet era — notably, offering prizes as an incentive and encouraging online collaboration to tap minds worldwide.
The lessons of the Netflix contest could extend well beyond improving movie picks. The researchers from around the world were grappling with a huge data set — 100 million movie ratings — and the challenges of large-scale modeling, which can be applied across the fields of science, commerce and politics.
The prize model is increasingly being tried on work like new science and freelance projects in design and advertising. The X Prize Foundation, for example, is offering multimillion-dollar prizes for path-breaking advances in genomics, alternative energy cars and private space exploration.
InnoCentive is a marketplace for business projects, where companies post challenges — often in areas like product development or applied science — and workers or teams compete for cash payments or prizes offered by the companies. A start-up, Genius Rocket, runs a similar online marketplace mainly for marketing, advertising and design projects.
“The great advantage of the prize model is that it moves work away from the realm of the beauty contest to being performance-oriented,” said Michael Schrage, research fellow at the Center for Digital Business at the Sloan School of Management at the Massachusetts Institute of Technology. “It’s the results produced that matters.”
The emerging prize economy, according to some labor market analysts, does carry the danger of being a further shift in the balance of power toward the buyers — corporations — and away from most workers.
Thousands of teams from more than 100 nations competed in the Netflix prize contest. And it was a good deal for Netflix. “You look at the cumulative hours and you’re getting Ph.D.’s for a dollar an hour,” Mr. Hastings said in an interview.
Netflix, Mr. Hastings said, did not do a crisp cost-benefit analysis of its investment in the contest. But several crucial techniques garnered from the contest have been folded into the company’s in-house movie recommendation software, Cinematch, and customer retention rates have improved slightly. Better recommendations, Netflix says, enhance customer satisfaction.
“We strongly believe this has been a big winner for Netflix,” Mr. Hastings said.
The prize winner was a team of statisticians, machine-learning experts and computer engineers from the United States, Austria, Canada and Israel, calling itself BellKor’s Pragmatic Chaos. The group was actually a merger of teams that came together late in the contest.
In late June, the team finally surpassed the threshold to qualify for the prize by doing at least 10 percent better than Cinematch in accurately predicting the movies customers would like, as measured against actual ratings. Under the contest rules, that set off a 30-day period in which other teams could try to beat them.
That, in turn, prompted a wave of mergers among competing teams, who joined forces at the last minute to try to top the leader. In late July, Netflix declared the contest over, and its online leader board showed two teams had passed the 10 percent threshold: BellKor and the Ensemble, a global alliance with some 30 members.
Netflix said the contest was too close to call, and the leader board showed a slight edge to the Ensemble. However, the teams’ software had to go through two data sets — one public, which was the basis for the leader board, and another hidden one, which determined the outcome of the contest.
The second data set was there to ensure that the winning solution really was the best at making better movie recommendations in general, and was not just tailored to get the best score from the public data set.
Win or lose, researchers agreed that they entered the contest in good part to get access to the Netflix data. “It was incredibly alluring to work on such a large, high-quality data set,” said Joe Sill, an independent consultant and machine-learning expert who was a member of the Ensemble.
Chris Volinsky, a member of BellKor, who is a scientist atAT&T Research, said Netflix “made a brilliant move by realizing that there was a research community out there that worked on these kinds of models and was starving for data.
“Netflix had the data, but only a handful of people working on the problem.”
Netflix was so pleased with the results of its first contest that it announced a second one on Monday. The new contest will present contestants with demographic and behavioral data, including renters’ ages, gender, ZIP codes, genre ratings and previously chosen movies — but not ratings. Contestants will then have to predict which movies those people will like.
Unlike the first challenge, the contest will have no specific accuracy target. Instead, $500,000 will be awarded to the team in the lead after the first six months, and $500,000 to the leader after 18 months.
The winners of the first contest said the money would be split seven ways, according to a formula they declined to disclose. The amounts each received, they said, would certainly help with a car, house payments or children’s college educations — but were not life-changing.
When asked if he planned to take on the second Netflix prize, Bob Bell, a scientist at AT&T Research, said, “I like the notion, but I think I’m too tired.”