Netflix Prize Update

I previously wrote about the Netflix prize: come up with a better system to recommend movies based on a large amount of data, and win $1 million. Tim Spey has a wonderful article on the dataset and the competition (though I can’t see a couple of the graphs he talks about). It is clear the dataset is pretty strange:

  • Customer 2170930 has rated 1963 titles and given each and every one a rating of one (very bad). You would think they would have cancelled their subscription by now.
  • Five customers have rated over 10,000 of the 17,770 titles selected – and presumably they also have rated some of the others among the 60,000 or so titles Netflix had available when they released the ratings. Are these real people?
  • Customer 305344 had rated 17654 titles. Even though Netflix make it easy to rate titles that you have not rented from them (so they can get a handle on your preferences) can this be real?
  • Customer 1664010 rated 5446 titles in a single day (October 12, 2005).

The main point of the entry, however, is that it is unclear that these sort of recommender systems can be useful in predicting consumer preferences. One striking point made is that a naive algorithm (predict the average value so far) is not that much worse than the Netflix system:

A simple algorithm that uses the average rating for each title as the prediction – “let’s see, the average rating for the 104,000 customers who rated Mean Girls was 3.514, so I predict you will give it a rating of 3.514” – gets an RMSE [Root Mean Squared Error] of 1.0540. Netflx’s Cinematch algorithm has an RMSE of 0.9525. Netflix set the prize target at a 10% improvement over that, which is an RMSE 0.8563. So the range that recommendation systems can realistically cover – from naively simple to cutting-edge research – seems to be [a] narrow band

To put that in perspective, here is the effect that sort of decrease in error has:

Anyway, if the errors followed a normal distribution (which they don’t, but we’re talking back-of-envelope here) then if a customer actually rated a title as 2 (poor), an algorithm with an RMSE of 1.0 would predict somewhere between 1 and 3 about 70% of the time. Not bad, but not startling. If the algorithm gave ten recommended movies, then it would get on average seven out of ten within one unit of the customer’s actual rating. Meanwhile, the RMSE=0.8563 algorithm would get 7.6 out of ten. While this is an improvement, and while it may be a remarkable technical accomplishment, it does not seem to be exactly a revolutionary leap compared to the really simple algorithms as far as customers go.

In short, would a customer even notice the difference? He concludes:

I’m no futurist, but I see little evidence from the first 300 days of the Netflix Prize that recommender systems are the magic ingredient that will reveal the wisdom of crowds.

This is an excellent blog entry that really goes to the heart of the value (or lack thereof) in these sorts of models.

EURO Gold Medal winner

The EURO conference in Prague has begun. I really like Prague: it is a beautiful city with great bars, restaurants, museums, and even an OK english-language bookstore or two.

The first order of business here is the opening session (somewhat after the truly first, 8AM technical talks). Awarded at the opening session is the EURO Gold Medal, bestowed as follows:

The EURO Gold Medal is the highest distinction within OR in Europe. It is conferred on a prominent person or institution, for an outstanding contribution to the Operational Research science. The award is officially bestowed in conjunction with a EURO Conference, if there is a suitable candidate.

One advantage of having a conference at a university (along with such disadvantages as funny room layout, confusing directions and an air of “this isn’t what we normally do”: I will say the organizers have done a great job in making things as easy as possible with transit passes, lots of student helpers and so on) is available wireless. So I am in the auditorium, ready to announce the EURO Gold Medal first on the blogosphere.

And the laureate is Aharon Ben-Tal of the Technion, Israel. He works in continuous optimization, so I am not particularly well versed in his work. This is from his web page:

Prof. Ben-Tal’s research work is mainly in the area of nonlinear optimization. His theoretical work is concerned with extremum principles for problems in a general setting, with regard to the underlying decision space, and the underlying smoothness of the functionals. He was among the first to develop a comprehensive theory of second-order optimality conditions for nondifferential problems.

Prof. Ben-Tal is also involved in research in stochastic mathematical programming. He introduced the concept of entropic-penalty for problems with randomness in the constraints, and developed a duality theory which established a link between stochastic programming and the Expected Utility principle in economics. Recently he developed, together with Prof. A. Nemirovski, the Robust Optimization methodology. The focus of Prof. Ben-Tal’s work in recent years is in computational methods for solving large-scale continuous optimization problems. The algorithms he develops are used in designing optimally complex engineering structures, water distribution networks, and techniques for medical image reconstruction. The above projects are carried out in the MINERVA Optimization Center, a 2 million DM endowed research center.

Prof. Ben-Tal was a member of the International Council of the Mathematical Programming Society. He served as Area Editor of the journal Mathematics of Operations Research, and is currently a member of the Editorial Board of the journals Convex Analysis and SIAM Optimization. He received Awards of Excellence from the Technion both for research and for teaching.

The work with the most impact is the work he did with Nemirovski in creating the area of robust optimization.

Sloan-Kettering wins 2007 Edelman Award

The Franz Edelman award is the most prestigious award for the practice of operations research, and each year’s competition is hotly contested. Nominees need to spend significant time preparing their presentations and almost all of them end up involving CEOs or other top executives in the firm.

This year, the winner of the Edelman Award is Sloan-Kettering for work entitled “Operations Research Answers to Cancer Therapeutics.” From the announcement

Yesterday was the first time that the association awarded the Edelman prize for a medical treatment. The Sloan-Kettering win demonstrates how operations research and mathematics are increasingly bringing improvements to health care, not only in the areas of policy, finance, and public health but in diagnosis and treatment, as well.

Dr. Marco Zaider, Attending Physicist in Medical Physics at Memorial Sloan-Kettering Cancer Center received the award together with Professor Eva K. Lee, Director of the Center for Operations Research in Medicine and HealthCare in the School of Industrial and Systems Engineering at Georgia Institute of Technology.

The 2007 Franz Edelman Award winner was announced at a special awards banquet during The Institute for Operations Research and the Management Sciences (INFORMS®) Conference on O.R. Practice in Vancouver. http://meetings.informs.org/Practice07/ The finalists included Coca-Cola Enterprises, Hewlett-Packard, DaimlerChrysler, and the U.S. Coast Guard.

Dr. Lee and Dr. Zaider devised sophisticated optimization modeling and computational techniques to implement an intra-operative 3D treatment planning system for brachytherapy (the placement of radioactive “seeds” inside a tumor) that offers a safer and more reliable treatment.

The real-time intra-operative planning system eliminates pre-operation simulation and post-implant imaging analysis. Based on the range of costs of these procedures, Prof. Lee estimated conservatively that their elimination nationwide could save $450 million a year for prostate cancer care alone.

I am hoping for some press coverage for this (the winning work is very important, as are the contributions of the other finalists), but not much so far (just a couple pickups from health-oriented web sites). Reporters: great opportunity here!

Papers associated with the Edelman finalists will appear in the January-February issue of Interfaces.

Do Operations Research, win $1 million

Art Geoffrion wrote me, pointing out that the Netflix Prize is a great opportunity for OR people to show their stuff. Netflix is offering up to $1 million for a system that predicts whether a customer will like a movie or not. They have made available a wonderful database of 100,000 ratings. Lots of people have used data mining methods on this database For me, the line between data mining and OR is very thin indeed, so it would be interesting to see what an OR approach can do with this.

The Wall Street Journal has an article on these types of prizes. There are a lot of good reasons for companies to provide these competitions:

Prizes prompt a lot of effort, far more than any sponsor could devote itself, but they generally pay only for success. That’s “an important piece of shifting risk from inside the walls of the company and moving it out to the solver community,” says Jill Panetta, InnoCentive’s chief scientific officer. Competitors for the $10 million prize for the space vehicle spent 10 times that amount trying to win it.

Contests also are a mechanism to tap scientific knowledge that’s widely dispersed geographically, and not always in obvious places. Since posting its algorithm bounty in October, Netflix has drawn 15,000 entrants from 126 countries. The leading team is from Budapest University of Technology and Economics.

Given the generality of OR, it is clear that our field can be competitive in many of these. Any takers?

Bloggers and OR

I try to keep up with people in OR who are blogging, but even with sites like technorati, I fear I am missing some. So if you are an OR blogger, don’t hesitate to let me know directly!

On that note, Scholarships around the US is offering a $5,000 scholarship for the best student blog. One requirement:

Your blog must contain unique and interesting information about you and/or things you are passionate about.

Well of course if you are blogging about OR, you are by definition writing about interesting things and you will naturally be passionate! I don’t know anything about this group, but they seem on the up-and-up (the site is certainly 95% useful and 5% ads, rather than the reverse).

Stockholm Prize in Criminology

Carnegie Mellon faculty member and operations researcher Al Blumstein has won this year’s Stockholm Prize in Ciminology, along with Terrie E. Moffitt. According to the prize announcement:

Blumstein’s analyses of the variations in the frequency of offending in careers of active criminals in US jurisdictions have also had a global influence on justice policies and practices, as well as on the rapid increase in the influence of developmental and life-course criminology.

The prize will be awarded in June 2007 (and is worth 1 million Swedish Kroner). Al has been President of all three of ORSA, TIMS, and INFORMS.

EURO Management Science Strategic Innovation Prize

A lot of the entries in this blog are about significant applications of operations research in practice. EURO has a prize on Management Science Stratetic Innovation (MSSIP). Their 2007 prize will be in the area of OR/MS in Logistics. From the announcement:

The prize is intended to recognize the role of Operational Research/Management Science in Logistics. Needless to say that Logistics or Supply Chain Management is taking centre stage in Operations. In the business world, Supply Chain Management has been a crucial discipline in a global world with an increasing number of players involved in bringing even the simplest of products to the customer. It is fair to say that a separate science of Supply Chain Management has been developed in the last fifteen years (even though Logistics goes back many centuries or even millennia).

Recent events have propelled Supply Chain Management into the boardroom in some businesses but the true strategic impact of Logistics is still poorly appreciated by numerous companies. This is even more the case among policy makers. Obviously, Logistics has always been central to successful military operations and the recent dramatic tsunami in Asia has convinced everyone of the central role of Logistics in disaster response as well. In spite of this rise in attention to logistics, many organizations (public or private) have not yet taken the logical next step of reorganizing their structures accordingly.

OR/MS has been a large contributor to the newly developed science of Logistics/Supply Chain Management. But there is a lot yet to do. The MSSIP 2007 price intends to recognize the OR/MS contribution but also to encourage researchers and professionals to continue and perhaps even increase their efforts in contributing to the content as well as the recognition of Logistics/Supply Chain Management as a core discipline and function in business and society.

One unusual aspect of this award is the breadth of definition of logistics:

So we will kindly regard innovative applications of OR/MS to new or unusual Logistics situations. Think for example about risk, robustness, agility, humanitarian organizations, new products/markets/services, interfaces and integration with other disciplines like accounting and finance, and the like.

The deadline on this is March 1, 2007, so there is plenty of time to put a paper together.