Skip to content

Crowdsourcing the Travelling Salesman

Despite some worries, the field of operations research is not exactly sitting on a street corner with a begging bowl.  There are lots of people out there who are willing to pay us for what we do.  Perhaps not as often as they should, or as much as we deserve, but “operations research” is a legitimate career path.  You can get jobs in government, academia or industry; you can work as a consultant; you can write (and get paid for) software; you can even write popular books on the subject!

But if you create an operations-research inspired movie, you might find it a rather tough go to get enough fannies in the seats in order to pay the bills.  While we certainly are captivated by the idea of “what would happen if P=NP”, most of the world seems far more captivated with “I wonder how James Franco would do as Oz”.  So when a movie does explore an operations research theme, it needs to be creative in its funding.

Traveling Salesman is a movie that explores what would happen if a group of mathematicians did find a fast algorithm for the Traveling Salesman Problem.  Now, I know what I would do if I found such an algorithm:  I would dance a jig on my Dean’s desk, and immediately look for the cushiest academic job I could find.  It appears that the writers of this movie went off in a slightly different direction, involving government agents, ominous music, and sweat-soaked brows.  Well, I suppose that might happen too.

The producers of the film are trying to get broader distribution, and are counting on the wisdom of the crowds to fund that distribution.  There is an IndieGoGo campaign with a very modest goal of raising $3500.  With that, they will make the film available through such outlets as (from their campaign):

  • Buy and Rent from iTunes
  • Buy and Rent from Amazon VOD
  • Buy and Rent from Google PLAY
  • Buy and Rent from Distrify
  • Netflix and Hulu are under consideration
  • Other outlets are also in negotiation

Personally, I love it just for the tchotchkes (that movie poster will look great in my office).

Right now they are at $2675:  is there interest in the operations research community to put them over the top?

Doing work on Metaheuristics and Optimization/Constraint Programming?

I’m putting together a track at the upcoming 10th Metaheuristics International Conference. The conference will be held August 5-8, 2013 in Singapore. The conference website is

The topic of the Track is Metaheuristics and Optimization/Constraint Programming. There has been a lot of work recently on combining metaheuristics with exact methods. I think it is a very exciting area: faster optimization codes and speedier computers make it easier to use optimization or constraint programming as part of a metaheuristic approach to problems.

If you have some work that you would like to present at the conference, I encourage you to submit that work to the track. Submissions can be in the form of 10 page full papers or 3 page extended abstracts. All submissions are thoroughly reviewed. More information on submission is at the general Call for Papers:

You will submit your paper through ConfTool at Note that you will have the opportunity to select the track “Special Session on Meta-Heuristics and Constraint Programming” (or similar: the track is not limited to CP).

The submission deadline is February 28. Please let me know if you have any questions.

Summer Internship at IBM Research, AI for Optimization Group

Just saw this announcement of a summer internship

A summer internship position is available for 2013 in the “AI for Optimization” group within the Business Analytics and Mathematical Sciences department at IBM Watson Research Center, Yorktown Heights, New York.  The internship will last for about 3 months and will be scheduled between March and October, 2013.

Candidates should be exceptional Masters or PhD students in Computer Science and related areas, and not have already received their PhD by the internship start date.  The successful candidate is expected to have strong interest and some experience in one or more of the following:

 +  Developing Novel Technologies based on AI and OR to advance the state of the art in combinatorial optimization (e.g., Heuristic Search, Mixed Integer Programming (MIP), Linear Programming (LP), Satisfiability (SAT))

 +  Robust Parallelization of search-based algorithms (e.g., using parallel Branch&Bound; information exchange) exploiting novel computing architectures such as BlueGene, multi-core end-user machines, large compute clusters, and the Cloud

 +  Advancing Simulation-Based Optimization approaches to solve real-world problems and/or to improve optimization methods themselves by, e.g., employing techniques from Machine Learning, Local Search, and Genetic Algorithms for automated algorithm selection and parameter tuning

 +  Applications of Optimization to Analytics, Production Modeling, Sustainability, Systems, and Databases

Interested candidates should send their CV as well as names and contact information of at least two references to all of the following:

Ashish Sabharwal []
Horst Samulowitz []
Meinolf Sellmann []

I wish I didn’t already have my PhD!

Easy and Hard Problems in Practice

David Eppstein of the blog 0xde has a list of his top 10 preprints in algorithms in 2012.  One particularly caught my eye:

Clustering is difficult only when it does not matter, Amit Daniely, Nati Linial, and Michael Saks,  arXiv:1205.4891. […] this represents a move from worst-case complexity towards something more instance-based. The main idea here is that the only hard instances for clustering problems (under traditional worst-case algorithms) are ones in which the input is not actually clustered very well. Their definition of a “good clustering” seems very sensitive to outliers or noisy data, but perhaps that can be a subject for future work.

This paper really hit home for me.  I have taught data mining quite often to the MBAs at the Tepper School and clustering is one topic I cover (in fact, research on clustering got me interested in data mining in the first place).  I generally cover k-means clustering (easy to explain, nice graphics, pretty intuitive), and note that the clustering you end up with depends on the randomly-generated starting centroids.  This is somewhat bothersome until you play with the method for a while and see that, generally, k-means works pretty well and pretty consistently as long as the data actually has a good clustering (with the correct number of clusters).  It is only when the data doesn’t cluster well that k-means depends strongly on the starting clusters.  This makes the starting centroid issue much less important:  if it is important, then you shouldn’t be doing clustering anyway.

There are other operations research algorithms where I don’t think similar results occur.  In my work in practice with integer programming, most practical integer programs turn out to be difficult to solve.  There are at least a couple of reasons for this (in addition to the explanation “Trick is bad at solving integer programs”).  Most obviously, easy problems typically don’t get the full “operations research” treatment.  If the solution to a problem is obvious, it is less likely to require more advanced analytics (like integer programming and similar).

More subtly,  there is a problem-solving dynamic at work.  If an instance is easy to solve, then the decision maker will do something to make it harder.  Constraints will be tightened (“What if we require at least 3 weeks between consecutive visits instead of just two?”) or details will be added (“Can we add the lunch breaks?”) until the result becomes a challenge to solve.  I have not yet had a real-world situation where we exhausted the possibilities to add details to models or to further explore the possible sets of constraints.  Eventually, we get to something that we can’t solve in a reasonable amount of time and we back up to something we can (just) solve.  So we live on the boundary of what can be done.  Fortunately, that boundary gets pushed back every year.

I am sure there is a lot of practical work in operations research that does not have this property.  But I don’t think I will wake up one morning to see a preprint: “Integer programming is difficult only when it doesn’t matter”.

Help Santa Plan His Tour (Twice!)

Just in time for the holidays, there is a very nice competition at Kaggle: The Traveling Santa Problem. The problem was devised by the ever-creative Bob Bosch, about whom I have written before. The problem is an interesting variant on the Traveling Salesman Problem. Given a set of points and distances between them, the TS(alesman)P is to find the shortest cycle through all the points.  The TS(alesman)P would not be a great competition problem:  you could save the effort and just send the prize to my soon-to-be-neighbor Bill Cook with his Concorde code.

The TS(anta)P asks for two disjoint cycles, with a goal of minimizing the longer of the two. Of course, there is a clear explanation of why Santa wants this:

Santa likes to see new terrain every year–don’t ask, it’s a reindeer thing–and doesn’t want his route to be predictable.

OK, maybe it not so clear. But what Santa wants, Santa gets.

There is just one instance to work with, and it is a monster with 150,000 points. It turns out that the points are not randomly scattered throughout the plane, nor do they seem to correspond to real cities. I won’t spoil it here, but there is a hint at the kaggle discussion boards.

There are prizes for the best solution by the end of the competition (January 18, 2013) and, most interestingly, at a random date to be chosen between December 23 and January 17.  The $3000 in prizes would certainly make a nice Christmas bonus (that is  7.5 Lego Death Stars!). You can check out the full rules, leaderboard, discussion, and more at the competition site.

I personally find this competition much more interesting than the data-mining type competitions like the General Electric sponsored Flight Quest (which is certainly not uninteresting). In Flight Quest, the goal is to predict landing times for planes. This is all fine and good, but as an operations researcher, I want to make some decisions to change those times to make the system work better.  Helping Santa avoid ennui might not be particularly realistic but it is much better than simply predicting when my Christmas toys arrive.

If we can get a good turnout for the Santa problem, perhaps we can see more optimization competitions, or better yet, competitions that combine data mining with optimization.

Which Average do you Want?

Now that I am spending a sentence as an academic administrator in a business school (the Tepper School at Carnegie Mellon University), I get first-hand knowledge of the amazing number of surveys, questionnaires, inquiries, and other information gathering methods organizations use to rank, rate, or otherwise evaluate our school. Some of these are “official”, involving accreditation (like AACSB for the business school and Middle States for the university). Others are organizations that provide information to students. Biggest of these, for us, is Business Week, where I am happy to see that our MBA program went up four positions from 15th to 11th in the recent ranking. Us administrators worry about this so faculty don’t have to.

Responding to all these requests takes a huge amount of time and effort. We have a full-time person whose job is to coordinate these surveys and to analyze the results of them. Larger schools might have three or four people doing this job. And some surveys turn out to be so time-intensive to answer that we decline to be part of them. Beyond Grey Pinstripes was an interesting ranking based on sustainability, but it was a pain to fill out, which seems to be one reason for its recent demise.

As we go through the surveys, I am continually struck by the vagueness in the questions, even for questions that seem to be asking for basic, quantitative information. Take the following commonly asked question: “What is the average class size in a required course?”. Pretty easy, right? No ambiguity, right?

Let’s take a school with 4 courses per semester, and two semesters of required courses. Seven courses are “normal”, classes run in 65 student sections, while one course is divided into 2 half-semester courses, each run in 20 student seminars (this is not the Tepper School but illustrates the issue). Here are some ways to calculate the average size:

A) A student takes 9 courses: 7 at 65 and 2 at 20 for an average of 55.
B) If you weight over time, it is really 8 semester-courses: 7 at 65 and 1 at 20 for an average of 59.4
C) There are about 200 students, so the school offers 21 sections of 65 student classes and 20 sections of size 20 for an average of 43.

Which is the right one? It depends on what you are going to use the answer for. If you want to know the average student experience, then perhaps calculation B is the right one. An administrator might be much more concerned about calculation C, and that is what you get if you look at the course lists of the school and take the average over that list. If you look at a student’s transcript and just run down the size for each course, you get A.

We know enough about other schools that we can say pretty clearly that different schools will answer this in different ways and I have seen all three calculations being used on the same survey by different schools. But the surveying organization will then happily collect the information, put it in a nice table, and students will sort and make decisions based on these numbers, even though the definition of “average” will vary from school to school.

This is reminiscent of a standard result in queueing theory that says that the system view of a queue need not equal a customer’s view. To take an extreme example, consider a store that is open for 8 hours. For seven of those hours, not a single customer appears. But a bus comes by and drops off 96 people who promptly stand in line for service. Suppose it takes 1 hour to clear the line. On average, the queue length was 48 during that hour. So, from a system point of view, the average (over time) queue length was (0(7)+48(1))/8=6. Not too bad! But if you ask the customers “How many people were in line when you arrived?”, the average is 48 (or 47 if they don’t count themselves). Quite a difference! What is the average queue length? Are you the store or a customer?

Not surprisingly, if we can get tripped up on a simple question like “What’s your average class size?”, filling out the questionnaires can get extremely time consuming as we figure out all the different possible interpretations of the questions. And, given the importance of these rankings, it is frustrating that the results are not as comparable as they might seem.

Registries To Avoid Publication Bias

I have been thinking about the issue of how a field knows what they know.  In a previous post, I wrote about how the field of social psychology is working through the implications of fraudulent research, and is closely examining the cozy interactions between journals, reviewers, and famous researchers.   And any empirical field based on statistical analysis has got to live with the fact that if there 1000 results in the field, some number (50 perhaps, if p=.05 is a normal cutoff and lots of results are just under that value) are going to be wrong just because the statistical test created a false positive.  Of course, replication can help determine what is real and what is not, but how often do you see a paper “Confirming Prof. X’s result”?  Definitely not a smooth path to tenure.

This is worse if malevolent forces are at work.  Suppose a pharmaceutical company has bet the firm on drug X, and they want to show that drug X works.  And suppose drug X doesn’t work.  No problem!  Simply find 20 researchers, sign them to a non-disclosure, and ask them to see if drug X works.  Chances are one or more researchers will come back with a statistically significant result (in fact, there is about a 65% chance that one or more will, given a p=.05).  Publish the result, and voila!  The company is saved!  Hurray for statistics and capitalism!

Fortunately, I am not the first to see this issue:  way back in 1997, the US Congress passed a bill requiring the registration of clinical trials, before the trials get underway.

The first U.S. Federal law to require trial registration was the Food and Drug Administration Modernization Act of 1997 (FDAMA) (PDF).

Section 113 of FDAMA required that the National Institutes of Health (NIH) create a public information resource on certain clinical trials regulated by the Food and Drug Administration (FDA). Specifically, FDAMA 113 required that the registry include information about federally or privately funded clinical trials conducted under investigational new drug applications (INDs) to test the effectiveness of experimental drugs for patients with serious or life-threatening diseases or conditions.

This led to the creation of (where I am getting this history and the quotes) in 2000.  This was followed by major journals requiring registration before papers could be considered for publication:

In 2005 the International Committee of Medical Journal Editors (ICMJE) began to require trial registration as a condition of publication.

The site now lists more than 130,000 trials from around the world.  It seems this is a great way to avoid some (but by no means all!) fraud and errors.

I think it would be useful to have such systems in operations research.  When I ran a DIMACS Challenge twenty years ago, I had hoped to keep up with results on graph coloring so we had a better idea of “what we know”:  then and now there are graph coloring values in the published literature that cannot be correct (since, for instance, they contradict published clique values:  something must be wrong!).  I even wrote about a system more than two years ago but I have been unable to find enough time to take the system seriously.  I do continue to track results in sports scheduling, but we as a field need more such systems.


Referees considered harmful

When doing empirical work, researchers often mess up either in the design of the experiment or in the analysis of data.  In operations research, much of our “empirical work” is in computational testing of algorithms.  Is algorithm A faster than algorithm B?  “It depends” is generally the only honest answer.  It depends on the instance selection, it depends on the computing environment, it depends on the settings, etc. etc.   If we are careful enough, we can say things that are (within the limits of the disclaimers) true.   But even a a careful experiment can fall prey to issues.  For instance, throwing away “easy” instances can bias the results against whatever algorithm is used to determine easiness.  And don’t get me started on empirical approaches that test dozens of possibilities and miraculously find something “statistically significant”, to be duly marked with an asterisk in the table of results.    It is very difficult to truly do trustworthy empirical work.  And it is even harder to do such work when researchers cheat or reviewers don’t do their job.

For some fields, these issues are even more critical.  Operations research generally has some theoretical grounding:  we know about polytopes and complexity, and so on, and can prove theorems that help guide our empirical work.  In fields like Social Psychology (the study of people in their interactions with others), practically all that is known is due to the results of experiments.   The fundamental structure in this field is a mental state, something that can only be imprecisely observed.

Social psychology is in a bit of a crisis.  In a very real sense, the field no longer knows what is true.  Some of that crisis is due to academic malfeasance, particularly that of an influential researcher Diederik Stapel.  Stapel has been found inventing data for dozens of papers, as described by a  “Science Insider” column.

Due to data fraud by Stapel and others, the field has to reexamine much of what it thought was true.  Are meat eaters more selfish than vegetarians?  We thought so for a while, but now we don’t know.  A Dutch report on this goes into great detail on this affair.

But overt fraud is not the only issue, as outlined in the report.  I was particularly struck by the role paper reviewers played in this deceit:

It is almost inconceivable that co-authors who analysed the data intensively, or reviewers of the international “leading journals”, who are deemed to be experts in their field, could have failed to see that a reported experiment would have been almost infeasible in practice, did not notice the reporting of impossible statistical results, … and did not spot values identical to many decimal places in entire series of means in the published tables. Virtually nothing of all the impossibilities, peculiarities and sloppiness mentioned in this report was observed by all these local, national and international members of the field, and no suspicion of fraud whatsoever arose.

And the role of reviewers goes beyond that of negligence:

Reviewers have also requested that not all executed analyses be reported, for example by simply leaving unmentioned any conditions for which no effects had been found, although effects were originally expected. Sometimes reviewers insisted on retrospective pilot studies, which were then reported as having been performed in advance. In this way the experiments and choices of items are justified with the benefit of hindsight.

Not infrequently reviews were strongly in favour of telling an interesting, elegant, concise and compelling story, possibly at the expense of the necessary scientific diligence.

I think it is safe to say that these issues are not unique to social psychology.  I think that I too have, as a reviewer, pushed toward telling an interesting story, although I hope not at the expense of scientific diligence.   And perhaps I could have worked harder to replicate some results during the reviewing process.

I don’t think we in operations research are in crisis over empirical issues.  I am pretty confident that CPLEX 12.4 is faster than CPLEX 4.0 for practically any instance you can throw at it.  And some journals, like  Mathematical Programming Computation have attempted to seriously address these issues.  But I am also pretty sure that some things I think are true are not true, either due to fraud by the author or negligence by reviewers.

One important role of a reviewer is to be on the lookout for malfeasance or bias and to avoid allowing (or, worse, forcing) authors to present data in an untruthful way.  And I think many of us are not doing a great job in this regard.  I would hate to have to rethink the foundations of my field due to these issues.

Operations Research and a Baseball Job

Analytics is getting to be more and more important in sports, and sports teams and leagues are looking to people with analytical skills to fill key roles in their organizations.   The MIT Sports Analytics conference is a big deal, attracting more than 2000 attendees, with an active job placement service.  The MBAs at my own school (the Tepper School) now has a sports analytics club, with a speaker series, case competition and more (including fun things like fantasy sports competitions) and many of these exceptionally bright and ambitious students are eager for jobs in the sports industry.  While some of this may be due to the success of Moneyball, much more of this is due to the fact that computers and decision making have gotten much, much better in the last years, making analytics a key competitive advantage.  And when you get past dashboards and basic data analysis and visualization, you move into using data to make better decisions.  In other words, you move into operations research.

It is clear that many clubs in Major League Baseball get it.  I see it when talking to people with my local team, the Pittsburgh Pirates (a team that I am sure will break .500 any year now!), and I just got a job announcement that shows that the next closest team to me, the Cleveland Indians, get it too.  They are looking for a VP-Technology, but it is clear that they see this as a job involving decision making, not just infrastructure.  From the ad, the primary purpose is:

The Vice President of Technology is responsible for developing, implementing, measuring and maintaining
plans that advance the organization’s achievement of its guiding commitments through enhanced
Baseball Operations and business decision-making tools, increased effectiveness of systems, hardware,
technology infrastructure and improved fan experience through fan-centric technology implementations.

I love the “decision-making tools” in that description.  Sounds just right for an operations research person who also understands technology.


The cutting plane method for matching is polynomial

Michael Mitzenmacher is a computer scientist at Harvard with a blog My Biased Coin.  As you might expect from the title, Michael works in the area of randomized algorithms, and even has a book on the subject.  His blog is an extremely useful guide to the what is happening in algorithms in CS (and what is happening in CS at Harvard, which is also quite interesting).  He often provides a summary of talks given at the big theory conferences (FOCS/STOC/etc.).  He just posted on this year’s FOCS (here and here).

There was one talk that caught my eye, summarized by a doctoral student:

[Editor: Fourth-year grad student Justin Thaler of Harvard contributes a summary of two unrelated talks.]

Paper Title: The Cutting Plane Method is Polynomial for Perfect Matchings.
Harvard’s own Karthekeyan Chandrasekaran talked about joint work with Laszlo A. Vegh and Santosh S. Vempala on cutting plane algorithms for matching problems. The cutting plane method is a popular algorithm for solving integer programs (IPs), used in commercial solvers. It works by starting with an LP relaxation of the given IP to obtain basic optimal solution x_0, and then iteratively adding constraints that are valid for integer solutions but violated by the basic optimum. It continues until the basic optimum is integral. The goal of this paper is to take a step toward explaining the practical efficiency of cutting plane methods, by giving an efficient cutting-plane algorithm for min-cost perfect matching (MWPM) –MWPM is known to be in P, but it was open (apparently for 30 years) whether there was a polynomial-time cutting-plane algorithm for this problem.
A brief summary of how they achieve this is as follows. They start with a natural, well-known LP relaxation of the MWPM problem, called the bipartite relaxation. This relaxation has the nice property that all basic optima x are half-integral, and the support of x is a disjoint union of edges and odd cycles. This makes it easy to find cuts (the cuts correspond to what are called blossom inequalities, see the paper for details). A major challenge, though, is that naively adding cuts will not preserve the half-integrality of intermediate LPs, so at each iteration they throw away some of the old cuts that were added earlier in the execution. They need to take considerable care in choosing which cuts to keep in order to guarantee half-integrality of intermediate LPs (and to ensure that their algorithm makes progress at a sufficiently high rate).
 This is pretty amazing.  First, it is wonderful that they were able to prove polynomiality.  It had bothered me that it seemed you might need an exponential number of cuts, even for something like matching.  I had looked at this 25 years ago when doing my doctorate, but didn’t have any particularly insightful ideas.
But the really amazing thing is that they were able to arrange their algorithm so they never had to work with anything worse than half-integral solutions.  This is astounding!  A bane of cutting plane approaches is the weird fractions that keep popping up, leading to numerical stability problems.  Here, they were able to keep things well under control.  And, by keeping to half-integral, maybe some old ideas I had about using generalized networks (networks with multipliers) might come back into play.   The approach certainly avoids the need for Gomory-Hu cut tree approaches to finding violated inequalities:  violated inequalities come straight out of  connected components.  This also harkens back to my dissertation where I had treated matching as a generalized network with side constraints.
So I checked out the paper that underlies the talk at arXiv, thinking I might try to implement some things this week and see how it works (CS theory people rarely implement:  you could make an entire career out of simply implementing what CS people suggest).  On the plus side, they reference my dissertation, so at least I am in the right ballpark.  On the down side:  it is looking a bit complicated!  Looks like I will have to reel in one of the bright doctoral students around to plow through this with me.
And I wonder what else this might be used for?