Dominic Cummings blog advert to bring different kinds of thinking into government. Seems loosely inspired by Philip Tetlock team of super forecasters. What can their story teach us about how governments could use statistical thinking. To get a head start on the future. Philip Tetlock is a professor at the University of Pennsylvania. I came across Philip Tetlock website because I read his earlier work and was curious about what he was up to.
In 2011 the director of national intelligence. Wanted to commission a series of forecasting turn events. To determine whether it would be possible. To identify talent making subjective probability judgments of important events. It was a response by the US intelligence community to previous intelligence failures. Most notably in the previous decade. 911 and the weapons of mass destruction controversy in Iraq.
So, he is looking for volunteers to be part of this research project because that is what they needed a big crowd. So, Tetlock signed up to be part of that.
You may have heard of a television programme ‘Who Wants to be a millionaire’?
The television game show originally produced by Toddla TV. Now owned and licenced by Sony Pictures Television. And now broadcast all around the world. If you are on that show and you want to be a millionaire, ask the crowd. The audience are really good judges as to what the answer is to a question.
Statistically the audience gives the right answer. Something like ninety three percent of the time. It is really impressive. The research projects task was to do better than ninety three percent.
They were other teams. For example, major universities such as University of Michigan and MIT. The challenge which team could come up with the most accurate predictions. About subjects of interest to US intelligence. They were posing a variety of different kinds of questions. Mainly about geo-politics, macroeconomics. Election outcomes in Zimbabwe. Whether they will be a recession in Taiwan and as it happens the likelihood of a disease outbreak too.
Tetlock was ready for this competition. He had spent a long-time perfecting technique for better forecasting. For example, greater cultural diversity, people from different backgrounds, weirdos.
He gave an example from a famous movie Zero Dark 30. A movie about the hunt for Osama bin Laden.
How likely is it that Osama is in a particular mystery compound in the Pakistani town of Islamabad.
He goes on to say I am about to look the president in the eye. What I would like to know from each of you is where do you stand. He goes around the table and he asks each of the analysts around the table and asks the question.
Is he there or is he not there?
Each of the analysts responds Mr Director I think there is a 70% chance he is there.
James Gandolfini sights the Iraq WMD case.
Apparently, the case for that was much stronger than this case.
The analysts respond by stating we do not deal in certainty. We deal in probabilities.
James Gandolfini asks do you ever agree on anything.
What should the director think?
Most would say the answer is rather obvious you should assume it is 70%. But you would only draw the 70% inference if the people around the table were clones of each other. But if one person around the table is drawing on satellite reconnaissance. Others draw upon cyber code breaking. Others draw upon human intelligence. If they are all converge on same answer.
Would you still say 70% now?
If people have different reasons for the 70% confidence, then you should be more confident. Than those with the same background and knowledge.
Tetlock team came from:
A wide variety of different backgrounds
The team did not know genders.
They didn’t know each other’s nationalities.
They did not know each other’s political beliefs.
Tetlock went on to say one person he had on his team for two years lived about a mile away and he did not know that.
Tetlock provides another example. Imagine there is a mosaic and this mosaic tiles slowly starts filling in. One person finds an important piece of information and shares it. Others in the team all bring different tiles to complete the mosaic. You might assume the combined skills of the team would help to complete the mosaic as early as possible. The competition taught Tetlock and his team something else.
In the first year that there were some people who are consistently better than the rest of the crowd. They called these people super forecasters and Warren Hatch would go on to become one.
They were not from specialist backgrounds. But it seemed they could out forecast the experts.
They all had a good understanding of probability and statistics.
But other qualities seemed even more important.
- They are curious.
- They think about their beliefs.
- They test propositions.
- They do not hold sacred possession.
- They are willing to change their minds in a fairly rapid way in response to new advents.
- And they are willing to take what Phil Tetlock calls the outside view.
Imagine that you are at a wedding. And someone has the bad taste to saddle up to you and ask you how likely, is this marriage to survive. Most people will say they look really happy. This is a joyous occasion I cannot imagine them getting divorced. But if forced to make a judgment you might say there is a high probability, they stay married. This perspective is called the inside view. You are looking at the marriage from the inside.
The outside view would take a statistic viewpoint. Let say for this couple’s social demographic group the likelihood of break up. In the first 7 years of marriage is 30%. They are much closer to being right. Than those who say, there is only a one percent chance of divorce because the couple looks so happy.
Provide the inside view with more information. For example, the groom is a psychopathic philanderer. The likelihood of divorce is going to be much higher than 30% maybe sixty, seventy, eighty percent.
So, what does this mean apart from do not invite this person to your wedding?
It means when somebody ask you a question for example how long a particular civil war is likely to last. Most will typically start with the population number. In the past civil wars typically last this period of time for a given population number. Then they begin to adjust for foreign interests. For example, the Syrian civil war. Is likely to last longer than the average civil war because of this and that, and that.
And for predicting the kind of challenges that the government might face it works. Not only did super forecasters win the competition. They also beat other teams they did not even know were in the competition. They found out later some of the other teams were intelligence analyst. With access to secret information and other experts. For some reason that factor took some time to be disclosed but the answer is a super forecaster beat them by 30%.
One thing you can learn from super forecasters is that certainty could lead you astray. The readiness to accept uncertainty. To go against received wisdom can give you a more accurate prediction about the future.
So, should we just let super forecasters run the government?
Sir David Spiegelhalter. States he has enormous respect for them. And they do have certain appealing characteristics. As Tetlock and others have made clear.
- They must be cold and analytic.
- They need to be clinical in their thinking.
- They really do not care about what happens in the future.
- Just what are the chances of specific events and this is a remarkably difficult thing to do.
Its remarkably difficult to separate our emotions about what we feel we want to happen. And what should happen and what we actually think will happen.
You really want them around. You want them there in the room making conclusions based on the cold evidence.
But do we want them in government?
Cummings call out for weirdos. Led to the hiring of super forecaster Andrew Sabisky. Who promptly resigned when his previous comments about eugenics came to light.
You don’t want super forecasters making the decisions. They have no concern about the consequences at all. Whereas politicians must look at the consequences.
Our school day is nearly over but before we grab our coats from the books and head for the gates what homework do, we need.
Reading the previous two articles.
- We learnt a better understanding of numbers would be beneficial.
- How to better present numbers would be helpful.
- A clearer grasp of uncertainty and probabilistic thinking may improve decision making.
- We looked at how powerful super forecasters can be better and forecasting. But we decided we don't want the forecasters making political decisions.
A good decision maker. Will recognise that they should rely on as much information as possible. Especially for critical decisions. A great example is when Eisenhower landed in Normandy. He had two letters with him. Letter number one was the one that he issued to the troops. We are going in. Victory is ours. Off we go. He had another letter in his back pocket. That said the failure of the invasion is my responsibility alone.
He was thinking probabilistically. Maybe this is not going to go all right. But when it came to motivating people to get things done, he needed to appear decisive.
Because in a crisis the challenge is how to acknowledge uncertainty. While grasping the nettle of decisions.
About five years ago. In the planning for a pandemic it was given in the national risk register. A one in twenty chance of a major flu pandemic happening in the next five years. And that is still fairly unlikely. However, by looking at the consequences. It suggests major planning should take place. In terms of stock piling protective equipment and so on. Which did not happen. So, there is an example of what might in the future might be seen as poor political decision by somebody. We do not know where it happened. Even though the analysis strongly suggested that this was a plausible event. It was more likely than not going to happen in the next five years, but it was still a plausible event. And it was the worst thing in the national risk register.
It looks as if we our planning. Even the best mathematical models do not lead to adequate PPE provisioning. For our political leaders to put data to best use we need both predictions and action.
Next time I am going to investigate the R value (reproduction number).
What is the R number and how is R calculated?