by Filip Palda
A rough analogy exists in sprinting where a tailwind of greater than a certain speed, or the presence of steroids invalidates world record times. You cannot say that the sprinter was the sole “cause” of the record because the tailwind was “positively correlated” with his or her forward motion. To credit the sprinter as being the sole cause of the record in such cases might be to bias attribution of success to his or her person and away from other causal variables such as wind and illegal drugs.
I say “might” because the other causal variables might work to cancel out each others’ effects. To protect themselves from such bias researchers may become overcautious and include a great many variables, with the risk that some have no business being in the regression at all. If you feed irrelevant variables into the regression formula this does not bias results, but it makes finding a causal link between relevant variables less likely. The problem is similar to applying to a dating service. The more criteria you specify, the less likely you are to find a partner.
Finally, the best specified regression may not show you any causal link if the random forces affecting performance are too great. To carry on with the sprinting analogy, if the wind buffets from many sides it will make it hard to judge whether the sprinter’s result is better or worse than it would have been without wind. Buffeting wind creates “noise” that obscures the link we are trying to make between effort and performance.
The role of models
TO HELP HIM cope with what variables to include in his regressions, Tinbergen did what generations of economists have done in imitation. He used a model as a best guess guide to selecting regression variables. Here is where The General Theory came to the rescue. If you believed in its soundness you could use its theoretical equations as a prescription for the practical search of cause and effect through regression.
Theory limits the choices of so-called explanatory or causal variables, and it can also limit the range within which you should look in your data for links and trends. Through the marriage of regression and the General Theory, the field of econometrics was born.
David Hendry summarizes the ideas described above as follows:
… econometrics commences an analysis of the relationships between economic variables (such as quantities and prices, incomes and expenditures, etc.) by abstracting the main phenomena of interest and stating theories thereof in mathematical form. The empirical usefulness of the resulting “models” is evaluated using statistical information of supposed relevance, and econometrics in the narrow sense (used hereafter) concerns the interpretation and analysis of such data in the context of “established” economic theory (1980, 388).
Now we can better understand the earlier quote by Sims that, “We are always combining the objective information in the data with judgment, opinion and/or prejudice to reach conclusions.”
Without a laboratory the economist is not learning uniquely from the numbers the way a physicist might, but is rather imposing on the numbers a certain range in which he or she believes beforehand a causal relation should exist.
Such an approach is contrary to the popular view of the scientist learning about causality through experiments. By stating a priori, that is, beforehand, what form the relationship between investment and interest rates should take, the economist is making an hypothesis about the form which causality might take. The regression technique he or she uses is just a recipe for finding patterns that seem to associate investment and interest and any other causal or “control” variables within the range specified by theory. In a different passage, Sims observes stoically that, “The best we can hope for is that econometricians are trained to confront the complexities and ambiguities that inevitably arise in non-experimental inference.”
How do economists justify this peculiar approach to analyzing data? They take the practical view that the world is not a laboratory where you can always control the environment to isolate the relationship between a cause and effect that might interest you. But that does not mean that useful information is absent from data the world generates. You just need to know how to look at the data. Looking at the data means having a lens that filters out what in this abstract discussion might be called glare, and other unwanted forms of light. Polarized lenses clarify a scene by allowing only light of a certain polarity to pass, eliminating scattered light which is not polarized, and even eliminating other forms of polarized light that interfere with the aspect of a subject you wish to examine. The economist takes the data and says, “My lens on the world is based on the notion that people make choices that maximize their well-being subject to constraints they face on their incomes, their time, and the limits placed on them by the law.”
Some call this prior view of the world a model. Others call it ideology. Whatever name one chooses, the economist’s approach presumes a relationship between investment and interest rates, advertising and smoking, the minimum wage and employment, or taxes and investment based on logical models of human behavior. A prior model narrows the search the economist needs to undertake by telling him or her which control variables to use. With the control variables in place, the economist can now use the regression technique to measure the quantitative impact of the causal variable of interest on the target variable. The problem is that if the economist’s model is incomplete or just plain wrong, the causal impacts measured may turn out to be meaningless or even misleading. This problem is akin to designing a lens for your glasses that either fogs up, or presents the world in too rosy or too dismal a color.
Despite these misgivings, Tinbergen and his colleagues were so gifted and visionary that their approach to data research dominated economics until the early 1980s. Then, in rapid succession, bad-boy econometricians Christopher Sims (1980), David Hendry (1980), and Edward Leamer (1983) launched what James Stock has described as a “scathing critique of contemporary econometric practice” (2010, 83). Leamer famously wrote that, “This is a sad and decidedly unscientific state of affairs we find ourselves in. Hardly anyone takes data analyses seriously” (37). Hendry wrote that, “It is difficult to provide a convincing case for the defense against Keynes’ accusation almost 40 years ago that econometrics is statistical alchemy since many of his criticisms remain apposite” (402).
The critiques were all based on the problems of finding the correct variables to put into the regressions. Each researcher suggested in his own manner that researchers needed to be less demanding of the data. Instead of insisting on finding the precise effects of certain economic forces, we might do better to settle for knowing some broad range into which these effects might fall. We also might not wish to disentangle causalities that ran both ways, but rather try to relate both variables to some fundamental feature of the economy. For example, interest rates could influence investment, but as machines were built and the economy became more productive, investors might demand more loans and this could drive up interest rates. Disentangling the effects of investment and interest rates on each other was so difficult that we should settle for understanding how each depended on some feature of the economy outside this causal loop.
These were difficult suggestions to understand and particularly to implement, but they had the merit of uniting economists in the belief that some simple, incontestable, non-subjective, that is, scientific way of analyzing data was needed. This realization led in the 1990s to what Joshua Angrist and Jörn-Steffen Pischke have described in a defining survey of the field as a “credibility revolution” in econometric work (2010, 4). The revolution was based not on fancy statistical theorizing and complicated formulas, but upon finding a way to simulate in economics the sort of controlled laboratory environment that physicists and other scientists take for granted. Let us examine why this development is so remarkable.
The credible path to control
TO THE SCIENTIST, control means that you block out any force, other than the one you are studying, from having an effect on the target of interest. Science does this by building laboratories that insulate their exp
eriments from vibrations and other contaminants. Economists and, to be fair, other social scientists such as psychologists, figured out that there was a way to mimic the laboratory. What you needed to do was find two groups of people who were similar in all ways that mattered to the target variable of interest. Then you subjected both groups to a different set of circumstances and then checked whether over time the target variables for each group differed. Then you attributed the difference between the two to the causal variable that differed between the groups.
Twins separated at birth were an early example of this procedure. Some were put into the care of rich families and others into the care of poor families in the same town and the same district. If the rich twins did better in life, you could assume that it was due to being brought up in luxury and not because of any mental or physical advantage over the other twin. Nurture, not nature, was responsible for the evolution of the target variable, which in this case is income. The twins example, taken from psychology, illustrates the point that similar groups may evolve differently if subjected to different influences. The differences in the performance of these groups can then be causally traced to the different circumstances in which they evolved.
Economic examples abound. Suppose you want to measure the effect of foreign aid on crop yields in Senegalese villages. You could just hand over the money and do a before and after comparison of yields. The problem is that you would not know if it was the aid or some other factor, uncontrolled for in your analysis, such as good weather, that influenced the change in yield. To avoid this confusion, before giving the aid you divide the villages into two groups. Each group has similar numbers of people, plot sizes, proximity to large towns, tractors, access to fertilizer, weather, and any other factor you think might influence the crop yield.
Now you give aid to only one subgroup of villages that in all other respects are similar to the remaining subgroup, and call it the treatment group. You do not give aid to another group and call it the comparison or “control” group. You wait a year, then measure how crop yields changed in each group. The difference between the yield chance of the group that received the aid, the treatment group, and the group that did not, the control group, gives an idea of the effect of the aid program. If both groups are the same in every respect except for the fact that the treatment group is the only one to receive aid, then the result of the treatment group will be the same as the result of the control group to which it is similar, augmented by the fact it received aid. If you subtract the increase in crop yield of both groups, then the influence of their similarities on yield cancel out and all you are left with is the effect of the only factor that differs between them: aid. Believe it or not, fortunes in aid dollars were spent, some under the guidance of economists trained in econometrics, without such an experiment ever being run, as the devastating exposé The Idealist by Nina Munk reveals. Norms for best practice are changing, but we will get to that a bit later.
For now, what we need to understand is that by creating similar treatment and control groups, you assure yourself that a part of the performance of each group will be similar because it is determined by the factors they share in common proportion. When you subtract the results of the control group from those of the target group, you filter out these common effects. All that is left is the net effect of what was different between them—in this case, receiving or not receiving aid. By subtracting the results of similar groups, economists filter out confounding effects to distill a unique cause. Thus they mimic the physicist’s laboratory where environmental control blocks out confounding effects.
The beauty of this method is that you do not need some complicated regression formula to tell you if foreign aid is having an effect on yields. All you need to do is know how to compare averages. Everyone understands averages. As long as you believe the experiment was properly carried out, then the results are understandable and credible. This is the revolution in data analysis that Angrist and Pischke are talking about, or almost.
Randomness cannot be controlled but can be measured
OUR DISCUSSION IS not complete because there is an important difference between the efforts of economists to control outside factors by filtering them out of their results and the control over the laboratory environment that physicists favor. The difference is that there is one effect economists can never filter out. Most people call it chance. Economists also call it randomness, stochastic shocks, or non-systematic influences. It is some force that acts differently on each member of the treatment and control groups, which the researcher cannot observe, and which would continue to act differently were the experiment run again. This is why it is sometimes called a non-systematic influence, as opposed to the systematic influence of factors such as age or education which presumably dispose the test and control subjects to a consistent level of performance.
The problem that chance poses for the economist is that it weakens his or her ability to conclude whether or not a government program was a success. Suppose the program whose effect you are trying to measure really increased the performance of the treatment group above that of the control group, yet by some unlikely quirk people in the control group were feeling good, or alert, or in some fleeting, unrepeatable way were “on”. Then the control group could end up having a better performance than the treatment group simply because luck favored it on that day. The consequence is that despite the logical beauty of filtering out other systematic effects by creating similar groups, economists must sort out whether the difference between groups they observe are due to a government program or due to the non-systematic effects of chance.
Physicists do not face this problem on anything but the smallest of atomic scales. What are economists to do? They have only one choice. They must try to quantify how certain they are that the net difference in performance between groups is not due to chance but rather due to the causal variable of interest, such as foreign aid to a village, or employment subsidies for workers.
Enter applied statistics, one of the least understood and most widely detested subjects imposed on generations of students. That it is so detested is a shame because applied statistics is about nothing more than getting a feel for how certain we are about something. More technically, it tells us the probability something will happen, or perhaps just how much more likely one event is than another.
Knowing how likely it is that something will happen can be of inestimable value. No wonder then that statistics had its origins in the calculations of gamblers who wanted to know the odds of dice, card games, and lotteries. The French philosopher Pascal formalized the insights of gamblers by creating the mathematics of probabilities, also known as the probability calculus. He too was a gambler of sorts, interested in the biggest bet a person could make. Should you devote your life to the worship of the Christian God? Pascal used probabilities to argue it was a rational bet to do so, and lived his life accordingly.
Later mathematicians used Pascal’s probability calculus to show that almost all of applied statistics could be developed from an understanding of one of the simplest games of chance: flipping a coin. The coin flip is known as a “binomial trial”. For a large number of coin flips you can use the mathematical formula for something called the “normal distribution” to approximate the likelihood of a long run of luck with heads. A distribution is simply a device for cataloguing how frequently some event takes place. In a thousand coin flips, the normal distribution can tell you how likely it is that you will see twenty heads, or three hundred heads, or even a thousand heads. Each of these is one event out of many possible events that could emerge from a large number of coin tosses. The normal distribution seems to pop up all over statistics. In addition to predicting binomial events in large numbers of trials it can also be used to predict runs of luck from games about whose odds of different outcomes we know little to nothing.
This discovery was made in the 19th century and came to be known as the Central Limit Theorem. You will not hear about it on science documentarie
s, but it is one of the most astounding intellectual discoveries. It says that if you take an average of the performance of many people, say on a driver’s exam, and so do not really know anything about the likelihood of how any one person will perform, the average performance will have many people who are very close to it and a few others who are further away.
More remarkably, the spread of the average will increasingly conform to a normal distribution as the number of people increases (this is quite different from the “law of large numbers”). So, starting without any knowledge of the chance acting on an individual unit, you can build a coherent picture of the chance acting on large groups of these units.
Hearing through the noise
WHAT DOES THIS have to do with treatment and control groups? Recall that a normal distribution is just a device for telling us how likely an outcome is. So if we see that the treatment group exceeds the control group by fifty per cent on average, we can refer to the normal distribution to see how likely a fifty per cent difference would be if only chance where generating it. If it is highly unlikely that chance alone was the generating force behind the difference, then the economist can conclude, with “statistical confidence” that the government program in question, instead of chance, had some role in the enhanced performance of the treatment group.
If this reasoning seems difficult then rest assured that you have used it all your life. Think of judging whether a coin is honest or not. You already know the statistical distribution behind the flip for an honest coin. It is fifty-fifty for heads and tails. If you flip it twice and see two heads, you cannot really deduce anything is out of order because there is a one in four chance of that happening. But if you flip the coin twenty times and observe twenty heads, then intuitively you know it is highly unlikely that an honest coin would behave in this manner (in fact, the probability calculus can help us find that the chance is slightly more than one in a million). So you would conclude with confidence that the coin has been tampered with.
