by Peter M Lee
1.2.3 A warning
In the case of Connecticut v. Teal [see DeGroot et al. (1986, p. 9)], a case of alleged discrimination on the basis of a test to determine eligibility for promotion was considered. It turned out that of those taking the test 48 were black (B) and 259 were white (W), so that if we consider a random person taking the test
Of the blacks taking the test, 26 passed (P) and the rest failed (F), whereas of the whites, 206 passed and the rest failed, so that altogether 232 people passed. Hence,
There is a temptation to think that these are the figures which indicate the possibility of discrimination. Now there certainly is a case for saying that there was discrimination in this case, but the figures that should be considered are
It is easily checked that the probabilities here are related by Bayes’ Theorem. It is worth while spending a while playing with hypothetical figures to convince yourself that the fact that is less than is irrelevant to the real question as to whether is less than – it might or might not be depending on the rest of the relevant information, that is on and . The fallacy involved arises as the first of two well-known fallacies in criminal law which are both well summarized by Aitken (1996) (see also Aitken and Taroni, 2004, and Dawid, 1994) as follows:
Suppose a crime has been committed. Blood is found at the scene for which there is no innocent explanation. It is of a type which is present in 1% of the population. The prosecutor may then state:
‘There is a 1% chance that the defendant would have the crime blood type if he were innocent. Thus, there is a 99% chance that he is guilty’.
Alternatively, the defender may state:
‘This crime occurred in a city of 800,000 people. This blood type would be found in approximately 8000 people. The evidence has provided a probability of 1 in 8000 that the defendant is guilty and thus has no relevance.’
The first of these is known as the prosecutor’s fallacy or the fallacy of the transposed conditional and, as pointed out above, in essence it consists in quoting the probability instead of . The two are, however, equal if and only if the prior probability happens to equal (E), which will only rarely be the case.
The second is the defender’s fallacy which consists in quoting without regard to . In the case considered by Aitken, the prior odds in favour of guilt are
while the posterior odds are
Such a large change in the odds is, in Aitken’s words ‘surely of relevance’. But, again in Aitken’s words, ‘Of course, it may not be enough to find the suspect guilty’.
As a matter of fact, Bayesian statistical methods are increasingly used in a legal context. Useful references are Balding and Donnelly (1995), Foreman, Smith and Evett (1997), Gastwirth (1988) and Fienberg (1989).
1.3 Random variables
1.3.1 Discrete random variables
As explained in Section 1.1, there is usually a set Ω representing the possibilities consistent with the sum total of data available to the individual or individuals concerned. Now suppose that with each elementary event ω in Ω, there is an integer which may be positive, negative or zero. In the jargon of mathematics, we have a function mapping Ω to the set of all (signed) integers. We refer to the function as a random variable or an r.v.
A case arising in the context of the very first example we discussed, which was about tossing a red die and a blue die, is the integer representing the sum of the spots showing. In this case, ω might be ‘red three, blue two’ and then would be 5. Another case arising in the context of the second (political) example is the Labour majority (represented as a negative integer should the Conservatives happen to win), and here ω might be ‘Labour 350, Conservative 250’ in which case would be 100.
Rather naughtily, probabilists and statisticians tend not to mention the elementary event ω of which is a function and instead just write for . The reason is that what usually matters is the value of rather than the nature of the elementary event ω, the definition of which is in any case dependent on the context, as noted earlier, in the discussion of elementary events. Thus, we write
for the probability that the random variable takes the particular value m. It is a useful convention to use the same lower-case letter and drop the tilde ( ) to denote a particular value of a random variable. An alternative convention used by some statisticians is to use capital letters for random variables and corresponding lower case letters for typical values of these random variables, but in a Bayesian context we have so many quantities that are regarded as random variables that this convention is too restrictive. Even worse than the habit of dropping mention of ω is the tendency to omit the tilde and so use the same notation for a random variable and for a typical value of it. While failure to mention ω rarely causes any confusion, the failure to distinguish between random variables and typical values of these random variables can, on occasion, result in real confusion. When there is any possibility of confusion, the tilde will be used in the text, but otherwise it will be omitted. Also, we will use
for the probability that the random variable takes the value m. When there is only one random variable we are talking about, this abbreviation presents few problems, but when we have a second random variable and write
then ambiguity can result. It is not clear in such a case what p(5) would mean, or indeed what p(i) would mean (unless it refers to where is yet a third random variable). When it is necessary to resolve such an ambiguity, we will use
so that, for example, is the probability that is 5 and is the probability that equals i. Again, all of this seems very much more confusing than it really is – it is usually possible to conduct arguments quite happily in terms of p(m) and p(n) and substitute numerical values at the end if and when necessary.
You could well object that you would prefer a notation that was free of ambiguity, and if you were to do so, I should have a lot of sympathy. But the fact is that constant references to and rather than to m and p(m) would clutter the page and be unhelpful in another way.
We refer to the sequence (p(m)) as the (probability) density (function) or pdf of the random variable m (strictly ). The random variable is said to have a distribution (of probability) and one way of describing a distribution is by its pdf. Another is by its (cumulative) distribution function, or cdf or df, defined by
Because the pdf has the obvious properties
the df is (weakly) increasing, that is
and moreover
1.3.2 The binomial distribution
A simple example of such a distribution is the binomial distribution (see Appendix A). Suppose, we have a sequence of trials each of which, independently of the others, results in success (S) or failure (F), the probability of success being a constant π (such trials are sometimes called Bernoulli trials). Then the probability of any particular sequence of n trials in which k result in success is
so that allowing for the ways in which k successes and n−k failures can be ordered, the probability that a sequence of n trials results in k successes is
If then k (strictly ) is a random variable defined as the number of successes in n trials, then
Such a distribution is said to be binomial of index n and parameter π, and we write
[or strictly ].
We note that it follows immediately from the definition that if x and y are independent and and then .
1.3.3 Continuous random variables
So far, we have restricted ourselves to random variables which take only integer values. These are particular cases of discrete random variables. Other examples of discrete random variables occur, for example, a measurement to the nearest quarter-inch which is subject to a distribution of error, but these can nearly always be changed to integer-valued random variables (in the given example simply by multiplying by 4). More generally, we can suppose that with each elementary event ω in Ω there is a real number . We can define the (cumulative) distribution function, cdf or df of by
As in the discrete case the df is (weakly) increasing, that is
and mor
eover
It is usually the case that when is not discrete there exists a function p(x), or more strictly , such that
in which case p(x) is called a (probability) density (function) or pdf. When this is so, x (strictly ) is said to have a continuous distribution (or more strictly an absolutely continuous distribution). Of course, in the continuous case p(x) is not itself interpretable directly as a probability, but for small
The quantity is sometimes referred to as the probability element. Note that letting this implies that
for every particular value x, in sharp contrast to the discrete case. We can also use the above approximation if y is some one-to-one function of x, for example
Then if values correspond in an obvious way
which on substituting in the above relationship gives in the limit
which is the rule for change of variable in probability densities. (It is not difficult to see that, because the modulus signs are there, the same result is true if F is a strictly decreasing function of x). Another way of getting at this rule is by differentiating the obvious equation
[strictly which holds whenever y and x are corresponding values, that is y=g(x). We should, however, beware that these results need modification if g is not a one-to-one function. In the continuous case, we can find the density from the df by differentiation, namely
Although there are differences, there are many similarities between the discrete and the continuous cases, which we try to emphasize by using the same notation in both cases. We note that
in the discrete case, but
in the continuous case. The discrete case is slightly simpler in one way in that no complications arise over change of variable, so that
if y and x are corresponding values, that is y=g(x).
1.3.4 The normal distribution
The most important example of a continuous distribution is the so-called normal or Gaussian distribution. We say that z has a standard normal distribution if
and when this is so we write
The density of this distribution is the familiar bell-shaped curve, with about two-thirds of the area between –1 and 1, 95% of the area between –2 and 2 and almost all of it between –3 and 3. Its distribution function is
More generally, we say that x has a normal distribution, denoted
if
where z is as aforementioned, or equivalently if
The normal distribution is encountered almost at every turn in statistics. Partly this is because (despite the fact that its density may seem somewhat barbaric at first sight) it is in many contexts the easiest distribution to work with, but this is not the whole story. The Central Limit Theorem says (roughly) that if a random variable can be expressed as a sum of a large number of components no one of which is likely to be much bigger than the others, these components being approximately independent, then this sum will be approximately normally distributed. Because of this theorem, an observation which has an error contributed to by many minor causes is likely to be normally distributed. Similar reasons can be found for thinking that in many circumstances we would expect observations to be approximately normally distributed, and this turns out to be the case, although there are exceptions. This is especially useful in cases where we want to make inferences about a population mean.
1.3.5 Mixed random variables
While most commonly occurring random variables are discrete or continuous, there are exceptions, for example the time you have to wait until you are served in a queue, which is zero with a positive probability (if the queue is empty when you arrive), but otherwise is spread over a continuous range of values. Such a random variable is said to have a mixed distribution.
1.4 Several random variables
1.4.1 Two discrete random variables
Suppose that with each elementary event ω in Ω, we can associate a pair of integers . We write
Strictly speaking, p(m, n) should be written as for reasons discussed earlier, but this degree of pedantry in the notation is rarely necessary. Clearly
The sequence (p(m, n)) is said to be a bivariate (probability) density (function) or bivariate pdf and is called the joint pdf of the random variables m and n (strictly and ). The corresponding joint distribution function, joint cdf or joint df is
Clearly, the density of m (called its marginal density) is
We can also define a conditional distribution for n given m (strictly for given ) by allowing
to define the conditional (probability) density (function) or conditional pdf. This represents our judgement as to the chance that takes the value n given that is known to have the value m. If it is necessary to make our notation absolutely precise, we can always write
so, for example, is the probability that m is 4 given is 3, but is the probability that is 4 given that takes the value 3, but it should be emphasized that we will not often need to use the subscripts. Evidently
and
We can also define a conditional distribution function or conditional df by
1.4.2 Two continuous random variables
As in Section 1.4, we have begun by restricting ourselves to integer values, which is more or less enough to deal with any discrete cases that arise. More generally, we can suppose that with each elementary event ω in Ω, we can associate a pair of real numbers. In this case, we define the joint distribution function or joint df as
Clearly the df of x is
and that of y is
It is usually the case that when neither x nor y is discrete there is a function p(x, y) such that
in which case p(x, y) is called a joint (probability) density (function) or joint pdf. When this is so, the joint distribution is said to be continuous (or more strictly to be absolutely continuous). We can find the density from the df by
Clearly,
and
The last formula is the continuous analogue of
in the discrete case.
By analogy with the discrete case, we define the conditional density of y given x (strictly of given ) as
provided . We can then define the conditional distribution function by
There are difficulties in the notion of conditioning on the event that because this event has probability zero for every x in the continuous case, and it can help to regard the above distribution as the limit of the distribution which results from conditioning on the event that is between x and , that is
as .
1.4.3 Bayes’ Theorem for random variables
It is worth noting that conditioning the random variable y by the value of x does not change the relative sizes of the probabilities of those pairs (x, y) that can still occur. That is to say, the probability p(y|x) is proportional to p(x, y) and the constant of proportionality is just what is needed, so that the conditional probabilities integrate to unity. Thus,
Moreover,
It is clear that
so that
This is, of course, a form of Bayes’ Theorem, and is in fact the commonest way in which it occurs in this book. Note that it applies equally well if the variables x and y are continuous or if they are discrete. The constant of proportionality is
in the continuous case or
in the discrete case.
1.4.4 Example
A somewhat artificial example of the use of this formula in the continuous case is as follows. Suppose y is the time before the first occurrence of a radioactive decay which is measured by an instrument, but that, because there is a delay built into the mechanism, the decay is recorded as having taken place at a time x> y. We actually have the value of x, but would like to say what we can about the value of y on the basis of this knowledge. We might, for example, have
Then
Often we will find that it is enough to get a result up to a constant of proportionality, but if we need the constant, it is very easy to find it because we know that the integral (or the sum in the discrete case) must be one. Thus, in this case
1.4.5 One discrete variable and one continuous variable
We also encounter cases where we have two random variables, one of which is continuous and one of which is discrete. All the aforementioned definitions and formulae extend in an obvious way to such a case provided we are careful, for example, to use integration for continuous variables but summation for discrete variables. In particular, the formulation
for Bayes’ Theorem is valid in such a case.
It may help to consider an example (again a somewhat artificial one). Suppose k is the number of successes in n Bernoulli trials, so , but that the value of π is unknown, your beliefs about it being uniformly distributed over the interval [0, 1] of possible values. Then
