CHAPTER 25

Mathematical Statistics

In probability theory we set up mathematical models of processes that are affected by “chance.” In mathematical statistics or, briefly, statistics, we check these models against the observable reality. This is called statistical inference. It is done by sampling, that is, by drawing random samples, briefly called samples. These are sets of values from a much larger set of values that could be studied, called the population. An example is 10 diameters of screws drawn from a large lot of screws. Sampling is done in order to see whether a model of the population is accurate enough for practical purposes. If this is the case, the model can be used for predictions, decisions, and actions, for instance, in planning productions, buying equipment, investing in business projects, and so on.

Most important methods of statistical inference are estimation of parameters (Secs. 25.2), determination of confidence intervals (Sec. 25.3), and hypothesis testing (Sec. 25.4, 25.7, 25.8), with application to quality control (Sec. 25.5) and acceptance sampling (Sec. 25.6).

In the last section (25.9) we give an introduction to regression and correlation analysis, which concern experiments involving two variables.

Prerequisite: Chap. 24.

Sections that may be omitted in a shorter course: 25.5, 25.6, 25.8.

References, Answers to Problems, and Statistical Tables: App. 1 Part G, App. 2, App. 5.

25.1 Introduction. Random Sampling

Mathematical statistics consists of methods for designing and evaluating random experiments to obtain information about practical problems, such as exploring the relation between iron content and density of iron ore, the quality of raw material or manufactured products, the efficiency of air-conditioning systems, the performance of certain cars, the effect of advertising, the reactions of consumers to a new product, etc.

Random variables occur more frequently in engineering (and elsewhere) than one would think. For example, properties of mass-produced articles (screws, lightbulbs, etc.) always show random variation, due to small (uncontrollable!) differences in raw material or manufacturing processes. Thus the diameter of screws is a random variable X and we have nondefective screws, with diameter between given tolerance limits, and defective screws, with diameter outside those limits. We can ask for the distribution of X, for the percentage of defective screws to be expected, and for necessary improvements of the production process.

Samples are selected from populations—20 screws from a lot of of 1000, 100 of 5000 voters, 8 beavers in a wildlife conservation project—because inspecting the entire population would be too expensive, time-consuming, impossible or even senseless (think of destructive testing of lightbulbs or dynamite). To obtain meaningful conclusions, samples must be random selections. Each of the 1000 screws must have the same chance of being sampled (of being drawn when we sample), at least approximately. Only then will the sample mean (Sec. 24.1) of a sample of size n = 20 (or any other n) be a good approximation of the population mean μ (Sec. 24.6); and the accuracy of the approximation will generally improve with increasing n, as we shall see. Similarly for other parameters (standard deviation, variance, etc.).

Independent sample values will be obtained in experiments with an infinite sample space S (Sec. 24.2), certainly for the normal distribution. This is also true in sampling with replacement. It is approximately true in drawing small samples from a large finite population (for instance, 5 or 10 of 1000 items). However, if we sample without replacement from a small population, the effect of dependence of sample values may be considerable.

Random numbers help in obtaining samples that are in fact random selections. This is sometimes not easy to accomplish because there are many subtle factors that can bias sampling (by personal interviews, by poorly working machines, by the choice of nontypical observation conditions, etc.). Random numbers can be obtained from a random number generator in Maple, Mathematica, or other systems listed on p. 789. (The numbers are not truly random, as they would be produced in flipping coins or rolling dice, but are calculated by a tricky formula that produces numbers that do have practically all the essential features of true randomness. Because these numbers eventually repeat, they must not be used in cryptography, for example, where true randomness is required.)

Representing and processing data were considered in Sec. 24.1 in connection with frequency distributions. These are the empirical counterparts of probability distributions and helped motivating axioms and properties in probability theory. The new aspect in this chapter is randomness: the data are samples selected randomly from a population. Accordingly, we can immediately make the connection to Sec. 24.1, using stem-and-leaf plots, box plots, and histograms for representing samples graphically.

Also, we now call the mean in (5), Sec. 24.1, the sample mean

We call n the sample size, the variance s² in (6), Sec. 24.1, the sample variance

and its positive square root s the sample standard deviation. , s², and s are called parametersof a sample; they will be needed throughout this chapter.

25.2 Point Estimation of Parameters

Beginning in this section, we shall discuss the most basic practical tasks in statistics and corresponding statistical methods to accomplish them. The first of them is point estimation of parameters, that is, of quantities appearing in distributions, such as p in the binomial distribution μ and σ and in the normal distribution.

A point estimate of a parameter is a number (point on the real line), which is computed from a given sample and serves as an approximation of the unknown exact value of the parameter of the population. An interval estimate is an interval (“confidence interval”) obtained from a sample; such estimates will be considered in the next section. Estimation of parameters is of great practical importance in many applications.

As an approximation of the mean μ of a population we may take the mean of a corresponding sample. This gives the estimate for μ, that is,

where n is the sample size. Similarly, an estimate for the variance of a population is the variance s² of a corresponding sample, that is,

Clearly, (1) and (2) are estimates of parameters for distributions in which μ or σ² appear explicity as parameters, such as the normal and Poisson distributions. For the binomial distribution, p = μ/n [see (3) in Sec. 24.7]. From (1) we thus obtain for p the estimate

We mention that (1) is a special case of the so-called method of moments. In this method the parameters to be estimated are expressed in terms of the moments of the distribution (see Sec. 24.6). In the resulting formulas, those moments of the distribution are replaced by the corresponding moments of the sample. This gives the estimates. Here the k th moment of a sample x₁, …, x_n is

Maximum Likelihood Method

Another method for obtaining estimates is the so-called maximum likelihood method of R. A. Fisher [Messenger Math. 41 (1912), 155–160]. To explain it, we consider a discrete (or continuous) random variable X whose probability function (or density) f(x) depends on a single parameter θ. We take a corresponding sample of n independent values x₁, …,x_n. Then in the discrete case the probability that a sample of size n consists precisely of those n values is

In the continuous case the probability that the sample consists of values in the small intervals x_j x x_j + Δx (j = 1, 2, …, n) is

Since f(x_j) depends on θ, the function l in (5) given by (4) depends on x₁, …, x_n and θ. We imagine x₁, …, x_n to be given and fixed. Then l is a function of which is called the likelihood function. The basic idea of the maximum likelihood method is quite simple, as follows. We choose that approximation for the unknown value of θ for which l is as large as possible. If l is a differentiable function of θ, a necessary condition for l to have a maximum in an interval (not at the boundary) is

(We write a partial derivative, because l depends also on x₁, …, x_n.) A solution of (6) depending on x₁, …, x_n is called a maximum likelihood estimate for θ. We may replace (6) by

because f(x_j) > 0, a maximum of l is in general positive, and ln l is a monotone increasing function of l. This often simplifies calculations.

Several Parameters. If the distribution of X involves r parameters θ₁, …, θ_r, then instead of (6) we have the r conditions ∂l/∂θ₁ = 0, …, ∂l/∂θ_r = 0, and instead of (7) we have

25.3 Confidence Intervals

Confidence intervals¹ for an unknown parameter θ of some distribution (e.g., θ = μ) are intervals θ₁ θ θ₂ that contain θ, not with certainty but with a high probability γ, which we can choose (95% and 99% are popular). Such an interval is calculated from a sample. γ = 95% means probability of being wrong—one of about 20 such intervals will not contain θ. Instead of writing θ₁ θ θ₂, we denote this more distinctly by writing

Such a special symbol, CONF, seems worthwhile in order to avoid the misunderstanding that θ must lie between θ₁ and θ₂.

γ is called the confidence level, and θ₁ and θ₂ are called the lower and upper confidence limits. They depend on γ. The larger we choose γ, the smaller is the error probability 1 − γ, but the longer is the confidence interval. If γ → 1, then its length goes to infinity. The choice of γ depends on the kind of application. In taking no umbrella, a 5% chance of getting wet is not tragic. In a medical decision of life or death, a 5% chance of being wrong may be too large and a 1% chance of being wrong (γ = 99%) may be more desirable.

Confidence intervals are more valuable than point estimates (Sec. 25.2). Indeed, we can take the midpoint of (1) as an approximation of θ and half the length of (1) as an “error bound” (not in the strict sense of numerics, but except for an error whose probability we know).

θ₁ and θ₂ in (1) are calculated from a sample x₁, …, x_n. These are n observations of a random variable X. Now comes a standard trick. We regard x₁, …, x_n as single observations of n random variables X₁, …, X_n (with the same distribution, namely, that of X). Then θ₁ = θ₁(x₁, …, x_n) and θ₂ = θ₂(x₁, …, x_n) in (1) are observed values of two random variables Θ₁ = Θ₁(X₁, …, X_n) and Θ₂ = Θ₂(X₁, …, X_n). The condition (1) involving γ can now be written

Let us see what all this means in concrete practical cases.

In each case in this section we shall first state the steps of obtaining a confidence interval in the form of a table, then consider a typical example, and finally justify those steps theoretically.

Confidence Interval for of the Normal Distribution with Known σ²

Table 25.1 Determination of a Confidence Interval for the Mean μ of a Normal Distribution with Known Variance σ²

Theory for Table 25.1. The method in Table 25.1 follows from the basic

Derivation of (3) in Table 25.1. Sampling from a normal distribution gives independent sample values (see Sec. 25.1), so that Theorem 1 applies. Hence we can choose γ and then determine c such that

For the value γ = 0.95 we obtain z(D) = 1.960 from Table A8 in App. 5, as used in Example 1. For γ = 0.9, 0.99, 0.999 we get the other values of c listed in Table 25.1. Finally, all we have to do is to convert the inequality in (6) into one for μ and insert observed values obtained from the sample. We multiply −c Z c by −1 and then by , writing (as in Table 25.1),

Adding gives or

Inserting the observed value of gives (3). Here we have regarded x₁, …, x_n as single observations of X₁, …, X_n (the standard trick!), so that x₁ + … + x_n is an observed value of X₁ + … + X_n and is an observed value of . Note further that (7) is of the form (2) with and .

Fig. 526. Length of the confidence interval (3) (measured in multiples of σ as a function of the sample size n for γ = 95% and γ = 99%

Confidence Interval for μ of the Normal Distribution with Unknown σ²

In practice σ² is frequently unknown. Then the method in Table 25.1 does not help and the whole theory changes, although the steps of determining a confidence interval for μ remain quite similar. They are shown in Table 25.2. We see that k differs from that in Table 25.1, namely, the sample standard deviation s has taken the place of the unknown standard deviation σ of the population. And c now depends on the sample size n and must be determined from Table A9 in App. 5 or from your CAS. That table lists values z for given values of the distribution function (Fig. 527)

of the t-distribution. Here, m (= 1, 2, …) is a parameter, called the number of degrees of freedom of the distribution (abbreviated d.f.). In the present case, m = n − 1; see Table 25.2. The constant K_m is such that F(∞) = 1. By integration it turns out that , where Γ is the gamma function (see (24) in App. A3.1).

Table 25.2 Determination of a Confidence Interval for the Mean μ of a Normal Distribution with Unknown Variance σ²

Figure 528 compares the curve of the density of the t-distribution with that of the normal distribution. The latter is steeper. This illustrates that Table 25.1 (which uses more information, namely, the known value of σ²) yields shorter confidence intervals than Table 25.2. This is confirmed in Fig. 529, which also gives an idea of the gain by increasing the sample size.

Fig. 527. Distribution functions of the t-distribution with 1 and 3 d.f. and of the standardized normal distribution (steepest curve)

Fig. 528. Densities of the t-distribution with 1 and 3 d.f. and of the standardized normal distribution

Fig. 529. Ratio of the lengths L′ and L of the confidence intervals (10) and (3) with γ = 95% and γ = 99% as a function of the sample size n for equal s and σ

Theory for Table 25.2. For deriving (10) in Table 25.2 we need from Ref. [G3]

Derivation of (10). This is similar to the derivation of (3). We choose a number γ between 0 and 1 and determine a number c from Table A9 in App. 5 with n − 1 d.f. (or from a CAS) such that

Since the t-distribution is symmetric, we have

and (13) assumes the form (9). Substituting (11) into (13) and transforming the result as before, we obtain

where

By inserting the observed values of and s² of S² into (14) we finally obtain (10).

Confidence Interval for the Variance σ²
of the Normal Distribution

Table 25.3 shows the steps, which are similar to those in Tables 25.1 and 25.2.

Table 25.3 Determination of a Confidence Interval for the Variance σ² of a Normal Distribution, Whose Mean Need Not Be Known

Theory for Table 25.3. In Table 25.1 we used the normal distribution, in Table 25.2 the t-distribution, and now we shall use the x²-distribution (chi-square distribution), whose distribution function is F(z) = 0 if z < 0 and

The parameter m (= 1, 2, …) is called the number of degrees of freedom (d.f.), and

Note that the distribution is not symmetric (see also Fig. 531).

For deriving (16) in Table 25.3 we need the following theorem.

Fig. 530. Distribution function of the chi-square distribution with 2, 3, 5 d.f.

Proof in Ref. [G3], listed in App. 1.

Fig. 531. Density of the chi-square distribution with 2, 3, 5 d.f.

Derivation of (16). This is similar to the derivation of (3) and (10). We choose a number γ between 0 and 1 and determine c₁ and c₂ from Table A10, App. 5, such that [see (15)]

Subtraction yields

Transforming c₁ Y c₂ with Y given by (17) into an inequality for σ², we obtain

By inserting the observed value s² of S² we obtain (16).

Confidence Intervals for Parameters
of Other Distributions

The methods in Tables 25.1–25.3 for confidence intervals for μ and σ² are designed for the normal distribution. We now show that they can also be applied to other distributions if we use large samples.

We know that if X₁, …, X_n are independent random variables with the same mean μ and the same variance σ², then their sum Y_n = X₁ + … + X_n has the following properties.

(A) Y_n has the mean nμ and the variance nσ² (by Theorems 1 and 3 in Sec. 24.9).

(B) If those variables are normal, then Y_n is normal (by Theorem 1).

If those random variables are not normal, then (B) is not applicable. However, for large n the random variable Y_n is still approximately normal. This follows from the central limit theorem, which is one of the most fundamental results in probability theory.

A proof can be found in Ref. [G3] listed in App. 1.

Hence, when applying Tables 25.1–25.3 to a nonnormal distribution, we must use sufficiently large samples. As a rule of thumb, if the sample indicates that the skewness of the distribution (the asymmetry; see Team Project 20(d), Problem Set 24.6) is small, use at least n = 20 for the mean and at least n = 50 for the variance.

25.4 Testing of Hypotheses. Decisions

The ideas of confidence intervals and of tests² are the two most important ideas in modern statistics. In a statistical test we make inference from sample to population through testing a hypothesis, resulting from experience or observations, from a theory or a quality requirement, and so on. In many cases the result of a test is used as a basis for a decision, for instance, to buy (or not to buy) a certain model of car, depending on a test of the fuel efficiency (and other tests, of course), to apply some medication, depending on a test of its effect; to proceed with a marketing strategy, depending on a test of consumer reactions, etc.

Let us explain such a test in terms of a typical example and introduce the corresponding standard notions of statistical testing.

Fig. 532. t-distribution in Example 1

This example illustrates the steps of a test:

1. Formulate the hypothesis θ = θ₀ to be tested. (θ₀ = μ₀ in the example.)
2. Formulate an alternative θ = θ₁. (θ₁ = μ₁ in the example.)
3. Choose a significance level α (5%, 1%, 0.1%).

4. Use a random variable whose distribution depends on the hypothesis and on the alternative, and this distribution is known in both cases. Determine a critical value c from the distribution of , assuming the hypothesis to be true. (In the example, and c is, obtained from P(T c) = α.)

5. Use a sample x₁, …, x_n to determine an observed value of . (t in the example.)

6. Accept or reject the hypothesis, depending on the size of relative to c. (t < c in the example, rejection of the hypothesis.)

Two important facts require further discussion and careful attention. The first is the choice of an alternative. In the example, μ₁ < μ₀, but other applications may require μ₁ > μ₀ or μ₁ ≠ μ₀. The second fact has to do with errors. We know that α (the significance level of the test) is the probability of rejecting a true hypothesis. And we shall discuss the probability β of accepting a false hypothesis.

One-Sided and Two-Sided Alternatives (Fig. 533)

Let θ be an unknown parameter in a distribution, and suppose that we want to test the hypothesis θ = θ₀. Then there are three main kinds of alternatives, namely,

(1) and (2) are one-sided alternatives, and (3) is a two-sided alternative.

We call rejection region (or critical region) the region such that we reject the hypothesis if the observed value in the test falls in this region. In the critical c lies to the right of θ₀ because so does the alternative. Hence the rejection region extends to the right. This is called a right-sided test. In the critical c lies to the left of θ₀ (as in Example 1), the rejection region extends to the left, and we have a left-sided test (Fig. 533, middle part). These are one-sided tests. In we have two rejection regions. This is called a two-sided test (Fig. 533, lower part).

Fig. 533. Test in the case of alternative (1) (upper part of the figure), alternative (2) (middle part), and alternative (3)

All three kinds of alternatives occur in practical problems. For example, (1) may arise if θ₀ is the maximum tolerable inaccuracy of a voltmeter or some other instrument. Alternative (2) may occur in testing strength of material, as in Example 1. Finally, θ₀ in (3) may be the diameter of axle-shafts, and shafts that are too thin or too thick are equally undesirable, so that we have to watch for deviations in both directions.

Errors in Tests

Tests always involve risks of making false decisions:

1. Rejecting a true hypothesis (Type I error).

   α = Probability of making a Type I error.

2. Accepting a false hypothesis (Type II error).

   β = Probability of making a Type II error.

Clearly, we cannot avoid these errors because no absolutely certain conclusions about populations can be drawn from samples. But we show that there are ways and means of choosing suitable levels of risks, that is, of values α and β. The choice of α depends on the nature of the problem (e.g., a small risk α = 1% is used if it is a matter of life or death).

Let us discuss this systematically for a test of a hypothesis θ = θ₀ against an alternative that is a single number θ₁, for simplicity. We let θ₁ > θ₀, so that we have a right-sided test. For a left-sided or a two-sided test the discussion is quite similar.

We choose a critical c > θ₀ (as in the upper part of Fig. 533, by methods discussed below). From a given sample x₁, …, x_n we then compute a value

with a suitable g (whose choice will be a main point of our further discussion; for instance, take g = (x₁ + … + x_n)/n in the case in which θ is the mean). If , we reject the hypothesis. If , we accept it. Here, the value can be regarded as an observed value of the random variable

because x_j may be regarded as an observed value of X_j,j = 1, …, n. In this test there are two possibilities of making an error, as follows.

Type I Error (see Table 25.4). The hypothesis is true but is rejected (hence the alternative is accepted) because Θ assumes a value . Obviously, the probability of making such an error equals

α is called the significance level of the test, as mentioned before.

Type II Error (see Table 25.4). The hypothesis is false but is accepted because assumes a value . The probability of making such an error is denoted by β thus

η = 1 − β is called the power of the test. Obviously, the power η is the probability of avoiding a Type II error.

Table 25.4 Type I and Type II Errors in Testing a Hypothesis θ = θ₀ Against an Alternative θ = θ₁

Formulas (5) and (6) show that both α and β depend on c, and we would like to choose c so that these probabilities of making errors are as small as possible. But the important Figure 534 shows that these are conflicting requirements because to let α decrease we must shift c to the right, but then β increases. In practice we first choose α (5%, sometimes 1%), then determine c, and finally compute β. If β is large so that the power η = 1 − β is small, we should repeat the test, choosing a larger sample, for reasons that will appear shortly.

Fig. 534. Illustration of Type I and II errors in testing a hypothesis θ = θ₀ against an alternative θ = θ₁ (> θ₀, right-sided test)

If the alternative is not a single number but is of the form (1)–(3), then β becomes a function of θ. This function β(θ) is called the operating characteristic (OC) of the test and its curve the OC curve. Clearly, in this case η = 1 − β also depends on θ. This function η(θ) is called the power function of the test. (Examples will follow.)

Of course, from a test that leads to the acceptance of a certain hypothesis θ₀, it does not follow that this is the only possible hypothesis or the best possible hypothesis. Hence the terms “not reject” or “fail to reject” are perhaps better than the term “accept.”

Test for μ of the Normal Distribution with Known σ²

The following example explains the three kinds of hypotheses.

Fig. 536. Curves of the operating characteristic (OC curves) in Example 2, case (c), for two different sample sizes n

Test for μ When σ² Is Unknown, and for σ²

Comparison of Means and Variances

25.5 Quality Control

The ideas on testing can be adapted and extended in various ways to serve basic practical needs in engineering and other fields. We show this in the remaining sections for some of the most important tasks solvable by statistical methods. As a first such area of problems, we discuss industrial quality control, a highly successful method used in various industries.

No production process is so perfect that all the products are completely alike. There is always a small variation that is caused by a great number of small, uncontrollable factors and must therefore be regarded as a chance variation. It is important to make sure that the products have required values (for example, length, strength, or whatever property may be essential in a particular case). For this purpose one makes a test of the hypothesis that the products have the required property, say, μ = μ₀, where μ₀ is a required value. If this is done after an entire lot has been produced (for example, a lot of 100,000 screws), the test will tell us how good or how bad the products are, but it it obviously too late to alter undesirable results. It is much better to test during the production run. This is done at regular intervals of time (for example, every hour or half-hour) and is called quality control. Each time a sample of the same size is taken, in practice 3 to 10 times. If the hypothesis is rejected, we stop the production and look for the cause of the trouble.

If we stop the production process even though it is progressing properly, we make a Type I error. If we do not stop the process even though something is not in order, we make a Type II error (see Sec. 25.4). The result of each test is marked in graphical form on what is called a control chart. This was proposed by W. A. Shewhart in 1924 and makes quality control particularly effective.

Control Chart for the Mean

An illustration and example of a control chart is given in the upper part of Fig. 537. This control chart for the mean shows the lower control limit LCL, the center control line CL, and the upper control limit UCL. The two control limits correspond to the critical values c₁ and c₂ in case (c) of Example 2 in Sec. 25.4. As soon as a sample mean falls outside the range between the control limits, we reject the hypothesis and assert that the production process is “out of control”; that is, we assert that there has been a shift in process level. Action is called for whenever a point exceeds the limits.

Fig. 537. Control charts for the mean (upper part of figure) and the standard deviation in the case of the samples on p. 1089

If we choose control limits that are too loose, we shall not detect process shifts. On the other hand, if we choose control limits that are too tight, we shall be unable to run the process because of frequent searches for nonexistent trouble. The usual significance level is α = 1%. From Theorem 1 in Sec. 25.3 and Table A8 in App. 5 we see that in the case of the normal distribution the corresponding control limits for the mean are

Here σ is assumed to be known. If σ is unknown, we may compute the standard deviations of the first 20 or 30 samples and take their arithmetic mean as an approximation of σ. The broken line connecting the means in Fig. 537 is merely to display the results.

Additional, more subtle controls are often used in industry. For instance, one observes the motions of the sample means above and below the centerline, which should happen frequently. Accordingly, long runs (conventionally of length 7 or more) of means all above (or all below) the centerline could indicate trouble.

Table 25.5 Twelve Samples of Five Values Each (Diameter of Small Cylinders, Measured in Millimeters)

Control Chart for the Variance

In addition to the mean, one often controls the variance, the standard deviation, or the range. To set up a control chart for the variance in the case of a normal distribution, we may employ the method in Example 4 of Sec. 25.4 for determining control limits. It is customary to use only one control limit, namely, an upper control limit. Now from Example 4 of Sec. 25.4 we have , where, because of our normality assumption, the random variable Y has a chi-square distribution with n − 1 degrees of freedom. Hence the desired control limit is

where c is obtained from the equation

and the table of the chi-square distribution (Table A10 in App. 5) with n − 1 degrees of freedom (or from your CAS); here α (5% or 1% say) is the probability that in a properly running process an observed value s² of S² is greater than the upper control limit.

If we wanted a control chart for the variance with both an upper control limit UCL and a lower control limit LCL, these limits would be

where c₁ and c₂ are obtained from Table A10 with n − 1 d.f. and the equations

Control Chart for the Standard Deviation

To set up a control chart for the standard deviation, we need an upper control limit

obtained from (2). For example, in Table 25.5 we have n = 5. Assuming that the corresponding population is normal with standard deviation σ = 0.02 and choosing α = 1% we obtain from the equation

and Table A10 in App. 5 with 4 degrees of freedom the critical value c = 13.28 and from (5) the corresponding value

which is shown in the lower part of Fig. 537.

A control chart for the standard deviation with both an upper and a lower control limit is obtained from (3).

Control Chart for the Range

Instead of the variance or standard deviation, one often controls the range R (= largest sample value minus smallest sample value). It can be shown that in the case of the normal distribution, the standard deviation σ is proportional to the expectation of the random variable R* for which R is an observed value, say, σ = λ_nE(R*) where the factor of proportionality λ_n depends on the sample size n and has the values

Since R depends on two sample values only, it gives less information about a sample than s does. Clearly, the larger the sample size n is, the more information we lose in using R instead of s. A practical rule is to use s when n is larger than 10.

25.6 Acceptance Sampling

Acceptance sampling is usually done when products leave the factory (or in some cases even within the factory). The standard situation in acceptance sampling is that a producer supplies to a consumer (a buyer or wholesaler) a lot of N items (a carton of screws, for instance). The decision to accept or reject the lot is made by determining the number x of defectives (= defective items) in a sample of size n from the lot. The lot is accepted if x c, where c is called the acceptance number, giving the allowable number of defectives. If x > c, the consumer rejects the lot. Clearly, producer and consumer must agree on a certain sampling plan giving n and c.

From the hypergeometric distribution we see that the event A: “Accept the lot” has probability (see Sec. 24.7)

where M is the number of defectives in a lot of N items. In terms of the fraction defective θ = M/N we can write (1) as

P(A; θ) can assume n + 1 values corresponding to θ = 0, 1/N, 2/N, …, N/N; here, n and c are fixed. A monotone smooth curve through these points is called the operating characteristic curve (OC curve) of the sampling plan considered.

Fig. 538. OC curve of the sampling plan with n = 2 and c = 0 for lots of size N = 20

Fig. 539. OC curve in Example 2

In most practical cases θ will be small (less than 10%). Then if we take small samples compared to N, we can approximate (2) by the Poisson distribution (Sec. 24.7); thus

Errors in Acceptance Sampling

We show how acceptance sampling fits into general test theory (Sec. 25.4) and what this means from a practical point of view. The producer wants the probability α of rejecting

Fig. 540. OC curve, producer's and consumer's risks

an acceptable lot (a lot for which θ does not exceed a certain number θ₀ on which the two parties agree) to be small. θ₀ is called the acceptable quality level (AQL). Similarly, the consumer (the buyer) wants the probability β of accepting an unacceptable lot (a lot for which θ is greater than or equal to some θ₁) to be small. θ₁ is called the lot tolerance percent defective (LTPD) or the rejectable quality level (RQL). α is called producer's risk. It corresponds to a Type I error in Sec. 25.4. β is called consumer's risk and corresponds to a Type II error. Figure 540 shows an example. We see that the points (θ₀, 1 − α) and (θ₁, β) lie on the OC curve. It can be shown that for large lots we can choose θ₀, θ₁ (> θ₀), α, β and then determine n and c such that the OC curve runs very close to those prescribed points. Table 25.6 shows the analogy between acceptance sampling and hypothesis testing in Sec. 25.4.

Table 25.6 Acceptance Sampling and Hypothesis Testing

Acceptance Sampling	Hypothesis Testing
Acceptable quality level (AQL) θ = θ0	Hypothesis θ = θ0
Lot tolerance percent defectives (LTPD) θ = θ1	Alternative θ = θ1
Allowable number of defectives c	Critical value c
Producer's risk α of rejecting a lot with θ θ0	Probability α of making a Type I error with (significance level)
Consumer's risk β of accepting a lot with θ θ1	Probability β of making a Type II error

Rectification

Rectification of a rejected lot means that the lot is inspected item by item and all defectives are removed and replaced by nondefective items. (This may be too expensive if the lot is cheap; in this case the lot may be sold at a cut-rate price or scrapped.) If a production turns out 100θ% defectives, then in K lots of size N each, KNθ of the KN items are defectives. Now KP(A; θ) of these lots are accepted. These contain KPNθ defectives, whereas the rejected and rectified lots contain no defectives, because of the rectification. Hence after the rectification the fraction defective in all K lots equals KPNθ/KN. This is called the average outgoing quality (AOQ); thus

Figure 541 shows an example. Since AOQ(0) = 0 and P(A; 1) = 0, the AOQ curve has a maximum at some θ = θ*, giving the average outgoing quality limit (AOQL). This is the worst average quality that may be expected to be accepted under rectification.

Fig. 541. OC curve and AOQ curve for the sampling plan in Fig. 538

25.7 Goodness of Fit. X²-Test

To test for goodness of fit means that we wish to test that a certain function F(x) is the distribution function of a distribution from which we have a sample x₁, …, x_n. Then we test whether the sample distribution function defined by

fits F(x) “sufficiently well.” If this is so, we shall accept the hypothesis that F(x) is the distribution function of the population; if not, we shall reject the hypothesis.

This test is of considerable practical importance, and it differs in character from the tests for parameters μ, σ², etc.) considered so far.

To test in that fashion, we have to know how much can differ from F(x) if the hypothesis is true. Hence we must first introduce a quantity that measures the deviation of from F(x), and we must know the probability distribution of this quantity under the assumption that the hypothesis is true. Then we proceed as follows. We determine a number c such that, if the hypothesis is true, a deviation greater than c has a small preassigned probability. If, nevertheless, a deviation greater than c occurs, we have reason to doubt that the hypothesis is true and we reject it. On the other hand, if the deviation does not exceed c, so that approximates F(x) sufficiently well, we accept the hypothesis. Of course, if we accept the hypothesis, this means that we have insufficient evidence to reject it, and this does not exclude the possibility that there are other functions that would not be rejected in the test. In this respect the situation is quite similar to that in Sec. 25.4.

Table 25.7 shows a test of that type, which was introduced by R. A. Fisher. This test is justified by the fact that if the hypothesis is true, then is an observed value of a random variable whose distribution function approaches that of the chi-square distribution with K − 1 degrees of freedom (or K − r − 1 degrees of freedom if r parameters are estimated) as n approaches infinity. The requirement that at least five sample values lie in each interval in Table 25.7 results from the fact that for finite n that random variable has only approximately a chi-square distribution. A proof can be found in Ref. [G3] listed in App. 1. If the sample is so small that the requirement cannot be satisfied, one may continue with the test, but then use the result with caution.

Table 25.7 Chi-square Test for the Hypothesis ThatF(x) is the Distribution Function of a Population from Which a Sample x₁, …, x_n is Taken

Table 25.8 Sample of 100 Values of the Splitting Tensile Strength (lb/in.²) of Concrete Cylinders

Fig. 542. Frequency histogram of the sample in Table 25.8

Table 25.10 Computations in Example 1

25.8 Nonparametric Tests

Nonparametric tests, also called distribution-free tests, are valid for any distribution. Hence they are used in cases when the kind of distribution is unknown, or is known but such that no tests specifically designed for it are available. In this section we shall explain the basic idea of these tests, which are based on “order statistics” and are rather simple. If there is a choice, then tests designed for a specific distribution generally give better results than do nonparametric tests. For instance, this applies to the tests in Sec. 25.4 for the normal distribution.

We shall discuss two tests in terms of typical examples. In deriving the distributions used in the test, it is essential that the distributions, from which we sample, are continuous. (Nonparametric tests can also be derived for discrete distributions, but this is slightly more complicated.)

25.9 Regression. Fitting Straight Lines. Correlation

So far we were concerned with random experiments in which we observed a single quantity (random variable) and got samples whose values were single numbers. In this section we discuss experiments in which we observe or measure two quantities simultaneously, so that we get samples of pairs of values (x₁, y₁), (x₂, y₂), …, (x_n, y_n). Most applications involve one of two kinds of experiments, as follows.

1. In regression analysis one of the two variables, call it x, can be regarded as an ordinary variable because we can measure it without substantial error or we can even give it values we want. x is called the independent variable, or sometimes the controlled variable because we can control it (set it at values we choose). The other variable, Y, is a random variable, and we are interested in the dependence of Y on x. Typical examples are the dependence of the blood pressure Y on the age x of a person or, as we shall now say, the regression of Y on x, the regression of the gain of weight Y of certain animals on the daily ration of food x, the regression of the heat conductivity Y of cork on the specific weight x of the cork, etc.
2. In correlation analysis both quantities are random variables and we are interested in relations between them. Examples are the relation (one says “correlation”) between wear X and wear Y of the front tires of cars, between grades X and Y of students in mathematics and in physics, respectively, between the hardness X of steel plates in the center and the hardness Y near the edges of the plates, etc.

Regression Analysis

In regression analysis the dependence of Y on x is a dependence of the mean μ of Y on x, so that μ = μ(x) is a function in the ordinary sense. The curve of μ(x) is called the regression curve of Y on x.

In this section we discuss the simplest case, namely, that of a straight regression line

Then we may want to graph the sample values as n points in the xY-plane, fit a straight line through them, and use it for estimating μ(x) at values of x that interest us, so that we know what values of Y we can expect for those x. Fitting that line by eye would not be good because it would be subjective; that is, different persons’ results would come out differently, particularly if the points are scattered. So we need a mathematical method that gives a unique result depending only on the n points. A widely used procedure is the method of least squares by Gauss and Legendre. For our task we may formulate it as follows.

To get uniqueness of the straight line, we need some extra condition. To see this, take the sample (0, 1), (0, −1). Then all the lines y = k₁x with any k₁ satisfy the principle. (Can you see it?) The following assumption will imply uniqueness, as we shall find out.

From a given sample (x₁, y₁), …, (x_n, y_n) we shall now determine a straight line by least squares. We write the line as

and call it the sample regression line because it will be the counterpart of the population regression line (1).

Now a sample point (x_j, y_j) has the vertical distance (distance measured in the y-direction) from (2) given by

Fig. 543. Vertical distance of a point (x_j, y_j) from a straight line y = k₀ + k₁x

Hence the sum of the squares of these distances is

In the method of least squares we now have to determine k₀ and k₁ such that q is minimum. From calculus we know that a necessary condition for this is

We shall see that from this condition we obtain for the sample regression line the formula

Here and are the means of the x- and the y-values in our sample, that is,

The slope k₁ in (5) is called the regression coefficient of the sample and is given by

Here the “sample covariance” s_xy is

and is given by

From (5) we see that the sample regression line passes through the point (), by which it is determined, together with the regression coefficient (7). We may call the variance of the x-values, but we should keep in mind that x is an ordinary variable, not a random variable.

We shall soon also need

Derivation of (5) and (7). Differentiating (3) and using (4), we first obtain

where we sum over j from 1 to n. We now divide by 2, write each of the two sums as three sums, and take the sums containing y_j and x_jy_j over to the right. Then we get the “normal equations”

This is a linear system of two equations in the two unknowns k₀ and k₁. Its coefficient determinant is [see (9)]

and is not zero because of Assumption (A1). Hence the system has a unique solution. Dividing the first equation of (10) by n and using (6), we get . Together with y = k₀ + k₁x in (2) this gives (5). To get (7), we solve the system (10) by Cramer's rule (Sec. 7.6) or elimination, finding

This gives (7)–(9) and completes the derivation. [The equality of the two expressions in (8) and in (9) may be shown by the student].

Confidence Intervals in Regression Analysis

If we want to get confidence intervals, we have to make assumptions about the distribution of Y (which we have not made so far; least squares is a “geometric principle,” nowhere involving probabilities!). We assume normality and independence in sampling:

Assumption (A2)

For each fixed x the random variable Y is normal with mean (1) , that is,

and variance σ² independent of x.

Assumption (A3)

The n performances of the experiment by which we obtain a sample

are independent.

κ₁ in (12) is called the regression coefficient of the population because it can be shown that, under Assumptions (A1)–(A3), the maximum likelihood estimate of κ₁ is the sample regression coefficient k₁ given by (11).

Under Assumptions (A1)–(A3), we may now obtain a confidence interval for κ₁, as shown in Table 25.12.

Table 25.12 Determination of a Confidence Interval for κ₁ in (1) under Assumptions (A1)–(A3)

Correlation Analysis

We shall now give an introduction to the basic facts in correlation analysis; for proofs see Ref. [G2] or [G8] in App. 1.

Correlation analysis is concerned with the relation between X and Y in a two-dimensional random variable (X, Y) (Sec. 24.9). A sample consists of n ordered pairs of values (x₁, y₁), …, (x_n, y_n) as before. The interrelation between the x and y values in the sample is measured by the sample covariance s_xy in (8) or by the sample correlation coefficient

with s_x and s_y given in (9). Here r has the advantage that it does not change under a multiplication of the x and y values by a factor (in going from feet to inches, etc.).

The theoretical counterpart of r is the correlation coefficient ρ of X and Y,

Fig. 544. Samples with various values of the correlation coefficient r

where (the means and variances of the marginal distributions of X and Y; see Sec. 24.9), and σ_XY is the covariance of X and Y given by (see Sec. 24.9)

The analog of Theorem 1 is

X and Y are called uncorrelated if ρ = 0.

Here the two-dimensional normal distribution can be introduced by taking two independent standardized normal random variables X*, Y*, whose joint distribution thus has the density

(representing a surface of revolution over the x*y*-plane with a bell-shaped curve as cross section) and setting

This gives the general two-dimensional normal distribution with the density

where

In Theorem 3(b), normality is important, as we can see from the following example.

Test for the Correlation Coefficient ρ

Table 25.13 shows a test for ρ in the case of the two-dimensional normal distribution. t is an observed value of a random variable that has a t-distribution with n − 2 degrees of freedom. This was shown by R. A. Fisher (Biometrika10 (1915), 507–521).

Table 25.13 Test of the Hypothesis ρ = 0 Against the Alternative ρ > 0 in the Case of the Two-Dimensional Normal Distribution

CHAPTER 25 REVIEW QUESTIONS AND PROBLEMS

1. What is a sample? A population? Why do we sample in statistics?
2. If we have several samples from the same population, do they have the same sample distribution function? The same mean and variance?
3. Can we develop statistical methods without using probability theory? Apply the methods without using a sample?
4. What is the idea of the maximum likelihood method? Why do we say “likelihood” rather than “probability”?
5. Couldn't we make the error of interval estimation zero simply by choosing the confidence level 1?
6. What is testing? Why do we test? What are the errors involved?
7. When did we use the t-distribution? The F-distribution?
8. What is the chi-square (X²) test? Give a sample example from memory.
9. What are one-sided and two-sided tests? Give typical examples.
10. How do we test in quality control? In acceptance sampling?
11. What is the power of a test? What could you perhaps do when it is low?
12. What is Gauss's least squares principle (which he found at age 18)?
13. What is the difference between regression and correlation?
14. Find the mean, variance, and standard derivation of the sample 21.0 21.6 19.9 19.6 15.6 20.6 22.1 22.2.
15. Assuming normality, find the maximum likelihood estimates of mean and variance from the sample in Prob. 14.
16. Determine a 95% confidence interval for the mean μ of a normal population with variance σ² = 25, using a sample of size 500 with mean 22.
17. Determine a 99% confidence interval for the mean of a normal population, using the sample 32, 33, 32, 34, 35, 29, 29, 27.
18. Assuming normality, find a 95% confidence interval for the variance from the sample 145.3, 145.1, 145.4, 146.2.
19. Using a sample of 10 values with mean 14.5 from a normal population with variance σ² = 0.25, test the hypothesis μ₀ = 15.0 against the alternative μ₁ = 14.5 on the 5% level. Find the power.
20. Three specimens of high-quality concrete had compressive strength 357, 359, 413 [kg/cm²], and for three specimens of ordinary concrete the values were 346, 358, 302. Test for equality of the population means,μ₁ = μ₂, against the alternative μ₁ > μ₂. Assume normality and equality of variance. Choose α = 5%.
21. Assume the thickness X of washers to be normal with mean 2.75 mm and variance 0.00024 mm². Set up a control chart for μ and graph the means of the five samples (2.74, 2.76), (2.74, 2.74), (2.79, 2.81), (2.78, 2.76), (2.71, 2.75) on the chart.
22. The OC curve in acceptance sampling cannot have a strictly vertical portion. Why?
23. Find the risks in the sampling plan with n = 6 and c = 0, assuming that the AQL is θ₀ = 1% and the RQL is θ₁ = 15%. How do the risks change if we increase n?
24. Does a process of producing plastic rods of length meters need adjustment if in a sample, 2 rods have the exact length and 15 are shorter and 3 longer than 2 meters? (Use the sign test.)
25. Find the regression line of y on x for the data (x, y) = (0, 4), (2, 0), (4, −5), (6, −9), (8, −10).

SUMMARY OF CHAPTER 25

Mathematical Statistics

We recall from Chap. 24 that, with an experiment in which we observe some quantity (number of defectives, height of persons, etc.), there is associated a random variable X whose probability distribution is given by a distribution function

which for each x gives the probability that X assumes any value not exceeding x.

In statistics we take random samples x₁, …, x_n of size n by performing that experiment n times (Sec. 25.1) and draw conclusions from properties of samples about properties of the distribution of the corresponding X. We do this by calculating point estimates or confidence intervals or by performing a test for parameters (μ and σ² in the normal distribution, p in the binomial distribution, etc.) or by a test for distribution functions.

A point estimate (Sec. 25.2) is an approximate value for a parameter in the distribution of X obtained from a sample. Notably, the sample mean (Sec. 25.1)

is an estimate of the mean μ of X, and the sample variance (Sec. 25.1)

is an estimate of the variance σ² of X. Point estimation can be done by the basic maximum likelihood method (Sec. 25.2).

Confidence intervals (Sec. 25.3) are intervals θ₁ θ θ₂ with endpoints calculated from a sample such that, with a high probability γ, we obtain an interval that contains the unknown true value of the parameter θ in the distribution of X. Here, γ is chosen at the beginning, usually 95% or 99%. We denote such an interval by CONF_γ {θ₁ θ θ₂}.

In a test for a parameter we test a hypothesis θ = θ₀ against an alternative θ = θ₁ and then, on the basis of a sample, accept the hypothesis, or we reject it in favor of the alternative (Sec. 25.4). Like any conclusion about X from samples, this may involve errors leading to a false decision. There is a small probability α (which we can choose, 5% or 1%, for instance) that we reject a true hypothesis, and there is a probability β (which we can compute and decrease by taking larger samples) that we accept a false hypothesis. α is called the significance level and 1 − β the power of the test. Among many other engineering applications, testing is used in quality control (Sec. 25.5) and acceptance sampling (Sec. 25.6).

If not merely a parameter but the kind of distribution of X is unknown, we can use the chi-square test (Sec. 25.7) for testing the hypothesis that some function is the unknown distribution function of X. This is done by determining the discrepancy between F(x) and the distribution function of a given sample.

“Distribution-free” or nonparametric tests are tests that apply to any distribution, since they are based on combinatorial ideas. These tests are usually very simple. Two of them are discussed in Sec. 25.8.

The last section deals with samples of pairs of values, which arise in an experiment when we simultaneously observe two quantities. In regression analysis, one of the quantities, x, is an ordinary variable and the other, Y, is a random variable whose mean μ depends on x, say, μ(x) = κ₀ + κ₁x. In correlation analysis the relation between X and Y in a two-dimensional random variable (X, Y) is investigated, notably in terms of the correlation coefficient ρ.