Significance Level of a Statistical Test

the probability of erroneously rejecting an initial hypothesis when it is true. In the theory of statistical testing of hypotheses the significance level is called the probability of an error of the first kind (Type I error).

The concept of the significance level arose in connection with the problem of testing the conformance of theory to experimental data. If, for example, the values of n random quantities X₁, …, X_n are recorded from observations and it is necessary to test, on the basis of these data, a hypothesis H according to which the joint distribution of the quantities X₁, …, X_n has some specific property, then the corresponding statistical test is constructed by means of an appropriately selected function Y = f(X)_l,f …, X_n). This function usually assumes small values when the hypothesis H is true and large values when H is false. In particular, if X₁…, X_n are the results of independent measurements of some known constant a and the hypothesis H represents a proposition regarding the absence of systematic errors in the results of the measurements, then to test H it is reasonable to select as Y the expression (2m − n)², where m is the number of those results of measurements of X_iL that exceed the true value of a. The large value of Y that is observed in experiments may be considered as a significant statistical deviation from the hypothetical agreement between the results of the observations and the hypothesis. The corresponding significance test is a rule according to which the values of Y that exceed a given critical value y are considered to be significant. In turn, the selection of the quantity y is determined by the given significance level, which, if hypothesis H is valid, coincides with the probability of the event [Y > y]

In selecting a significance level one should take into account the loss that inevitably occurs when any significance test is used. Thus, for example, if the significance level is too great, most of the loss will occur as a result of erroneous rejection of the correct hypothesis, and, on the other hand, if the significance level is too low, the loss will generally result from erroneous acceptance of a hypothesis when it is false. In ordinary statistical calculations, a probability within the range 0.01–0.1 is chosen as the significance level in practice. Values of the significance level lower than 0.01 are used, for example, in statistical decisions involving toxic medal preparations and in other special cases when protection against erroneous rejection of a hypothesis is of primary importance.