See Steel, Torrie and Dickey, Chapter 8, and Multiple Comparisons and Multiple Tests using the SAS System, Westfall et al.

Pre-planned comparisons that are linearly independent of one another, hence involving no more contrasts than the appropriate degrees of freedom, can be made using either t-tests or F tests (although it should be noted that this does not, of itself address the issue of multiple comparisons). If we want to test all the possible differences, or if we wish to make tests suggested by the data then simple t-tests or F tests are no longer appropriate, since the overall probability level will likely be toohigh (the risk of false positives). We need tests appropriate to multiple comparisons. Multiple comparison tests are characterised by considering the number of tests that could be made.

There are a number of multiple comparison tests :

• Scheffé's test
• Tukey's w procedure
• Bonferonni's test

Here we look at Scheffé's test; because it is a valid, fairly conservative test, sufficiently generalised to be applicable to unequal designs. It is very general, so that all possible contrasts can be tested for significance, or confidence intervals contructed for corresponding linear functions of parameters.

Comparison of treatment means from a Completely Randomised Design

Consider the example from the Completely Randomised Design, One-way ANOVA.

	Treatments
	3DOk1	3DOk5	3DOk4	3DOk7	3DOk13	composite
means	28.82	23.90	14.35	19.92	13.26	18.7

We shall look at the multiple comparison test due to Scheffé. We can use it for computing a Confidence Interval, and also for computing a Critical Difference, to allow us to determine whether a difference can be considered to be statistically significantly different,or not.

First of all we need to decide on our Hypothesis; our Null Hypothesis (H_o) and our Alternative Hypothesis (H_A).

Our Null Hypothesis (H_o) will be that there is no difference, i.e. that the difference = Zero.

Our Alternative Hypothesis (H_A) will be that there is a difference, i.e. that the difference is not equal to Zero.

Basically it involves estimating the difference of any particular desired contrast, and the standard error, using the appropriate methodology, i.e. using our matrix k`!

We then need to determine the F_tabulated, which will depend on our probability level, numerator and denominator degrees of freedom, hence the importance of being able to compute the correct tabulated F value for any given numerrator and denominator degrees of freedom (see the section on computing F vaalues).

Then we compute the critical difference as :

Then, if the absolute estimated difference is greater than the critical difference, we can declare that the difference is statistically significant, i.e. reject H_o, where H_o is that the difference is zero.

Thus, for example, suppose that we have analysed these data and see that there appears to be a difference between the 5 straight inoculants and the mixture. So, we wish to compare the average of the 5 inoculants vs. the 6^th (which is the combined mixture of the 5 inoculants), then the difference is:

The tabulated t value for 22 residual degrees of freedom at the 5% level is 2.074. Thus, using a simple (inappropriate) t statistic we would require a critical difference (estimate) of at least t * s.e.; i.e. 2.074 * 1.77 = 3.67

If we did not have this as a pre-planned comparison, but rather after our analysis we noted this difference and wanted to know if it was significant we should be using a multiple comparison test, e.g. Scheffé's test. The calculations are as above, to compute the estimate of the difference and the standard error. We have 6 treatments, so s = 6, and s - 1 = 5, the degrees of freedom for treatments.

F 5%, 5 d.f., 22 d.f.	= 2.66
	= 3.6469

Then the critical difference = 1.77 * 3.6469 = 6.45. Thus the estimated difference must exceed 6.45 to be considered statistically significant, at the 5% level!! Therefore in this case we would accept the null hypothesis, that the treatments do not in fact differ significantly from one another.

We can also use the Sheffé's multiple comparison method to compute a Confidence Interval; which will simply be our estimate +/- the Critical difference! This will apply equally well to an estimate of a difference, or a Least Square mean.

How do we do this with SAS (or any other statistical package)?
With SAS we can use the estimate statement, after the model statement, to compute the estimate of the difference between two 'treatments' or levels together with the standard error of the estimate. Then a little bit of work by hand with a calculator will give us the critical difference.

In addition, if we have carried out an experiment and then made the statistical analysis and want to compare all the treatments (for example in the above experiment) we are now able to have SAS do the Scheffé's test for us when we compare the Least Squares Means. We shall not obtain the estimates of the differences, nor a critical difference. Rather SAS will provide us with a table of the probabilities of the differences, adjusted for the multiple comparisons, using the method of Scheffé. The SAS statements for the CRD would be:

proc glm data=crd1;
classes trt;
model y = crd;
lsmeans trt/stderr pdiff;
lsmeans trt/stderr pdiff adjust=scheffe;  /* Scheffe's test  */
lsmeans trt/stderr pdiff adjust=bon;      /* Boneferoni's test  */
estimate 'trt1-trt2' trt 1 -1 0 0 0 0;
estimate 'trt1-trt3' trt 1 0 -1 0 0 0;
estimate 'trt1-trt4' trt 1 0 0 -1 0 0;
estimate 'trt1-trt5' trt 1 0 0 0 -1 0;
estimate 'trt1-trt6' trt 1 0 0 0 0 -1;
estimate 'trt2-trt3' trt 0 1 -1 0 0 0;
estimate 'trt2-trt4' trt 0 1 0 -1 0 0;
estimate 'trt2-trt5' trt 0 1 0 0 -1 0;
estimate 'trt2-trt6' trt 0 1 0 0 0 -1;
estimate 'trt3-trt4' trt 0 0 1 -1 0 0;
estimate 'trt3-trt5' trt 0 0 1 0 -1 0;
estimate 'trt3-trt6' trt 0 0 1 0 0 -1;
estimate 'trt4-trt5' trt 0 0 0 1 -1 0;
estimate 'trt4-trt6' trt 0 0 0 1 0 -1;
estimate 'trt5-trt6' trt 0 0 0 0 1 -1;
estimate 'trt1-5 vs 6' trt .2 .2 .2 .2 .2 -1;
run;

The output produced from the above analysis is shown here:

Comparison of treatment means from a Factorial Design

Consider the example from the Factorial Design.

We have a simple Factorial Design, with both factors being fixed effects, Diet with 3 levels and Sex with 2 levels, so that the effects are all being tested against the Residual Mean Square. The interaction effect was statistically significant (Pr < .0034). Thus the main effects loose their importance and we should be looking at the 'Simple Effects'. From this 3 X 2 Factorial we therefore have 6 Diet X Sex combinations, or 'Simple Effects'. Therefore in terms of 'Simple Effects', if we wish to make multiple comparisons or ad hoc, post priori tests we have 6 'levels', hence 5 degrees of freedom. These 5 degrees of freedom are equal to the 2 d.f. for Diet + 1 d.f. for Sex + the 2 d.f. for the Diet X Sex interaction.

Simple effects	Estimate ± s.e.
µ + a1 + b1 + ab11	105.25 ± 1.40
µ + a1 + b2 + ab12	95.00 ± 1.40
µ + a1 + b3 + ab13	91.25 ± 1.40
µ + a2 + b1 + ab21	102.25 ± 1.40
µ + a2 + b2 + ab22	102.25 ± 1.40
µ + a2 + b3 + ab23	89.50 ± 1.40

Using the same principles as above, for a multiple comparison test using Scheffé's test, we have 6 'treatments' (combinations); therefore s = 6, and s-1 = 5. The residual degrees of freedom are 18.

F 5%, 5 d.f., 18 d.f.	= 2.77
	= 3.7216

Consider the difference between A₁B₂ ( 95.00) and A₂B₂ (102.25).

Then the critical difference is 1.98*3.7216 = 7.369

Thus an aposteriori test, à la Scheffé's test, would accept the null hypothesis, that there is no difference, since the difference is less than the critical difference; whereas a simple t-test would reject the null hypothesis :

and the tabulated t value for 5% and 18 d.f. is 2.101; less than our computed t value.

How to do this using SAS? The approach to use is much the same as that described above for a CRD design, we use the SAS estimate statement to get SAS to compute the estimate and standard error of the particular contrast that we are interested in; the rest we do by hand!

Multiple Comparisons in a Nested Analysis

The above approaches can be extended to Nested Designs and Analyses, effectively the only difference is that in a Nested (Subsampling) Analysis the Residual Mean Square (Error) is replaced by the appropriate Mean Square, the same one as used in the Analysis of Variance.