Power/Sample-size GUI for balanced ANOVA models

Creating a GUI

Use the form above to enter the information needed to create a graphical user interface (GUI) for the ANOVA model of your choice. The GUI is useful for examining power, detectable effect size, or sample size for the F tests. The required information is: When everything is entered, click on the Create GUI button; a new window then pops up with the user interface for the model you specified. It may take awhile for the window to appear (especially the first time). You can revise the information (or not) and open as many additional GUIs as you like. Note: In the Levels and Random windows, only factors should be listed. Give only the root name of the factor.

This dialog is not very robust. If you misspell a factor name, fail to put "+" signs between the terms, etc., the model structure will not be built. You may refer to the Java console (on Netscape, use "Options/Show Java console") to see error messages. These messages are Java-generated but they often display the string that it found troublesome.

Examples

Randomized complete-blocks design

Suppose we have n subjects (we'll start with n=10) and each subject is to be tested once with each of 4 different treatments (in separate random order for each subject). Then use:
        Title:   RCB design
        Model:   SUBJ + treat
        Levels:  SUBJ 10   treat 4
        Random:  SUBJ
        Reps:    1
Giving reps = 1 sets it up so that an unreplicated design is assumed.

Two-way ANOVA

Here we'll leave the Levels and Random fields blank; we can always change them in the GUI. A standard 2-way model with interaction is used.
        Model:   Temp | Speed     (expands to TEMP + SPEED + TEMP*Speed)
        Reps:    5                (or any number >= 2)

Nested factorial

Subjects are divided into 4 groups, each identified with a particular drug. Subjects are tested with 3 different dosages of the assigned drug.
        Model:   drug + SUBJ(drug) + dose + drug*dose
        Levels:  Drug 4   Dose 3   Subj 6
        Random:  subj
Note: Case is ignored, so "Subj" and "subj" get identified with "SUBJ". The model statement dictates what gets displayed. Note in the Levels and Random fields, only "subj" is given, not "subj(drug)".

Using the GUI

The ANOVA power GUI contains familiar elements such as radio buttons and input windows, as well as several unfamiliar bar-graph-like elements that work a little bit like scrollbars.

To use the bar-graph interface

lick the mouse at any point along the centerline of a bar, and the value of that parameter will be changed to that number. (The exception are the bars that display power; inputs on those are disabled.) Alternatively, you may enter a value in the associated input window and hit the Enter (or Return) key. You may also drag the end of a bar with the mouse. (This can keep your computer pretty busy calculating noncentral F probabilities; so drag cautiously.)

Below each set of bars is a numerical scale. If the range of that scale is not to your liking, click at any point along that scale and drag to a new position for that scale value. For example, to double the range of the scale, you might click at "2" and drag left to "1" before letting go. Scales are always updated to make room for all the values in a set.

Solving power/sample-size problems

The top section of the window is used for changing the sample size (or number of levels) and selecting which factors are fixed and random. The bottom section has bar graphs for effect size and power. There is also an "alpha" window for setting the significance level (for all of the F tests). To obtain sample size, input the effect sizes (see below for details) and try different sample sizes until the desired powers are achieved. There are potentially several tests, with different powers. Keep in mind that, for equal effect sizes, the biggest player is the number of observations at each level (or combination of levels) of a term. So, for example, tests of interactions are less powerful than tests of main effects, all other things being equal. Often, sample size is limited by time or budget. Then you would probably enter the budgeted sample size(s), and either observe the power, or vary effect size to see what can be detected with a reasonable power.

Effect size

Effect size is quantified by the standard deviation (SD) of the associated effects in the ANOVA model. For a random effect, the effect size, squared, is thus the variance component for that term. Note in particular that the effect size for "Within" or "RESIDUAL" is the error SD. Typically, one uses existing data or does a pilot study ahead of time to estimate these standard deviations.

For a fixed effect (meaning that all factors involved have fixed levels), there is a model term of the form tau_{i,j,...}. Let Q denote the sum of the squares of all of these taus; then the effect size is sqrt(Q/d), where d is the degrees of freedom associated with the model term (not the error df!).

More development is afoot for providing more flexibility in the ways that effect sizes can be specified. (Suggestions are welcome!)

Technicalities

All tests are based on the "unrestricted model" for the analysis of variance, whereby it is assumed that all random or mixed terms are iid normal random variables. This is the same model used by SAS in determining expected mean squares. There are other possible models, the most popular being the "restricted model" where a mixed effect is constrained to sum to zero over any subscript associated with levels of a fixed factor.

Sometimes, the correct error term (denominator) for an F ratio cannot be found, in which case an appropriate linear combination of mean squares is constructed, and Satterthwaite approximation is used to obtain the degrees of freedom. Such tests are only approximate.

It is extremely important to remember that the effect sizes of random and mixed terms can affect the powers of other tests! That is because they are used as values of the effect standard deviations that may appear in the error terms of other tests. To dramatize this, try a model like A | B | C | D and watch what happens to the power when you change one of the factors to random.

Disclaimer

Please understand that, though I have made every effort to write correct code, there may still be errors. I do not guarantee the correctness of any results you obtain and do not take responsibility for any losses incurred as a result of those errors.
Russell V. Lenth
Department of Statistics and Actuarial Science
University of Iowa
Iowa City, IA 52242
Russell-Lenth@uiowa.edu