PPA 696 RESEARCH METHODS
EXPERIMENTAL DESIGNS FOR RESEARCH
CausalityExperimental Designs
Control Group Pre-test/Post-test Design
Threats to Internal Validity
Threats to External Validity
Post-Test only Control Group Design
CAUSALITY
To establish whether two variables are causally related, that is, whether a change in the independent variable X results in a change in the dependent variable Y, you must establish: 1) time order--The cause must have occurred before the effect; 2) co-variation (statistical association)-- Changes in the value of the independent variable must be accompanied by changes in the value of the dependent variable; 3) rationale-- There must be a logical and compelling explanation for why these two variables are related; 4) non-spuriousness-- It must be established that the independent variable X, and only X, was the cause of changes in the dependent variable Y; rival explanations must be ruled out. To establish causality, one must use an experimental or quasi-experimental design. Note that it is never possible to prove causality, but only to show to what degree it is probable.EXPERIMENTAL DESIGNS
True experimental designs include: Pre-test/Post-test control group design Solomon Four-Group design Post-test only control group designPre-test/Post-test control group design
This is also called the classic controlled experimental design, and the randomized
pre-test/post-test design because it:
R O1 X O2
R O1 O2
This diagram can be expanded upon as in the following table:
Scientific Random Assignment of Subjects to: | 1st observation (measurement) of the dependent variable O1 = Pre-test | Exposure to the Treatment (X) (independent variable) | 2nd observation (measurement) of the dependent variable O2 = Post-test |
Experimental Group | Experimental Group's average score on the dependent variable | X | Experimental Group's average score on the dependent variable |
Control Group | Control Group's average score on the dependent variable | Control Group's average score on the dependent variable |
The difference in the control group's score from the pre-test to the post-test indicates the change in the value of the dependent variable that could be expected to occur without exposure to the treatment (independent) variable X.
Control group - control group = control group difference
pre-test score post-test score
on the dependent variable
The difference in the experimental group's score from the pre-test to the post-test indicates the change in the value of the dependent variable that could be expected to occur with exposure to the treatment (independent) variable X.
Experimental group - experimental group = experimental group difference
pre-test score
post-test score on the dependent variable
The difference between the change in the experimental group and the change in the control group is the amount of change in the value of the dependent variable that can be attributed solely to the influence of the independent (treatment) variable X.
Control group difference - experimental group difference = difference attributable to X
This can be illustrated by the following experiment to see whether participation in small group discussions would improve medical students' ability to respond to emotional needs of patients:
Scientific Random Assignment of Medical Students to: | How many times did students use emotional words to describe patients | Exposure to the Treatment (X) (independent variable) | How many times did students use emotional words to describe patients |
Small group discussions (experimental group) | Average of .68 times per student in 3 case studies | Attended small group discussions plus regular course work | Average of 2.02 times per student in 3 case studies |
Control Group | Average of .89 times per student in 3 case studies | Regular course work only | Average of 1.13 times per student in 3 case studies |
The control group used emotional words an average of .89 times per student (in three case studies) on the pre-test and an average of 1.13 times per student (in three case studies) on the post-test. The difference in the control group's score from the pre-test to the post-test is +.24 times per student. This indicates the change in using emotional words that could be expected to occur with regular course work only.
The experimental group used emotional words an average of .68 times per student (in three case studies) on the pre-test and an average of 2.02 times per student (in three case studies) on the post-test. The difference in the experimental group's score from the pre-test to the post-test is +1.34 times per student. The experimental group's score from the pre-test to the post-test indicates the change in using emotional words that could be expected to occur with regular course work plus the small group discussions.
The difference between the change in the experimental group (+1.34) and the change in the control group (+.24) is +1.10. This is the amount of change in using emotional words that can be attributed solely to the influence of the small group discussions.
The controlled or true experimental design allows the researcher to control for threats to the internal and external validity of the study. Threats to internal validity compromise the researcher's ability to say whether a relationships exists between the independent and dependent variables. Threats to external validity compromise the researcher's ability to say whether this study's findings are applicable to any other groups.
Controlling for Threats to Internal Validity
1) History: did some other current event effect the change in the dependent variable? No, because both groups experienced the same current events.2) Maturation: were changes in the dependent variable due to normal developmental processes? No, because both groups experienced the same developmental processes.
3) Statistical Regression: did subjects come from low or high performing groups? Differences between the two groups that could influence the dependent variable would be controlled for as subjects were generally equivalent at the beginning of the research.
4) Selection: were the subjects self-selected into experimental and control groups, which could affect the dependent variable? No, the subjects were assigned by strict random selection and all had equal chance of getting the treatment or control condition.
5) Experimental Mortality: did some subjects drop out? did this affect the results? About the same number of students made it through the entire study in both the experimental and control groups, so there appears to be no bias.
6) Testing: Did the pre-test affect the scores on the post-test? Both groups got a pre-test; but a pre-test may have made the experimental group more sensitive to the treatment.
7) Instrumentation: Did the measurement method change during the research? The measurement method and instruments did not change.
8) Design contamination: did the control group find out about the experimental treatment? did either
group have a reason to want to make the research succeed or fail? The researcher must do some qualitative investigation to find out if there was design contamination.
Controlling for Threats to External Validity
1) Unique program features: There may have been an unusually motivated set of facilitators for the small group discussions.2) Effects of Selection: Probably applicable to other medical students.
3) Effects of Setting: Medical schools have their own cultures; doubtful if this would be applicable to other types of students.
4) Effects of History: No information given
5) Effects of Testing: No information given
6) Reactive effects of experimental arrangements: It would be best to replicate the results in other medical schools.
Post-Test Only Control Group Design
This design follows all the same steps as the classic pre-test/post-test design except that it omits the pre-test. There are many situations where a pre-test is impossible because the participants have already been exposed to the treatment, or it would be too expensive or too time-consuming. For large enough groups, this design can control for most of the same threats to internal and external validity as the classic controlled experimental design. For example, it eliminates the threat to internal validity of pre-testing by eliminating the pre-test. It may also decrease the problem of experimental mortality by shortening the length of the study (no pre-test).For small groups, however, a pre-test is necessary. Also, a pre-test is necessary if the researcher wants to determine the exact amount of change attributable to the independent variable alone.
Public
administrators would like to be able to use experimental designs for policy and program evaluation. Did a regional economic development policy bring more business to the economically depressed region? Did the Women-Infants-and-Children (WIC) program lower the rate of malnutrition in young children?