Describing a study whose results have been reproduced when the study was repeated, or replicated. Also called reproducible. See also conceptual replication, direct replication, replication-plus-extension. (page 425)
A replication study in which researchers repeat the original study as closely as possible to
see whether the original effect shows up in the newly collected data. See also conceptual replication, replication-plus-extension. (page 426)
A replication study in which researchers examine the same research question (the same conceptual variables) but use different procedures for operationalizing the variables. See also direct replication, replication-plus-extension. (page 427)
replication-plus-extension
A replication study in which researchers replicate their original study but add variables or conditions that test additional questions. See also conceptual replication, direct replication. (page 428)
A family of questionable data analysis techniques, such as adding participants after the results are initially analyzed, looking for outliers, or trying new analyses in order to obtain a p value of just under .05, which can lead to nonreplicable results. (page 432)
As part of a study's publication process, the practice of sharing one's data and materials freely so others can collaborate, use, and verify the results. (page 432)
The practice of posting a study's method, hypotheses, or
statistical analyses publicly, in advance of data collection. (page 432)
A series of related studies, conducted by various researchers, that have tested similar variables. Also called literature. (page 433)
The extent to which the tasks and manipulations of a study are similar to real-world contexts; an aspect of external validity. Also called mundane realism. (page 439)
A researcher's intent for a study, testing association claims or causal claims to investigate support for a theory. See also generalization mode. (page 440)
The intent of researchers to generalize the findings from the samples and procedures in their study to other populations or contexts. See also theory-testing mode. (page 442)
A subdiscipline of
psychology concerned with how cultural settings shape a person's thoughts, feelings, and behavior, and how these in turn shape cultural settings. (page 444)
A real-world setting for a research study. (page 449)
The extent to which a laboratory experiment is designed so that participants experience authentic emotions, motivations, and behaviors. (page 449)
A way of mathematically averaging the effect sizes of all the studies that have tested the same variables to see what conclusion that whole body of evidence supports. (page 433)
A problem relating to literature reviews and meta-analyses based only on published literature, which might overestimate the support for a theory because studies finding null effects are less likely to be published than studies finding significant results, and are thus less likely to be included in such reviews. (page 436)
Direct replication
•Conceptual replication
•Replication-plus-extension
The Replication Debate in Psychology
The “replication crisis”•Why might replication studies fail?•Improvements to scientific practice.
Why Might Replication Studies Fail?
Contextually sensitive effects
•Number of replication attempts
•Problems with original study –Sample Size –Harking –P-Hacking
Improvements to Scientific Practice
Larger sample sizes •Report all analyses and variable •Open Science Collaboration •Preregistration
Strengths and Limitations of Meta-Analysis
File drawer problem •Null results and opposite results rarely published. •Example: antidepressants and depressive symptoms
Replicability, Importance, and Popular Media
Journalists do not always consider the importance of replicability.
•A responsible journalist will not only report on the latest and most exciting studies but will also summarize the body of literature on the topic.
Generalizing to Other Participants
It’s a population, not thepopulation. •External validity comes from how, not how many. •Just because a sample comes from a population doesn’t mean it generalizes to that population.
Generalizing to Other Settings
Ecological validity: an aspect of external validity in which the focus is on whether a laboratory study generalizes to real-world settings
Examples –The contact comfort theory–“Parent-as-grammar-coach” theory
Frequency claims: always in generalization mode
•Association and causal claims: sometimes in generalization mode
Cultural Psychology: A Special Case of Generalization Mode
Theory Testing Using WEIRD Participants
Most research in psychology has been conducted on North American college students.
•WEIRD = Western, educated industrialized, rich and democratic
Does a Study Have to Take Place in a Real-World Setting?
External validity and the real world •Generalization mode and the real world •Theory-testing mode and the real world
External Validity and the Real World
When a study takes place it the real world, it occurs in a field setting and has high external validity.
•Laboratory research can be just as
realistic as research conducted in the real world, which is known as experimental realism.
Generalization Mode and the Real World
External validity is important when in generalization mode, so having a representative sample is paramount. •Ecological validity is also important to consider in terms of generalizing to nonlaboratory settings.
Theory-Testing Mode and the Real World
When in theory-testing mode, external validity and real-world applicability are not as important.
Which of the following types of study support external validity?
NOT A study that is a direct replication
In a previous correlational study, Dr. Lafayette has found that owning dogs is associated with lower levels of daily stress in a sample of returning war veterans. Which of the following would constitute a shift from theory-testing mode to generalization mode?
Conducting a
study on dog ownership using a sample of elderly participants from a local retirement home
RESEARCH STUDY 14.1: Mark conducts a study for his research methods class to determine if consuming caffeine causes people to perform better on cognitive tasks. In his study, he gives half of his participants a glass of water and half of his participants a glass of cola and then has them attempt to solve 15 math problems. He finds that people
who drink caffeine beforehand solve more math problems than those who drink water beforehand. Mark decides to conduct a variation of the original study in which he measures participants’ ability to solve verbal analogies as the cognitive task instead of measuring their ability to solve math problems. He finds that people who drink caffeine beforehand solve more verbal analogies than those who drink water beforehand, suggesting caffeine consumption causes improved cognitive performance. This
study is known as a:
Which of the following is true of theory-testing mode?
It is related to the theory-data cycle.
Which of the following is true of p-hacking?
It can involve
running a different analysis than originally proposed.
In open science, open data are more concerned with _________, while open materials are more interested in _________.
statistical validity; replication
Dr. Fern recently collected data
for a study examining whether an intervention targeting coping strategies can reduce depression. After analyzing the results, she finds no effect of her intervention on depression symptoms. Which of the following would you recommend she do next?
Contact journal editors to see if they will publish null results.
When generalizing the
results of a sample to a population, which of the following is the most important question to ask?
How were the participants sampled?
Which of the following studies is most likely to be conducted in generalization mode?
Determining how frequently people report fighting
with their bosses
RESEARCH STUDY 14.2: Cindy decides to conduct a meta-analysis examining the relationship between daily stress and cardiovascular health for her senior psychology research project. Cindy’s advisor recommends that she contact several researchers in the field for articles that were not published and/or that found null effects. Doing this will address which of the following?
A study conducted in the “real world” is often said to be conducted in:
Which of the following studies would be least likely to be influenced by cultural differences?
A study examining motor responses of newborns
According to the textbook, which of the
following is a component of WEIRD people?
Which of the following is another term for ecological validity?
Who determines the population to which a study’s findings generalize?
Some studies may not replicate because the original study used questionable scientific practices. Which of the
following is a way open science practices attempt to deal with this problem?
Requiring scientists to make their data and materials available for review
According to the textbook, what is the problem with WEIRD samples?
They are not very representative of the world’s
population.
What does it mean when a study cannot be replicated by an independent researcher?
The importance of the finding should be interpreted cautiously.
The text states that the first step in establishing a study’s importance is to establish its _________.
Which of the following types of replications investigate the same research question but use different procedures?