What are four common threats to internal validity when conducting experimental research designs?

3 Threats to Internal Validity

The literature on internal validity consists largely of detailed lists of validity ‘threats.’ Internal validity threats are generic categories of causal forces that may frequently obscure causal inferences. Take as an example, once again, a researcher's efforts to determine whether an anger management program reduces aggressive behavior in a middle school. ‘History’ refers to the possibility that specific events, other than the intended treatment, may have occurred between the pretest and post-test observations and may obscure the true treatment effect. If the researcher observed the level of aggressive behavior on the playground before the anger management program, and again afterward, history would be a problem if a different, stricter teacher became playground monitor in the interim. ‘Maturation’ refers to the possibility that natural processes which occur over time within the study participants, such as growing older, hungrier, more fatigued, wiser, and the like, may create a false treatment effect or mask a real one. Less aggression may occur at the post-test simply because the children are older than at the pretest, for instance. ‘Attrition’ refers to the possible loss of participants in a study. For example, if children from troubled families are more likely to drop out of school or to move away in the middle of the school year, then attrition could cause a decrease in aggression from the pretest to the post-test. ‘Instrumentation’ arises as a validity threat when a change in a measuring instrument causes erroneous conclusions about the effects of an intervention. For instance, if observers' standards shifted over time, such that later incidents had to be more violent to be rated as aggressive, this could cause the appearance of a treatment effect when in fact there is none.

‘Selection’ refers to the possibility that post-test differences between a treatment group and a control group may be due to initial differences between the groups rather than to a treatment effect. Selection problems might occur if a researcher attempted to assess the effectiveness of an anger management program by comparing the level of playground aggression in two middle schools, one of which had implemented the program. In addition, more complex internal validity problems can occur, whereby some threat operates only (or more powerfully) in one group than another. For instance, ‘selection by maturation’ indicates that participants in the treatment condition are maturing at a different rate than those in the control condition. See Cook and Campbell (1979) for additional discussion of internal validity threats, including the threats of testing and regression to the mean.

7.6 Threats to Experimental Validity

8.2 Internal validity

An internal validity threat concerns the data extraction from the set of included studies. Both authors are incorporated in this process. Following the guidelines of Peterson et al. [7], a data extraction form is designed by the first author, discussed and updated after a deep discussion with the second author. Each author has filled the form independently; after solving the disagreements between the two authors, a unique and final form is adopted for the rest of the study. Following the guidelines of Kitchenham et al. [6], the kappa coefficient is estimated and found equal 0.94 which is evaluated as very good or almost perfect score according to [6].

6.7 Threats to validity

The internal validity threat in this example would be to any change in the design of the example, such as a lack of explanation for the concepts to be assessed in the running example. To mitigate the risk of this threat the principal researchers conducted an example of the proposed reference model. An external validity threat is articulated at the level of the results.

The proposed reference model of software security requirements was illustrated using only the requirement specifications of the withdrawal process for an ATM machine system. To mitigate the risk of this validity threat further examples should be conducted using requirement specifications of different types of software products (i.e., real time embedded software, business application software, or even a hybrid of both types).

6 Threats to validity

The validation process is essential for any kind of empirical study. Wohlin et al. [70] defined four different types of validity threats: construct, internal, external, and conclusion.

Construct validity threats: According to [70], construct validity threats are the threats linked to the generalization of the results to the concepts behind the study. During our review, we could see that a few SOAMMs have very detailed documentation whereas others do not. In a few models, they use different terminologies or the details are not explicitly mentioned. In such cases, we concluded with our assessments. In order to minimize this kind of threat, we focused only on SOAMM design and usage aspects and the second author verified the extracted data.

Internal validity threats: Internal validity threats may affect the results due to incorrect conclusions. In order to mitigate this kind of risk we followed the SLR guidelines provided by Kitchenham [25] and conducted the study systematically. We defined a review protocol to include all relevant SOAMM studies in the search. We included the major digital libraries to cover all the publications in SOA maturity.

External validity threats: These threats are related to the generalization of results of the review to the real world scenarios [70]. In this review, there may be chances to exclude the SOAMMs, which may not be part of academic publications or related works of selected primary studies. We also defined inclusion/exclusion/quality criteria to mitigate the risks associated with primary study selection.

Conclusion validity threats: The threats related to the inaccurate conclusions come under this category. This could be the small set of primary studies we have collected. In order to mitigate the risk of excluding primary studies we did the following: (i) the search strings are defined by considering the concepts and their acronyms, (ii) the multiple electronic libraries are used as a search base, (iii) the related works of all the primary studies are carefully analyzed. This way, we could find three more new SOAMMs from the related works of selected models. In addition, to mitigate the risks associated with study duplication we used Microsoft Excel's sorting and filtering options based on the title, author, publisher, and year of publication. The selected papers are carefully read through to mitigate the misinterpretations of title/abstract.

2.1 Threats to validity in empirical software engineering

Threats to validity have been often categorized in the literature of general research methods in different types. Initially, Cook and Campbell [8]2 recorded four types of validity threats in quantitative experimental analysis: statistical conclusion validity, internal validity, construct validity of putative causes and effects and external validity. Concerning qualitative research, Maxwell [29] provided a general categorization of threats that can be mapped to Cook and Campbell's categorization as follows: theoretical validity (construct validity), generalizability (internal, external validity), and interpretive validity (statistical conclusion validity). An additional threat category, mentioned by Maxwell [29], is descriptive validity, which is relevant only for qualitative studies. Descriptive validity reflects the accuracy and objectivity of the information gathered. For example, when researchers collect statements from participants, threats to validity can be related to the way that researchers recorded or transcribed the statements. Other types of validity threats that are found in literature are: reliability [38,51], transferability, credibility and confirmability [27], uncontrollability, and contingency [14].

In the empirical SE community there are two main schools on reporting threats to validity: (a) Wohlin et al. [47] who adopted Cook and Campbell's [8] categorization of validity threats and presented four main types of threats to validity for quantitative research within software engineering: conclusion, internal, construct, and external validity; and (b) Runeson et al. [38] who discussed four main types of validity threats for case studies within software engineering: reliability, internal, construct, and external validity. The threats of Runeson et al. [38] are similar to those of Wohlin et al. [47] with the exception of reliability replacing conclusion validity.

Biffl et al. [4] argue that researchers should also consolidate actual experimental research on a specific topic to complement existing generic threats and guidelines when performing their research. The tradeoff between internal and external validity has been addressed by Siegmund et al. [40], where the authors performed a survey and concluded that externally valid papers are of greater practicality while internally valid studies seem to be unrealistic. Additionally, the study examined the impact of replication studies and found that although researchers realize the necessity of such studies they are reluctant to conduct or review them mainly due to the fact that there are no guidelines for performing them [40]. A list of definitions of the union of the aforementioned categories of threats to validity (i.e. from [38] and [47]) are presented in Table 1.

Table 1. Categories of Threats to Validity in ESE Research.

Petersen et al. [35] based on the categorizations of threats to validity suggested by Maxwell, suggested a check list that can help researchers identify the threats applicable to the type of research performed by reporting first their world-view and then the research method applied. A secondary study attempting to assess the practices in reporting validity threats in ESE [12] concluded that more than 20% of the studied papers contain no discussion of validity threats and the ones that do discuss validity threats on average contain 5.44 threats.

Regarding threats to validity for secondary studies in software engineering, we have been able to identify only one related work. In particular, Zhou et al. [53] have performed a tertiary study on more than 300 secondary studies until 2015. The authors have identified 23 threats to validity for secondary studies, and organize the consequences of these studies into four categories: internal, external, conclusion, and construct validity. To alleviate these threats the authors maps the threats and possible consequences to 24 mitigation strategies. This paper shares common goals with our study, however, ours is broader in the sense that: (a) it covers a wider timeframe (until 2017 instead of middle of 2015); (b) it focuses only on top-quality venues, which are expected to pay special attention in the proper application of methodological guidelines, such as the proper reporting of threats to validity, a fact that increases the quality of the obtained data; and most importantly (c) our study answers two additional RQs, providing a classification schema and a checklist for identifying, mitigating, and reporting threats to validity. In addition to this, as indirect related work (especially in terms of mitigation actions), in Section 2.3 we present a review of guidelines on secondary studies in software engineering.

8 Threats to validity

Most of SLR results were extracted by the first author of this study. It might be a threat towards the validity of this study because the results of a single researcher could be biased and may constantly extract the wrong data. However, this threat was tried to be reduced by the participation of the other authors to arbitrarily examine the SLR results in order to find any issues that might exist.

Most of the selected primary studies have not discussed the key causes of the identified success factors and it could be the internal validity threat to this study. It is possible that in certain studies, there might be a trend to report a particular type of factors. Furthermore, the majority of the researchers of the 54 selected primary studies are in the academic field; therefore, they might have lack of knowledge about the current practices of RCM processes in the software development industry.

We have noticed that only 34 primary studies out of 107 have provided organization size information and due to this limitation, we were not able to generalize the organizational size information of all the selected primary studies. It might be a possible threat for the results of RQ2 that focused on the size of the organizations.

With the increasing number of studies on RCM, certain relevant studies may have been inadvertently omitted from our SLR process. However, similar to other SLR researchers, this omission was not systematic [16,18,23].

9 Threats to validity

In this section, we report the validity threats of our study and the corresponding mitigation strategies following the guidelines proposed by [42,58,59].