Brief overview of the replicability of studies

The publication in high-impact journals of a large number of questionable scientific studies heightened a controversy over the quality criteria of scientific production that was already pre-existing (^{Gelman & Loken, 2014}). These controversies in psychology have stimulated conversations about the nature and quality of psychological research. (^{Makel et al., 2012}). Therefore, a large number of psychologists are concerned about a possible excess of false positives in the scientific literature(^{Maxwell et al., 2015}) and this concern is not recent (^{Agnoli et al., 2017}; John PA ^{Ioannidis, 2005}). ^{Simmons et al. (2011}) would affirm that the probability that a researcher finds evidence that there is an effect is higher than the probability of correctly finding the evidence that it does not, which leads to the conclusion that there is a crisis of confidence in psychological science (^{Pashler & Wagenmakers, 2012}) which has generated a repeated discussion about the importance of replication (^{Earp & Trafimow, 2015}).

To a large extent, the credibility of a science depends on the possibility of replicating its findings, that is, reproducing the effects that are reported in original studies. In this sense, replicability becomes an essential procedure and one of the criteria used in scientific research to guarantee the validity of knowledge. (^{Blanco et al., 2018}). For example, a replication experiment that demonstrates that the same results can be obtained elsewhere or with another researcher is conceived as the operationalization of objectivity, and although replication is of great importance(^{Schmidt, 2009}), replication studies rarely appear in psychology journals (^{Association for Psychological Science, 2013}), since only a minimum percentage (1.07%) corresponds to replicated studies (^{Makel et al., 2012}). This situation is due to a lower assessment of replication studies compared to original (or novel) studies by the editorial board of the journals, generated by an emphasis on novelty (^{Stevens, 2017}).

In view of this situation, one of the largest projects that at- tempted to quantify replicability problems in psychology was the Reproducibility Project: Psychology (^{Open Science Collaboration, 2015}) where they tried to replicate 100 studies drawn randomly from three of the most prestigious journals, of which 97 reported significant results. However, only 36% of the replication studies did. Furthermore, the magnitude of the effect of the aftershocks was half that observed in the original studies. These results fueled the debate on the conclusions of the low rate of replicability in psychology (^{Protzko & Schooler, 2020}), showing that the replicability rate in social psychology is only 25% and in cognitive psychology 50% (^{Open Science Collaboration, 2015}). On the other hand, although clinical and school psychology have not been included very frequently in studies and discussions on replicability and are even considered “isolated”, it is unlikely that they are immune to this crisis.(^{Tackett et al., 2017}). Even in view of the heterogeneity of the educational systems in the different countries, it is highly probable that the replicability rate is even lower.

The results of the replicability studies have not been encouraging, however, the authors themselves point out the need to carry out these studies in other contexts in order to determine the influence that could exist due to cultural differences.

Types of replication

In previous paragraphs it was indicated that replicability is the cornerstone of science (^{Open Science Collaboration, 2012}); However, this does not coincide with the number of replications observed in the literature, and although in recent years researchers have shown more and more concern to know if the findings published in the literature are actually replicable (^{Świątkowski & Dompnier, 2017}), the researchers themselves do not conduct replication studies. This situation opens up a series of questions: What are the reasons why, despite knowing its importance, a considerable number of these studies are not reflected in the scientific literature? and an even more important point, how can it be possible to promote the development of more replication studies?

It is important to know that a replication study is a procedure used to verify the veracity and accuracy of reported results. In a review of the literature it can be found that several types of replication were identified. In psychology, they are classified into direct replicas and conceptual replicas (^{Schmidt, 2009}). When researchers repeat all relevant aspects of an original study and try to be as similar as possible to this one, it is defined as direct replication, which must be distinguished from conceptual replications, which are those that seek to test the hypothesis of previous studies using a different research design (^{Makel et al., 2012}). Direct replication is rare (^{Patil et al., 2016}) and although conceptual replicas can be found occasionally, they often lack a systematic approach (Schmidt, 2009). This has led to the conclusion that there is a replicability crisis in psychology (^{Anvari and Lakens, 2018}), which has prompted a process to review the scientific evidence to generate the opportunity to reconsider some current practices that could be part of the origin of this crisis.

What prevents the replication of studies?

Some of these causes have been mentioned by various authors, many of whom have agreed that there are individual factors(^{Blanco et al., 2018}), decision-making or degrees of freedom of the researcher(^{Gelman & Loken, 2014}) as well as contextual factors, among which are included the pressure of some journals for the publication of positive studies, as well as the competitiveness and need for novelty in research and the lack of acceptance and adequate incentives for replicability studies (^{Blanco et al., 2017}; ^{John et al., 2012}; ^{Patil et al., 2016}; ^{Świątkowski & Dompnier, 2017}).

Starting with individual factors, some studies connect this cri- sis with the serious flaws of the null hypothesis significance tests (NHST; ^{Cumming, 2014}; ^{Savalei & Dunn, 2015}), identifying as an underlying cause the dependence that would still have on these significance tests (^{Collins & Tabak, 2014}), to the imperative of achieving statistical significance and the adoption of important search behaviors from various fields, selectively highlighting positive results(^{Brunner & Schimmack, 2016}). For example, one of the things that happens in this regard is that researchers can become victims of confirmation bias and focus only on positive hypothesis confirmations, since by analyzing the data, researchers could report results that confirm their hypotheses while ignoring others who don’t (^{Bakker et al., 2012}; ^{Bakker and Wicherts, 2011}). These practices are related to the behavior and decision of the researchers themselves; However; Is it just a publication bias at the individual academic level? The refusal by the researchers to send null results for publication in addition to being able to signify conflict with the theoretical models, represent for them the belief that they will not be published (^{Ferguson and Heene, 2012}); which would relate this individual factor to a publication bias, and the rejection of null results.(^{Świątkowski and Dompnier, 2017}; ^{Wagenmakers et al., 2011}; ^{Wetzels et al., 2011}).

Likewise, it is also possible to observe that in some cases the hypotheses “emerge” as the data are explored instead of being formulated a priori, analyzing the data as necessary until the positive result is found to be published. This practice is known as p-hacking (multiple undisclosed tests without ad- justment;^{Wicherts et al., 2016}) that generate an excess of re- ports of significant results (^{Stevens, 2017}). These practices are disclosed in the scientific literature as questionable research practices (PIC; ^{John et al., 2012}; ^{Protzko & Schooler, 2020}) that weaken the credibility and reproducibility of research results are very common today (^{Agnoli et al., 2017}) and it brings with it the possibility of losing objectivity, maintaining theories in the face of inadequate tests, and therefore distorting the real scientific evidence.

Therefore, the statistical power would then be recognized as one of the reasons why replication studies fail to find the original effect, coupled with the fact that several studies have dis- covered direct and indirect evidence of the use of PIC among psychologists, such as biases selective in the information of methods and results, information of exploratory analyzes as confirmatory and generalized errors of portability of statistical results, demonstrating that the prevalence of PIC in different countries that include up to ten of these practices reaches more than a quarter of the all those surveyed in the studies, and those who have used a PIC more than 80% (^{Agnoli et al., 2017}; ^{Fiedler & Schwarz, 2016}; ^{John et al., 2012}). For this reason, a possibility of generalization arises. However, it would be necessary for more studies to be carried out to determine the extent of these practices, especially in Latin American countries.

In addition to the individual factors, contextual factors are added that currently continue to respond to the lack of replications in psychology, but above all to the absence of replication studies, and the presence of the PICs already mentioned. Is there currently pressure to publish positive and novel results? Several authors over the years have affirmed this and it is that the systematic publication of studies that depend on their result, and where the published articles only include confirmatory results of the hypotheses presented, represent 90% of the volume of publication in psychology (^{Blanco et al., 2018}; ^{Fanelli, 2010}). Statistically significant positive results are published more easily than statistically non-significant negative results, this problem is known as the “file drawer”(^{Fanelli, 2012}; ^{Ferguson and Heene, 2012})and it is related to making decisions based on the p-value, one of the limitations of the NHST procedure mentioned above. A significant p-value leads to the rejection of the null hypothesis, however, a non-significant p-value does not lead to the acceptance of the null hypothesis, which makes it inconclusive and difficult to interpret results.(^{Cohen, 1994}); this being a reason why reviewers may prefer positive results and for them more conclusive.

If there is a pressure to publish positive results and a publication bias against negative results; researchers will be able to focus their interest more on obtaining “successful” results (^{Giner-Sorolla, 2012}), although this involves the use of PIC (^{John et al., 2012}; ^{Schimmack, 2012}; ^{Simmons et al., 2011}), and set aside negative results; With all this, it would not be surprising if the publication of null results decreased over time (^{Fanelli, 2012}). In addition, not only the significant results, but those that are considered novel are the ones that are more easily published. (^{Klein et al., 2014}). This publication bias represents the direct distortion of scientific knowledge, all results regardless of their additions are relevant to science, provided they have been produced by solid methods (^{Fanelli, 2010}). Is it important to publish negative results? Some authors mention that they represent 50% of knowledge, therefore their level of importance would be equivalent to that of positive results (^{Culebras, 2016}); and the fact is that its disclosure would represent an increase in the knowledge of other researchers, by making it easier for them to know the negative results of research in the field of interest; it would also provide an optimization of expenses in human and material resources since in many cases it would mean that researchers can make modifications in their studies if what has already been discovered were disseminated (Tárraga and Rodríguez, 2016); Furthermore, if negative results are systematically omitted from the scientific literature, it is distorted and prevents its clarity for future studies or procedures such as analytical tests.(^{Świątkowski and Dompnier, 2017}). Added to this, if there is an emphasis on novelty, a cumulative science construction is prevented; if journals today focus their acceptance criteria on studies with novel information, the importance of replications is neglected (^{Makel et al., 2012}) despite the fact that the research community would collectively benefit if re- searchers made replicas (^{Koole and Lakens, 2012}), the current situation of what can be published introduces biases that prevent researchers from taking the risk of taking this path.

To exemplify this situation, the 45 Latin American psychology journals included in the Scimago Journal & Country Rank for the year 2019 were reviewed and it was found that a considerable percentage (around 40%) discourages contributions in the form of replies, including as part of their evaluation criteria the theoretical and empirical novelty and originality in the studies that aspire to be published. Likewise, around 55% of these journals do not include information in their editorial evaluation criteria about replications, thus granting minimal chances of acceptance. Finally, only a small percentage leaves open the possibility of receiving replicate studies.(^{van Dijk et al., 2014}).

Therefore, if these journals prioritize novelty, researchers will maximize their number of publications by concentrating on finding novel results.(^{Higginson & Munafò, 2016}) Leaving out replication jobs, since they could risk immediate rejection if they submit these types of studies, so they are unlikely to be motivated to ever run them.

Why is this a problem?

The replica is a mechanism that allows to verify empirical experiences and knowledge exploration that enrich the different approaches to a problem (^{Ordoñez-Morales, 2014}). If there is a crisis of replicability in psychology, not only is a process of adaptation of original designs limited, but also the new learning spaces that these can generate, as well as critical analysis, rethinking of problems and results that in greater Sometimes they are presented as the last word that other researchers, psychologists and students must accept unconditionally.

For this reason, the lack of replicability represents not only one but several problems in the different areas of psychology, the absence of this criterion obstructs the verification of the evidence. In clinical psychology, should a therapy or intervention that is not empirically supported be applied? The answer could be no, however, if you only take into account the positive results of an investigation that concludes the validity of a procedure or method, and thereby ensure its full validity, you are going against the advancement of psychology.

In previous paragraphs it was mentioned that clinical psychology and related areas have been little participants in the replicability discussions, representing a loss of opportunity for clinical scientists and the field in general, and although possible causes of this isolation have been attributed, has emphasized the concern that exists for this, given that currently the critical research evidence would not have a greater rigor, for example, the criteria of empirically supported therapies (ERT) in psychological interventions for specific disorders, only require two positive results to give this support, regardless of the number of negative results that may be obtained; in the context of negative results, these positive results may be attributable to chance(^{Tackett et al., 2017}). So, if this information reaches other professionals as endorsed, they will choose to use it in their various interventions without a verification of procedures, methods or theories. However, clinical scientists are likely to have greater difficulties in implementing changes related to the points mentioned above, clinical research would benefit from the implementation of recommendations to improve replicability in the field by al- lowing their progress with empirically supported evaluation and intervention.

Likewise, in the field of psychometry it is generally accepted that a single instrumental study concludes that the instrument studied has sufficient metric qualities, without considering that obtaining validity evidence is a continuous process and various studies are required to consolidate the findings (^{American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014}).), Since it is possible that, for example, the internal structure found obeys to a greater extent the characteristics of the sample, but does not represent the construct. In addition, the use of outdated or inadequate procedures is also common (see ^{Lloret, Ferreres, Hernández, & Tomás, 2014}), so additional studies would serve to contrast and refute these findings (see Dominguez-Lara & Navarro-Loli, 2018).

In many cases, the results of research can fall into the points mentioned above, and be overestimated in magnitude, which implies the need for replicability studies that allow greater rigor to the research evidence.

What alternatives are there?

Some authors responding to the factors that contribute to this so-called replicability crisis describe methods and recommendations for this to change. Among these proposals is to implement statistical approaches that emphasize the estimation of parameters instead of the NHST (^{Asendorpf et al., 2013}). In this sense, Bayesian statistics are presented as an alternative to remedy the bias against publications of non-significant results. (^{Świątkowski & Dompnier, 2017}). Currently, more journals are added to require the approach of statistical power and sample size, allowing the use of new approaches such as Bayesian statistics and meta-analysis.

Consideration should be given to restructuring the incentive to replicate publications, since if the replicas remain unknown to the scientific community, researchers will not be motivated to carry out this type of study. Therefore, there must be outlets so that they can publish, either with separate space for replications in scientific journals or open access to the data as online material together with the original studies. (^{Asendorpf et al., 2013}; ^{Koole and Lakens, 2012}). Likewise, a meta-analytic view of scientific knowledge could be considered (^{Cumming, 2014}), considering that the value of a publication is many times determined by the amount of citations it can get (^{van Dijk et al., 2014}), since quoting an original article could make a cocitation of the replication file; this could mean that at the same time as replicas get more citations, the original studies also benefit in the same way (Koole and Lakens, 2012).

Finally, the development of editorial policies that generate access to pre-registration and open data should be encouraged; which includes pre-registering hypotheses, designing, analyzing and making the collected data open access (^{Gelman & Lo- ken, 2014})for example; archiving data, scripts, analysis and a description of them in a public way, would allow full access to other researchers and, therefore, facilitate replication. In addition, reproducible publishing workflows should be considered, allowing for process monitoring, and providing researchers with a space to store IRB materials, data, analysis scripts, etc. (Nosek et al. 2012); such as the Open Science Framework, which pro- vides research with a hypothesis pre-registration platform and collection and analysis using anonymous data available in public mode after the study, which allows the materials to be available to other researchers (^{Tackett et al., 2017}). If the journals implemented these spaces, it would prevent the initial research hypotheses of the studies from being modified at their convenience, the ICPs among researchers would be reduced, and the rate of false positives in the scientific literature would be lower. It should be considered that the registry can be the most ac- curate, transparent and reliable; however, the evidence shows how far we are from this ideal, and continues to provide studies that cannot be reproduced or evaluated; if the registration of a study is not adequate, despite this being very good it becomes unusable (^{Zúñiga, 2019}). As a possible solution to this problem, it is important that authors and editors rely on tools that guide the writing and publication of scientific papers, as indicated by the EQUATOR network (Enhancing the Quality and Transparency of health Research), which will allow improvement the reliability and value of the research literature. The EQUATOR network is an international initiative established to promote high-quality reports on health research studies (^{Simera et al., 2010})and pro- vides tools for authors including reporting guidance resources; guidelines for planning, writing, ethical considerations, sharing and publishing data. It also contains tools for peer reviewers evaluating research manuscripts, as well as for editors considering implementing policies that help improve the accuracy and integrity of reports; and value a content that guarantees the result of a transparent, reproducible and quality investigation. Some journals have taken steps to improve the quality of re- search they publish (^{Ioannidis et al., 2010}), thus contemplating the possibility of publishing replicas or null results; for example Adavances in Methods and Practices in Psychological Science has incorporated a section destined to compile replication studies; The new type of article, Registerd Replication Reports, aims to strengthen the foundation of psychological science by encouraging the publication of replications based on a shared and vetted protocol.

CONCLUSION

For research in science in general replicability is a prerequisite for obtaining valid conclusions, so it is necessary to incentivize these studies in psychology, and although there are barriers and challenges to implement them, and even more so in the field of scientific publications, these changes will be part of an improvement in psychological science as it would improve its quality and reputation as confidence for current publications increases. It is important to discover, but equally so is replicating, and in this way psychology could be regarded as a more rigorous and solid science.