Investigators want to have their work accepted for publication through peer review. Motivation is good if it pushes them to develop their best product, but it can be problematic if it encourages them to focus on relatively unimportant elements just to improve the likelihood of acceptance. One of the problems that can be reinforced by some reviewer and even editor attention is an unreasonable reliance on statistical significance.
There is a fairly common perception that studies describing statistically significant differences are easier to get accepted for publication. This may sometimes be true, but it is based on misplaced logic. The value of hypothesis-driven research is to effectively test hypotheses, not to worry about the direction of findings.
Scientifically, the outcome of a statistical test is probably the least important part of the process. Most of the vital elements come before that point. The critical first step is to pose a reasonable question, one that is meaningful and testable. The concept of meaningful is somewhat subjective, but there must be sound logic behind the question and sufficient interest in the answer to justify the effort. The research question is operationalized by developing it as one or more simple hypotheses that can be tested objectively.
The research design is the next critical step. It must make it possible to answer the question posed, without confounding by uncontrolled variables. At the same time, the design needs to have sufficient relationship with the real world to allow at least some generalizability beyond the sample group studied.
Data collection is achieved through execution of the study design, the step in which a lot of research goes awry. This may be due to inadequate design or the real world compromises that creep into most projects. Subject selection, scheduling, test measures, inter-trial intervals, and a host of considerations specific to each study can be well planned but still compromised in execution.
Once data are collected, the soundness of the analytical procedures and interpretations are crucial. The analytical procedures should have been developed as part of the research design to ensure that appropriate data are captured for analysis, but there is also the need for mindful evaluation and interpretation to ensure propriety. Minor or modest shortcomings can often be mitigated by documentation in the limitations section of a manuscript, but serious shortcomings are more problematic. They can drive the need for additional data collection or alternative analyses, but they can also invalidate the findings if they reach the level of fatal (insurmountable) flaws.
Assuming an appropriate question, a well-designed and executed study, sufficient and valid data, and proper analysis and interpretation, all that remains to make work publishable is sound presentation in manuscript form. This will include having all the right pieces in the right places—introduction, methods, results, discussion, conclusions, and references.
The outcome of statistical testing was not mentioned in the above steps because it should not be a factor in determining publishability. It is the process of getting to the finding that is important. Objectively, any outcome is valid as long as it is appropriately derived. Effectively, determining the absence of a relationship can be as important as determining the presence of one for a meaningful open question.
Authors should not be afraid to submit findings from sound research in which a null hypothesis could not be rejected. At the same time, they should not get excited about statistically significant differences that are not practically or clinically important. Statistical “trends,” differences approaching but not reaching significance, may or may not be considered, depending on practical or clinical importance, the statistical power of the analyzed data, and proximity to the threshold.
Every piece of work has an appropriate context, and this should be based most firmly on the real world relevance of the findings, and much less on arbitrary rules or unimportant findings. Analysis and interpretation are best completed by thinking about research data as the product of someone else, with the intent of minimizing personal bias in the outcome.
Writing up research work appropriately requires mindfulness that all research is imperfect. The goal should not be to make everything seem as positive as possible. Fair documentation of strengths and weaknesses of individual efforts is much more useful to help strengthen future efforts. Limitations should be stated clearly, openly, and generally without apology. They should not be immediately discounted by comments suggesting that they were not really important. Critical readers will often have greater respect for authors who present limitations objectively and make sure that interpretations and conclusions are mindful of them.
Burying flaws is counterproductive for science. If fatal flaws are discovered at any point, the effort to publish research data should stop. There are, though, still ways to make the experience useful. If the flaw is one that others could easily stumble into, a letter to the editor could be of service to the community. On a personal level, putting flawed work on a highly visible, if metaphorical, shelf can help remind investigators and their colleagues and protégés to not make similar mistakes in the future.
Article info
Publication history
Published online: August 07, 2019
Accepted:
June 11,
2019
Received:
June 11,
2019
Identification
Copyright
© 2019 Direct Marketing Educational Foundation, Inc., dba Marketing EDGE. All rights reserved.
