The most commonly reported statistic in research papers may also be the most misunderstood and misused. We’re talking about the p value.
Recently, the American Statistical Association (ASA) released the “Statement on Statistical Significance and P-Values,” outlining six principles pertaining to adequate use and interpretation of p values. The utter statement is available here.
Let’s take a look at the ASA principles, and how they apply to research.
1. P values can indicate how incompatible the data are with a specified statistical model.
Here, the significant word is “specified.” Recall that in any examine or analysis, the researchers are trussed to have made certain assumptions when creating a statistical model. According to statisticians, a p value of 0.05 does not mean that there is a 95% chance that a given hypothesis is correct. Instead, this value means that if the null hypothesis is true and if all other assumptions made are valid, we have a 5% chance of obtaining a result at least as large as the result presently obtained.
Two. P values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
Researchers often erroneously interpret smaller p values to mean that the null hypothesis is false. Actually, p values only indicate the probability of obtaining results at least as large as those observed if the null hypothesis was true.
Trio. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
“P < .05” is not a ensure that something is true. Ultimately, a p value is just a statistic and not a sign from heaven. A p value can be influenced by many aspects of a probe, especially sample size. If a sample is particularly large, the p value may almost always be significant (tho’ the effect size may be negligible), unless there is absolutely no effect. Hence, it’s common sense that you can’t make a practical decision on the basis of a p value alone.
Four. Decent inference requires utter reporting and transparency.
Often, the only results reported are those with p value below .05. The ASA strongly discourages this kind of “cherry picking.” Instead, it recommends that authors report all hypotheses explored, all statistical analyses conducted, and all p-values obtained, whether significant or not. Only then can authors draw valid conclusions on the basis of their data.
Five. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
Some authors label findings with a very low p value (<.001) as “very significant” or “highly significant.” However, a low p value does not mean that the result is of practical or clinical importance.
Let’s assume you’ve found a statistically significant relationship inbetween enlargened energy drink consumption and positive figure picture in women. This doesn’t mean that you should design an intervention to improve bod picture in which damsels are given free energy drinks! Instead, you should look at the strength of the relationship (e.g., correlation coefficient, regression coefficient). If the relationship is powerless (e.g., a correlation coefficient of just 0.1), your intervention will most likely be more effective if you consider other factors that have a stronger relationship with bod pic (e.g., general self-esteem, frequency of fat talk).
It’s also significant to consider context when determining the importance of a result. Petite differences inbetween large groups can be statistically significant but unimportant practically, while large differences inbetween petite groups can be significant in practical terms even if they are not statistically significant. For example, a mean increase of 1.Five points in scores on a 100-point math test after an educational intervention may be statistically significant, but the intervention itself may not be particularly beneficial or useful in real life.
6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
Authors should avoid reporting just p values in their results. A puny p value doesn’t indicate that the null hypothesis is false, nor does a large p value mean that the null hypothesis is true. In research, there could be a multitude of hypotheses that are just as consistent with the observed data. Hence, a p value is not the only form of statistical support for the model or theory being tested, and the value of a probe does not depend solely on the p values found.
In summary, albeit p values can be useful, they are not the yardstick by which a examine becomes valuable and significant, and they should not be treated as such. Statistical significance is not the same as scientific, practical, or clinical significance.