Introduction
A key driver of statistical significance is sample size. One issue with statistical significance is that with a large population, you will most likely determine statistical significance (i.e., any difference or any correlation will be significant). The differences between any sample means will be significant if the sample is large enough. However, when conducting real-world research, statistical significance does not always equate to practical significance (or vice versa). You need to ask the question, “Can you use the statistically significant (or insignificant) results in a practical, real-world application?”
Significance tests are not always valid due to faulty data collection, outliers, or other variables that might invalidate your data. You typically use costs, timing, skills, resources, or your research objectives to determine practical significance. Just because your results indicate statistical significance (i.e., the p-value is lower than the significance level (a)), the data might not be valuable for decision-making.
A statistically significant result only determines your evidence against the null hypothesis, not necessarily if the results apply to real-world decision making. Statistical significance does not guarantee practical significance. A few questions to ask when determining the applicability of results is:
How much significance does the difference have statistically and practically?
How will the results affect the business?
Is the difference large enough to be valuable in the organization?
Are the differences between samples big enough to have real-world meaning?
Can you afford to make changes based on the findings?
Can you afford not to make changes based on the findings?
For example, you set your significance level at 0.01. The resulting p-value is 0.015. This p-value is not statistically significant (0.015 > 0.01), and you do not reject the null hypothesis. However, the difference between 0.015 and 0.01 is not excessive regarding your research topic. For practical application, you determine to use the findings (i.e., reject the null hypotheses). The result is not statistically significant, but can still be practical in addressing your real-world research objective.
Practicality
Moreover, the opposite can also be true. Your results can be statistically significant, but not practical. For example, you want to know if targeting college students between the ages of 18 and 21 years will increase sales. Your p-value demonstrates a statistical significance; however, you decide the effort and cost to pursue these potential target customers is not practical. The possibility of a 3% increase in sales does not greatly affect profitability regarding the effort and costs of acquiring these customers.
Conclusion
Even though the data resulted in a statistical significance, it is not practical to pursue. Do not let your statistical results drive your decision-making. You must take a holistic view of the data, along with subject-matter knowledge, intuition, and your exploratory research findings to make practical decisions that will allow you to achieve your research objectives.