Plan Your Data Analysis in 3 Steps

Back when I was doing psychology research, one of my biggest challenges was having enough data. And while this is probably still true for many experimental researchers, with the internet’s ability to make data sets accessible, it seems data is reproducing faster than rabbits.

Now I see many researchers grappling with overwhelm at managing and analyzing enormous data sets.

Even a moderate number of variables could lead to endless variations on analyses. “Hmm, I wonder if the stem length is correlated to the wind direction. What about this measure of plant size—leaf area? Is that correlated with wind direction?” This can go on forever.

According to Frank Scarpaci, owner of Project Designworks, there is a

“1:10:100 Rule:

Every dollar spent on planning and preparation saves $10 onnproject work or $100 on fixing problems after the project is done.”

And since, even in academia, time is money (or even more precious) planning your statistical analysis will save endless time and frustration. I mean, you’d rather spend an hour now planning the analysis than redoing it in a year after the reviewers rip it to shreds, right?

The best time to plan the analysis is before collecting data. This prevents those (all too common) situations where you realize you needed another variable or you should have measured something on a different scale. Grant applications force you to do this, but every study would benefit.

How do you plan it? You base it on the results you will report. You should already know the questions you want to answer in this study, but having it written down in a list keeps you on track. You do not need to (and should not) answer every question the data could answer in this study.

I find a great outline for a simple analysis plan comes from a brilliant article written by Darryl Bem about writing journal articles. The entire article is excellent (and I highly recommend it), but most helpful for planning is the section, “Presenting the Findings”. This section outlines 7 steps for reporting each finding. For planning purposes, I condense these into three:

1. State the conceptual hypothesis you are asking
2. Restate this hypothesis in the terms of the variables that measure the concept
3. List the statistical test or method that will answer this questio

Simply repeat these three steps for all hypotheses the study is set up to answer. Start with the most general and important, and work down from there.

Doing these three steps before you sit down to analyze has three advantages:

1. It forces you to choose the variables you will use. Choosing early allows you to take unwanted variables from large data sets, making processing time much faster. It also defines the variables on which to conduct univariate analysis, and precisely defines which variables to collect.
2. It discourages you from performing irrelevant analyses, saving time, energy, and frustration as well as making your article clearer and more logical.
3. It makes writing the results section a breeze.

It is certainly true that there is a place for exploratory data analysis and some surprises always pop up—missing data, an unexpected skew, and so on. But getting back on track is always easier if you know the direction in which you’re heading.

For more information, see:

Bem, D. (2003). “Writing the Empirical Journal Article.” In The Compleat Academic:A Practical Guide for the Beginning Social Scientist, 2nd Edition. Darley, J. M., Zanna, M. P., & Roediger III, H. L. (Eds) Washington, DC: American Psychological Association.