Don’t Compute the Statistical Power of Your Experiment…Even if SPSS Allows It And Your Editor Requires It!

A refresher on hypothesis testing

We must first recall the basics of hypothesis testing.

Examples:

• We run an AB test and want to know if the A channel converted at a higher rate than the B channel.
• We test a treatment vs. a placebo and want to know if the treatment improves a health outcome, e.g. reduces blood-pressure.

We are interested in \(\mu_A\) and \(\mu_B\), the means of our groups. We do the following hypothesis test: \[\cases{H_0 : \mu_A = \mu_B, \text{ both groups have the same mean}\\ H_1 : \mu_A \neq \mu_B, \text{ means are different.}}\]

We collect data (using sound methodology, which is way harder than it seems) and want to reject the null hypothesis \(H_0\).

Goal of hypothesis testing: We want our data to convince us that we can comfortably affirm that the means are not equal in both groups, using some statistical test, often the two-sample t-test (see below).

Power

Power depends on:

1. How large the true difference \(\delta\) is
   • A larger difference is easier to detect.
2. How uncertain is our estimate of this difference
   • A less variable estimate (i.e. smaller variances \(\sigma_A^2, \sigma_B^2\) for groups \(A\) and \(B\), respectively) gives us more confidence
3. How much data (i.e. \(n_A, n_B\) data points for group \(A\) and \(B\), respectively) is available in each group to run the test.
   • More data gives us more confidence.

Mathematically, we write \[\mathbb{P}(\text{reject } H_0|H_0 \text{ is false}) = f(\delta, \sigma^2, n),\]

where \(f\) is some function that depends on the test.

In practice, we summarize all this in an effect-size (there are many ways to do this).

Power analysis, the right and wrong ways

Simulation Parameters

You can safely skip these sections and head directly to the plots if you’re not familiar with code or technical probability and statistics stuff.

We fix identical parameters for the two groups: there are no differences to detect with the test.

• \(x_A \sim \mathcal{N}(\mu_A, sd_A),\quad x_B \sim \mathcal{N}(\mu_B, sd_B)\), where
  • \(\mu_A = \mu_B = 0\)
  • \(s_A = s_B = 1\)
  • \(n_A = n_B = 20\)

set.seed(1) #for reproducibility
n_sim <- 10000 #We will simulate a large number of experiments
#normal variate simulation parameters
n_A <- 20  #group A
n_B <- 20 #group B
mu_A <- 0 #group A's real mean we are trying to estimate
mu_B <- 0 #group B' real mean we are trying to estimate
sd_A <- 1 #group A's standard deviation
sd_B <- 1 #group B's standard deviation

Simulation

#t-test critical value, we will reject H_0 when the absolute value of the t-statistic exceeds this threshold
crit <- qt(0.975, n_A + n_B -2) 

#simulation
significant <- c()
t.stat <- c()

for (simulation in 1:n_sim){
  x_A <- rnorm(n_A, mu_A, sd_A) #we simulate our group A
  x_B <- rnorm(n_B, mu_B, sd_B) #we simulate our group B
  diff.means <- mean(x_A) - mean(x_B) #we estimate  the difference in means
  # print(diff.means)
  s <- sqrt((var(x_A)/n_A + var(x_B)/n_B)) #we estimate s
  
  t.stat[simulation] <- diff.means/s #we compute the t-statistic
  significant[simulation] <- abs(t.stat[simulation]) >= crit #we check if the test is significant
}

Simulation Results

From our 10000 simulations, there is no difference to detect between our groups. We therefore expect about \(5\%\) of them to be significant, by definition of testing at the \(5\%\) level.

## [1] "5.02 % of simulations are significant, when the null hypothesis is true."

Power and P-Values

Observed power is then computed for every simulation by using the estimated t-statistics.

#what about power?
observed.power <- 1-pt(crit,n_A + n_B - 2, ncp=abs(t.stat))