Skip to main content

15  Simulations

What is a simulation study? We can think of a simulation study as an experiment using synthetic data, generated from a know (reproducible) data generating mechanism. The experiment is designed to answer a specific questions, for example:

A simulation study can become very complex. As with other areas, it is a good idea to start with a minimum setup and build from that. Below, we introduce the basics!

15.1 How to do simulations studies?

R has a set of functions used to generate (pseudo) random numbers, such as rnorm(). The rnorm() function samples data from a normal distribution. Similarly, functions listed in the table (Table 15.1) uses other distributions as a source for simulating data.

Table 15.1: Functions for generating pseudo-random numbers in R.
Function Distribution Data type Example use
rnorm(n, mean, sd) Normal Continuous, unbounded Body mass, strength, change scores
rlnorm(n, meanlog, sdlog) Log-normal Continuous, positive, right-skewed Reaction times, biomarker concentrations
rt(n, df) Student’s t Continuous, unbounded, heavy tails Data with outliers
runif(n, min, max) Uniform Continuous, bounded Random allocation, bounded measurements
rgamma(n, shape, rate) Gamma Continuous, positive, skewed Waiting times, costs
rexp(n, rate) Exponential Continuous, positive Time to event
rweibull(n, shape, scale) Weibull Continuous, positive Survival / time to failure
rbeta(n, shape1, shape2) Beta Continuous, proportion (0–1) Percentages, fractions
rbinom(n, size, prob) Binomial Count out of fixed total (binary if size = 1) Successes/failures, yes/no outcomes
rpois(n, lambda) Poisson Count, unbounded Number of events, repetitions
rnbinom(n, size, mu) Negative binomial Count, overdispersed Counts with variance > mean
rgeom(n, prob) Geometric Count (trials before first success) Attempts until success
rmultinom(n, size, prob) Multinomial Categorical counts Counts across several categories
sample(x, size, replace, prob) Discrete (user-defined) Categorical / ordinal Group allocation, Likert-type responses
MASS::mvrnorm(n, mu, Sigma) Multivariate normal Continuous, correlated variables Repeated measures, correlated outcomes

15.1.1 A simple example

We want to simulate data from a normal distribution, consider the model \(y \sim \operatorname{Normal}(\mu, \sigma)\). We read this as, y is distributed as Normal, with mean μ and standard deviation, σ. We can add some numbers, let’s say that \(\mu = 0\) and \(\sigma=1\). Using these parameter values we can use rnorm to simulate a data set.

d <- data.frame(y = rnorm(10, 0, 1))

The data frame saved in d contains (pseudo-) random numbers from a normal distribution, according to our specification.

d
            y
1  -0.8492836
2   0.8521449
3   0.4854341
4   0.1280756
5   0.2604764
6   1.1291843
7  -0.9970583
8   0.4227155
9   0.8027113
10 -0.2690905

We can expand the model to include a linear predictor (Equation 15.1).

\[ \begin{align} y_i &\sim \operatorname{Normal}(\mu_i, \sigma) \\ \mu_i &= \alpha + \beta x_i \\ x_i &\sim \operatorname{Uniform}(-1,1) \\ \alpha &= 0,\quad \beta = 0.2,\quad \sigma = 1 \end{align} \tag{15.1}\] for \(i = 1,\dots, n\)

The model can translated to code by first setting the size of the sample (n), sampling a predictor variable and setting the parameter values.

# The number of observations
n <- 5000
# Sampling the predictor variable 
x <- runif(n, -1, 1)

# Parameters
alpha <- 0
beta <- 0.2
sigma <- 1

# Generate data
y <- rnorm(n, alpha + beta * x, sigma)

# Save the data
d <- data.frame(x, y)

The

head(d)
           x          y
1 -0.8612270 -1.5739348
2  0.1857432 -0.1673838
3 -0.5649543 -0.8876441
4 -0.8354950  0.5089536
5 -0.8582871 -0.1210711
6  0.4973516  0.2366930

The generated data can be used to see if we can retrieve the simulated effect:

# The model
m <- lm(y ~ x, data = d)

# Estimated parameters
ahat <- coef(m)[1]
bhat <- coef(m)[2]
shat <- sigma(m)

paste("Intercept: ", round(ahat,3))
[1] "Intercept:  0"
paste("Slope (x): ", round(bhat,3))
[1] "Slope (x):  0.2"
paste("Residual variation: ", round(shat,3)) 
[1] "Residual variation:  0.99"

15.1.1.1 Power analysis through simulations

Let’s say we want to determine the sample size for a observational study. Participants are either in group A or group B, we want to determine if strength is associated with group. We want a study design (number of participants recruited) that will find a small effect (\(\beta = 0.2\sigma\), where \(\sigma\) is the within group standard deviation).

# Set a seed for reproducibility
set.seed(1)

# Total sample sizes (split equally between groups)
n <- c(50, 100, 200, 400, 800, 1200)

sim_fun <- function(n, alpha, beta, sigma) {

  # Balanced groups, coded 0 (A) and 1 (B)
  x <- rep(0:1, length.out = n)
  y <- rnorm(n, alpha + beta * x, sigma)
  data.frame(x, y)

}

sim_analyse <- function(d) {
  
  # Fit the model
  m <- lm(y ~ x, data = d)
  # get the p-value
  pval <- coef(summary(m))[2,4]
  # return the p-val
  return(pval)
}


iter <- 1000
results <- data.frame(sample_size = rep(NA, iter * length(n)),
                      pval = rep(NA, iter * length(n)))

for(k in seq_along(n)) {

  for(i in 1:iter) {

    # Add a row counter
    row <- (k - 1) * iter + i
    d <- sim_fun(n[k], alpha = 0, beta = 0.2, sigma = 1)
    results$sample_size[row] <- n[k]
    results$pval[row] <- sim_analyse(d)

  }

}


library(tidyverse)
# Power: proportion of p-values below 0.05 per sample size
results |>
  mutate(sig = if_else(pval < 0.05, 1, 0)) |> 
  summarise(.by = sample_size, 
            power = sum(sig) / n()) |> 

  ggplot(aes(sample_size, power)) + geom_line() + 
  geom_hline(yintercept = c(0.8, 0.9)) + 
  annotate("text", 
           y = c(0.83, 0.93), 
           x = 10, 
           label = c("80%", "90%")) + 
  labs(x = "Sample size", 
       y = "Power")

The above serves an illustration of simple example.