The goal of inference

The goal of inference

The scientific process.

The goal of science is mostly to study some phenomena and then communicate how the world works. This process may include describing a phenomena, doing an experiment, observing specific natural events…

P < 0.05

Surprisingly often, the goal of science seems to be the production of a number, that for some reason we want to keep below 5%!

Everybody in the room is happy when significance is high.

The p-value has been used to produced a new discovery, or proof or even the truth.

The problem with
inference

• All swans are white (P < 0.05)

JJ Harrison (https://www.jjharrison.com.au/) - Own work CC BY-SA 4.0

It turns out that it is not that simple.

The P-value did not presented us the truth. It was just a number calculated from a laborious process that included sampling of data, calculation of summary statistics and P-value.

The P-value might not even mean what we think it means?

What does the p-value mean?

(Svetkey 2008)

Svetkey et al. performed an intervention where overweight or obese individuals that had lost at least 4 kg in a first weight loss phase were randomized to maintain their weight on their own or receive either “personal contact” or “interactive technology” in a follow up phase.

What does the p-value mean?

All groups regained weight after randomization by a mean of 5.5 kg in the self-directed, 5.2 kg in the interactive technology–based, and 4.0 kg in the personal-contact group… Those in the personal-contact group regained a mean of 1.2 kg less than those in the interactive technology–based group (95% CI, 2.1-0.3 kg; P=.008).

What does the p-value mean? Svetkey at al…

1. …have absolutely disproved the null hypothesis (that there is no difference between the population means).
2. … have found the probability of the null hypothesis being true.
3. … have absolutely proved their experimental hypothesis (that there is a difference between the population means).

Questions adopted from (Dienes 2008).

What does the p-value mean? Svetkey at al…

4. … can deduce the probability of the experimental hypothesis being true.
5. … know the probability that you are making the wrong decision, if you decided to reject the null hypothesis.
6. … have a reliable experimental finding in the sense that if, hypothetically, the experiment were repeated a great number of times, you would obtain a significant result on 95 % of occasions.

Questions adopted from (Dienes 2008).

Statistical inference

Scientific studies only occasionally gather all available data regarding some phenomena under study. All possible data are not available to us due to, e.g., practical or economic reasons.

We, therefore, have no direct way of studying all occurances of a phenomena.

Thus, the scientist faces the challenge of concluding something about the population based on a limited set of observations, a sample.

In order to conclude something, we often make some kind of model. A simple case can be the mean of a sample. We can use it to make a statement about the population.

Models can become more complex…

Statistical inference

In statistical lingo, the population has parameters, these are for example the population mean, standard deviation, proportion, all with the N, the number of possible observations.

The sample can be described using a sample statistic such as the sample mean, standard deviation, proportion, all described using the sample size (n)

Collecting a sample from the population is the process of sampling. Using random sampling, we are The process of drawing conclusions regarding the population based on the data/model is called inference.

Drawing inference from a sample

• Participants are recruited to an intervention and randomized to either receive a RED or BLUE pills.

• Systolic blood pressure after the intervention

• Do RED and BLUE pills affect blood pressure differently?

We recruit participants to a trial were they will, by random allocation, receive either a blue or a red pill

We are interested in the systolic blood pressure as a result of the intervention, to be able to say something about the population we want to know if the groups differ?

The results

We have measured BP in both groups after the intervention.

It turns out that we have a difference between the groups. But how can we be sure that is a difference that is due to the different pills? We would like to have a test were we compare our results to what can be expected if there were no differences between the pills.

We can achieve this by random permutations of the observed data…

A permutation experiment

The observed data is re-ordered to create a new data set where differences between groups is not due to the randomization of pills. From these groups, we calculate the average difference between our new groups. This follows by another re-allocation of participants, and a new average difference.

We repeat this process 10000 times.

10 000 Permutation

Most differences, out of our 10000 permutations are close to 0. Some are greater than 2.5…

10 000 Permutation and our result

Our observed difference is 2.5. This is located in the tail of the distribution of average differences. But how unlikely is our results among a collection of randomly allocated participants where the intervention had no effect?

How unlikely is our result if the pills had no effect?

• The average observed difference was 2.47.
• 167 reshuffled averages where equal to or greater than our observed result.
• This represent a small fraction of the reshuffled differences, in fact…
• \(p = \frac{167}{10000} = 0.0167\)

To account for both extremes

• When allocating the pills we did not really know what to expect, we did not account for the direction of the effect in our last test.
• We should compare our result to extreme results in both directions.
• 167 averages were equal to or greater than our observed result, and 169 averages were equal to or smaller than an effect corresponding to the observed in the other direction.
• \(p = \frac{167 + 169}{10000} = 0.0336\)

What is enough evidence for inference?

• We have produced a P-value!

• So far, given our experiment, how do you define the P-value?

• Is the P-value low enough to conclude anything about our pills?

References

Dienes, Zoltan. 2008. Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. New York: Palgrave Macmillan.

Svetkey, Laura P. 2008. “Comparison of Strategies for Sustaining Weight LossThe Weight Loss Maintenance Randomized Controlled Trial.” JAMA 299 (10): 1139. https://doi.org/10.1001/jama.299.10.1139.