How to check statistical significance of calculation results?

Mathematically significance and reliability can be checked by measuring confidence interval.

Confidence interval — is a range of sample where the true value of the parameter belongs to with some confidence level.
In attribution calculation results, step probability is the parameter.

Confidence interval = z*SQRT((p*(1-p))/n),
where z — confidence level. For 80% confidence level it's 1.28, for 90% — 1.64, 95% — 1.96, 99% — 2.58.
     p — step probability,
     n — count of previous step sessions.

Example

Lets measure confidence interval with 90% confidence level for a step where were 590 visits. There were 1,000 visits on the previous step, i.e. step probability is 59%.

Confidence interval = 1.64*SQRT((0.59*(1-0.59))/1000) = 2.55%.
That means, in 90% of cases the step probability is 59% ±2.55%, i.e. it's in range from 56.45% to 61.55%.

We consider calculation results to be statistically significant if the confidence interval value is not more than 1/10 of step probability value.
In the example below, confidence interval value below 5.9% would be acceptable.

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.