When measuring practical significance, what is big enough? What is too small? Many guides exist, but most will not apply well to your individual research project. The last post covered how to calculate your effect size. The question still remains: what kind of size are you looking for in your study?

This is a continuation of a previous post on effect sizes. It covers the basics of p-values and effect sizes, along with a separate topic on standardization in effect sizes.

Past guides and problems

Jacob Cohen is most frequently cited for his thresholds for correlation effect sizes: r = .10, .30, and .50 for small, medium, and large effects, with corresponding conventional labels at 0.2, 0.5, and 0.8 for Cohen’s d. These are what you’ll find most prominently in current discussions within UX research (though the authors there also discuss the controversy subsequently).

In later personal communications, Cohen regretted proposing them in the first place. Let’s go back to the paper by Funder and Ozer, the same paper as the previous post. I think they would agree with Cohen. They call such guidelines “meaningless in the absence of a frame of reference.” I agree with Funder and Ozer: we’d be better off doing away with the Cohen guidelines entirely.

According to the discussion by Funder and Ozer, a meta-analysis of correlations in social and personality psychology placed Cohen’s “medium” of r = .30 near the 75th percentile. That is a benchmark just from sub-areas of academic psychology. What do we know about benchmarks in UX measures?

Right now, we don’t have a clear guideline for how small or large UX measures’ effect sizes should be. Sauro and Lewis indicate that they plan to publish some guidelines in the future, which would be a huge boon to the field. With those data in hand, I still think each practitioner in UX research should have some sense of how to draw a line for expected effect sizes in their own context. Contexts differ substantially: panel vs. first-party data, behavioral vs. attitudinal measures, consumer vs. business products, and more.

This is the part of quantitative UX research that is more art than science. We can use numbers, but the numbers ultimately are informed a researcher’s intuition for what matters. Funder and Ozer describe two routes to build this intuition: benchmarks and consequences.

Benchmarks

It is feasible in UX research to get effect size benchmarks, a typical size one might find in their work. As awareness grows of this problem, I hope to see more institutions or researchers follow MeasuringU’s lead by publishing their own data and guidelines. Given corporate confidentiality, this will always be less frequent than I would ideally see. Therefore, we have to get creative at the moment when thinking about benchmarking.

General benchmarks

We can turn to contexts to UX that are still intuitively informative for how people might think about the strength of an effect. Meyer and colleagues published a table in 2001 comparing dozens of well-known relationships on a common correlation scale. No one would argue that coronary bypass surgery is ineffective, but its five-year survival rate only shows an effect size of r = .08. No one is fighting the critical connection between smoking and lung cancer, but having ever been a smoker predicts lung cancer within 25 years at r = .08. Earlier in my career, .08 would have felt like a “small” effect, given how little variance it explains. In real-world terms, this can be a hugely important effect.

While I think this is fascinating, I don’t imagine it will sway stakeholders. “What does coronary bypass have to do with our new product offering?” It’s likely a bit too theoretical for stakeholder communication, but it may help you refine your intuition as a researcher.

Internal benchmarks

The most promising option that is in your control is an internal benchmark analysis. It would take some time, but you likely have the data to create your own distribution of effect sizes. Look at the results in your work and your team’s work, including null and negative findings, to map out what effect sizes you have found. Compare like metrics, populations, and study designs rather than pooling every score into one distribution. This distribution comes from your own study populations, tools, and product context, which makes it valuable. It also is less subject to the publication biases you may see in industry. Internal archives can still be selective about which studies and outcomes get retained, so include all eligible planned results, not just successful studies or favorable findings.

To get even more sophisticated, you could link the estimates in the distribution to subsequent decisions and outcomes. At what threshold did the team act? What happened in later experiments? Establishing causal effects is challenging, but it would be beneficial in the long term if you can pull it off on your team.

If you don’t have access to benchmarks, there is yet another path to assessing how big your effect sizes should be.

Consequences

What if you don’t have time to wait for benchmarks or to create your own? The authors recommend looking at the consequences of the effect. There are two broad ways to do this: put it into real terms, or consider the long-term cumulative process of the effect.

Put it into real terms

Let’s start with making effect sizes concrete. You can investigate the practical meaning of your effect. In other words, you can estimate who is going to succeed or fail on whatever the metric of success is.

The authors recommend binomial effect-size display (BESD)¹ to explain results in concrete terms of how many people or cases would be affected. However, BESD has come under scrutiny for some major flaws². BESD can be an illustrative option when you have only the correlation and not the raw data, provided you make its assumptions clear. When you’re doing your own research, you can usually report the observed outcomes directly.

Instead, it’s best to just report the raw data (conceptually similar to the last post). Suppose a hypothetical randomized comparison found onboarding task completion of 62% on the current design and 71% on the new one. The difference is 9 percentage points³. That 9 pp difference is how you can report it to stakeholders, who tend to be familiar with this type of percentage value. If you haven’t been concerned with formal effect size approaches, there is a good chance you’ve already been doing this anyway, so just keep doing it!

One addition you can also make is something borrowed from clinical research. The number needed to treat is the reciprocal of that 9 pp difference. It converts the percentage-point gap into a tangible count of people.

Number needed = 1 / (difference in proportions) 
              = 1 / 0.09 
              = 11.1, which rounds up to 12 (by rounding convention)

The estimated difference corresponds to about one additional onboarding completion for every 12 users assigned to the new design rather than the current design. Here are a few ways to say that:

Simple, but a bit wordy: For every 12 users who attempt the onboarding task, we estimate about one additional completion with the new design rather than the current design.

Scaled to real volume: At 5,000 onboarding attempts a month, the new design would yield about 450 additional completions (if the estimated difference holds at that volume and in that population).

Traffic-based: Using these estimated rates, reaching the same number of onboarding completions on the current design would require about 15% more attempts.

All of these approaches have their strengths, depending on your stakeholders and context. Compared to Cohen’s d, they make the outcomes quite tangible and, therefore, better for decision-making.

As a final note, if you have continuous data, you can use the common language effect size⁴. I won’t spell it out here, but you can use this formula to achieve similar reads on the concrete effects of your models.

Go here for all the math in this section

The same effect over many occasions

Note: while I was writing this post, I found a paper published in July that directly affects this section, Effects May Both Increase or Decrease in the Long Term: A Statistical Illustration. This adds an important qualification to what I had already written: repeated observations do not guarantee a larger long-term effect. Long-term effects can weaken even without habituation. If the value of an effect rests on its long-term consequences, we need to examine those consequences over time. It’s cool to see science happening in real time! So read on with this caveat.

The last piece is more of a concept than a calculation. A small effect size may be hugely influential because the small effect may happen many times. Funder and Ozer give a few examples, but we already looked at a good hypothetical in the first effect size post we can reuse.

In the hypothetical, a new version of a support-system task yielded a 17-second lower completion time on average. In UX research, we naturally scale up our thinking already. We know we’re not dealing with a single 17-second decrease. In the first post’s example, support employees perform the task 100 times a day across 250 workdays. If the time saving persists across those executions, it could save about 118 hours a year.

This example was handy because the number of task executions and the time horizon would be straightforward to calculate. In some contexts, this may take more estimation or guesswork.

Perhaps you measured trust in a checkout flow. Getting the exposure and frequency would require linking trust to actual checkout behaviors. Or perhaps you look at how connected someone feels to their friends on a feed. You can quickly estimate how often a feed is checked, but the moments that relate to more or less connectedness would have to be considered in your calculation. In any case, I think this is a useful way to reason about potential consequences. An r = .05 satisfaction effect may matter differently depending on how many users encounter each feature and how often.

The last consideration for repeated exposures in your effect size is how those exposures stack (or don’t), as detailed in the caveat above. Human perceptual processes can habituate, novelty effects can wear down salience, or a ceiling could be reached for the measurable effect. For your own rigor, consider (or better yet, measure) how the effect you are measuring would change over time for the user.

Begrudging guidelines

Funder and Ozer strongly argue against generic thresholds, but they do propose some in conjunction with their main recommendations. I think this is worthwhile to review these, given we already have a widespread generic set of thresholds (à la Cohen), and those existing guidelines are poor by Cohen’s own admission. Their guidelines can still serve as a rule of thumb that is more useful than Cohen’s, because they consider a broader set of empirical data and incorporate their thinking around single and repeated events.

• r = .05 is very small for a single event and potentially consequential over many.
• r = .10 is still small per event and more consequential in the aggregate.
• r = .20 is medium and useful even in the short run.
• r = .30 is large and potentially powerful in both the short and long run.
• r = .40 or above in psychological research is likely a gross overestimate that will not survive replication or a larger sample.

You’ll notice that r = .40 is smaller than the “large” size from Cohen’s guidelines, yet Funder and Ozer urge skepticism about unusually large estimates in the psychological research they discuss. This shows how different both sets of guidelines are in practice, and why I think it’s worth updating what you use.

Producing stable estimates of effect size

Small effects may be more important than we previously thought, but they do require careful estimation. The standard sample sizes for UX research are greatly changed when we want to estimate an effect size of a certain magnitude rather than just general uncertainty of a point estimate.

For certain UX contexts, large samples would be hard to achieve, so it’s important to be realistic about what effects you can expect to detect and how precisely you can estimate them. In my next post, I’ll discuss testing results against a threshold and planning a study around that decision.

Wrap up

There are two ways to judge whether an effect is big enough. Benchmarks give you a distribution to compare against. However, published UX data is sparse right now. You can use effect sizes from unrelated contexts to calibrate your own intuition. You can also take a more hands-on approach by doing an internal review to benchmark your existing data.

Investigating the consequences gives you a way to state the effect in terms of people and outcomes your team cares about. This can be done by looking at raw differences in percentage points and using an analog to “number needed to treat.” It’s also useful to consider whether the event being measured happens once or many times over a period of time. A repeated effect can have greater total consequences, provided it persists across relevant exposures.

For all of what we’ve just discussed, we are getting the number and then deciding what is big or small. The problem here is that it’s easy to say the number is or isn’t big enough based on what you want the result to be. How do we resolve that? On the surface, it’s quite simple. We need to choose the “big enough” number, an action threshold, beforehand. More to come!

Appendix

1. Here is a separate hypothetical BESD illustration, not the onboarding comparison above. Take 200 people, split them into two groups of 100, and assume half succeed overall. Under these balanced margins, r = .30 corresponds to 65 successes and 35 failures in one group, versus 35 successes and 65 failures in the other: 65% versus 35% success. The .30 correlation explains 9% of the variance in this constructed table. Those success rates illustrate BESD under its assumptions; they are not observed rates inferred from an arbitrary correlation. ↩︎
2. See Hsu (2004), Biases of success rate differences shown in BESDs and Thompson & Schumacker (1997), An evaluation of Rosenthal and Rubin’s BESD. The usual balanced BESD assumes equal-sized groups and 50% overall success. Its illustrative success-rate difference can misrepresent observed outcomes when the underlying variables or base rates do not match those assumptions. ↩︎
3. It’s important to make sure you’re talking about the right kind of percent: relative, absolute, or net/point. ↩︎
4. For two independent groups with normally distributed outcomes and equal population variances, CL = Φ(d / √2), where Φ is the standard normal cumulative distribution function. In plain terms, CL is the probability that a randomly chosen observation from one group exceeds a randomly chosen observation from the other. This conversion is not universal for continuous data. With raw observations, a rank-based probability of superiority can be estimated without assuming normality, with an explicit convention for ties. ↩︎