Are A/B Tests Reliable for Accessibility

A/B tests are reliable for confirming something that already works, and unreliable for discovering whether an interface is usable or accessible in the first place.

The reason is structural. Most published programmes report that only 10 to 20% of A/B variants produce a positive, statistically significant result, and a test only moves the metric if you already had a good hypothesis. Accessibility and cognitive-load problems rarely announce themselves as a clean hypothesis. You cannot A/B your way to discovering that the error copy is confusing for someone with dyslexia if you never suspected it.

For discovery questions, qualitative behavioural research is the cheaper and faster diagnostic. You watch a handful of real users hit the friction, you see why, and you fix the cause. Then, if you want, you A/B the fix to confirm the lift.

Use both, in that order: behavioural research to find the problem, A/B testing to confirm the solution.