Glossary
Multivariate testing
What is multivariate testing?
Multivariate testing compares combinations of variants across several design elements to estimate their effects on an outcome.
How it works
A full factorial test of three headlines, two images and two button colors has twelve combinations. A/B testing can compare whole designs that differ in several elements; it is not restricted to one changed element. Factorial designs can estimate interactions when the design and sample support them. Fractional designs use fewer combinations with additional assumptions and limits.
Applied example
A product-page experiment could test whether a button-color effect differs by headline. Does the button help only with one headline, or across all three? That is the interaction question. A simple sequence of isolated changes may miss interactions, but alternative experimental designs can also investigate them.
Why it matters
Multivariate testing helps study combinations and interactions. It does not guarantee a global optimum, and the number of combinations affects the data needed.
Evidence note: Aurora Harley: Estimated effects depend on the tested combinations, sample and analysis.