Multivariate Testing (MVT)

What is Multivariate Testing (aka MVT)?

Unlike A/B testing, the goal of MVT (multivariate testing) is to determine not which page, but which combination of variations is the most effective in improving the target metrics. The number of variations can vary, but you can test elements like the pictures on a page, the buttons, the headlines, the fonts, and you can create two or more variations for each element, then test all the combinations to find the winning one.

Multivariate testing requires more time and traffic than A/B testing, as you have to prepare multiple combinations of variables, but the results can reveal more information about each change and about the impact it has on conversions or other metrics. 

Doing MVT experiments will draw the macro picture of what visitors like more and what type of design makes them convert. Out of all possible combinations, MVT points out the ones that perform better and score higher conversion rates. Multivariate testing needs more traffic on site than A/B Testing and it doesn’t work very well for new companies and beginners.

A/B or Multivariate Testing?

Depending on your marketing goals, you may have to choose between implementing A/B testing or Multivariate testing. Here are the main pros and cons to take into consideration before deciding which way to go:

MVT Advantages

The main advantage of an MVT is that it allows you to test different combinations of elements on the same page in order to see which one gets better results. 

MVT Disadvantages

• Since a multivariate test involves many possible combinations, the high amount of traffic needed for the experiments to be relevant is a big downside for MVT. Compared to an A/B Test, where most of the time the traffic is split in two, in an MVT you can split the traffic in 3,4,5, etc. variations, depending on the number of combinations tested. 
• Due to this high amount of variations, multivariate tests also tend to have a higher amount of false positives. This is something to keep in mind. 

A/B Testing Advantages

• A/B Testing works really well when testing different layouts of the same page.
• It’s easy to implement and track.
• It does not require vast amounts of traffic. 

A/B Testing Disadvantages

In a single A/B Test, you can only change one variation at a time and it might take a while to test all the elements you want on a web page.

Multivariate Test Example

This is an example of how you get the total number of versions when you run a multivariate test.

Variables:

• image placement
• headline font 
• headline color

A number of versions for each variable:                                                                            

• Image Placement: Left side of the button, Right side of the button
• Headline Font: Arial, Calibri
• Button color: Blue, Grey

The total number of versions to be tested:

• Left, Arial, Blue
• Left, Arial, Grey
• Right, Arial, Blue
• Right, Arial, Grey
• Left, Calibri, Blue
• Left, Calibri, Grey
• Right,  Arial, Blue
• Right, Calibri, Grey

Multivariate Testing Statistics

Statistics show that multivariate testing is considered highly valuable by most companies that are focused on optimizing the conversion rate. A study made by Econsultancy and Red Eye showed that, in 2011, Multivariate testing was considered more valuable than usability testing and customer journey analysis. Though, not so many companies use multivariate testing as a method to improve the conversion rates on their websites.

When respondents were asked what methods do they intend to use in order to optimize the conversion rate, they placed multivariate testing bellow methods like copy optimization, usability testing or cart abandonment analysis.

What methods do companies intend to use for optimizing the conversion rate

The most valuable methods considered by companies valuable for conversion rate optimization.

So, if you want to make multivariate testing work for your conversion rate optimization strategy:

• make sure that you have enough traffic on your website: a large number of versions tested needs a proper number of users;
• you will test subtle changes on your website, you will not make a complete change of its design; don’t forget that the role of MVT is to get more detailed information concerning your users/customers behavior;
• run a test that makes a difference; use multivariate testing on the most important pages on your website; e.g.: a landing page where you test the length of a form combined with different shapes or copy for headers and footers.

FAQs