Which Multivariate?

The ultimate comparison of split-testing
and multivariate software

The chart below lets you compare the most popular split-testing tools.

	initial set-up cost	ongoing costs	technology compatibility Hover over the text below to see whether the software will be technologically compatible with your website.	a/b or mvt? In an A/B test, your visitor sees Version A of a page or Version B of it. (Note, a test with three page versions would be called an A/B/C test. “A/B/n” is often used as shorthand for a test that has n different page versions.) Multivariate (MVT) tests are simply ones in which you carry out A/B tests (or A/B/n ones) on several independent page elements simultaneously.	type of mvt There are many different ways of carrying out multivariate (MVT) tests: You can show every single possible combination of page elements. This is called “full factorial”. Or you can show only a fraction of them. This is called “fractional factorial”. There are loads of types of fractional factorial test, and they all have complicated-sounding names. (Taguchi is one of them). You can also choose to display losing combinations less frequently. The data can be analyzed by the software in different ways too.	can it be self hosted You may be happy for the software vendor to host the software themselves, or you may have reasons why you’d prefer to host it yourself—perhaps for privacy reasons or technological reasons.	can multiple goals be set ? Sometimes you may want to track more than one goal. For example… (i) You may want to know the effect that your experiment has not only on orders, but also on leads submitted. (ii) You may be concerned that you’re increasing the sale of one product at the expense of the sales of others. In this case, you’d want to track the effect that the experiment has on the sales of each product. (iii) You may wish to count conversions at several stages down the conversion funnel. That way, if you’re aiming to increase, say, the number of orders, but your test doesn’t reach statistical significance, you may instead choose to consider an intermediate goal such as click-throughs (such as add-to-cart actions), and make the assumption that the number of orders would be proportional to the number of click-throughs.	can traffic be segmented ? You may discover that different types of traffic prefer different versions of your page. For example, new prospects and existing customers may prefer different versions of a webpage. Some software vendors recommend that their software can act as a “traffic cop”, forever directing different types of visitor to different optimal versions of the webpage long after the test has concluded. Though this feature may have great benefits for you, be aware that a vendor benefits greatly from being able to “lock you in” in this manner, so they may be tempted to over-enthusiastically sell the benefits of this feature.	used on email campaignsHTML emails can be tested in the same way as webpages. Because an email campaign happens within a short space of time, you probably want the software to automatically promote the winner as soon as enough data has been collected.	has a platform for creating webpages Some of the software tools have in-built platforms for creating and editing webpages — a bit like having their own WordPress interface. They can be useful if the thought of involving your company’s web development team gives you the shudders.	customer support by phone	self service or full service “Self service” means you just get the software (and maybe some support). “Full service” usually involves * technical support * assistance in deciding “what to test” * designing the experimental plans * interpreting the data
	~$33,000	$160,000-$320,000	Client-side	Both	Several (see details)				(see details)		(see details)	Both
	Contact Adobe	Usually a four-figure monthly fee	Client-side	Both	Taguchi	(see details)						Both
	Contact Optimost	Usually a four-figure monthly fee	Client-side	Both	Optimal design							Both
	Free trial	From €300	Client-side	Both	Full factorial							Both
	None	$2,500-$7,500 /month	Client-side	Both	Modified Taguchi (see details)			(see details)				Self service (see details)
	None	From $99 per month	Client-side	Both	Full and fractional factorial							Both
	None	Usually a four-figure monthly fee	Client and Server-side	Both	Full factorial (see details)	(see details)	(unlimited)					Both
	None	From $500 per month	Client-side	A/B	N/A							Both
	From £2,000	Usually a four-figure monthly fee	Client-side	Both	Proprietary adaptive full factorial							Both
	None	None	Client-side	A/B	N/A						(see details)	Both (see details)
	None	From $99 per month	Client-side	Both	Proprietary adaptive fractional factorial							Both
	None	From $1295 per month	Client-side	Both	Full factorial		(unlimited)					Both
	None	Usually a four-figure monthly fee	Client-side	Both	Several (see details)							Both
	Contact Monetate	Flat monthly fee based on previous 12 months’ traffic	Client-side	Both	Several (see details)							Both
	Contact mPath for details.	Monthly fee (Contact mPath for details.)	Client and Server-side	Both	Full factorial (see details)	Contact mPath for details.	(see details)					Self service
	None	$7.5k+/year	Server-side	Both	Non-parametric MVT (see details)			(see details)				Both
	Free trial	$19-$399 /month	Client-side	Both	Full factorial							Self service
	See details	See details	Proxy server method	Both	Full and fractional factorial.							Both
	None	$1k-$2.5k per test	Client-side	Both	Proprietary non-parametric technology		(see details)					Both
	$297	Two additional payments of $297	Server-side PHP and MySQL	Both	Taguchi			(see details)				Self service
	Free	Free or $29-$999/month, depending on traffic	Server-side	A/B	N/A							Self service
	Contact Vertster	Monthly fee	Client-side	Both	Full factorial, Taguchi and Treadmill MVT							Both
	None	$49-$729/month	Client-side	Both	Full factorial							Self service
	Contact Webtrends	Usually a four-figure monthly fee	Client-side	Both (see details)	Several							Both

Full disclosure: We are authorized consultants for Google Analytics. However, we don’t receive any remuneration for recommending it. In fact, we don’t receive payment for recommending any software.

So, how can we help you?

How-to guides

Our selection of easy reading will familiarize you with important tools and topics.

To explore these helpful resources, click here.

Work with us

If you want us to work on your website personally and dramatically improve its profitability, visit our "Services" page and then contact us for more information.