A/B Testing

What is A/B Testing?

A/B testing offers operators of websites, mobile apps or online advertising the opportunity to gradually optimize their “appearance” by comparing two variants (A and B) of individual details. It is important to make only small changes during each test in order to obtain meaningful results. This is followed by an analysis to determine which version promises more success in the long term.

What are typical applications for A/B testing?

The things that are tested using A/B testing are manifold. Most often the webmaster subjects the following elements of his website, app or advertisement to a test:

  • General design and layout (color, font size, font etc.)
  • Headings on main or subpages (wording, length, etc.)
  • Call-to-Action (concrete wording, length, color, placement, etc.)
  • Category or product description (style, length, keywords etc.)
  • Images, graphics or videos (type, size or length, scope etc.)
  • Prices or discounts (including psychological factors)
  • Fields to be filled out in forms (type, number, arrangement, etc.)

Here are some examples in a more detailed form:

a) A/B test of a new web design

Is the newly designed web design better or worse received by customers than the old version? This question can be answered using A/B testing. When creating a new template, the new and the old design are often put online to find out which one appeals more to the users. The focus is on improving usability on the one hand and increasing the conversion rate on the other.

b) A/B tests of special features

Webmasters who want to revise the navigation and user guidance of their website or introduce new features such as filters, search functions or purchase advisors often make use of A/B testing. This makes it clear whether the changes really make sense and increase success. Such a check makes it possible, for example, to find out about operating errors and correct them.

c) A/B tests in relation to landing pages

If a site operator doubts whether a certain landing page of his Internet presence already exploits the maximum conversion potential, he can switch a second landing page in parallel, which has a slightly different structure than the original one. A/B testing then answers the question as to which of the landing page versions brings the higher conversion rate. The loser variant is then removed.

d) A/B test for more downloads

App Publisher always strives to increase the number of installs. A/B testing can be used to find out what prevents/motivates landing page visitors in the App Store to download. For example, the type of visuals used, the visual appearance of the screenshots or the call-to-action formulation in the first 200 signs can be put online for a certain time in two variants. If you want more inspiration, check out the guide on App Store Optimization.

How does A/B-Testing work?

The website operator programs the A/B test in such a way that the incoming traffic is split. While half of the traffic still lands on the original version of the website, the other half is diverted to the modified page. It is crucial that this splitting takes place involuntarily and that the user himself knows nothing about it.

Start Android App Advertising Now

Specifically, there are two ways to set up A/B testing:

a) Exchange of the elements to be tested before loading the website (classic A/B testing)

If a single element of a website is tested, for example, the login button, the webmaster has to create two variants of this button with his A/B testing software. If the test is activated, the software randomly selects which visitor is shown which button version.

b) Redirection to another page (split URL testing)

Does the A/B test refer to a complete page (example: is green or red better as the basic color on the page?), the website operator must create a new page and put it online in addition to the actual page. Again, the A/B testing software randomly directs visitors to one of the two pages.

Again, the A/B testing software randomly directs visitors to one of the two sites.

What are the steps for the webmaster to take?

In order to implement an A/B test correctly, a hypothesis must first be formulated. Webmasters who know their target group will have an advantage here. In principle, a hypothesis for such a test is an assumption about the target group related to a certain component of the page, app or advertisement.

So-called tracking – carried out in advance – helps to find out which components are worth testing at all. This tracking includes eye tracking, heat maps or neural networks.

The general formula of a hypothesis for an A/B test is something like this:

“If on the basis of observation X, I adjust component A as a result of my target group acceptance Z through Measure B, I can achieve my conversion target C.”

In this sense the website operator needs:

  • a conversion target (C)
  • a target group acceptance (Z)
  • one observation (X)
  • one measure (B)
  • a test component (A)

A is the original version, B the new version.

What is the significance of temporal factors for A/B testing?

In A/B testing, both variants must always be tested at the same time – this usually means that a visitor to the page will alternately see version A and version B. This means that the visitor to the page will always see both versions at the same time. If the webmaster makes the change at longer intervals (e.g. every hour, after a day or after a certain number of visitors), the results may be falsified. It doesn’t only depend on the design of the website how many people call it up and which willingness to register or buy it they bring with them. Various factors are important – both within people and “outside”. External variables include, for example, the day and the exact time of day (working-day weekend, morning afternoon evening, working time evening, etc.).

Note: In addition to A/B testing, there is also Multivariate Testing (MVT). With this instrument, several changes can be tested simultaneously (e.g. headline, description, and pictures). This special form tests all possible combinations of all variations and determines the best composition. But: Multivariate testing requires a very high traffic level on the website/in the app.

What are the advantages and disadvantages of A/B tests?

A/B testing has some advantages, but also several disadvantages. The advantages relate to the benefits of the test instrument, the disadvantages essentially to the fact that a lot has to be taken into account during testing in order to achieve real meaningful results.




Often the personal view of the webmaster differs from the view of the target group as far as the details of the website are concerned. A/B testing gives the operator the opportunity to compare his own feelings with those of the visitors. This is great in combination when you buy app downloads, as you will have enough visibility and testing potential in a short time frame.A/B testing only provides clear insights if the two versions to be compared differ from each other in only one detail. If the website operator tests several sites simultaneously, he has no chance of determining the direct trigger of success or failure when evaluating the results. Thus, the webmaster has to perform many individual tests if he wants to optimize various areas of his appearance. Under certain circumstances, this can greatly increase the effort involved.
In short: With every A/B test the website operator deepens the knowledge about his audience: What does he like, what does he look for, what needs and habits do he have?In A/B testing, there is always only one winner. This means that only one variant is followed up, while the other is completely eliminated after the test. Half of the information is lost. This always leaves some uncertainty if the winning variant is subjected to further A/B testing afterward (another detail varies). It remains unclear how the previously eliminated version would have performed with the change of the new test in terms of conversion.
Using suitable tools, A/B testing is easy to perform and usually does not require much effort. In addition, it does not necessarily require prior technical knowledge.In principle, the webmaster should proceed cautiously with A/B testing. Excessive changes, which are reversed shortly afterwards, can lead to enormous confusion for existing customers. It is therefore advisable to conduct the test exclusively with new visitors.
Correctly performed, the analysis of the A/B test provides clear results. After the end of the test, the website can be changed or adjusted in such a way that the visitors only get to the more promising version.In order to actually be meaningful, the data should achieve statistical significance. This requires a large number of visitors. For smaller websites with rather few visitors or low conversion, this can take a while.
There is no longer any room for gut decisions in the marketing strategy, instead, everything is tested that has a positive effect on the conversion rate.Before it is possible to operate playfully and safely with A/B testing, it is essential to familiarize oneself intensively with the topic or to get comprehensive advice from experts.
Correctly performed, the analysis of the A/B test provides clear results. Thus the website can be changed or adjusted immediately after the end of the test so that the visitors only get to the more promising version.
Through the results of A/B tests and the following measures, the website operator invests his time and money in what appeals most to his visitors.
A/B testing is versatile. It is suitable for companies with many visitors per day as well as for “smaller” blogs or similar with comparatively few guests.

Specific tips for A/B testing

As a general rule, the A/B test should not be aborted or terminated too early. A large number of visitors are necessary to obtain a meaningful result. For the significance check, there are online calculators (some of them free of charge) that target A/B testing and help the webmaster to interpret the results correctly. In any case, conclusions must not be drawn too early.

However, the test should not run for too long either, otherwise, the company may miss further sales or registrations because the “worse” version is still displayed.

Apart from the fact that the webmaster would do well not to include existing customers in A/B testing, it also makes sense to always guide the returning new visitors to the same variant (this can be defined accordingly in the tool).

Another crucial point: The website should be consistent in itself – that means the changed detail (e.g. the registration button for the newsletter) must be displayed everywhere on the page in the same form. Otherwise, the webmaster will cause confusion among the visitors. This can also lead to impaired results.

What is the general benefit of A/B testing for companies?

User behavior on websites is extremely important for successful conversions. In addition, signals such as the time spent on the respective page or bounce rates are important criteria for search engines to determine the quality of the target page. For these reasons, A/B testing is becoming increasingly indispensable. Through testing, key visitor indicators can be optimized in the long term. For websites that are “at home” in e-commerce, A/B testing helps to increase sales.

Changes to a website as a result of an A/B test can have an immediate effect on traffic. Among other things, the length of stay and bounce rate adapt; in some cases, the values of these components even make great leaps. A/B testing enables a step-by-step optimization of the success-relevant factors. Depending on the individual objective, an increase in the conversion rate or an increased number of subscribers to a newsletter counts as success.

Fictitious Example for A/B Testing in the App Store

The App “Monkey” has the goal to increase the conversion rate from visits of there App Store Page after they receive a good organic amount of visitors each day from focussing on the important app store ranking factors. By means of A/B testing, the publisher wants to find out whether a color variation on the Screenshots has a meaningful impact. He thus creates a second variant of the App Store Page with the Google Developer A/B Testing Tool and adds the color variation for the Screenshots.

Contrary to expectations, the change in color does actually reduce the number of installs for this variant. It seems to be unappealing for visitors. The result is that the conversion rate for the version with the old color variation is 400 percent higher than for the version with the variation. The Publisher, therefore, removes the latter variant and leaves it at the classic page the color variation.

A/B Testing Tools for Apps

apptentive logo


Apptentive as a platform gives you great analytics and the chance to get in contact for your users. This includes the gathering and breaks down on feedback the user provides.




Splitmetrics logo


SplitMetrics gives you the possibility to test for conversion rates, conversion costs, increased organic traffic and increase Return on investment.

Website: https://splitmetrics.com/

Related Topics You Might Like: