A/B test - WEBSALE Dokumentation

Whether the green or the orange order button works better cannot be settled by thinking about it. An A/B test answers the question with numbers: over a defined period, each visitor is randomly shown one of the variants, and the shop counts how often the goal is reached afterwards. The goal is either a completed order or the call-up of a specific page. At the end, the statistics show which variant led to the goal more often.

How the test works

Aspect	Behavior
Distribution	Random; on average, each test case receives the same number of visitors
Binding	Always the same test case per session, even if the same page is called up several times
Counting of impressions	Once per session, not per page view
Counting of conversions	Only if the visitor saw a test case in the same session
Scope	Two to ten test cases per test
Location	Any template; a test is not intended for configurations

Search engine crawlers are also randomly shown a variant. This is intentional and prevents cloaking, that is, serving different content to the crawler than to the visitor. These page views are not counted, so the results remain unaffected.

Because the shop counts per session and not per page view, the statistics measure visitors, not page views. A visitor who views the page five times appears once.

Rules for planning

Always test just one feature in different variants, never several at the same time. Otherwise, at the end you will not know which change had an effect.
Expect at least 200 measurements per test case and subshop, preferably more than 1,000. Statistics based on thin data are not a basis for decisions.
Plan the test completely before you start it.

Once a test has started, it can no longer be changed or extended. Even a test that is ended early remains locked. A forgotten test case therefore means starting a new test from scratch.

Creating a test

In the online service area, select your shop ID, then the “A/B-Test” service and there “Neuen A/B-Test anlegen” (create new A/B test).

Step 1: Basic settings

Field	Meaning
Count conversion	”Automatisch bei Bestellung” (automatically on order) counts when the order confirmation is reached, “Durch Tag auf Zielseite” (via tag on target page) counts when a page of your choice is called up
Test ID	Identifier of the test, which you later specify as a parameter in all tags
Description	The test scenario in up to 1,000 characters, for looking back in three months

A test only counts what you have set here. A test with a target page does not count orders, and a test with orders does not count page views.

Step 2: Test period

Set the start and end using the calendar icons, and select the shop’s time zone.

The time zone determines not only the start and end, but also the day boundary of the daily charts in the evaluation.

Step 3: Test cases

Use the document icon next to “Testfälle” (test cases) to create one row per test case. Each row gets a label and a description.

Setting	Purpose
Test case ID	Assigned automatically, `CaseA` to `CaseJ`, and later forms the name of the section tag
Label	Caption in charts and legends, a maximum of 25 characters
Description	What is different about this test case

Keep the labels short. They later appear on the axes and in the legends of the charts, where long text becomes unreadable.

Finally, save.

Adding the test cases to the template

Two main sections are created in the template of the affected page. The first encloses the variants and is only output during the test period; the second contains the version for the time before and after.

{ABTest-active(Meine_Test-ID)}

   {ABTest-CaseA}
      <!-- Variante für den Testfall CaseA -->
   {/ABTest-CaseA}

   {ABTest-CaseB}
      <!-- Variante für den Testfall CaseB -->
   {/ABTest-CaseB}

{/ABTest-active(Meine_Test-ID)}

{!ABTest-active(Meine_Test-ID)}
   <!-- gilt vor und nach dem Test und beim Besuch durch eine Suchmaschine -->
{/!ABTest-active(Meine_Test-ID)}

Within the first section, the shop selects exactly one of the test case sections when the page is called up.

The second section is not optional. Without it, nothing appears at this point outside the test period.

Reference: ABTest-active() · ABTest-CaseA…J

Setting the conversion counter

If you selected “Automatisch bei Bestellung” as the goal, there is nothing else to do. The counter increases as soon as a visitor reaches the order confirmation after going through a test case. With “Durch Tag auf Zielseite”, you set the counter yourself, at the point in the target page’s template whose reaching should count as a success.

~ABTest-TriggerConversion(Meine_Test-ID)~

The evaluation distinguishes two figures: how many sessions reached the goal, and how often it was triggered in total. If a visitor calls up the target page three times, the first figure counts once and the second three times.

Reference: ABTest-TriggerConversion()

Checking the test cases

So that you do not have to rely on chance when you want to view all variants, there is a dedicated test mode. For this, the shop must be in the general test mode.

https://www.ihr-shop.de/?test=on&ab-test-id=<Test-ID>&ab-test-case=<A bis J>

Switching off:

https://www.ihr-shop.de/?test=on&ab-test-id=off

Guide: Switching the shop’s test modes on and off

Reading the evaluation

In the “A/B-Test” service, the bar chart icon at the end of the test row leads to the statistics. They are available during the test period as soon as the first measurements are in.

Each evaluation begins with the same control bar: a chart showing the frequency and even distribution of the test cases, a slider for the time period under consideration, and checkboxes for the subshops included.

Below this follow the evaluations, depending on the conversion goal:

Conversion goal	Sections
Durch Tag auf Zielseite	One section on the target page views
Automatisch bei Bestellung	Order conversion, revenue, line items, products

All four sections have the same structure: a bar chart with the average per test case and a line chart with the daily values.

Section	What is compared
Order conversion	Share of sessions with an order, in percent
Revenue	Order revenue per day
Line items	Number of line items per order
Products	Number of products per order

Looking at conversion alone is misleading. A variant can lead to orders more often while producing smaller baskets. That is why revenue, line items and products are shown alongside it with equal weight.

To zoom in, drag a frame around the desired area of a chart with the mouse; a double-click restores the original size. Clicking a legend text hides and shows the corresponding line.

Guide: Switching the shop’s test modes on and off
Guide: Integrating tracking

​How the test works

​Rules for planning

​Creating a test

​Step 1: Basic settings

​Step 2: Test period

​Step 3: Test cases

​Adding the test cases to the template

​Setting the conversion counter

​Checking the test cases

​Reading the evaluation

​Related topics

How the test works

Rules for planning

Creating a test

Step 1: Basic settings

Step 2: Test period

Step 3: Test cases

Adding the test cases to the template

Setting the conversion counter

Checking the test cases

Reading the evaluation

Related topics