OnSite > AB Testing

AB Testing

OneSpot OnSite has built in AB test and multivariate testing capabilities. This is used for two purposes: first to quantify what improvements in content engagement we are making on client sites, and second to test variations in content selection logic and display formats. This document describes our methodology and metrics for measuring lift.

When measuring lift using OnSites testing framework, the testing and results reporting are managed completely OneSpot, there is no setup required by the client

OneSpot’s OnSite AB testing framework will calculate engagement metrics for multiple measures of engagement in different categories. We have chosen these metrics to reflect the overall engagement goals of an OnSite recommendation product like OnSite. For each metric we measure the difference between the engagement levels of a control group (that sees no OnSite recommendations) and a test group (that is exposed to OnSite recommendations).

We calculate a metric value, 95% confidence interval of error, chance to beat, % lift in metric, and a 1 yr projection of overall lift in engagement from OnSite for that metric. Lift is calculated on a per session basis and the projected overall lift is based on applying that lift to the number of sessions projected for a 12mo period.

We also segment overall lift, and lift by device type (computer, mobile, tablet).

While initial tests focus on measuring the lift or improvement OnSite provides clients, we also will run multivariate tests to try different algorithmic settings for content selection, or different display options to optimize engagement.

Total page views
Based on count of page views (by default page-loads, does not take virtual page views into account)

Number of distinct content pages in session
Based on page canonical url, will remove the effects of users using the back button or refreshing their pages.

Active engaged time with content
Based on a timer that tracks user activity (mouse movement, scrolling, active browser tab)

We take three different cuts of these metrics, as follows:

Overall traffic

Unique content pages (based on unique canonical urls)
Only measures engagement (time, views, etc) for distinct pages, to take out the effects of users hitting back buttons, or reloading

Recirculation
Only counts engagement for pages that have referal medium ‘internal’ meaning that they navigated from another internal page, removes traffic that comes in from external referers
Since the goal of OnSite is to keep users on-site longer, measuring recirculation specifically targets our ability to do that.

If clients want to control their own AB tests:
  • Clients will need to control if and when OnSite units get rendered by using the OnSite SDK (see documentation in the appendix)
  • Clients will need to manage the setup of AB tests in their own system, and also manage the data collection and metrics reporting for lift. Many of the metrics listed above that are specific to on-site content recommendations may not be measurable in a 3rd party testing tool.
  • Clients will not be able to fire off different variants of recommendation configuration for things like algorithm or display variants.