A/B Testing with Nonfig

Introduction To A/B Testing

What is A/B Testing

A/B testing is also known as Split Testing and Bucket Testing. It is a testing technique used to test two variants of a particular version. A/B testing is used in many professions for different purposes, researchers use it to test the efficacy of drugs, marketers use it for improving customer experience, politicians use it to attract what attracts voters and the list goes on.

A/B testing splits traffic 50/50 between control and variation. As a marketer, you can apply A/B testing at any digital platform where you want to apply to such as email, website, messages, announcements, app, CTA etc. For example, if you are running a simple A/B test, you will split 50% of your traffic to one version and 50% of your traffic to another version.

A/B Testing and Conversion Rate

In the online world, people who are visiting your website are your potential customers or a potential opportunity that you can capture. Business wants those customers to take the desired action or convert them to improve the conversion rate. A/B testing helps a business to improve the conversion rate. Basically, of the two versions, the winning version is adopted by the business for better results.

The conversion metrics differ from business to business. If you are a B2C business, conversion may mean the sale of products for you, whereas if you are B2B business it may be generating qualified leads. A/B testing is one of the important factors that lift up your Conversion Rate Optimization and helps you gather qualitative and quantitative insights about customers through which you can improve your conversion funnel based on the data.

Moreover, the process of A/B testing considers business goals. It weighs gains against losses, carrot against stick, science against business. Keeping them in consideration, a business makes a decision.

Businesses also apply tests with more than two variations. Those tests are known as A/B/n tests. If your business has enough traffic, you can test as many variations as you like. Here, you can divide traffic for different variants at whatever percentage you like to. Moreover, higher traffic is the requirement for the test so that you can easily distribute it among different pages.

How To Prioritize A/B Test Hypothesis

You can find a number of frameworks and ways to prioritize your A/B tests. You can even design and invent your own framework. Once you have prioritized your A/B tests you will find issues ranging from minor to major. You can allocate every issue under the following headings:

  1. Test: Here you will place stuff for testing
  2. Instrument: Issues under the heading require to be fixed, improved or added in analytics
  3. Hypothesis: Here, you will place process, page or widget that’s not working well and doesn’t reveal a clear solution
  4. Just do it: List for no-brainers
  5. Investigate: You need to dig deeper for these issues

You can rank the issues from 1 to 5 (1=minor, 5=critical). There are two criteria that are more important than others when giving a score:

  1. Ease of implementation(complexity/risk/time): Don’t start by developing the feature, because sometimes your data asks to develop a feature which might consume months.
  2. Opportunity: Score issues subjectively based on how big a lift or change they may generate.

How To Set-up A/B Test

Set up a prioritized list of ideas which you want to A/B test. After, form a hypothesis and run an experiment.

A hypothesis is a testable statement of what the researcher predicts the outcome of the study will be or a hypothesis is that defines why you believe a problem occurs. Moreover, a good hypothesis should always:

  • Explain what you expect to happen
  • Be clear and understandable
  • Testable
  • Measurable
  • Contain an independent and dependent variable

When a marketer creates a hypothesis, it should solve the conversion problem and must provide market insights which means it should give you information about the customers.

A hypothesis kit provided by Craig Sullivan has simplified the process for us:

  1. Because we saw (data/feedback),
  2. We expect that (change) will cause (impact).
  3. We’ll measure this using (data metric).

And the advanced one:

  1. Because we saw (qualitative and quantitative data),
  2. We expect that (change) for (population) will cause (impact[s]).
  3. We expect to see (data metric[s] change) over a period of (X business cycles).

Picking Up a Tool For A/B Testing

Now it is time to pick a tool to carry out tests for you. Just like strategy and statistical knowledge, picking up the right tool also plays an important role.

Before picking up a tool you must know the difference between two types of tools, Server-side and Client-side tools.

A Client-Side tool creates variation by manipulating your browser via clever JavaScript. The changes aren’t done on your server, instead, all the changes occur on the visitor’s server. Your server sends the default version, as it normally does, however, all the changes are reflected by the visitor’s browser.

However, in Server-Side tools, no modification occurs at the visitor’s browser level. When a visitor land on your page, a randomly picked version of your test is sent from your server to the browser. Your developers are involved in the process, as a result, it gives you more flexibility.

So why do you need to know about both types of tools?

If you’re a small team or your business lacks technical resources and you want to save your time for something more valuable, client-side tools are the best option for you. However, if you want to be more flexible, go for server-side tools.

How To Analyze A/B Test Results

So, you created a hypothesis correctly, did research, set up your test correctly, and implemented it well. Now you have the results in front of you. Analyzing results is not about giving a look at the numbers and graphs given from the tool you used. Your testing tool might be recording the data incorrectly and giving you incorrect results. You can never trust a single source of data, rather you must create multiple sources of data.

That said, you should also analyze your results on Google Analytics. You can now be more confident in the decision making with your wide analysis capabilities.

Segment Your Data

When analyzing your data you must segment your data to produce the best possible results. For example, you tested your data for two segments X and Y. The result might show that X wins in the overall result from Y, however, when you compare the results on individual segmentation basis, you might see that Y beats X in individual segments phenomenally.

In what ways can you segment your data? Here are a few example segments:

  • New – Returning visitors
  • New – Repeat purchase
  • Power-user – Casual user
  • Men – Women
  • Age
  • Logged in – Logged out users
  • Mobile – Desktop – others
  • Browser
  • Source

There are hundreds of other segments that you can make depending on the business model and the type of target market you are catering to. Moreover, you must look over the following three segments:

  • Device used by users: Desktop – Mobile – others
  • New – Returning
  • Traffic that lands on the page – Traffic from internal links


A/B testing is a very critical resource for every decision-maker, who aims to increase the market share. With a little bit of knowledge and consistent effort, you can mitigate the risks that businesses face.

In the online environment, if you believe in the power of A/B testing. You can gain a competitive edge in the market. Many businesses that are said to be the market leaders in the industry used A/B testing and improved their conversion rate. Start testing with Nonfig!

Leave a Reply

Your email address will not be published. Required fields are marked *