A/B Testing

Perform experiments and continuously optimize your communication campaigns with A/B testing. Testing audiences in the Flow tool enables you to see how they react to your content, what channel they prefer, or what is the best time to send them communication.

You can do the following:

• Compare how different channels perform with your audience.
• Perform A/B testing for all available channels.
• Manually pretest different versions and choose the winner.
• To test variations in versions, change any parameter or attribute of a message for the different test versions.
• Compare the testing results for different versions.

  • The statistics are channel-specific. However, the Number of Engaged Persons and Performance Ratio are available for all channels.

Set Up an A/B Test

We will show an example with different channels involving similar content - a discount code that a company would like to communicate to its customers. After a few hours or days (depending on the timeframe specified for this communication), the metrics will show which channel would be the best choice in the future.

1. In the Moments module, select an entry point for your communication (for this example case, we will use Item Added to Wishlist event as our communication trigger), and then add the Split Audience element - you can select how many audiences you want and split them in proportions of your choice. Separate branches for each subset audience will be created as seen in the image below, with people randomly being assigned to these branches.
    

 

2. Select channels and messages that will best serve your desired testing scenario.

3. Next, include the Evaluate Message element with keywords that will let you know the customer reaction. For Email, select link clicked instead of using the keyword.

4. If the customers responded using the keyword, send another SMS and Viber message with instructions involving the promo code. Also, you can send a message to customers who clicked the link in the email message. 

5. Use an event such as Item Purchased. This event will let you know whether the customer has reacted to your message and completed the purchase. Think about how long you want to wait before sending another message to them based on their actions. For example, you could wait 2 days.

 

6. Finally, if you want, you can send a thank you message to those who made a purchase. You could also decide to continue the Flow by sending another reminder to those who have not reacted to your communication. 

7. After you Validate and Launch the Flow, you can go back to it and check the Flow Performance.  

Test Results

The outcome of the A/B Testing shown in this article offers a couple of possibilities, including the continuation of the same campaign or planning future communication with customers.

Pretest and Send Winner Message

Another A/B Testing option in Flow is to add a tag to your communication. The tag would represent a winning segment (tag can have any name). The first three audience branches are used for pretesting, and the best performing message you can later send to the remaining audience branch via a new campaign that you would create in Flow. This we will explain in the following paragraphs.

 

New Campaign in Flow

Go back to the Moments page and create another flow same as the winning one or duplicate the existing one. But this time use the tag that involves the remaining audience. 

1. Select the Predefined Audience starting point and select tag winning_segment.
2. Add the SMS message. 
3. Add Evaluate Inbound SMS.
    

 

4. Add another message and a Wait for an event element; in the side panel, select Item Purchased event. Select the wait time 2 days.
    

 

5. The final Flow would look something like this: