How to Set Up Shipping Tests in the App

Intelligems enables your organization to run real-time split tests on your store's shipping rates and thresholds. This article will walk you through how to set up a shipping test with Intelligems, as well as some best practices to keep in mind.

---

In this article: 

• Prerequisites: How shipping tests work.
• Step 1: Create a new test.
• Step 2: Select the shipping profiles and zones to test.
• Step 3: Select the rates you want to include for this test.
• Step 4: Create your test groups.
• Step 5: Configure the Intelligems rate(s).
• Step 6: Review your rates.
• Step 7: Configure advanced settings.
• Step 8: Save your test.
• What happens next?

---

Prerequisites: How shipping tests work.

When you create and save your shipping test, Intelligems will automatically be added as a third party rate carrier in your Shopify admin. With the Intelligems rate carrier installed, once you start your test, Intelligems will provide rate(s) to shoppers at checkout based on their test group, cart value and item weight (if relevant). It will only calculate and display rates for shipping profiles and zones it is added to and will appear at checkout along with any other applicable rates configured in your Shopify admin that are not hidden for the test. You can learn more about Shopify shipping settings here.

This article will take you through how to:

• Select profiles / zones to add the Intelligems rate to
• Choose existing rates to hide during the test (those that conflict with the Intelligems rate)
• Create test groups and configure which rate(s) Intelligems will display for each group
• Review rates for the combined shipping profiles in your test
• Configure advanced options for your test

If you are setting up your first shipping test, we recommend reading through the full article before starting setup.

Step 1: Create a new test.

Navigate to the 'A/B Tests' tab in the menu on the left-hand side of the Intelligems app. Once there, click 'Create New Test' above the experiments table. Select 'Shipping Test' and 'Create New Test.'

Step 2: Select shipping profiles and zones to test.

Using the expander for each shipping profile, select the zone(s) where the Intelligems rate will be added. The Intelligems rate will only apply to the profiles and zones selected here and will apply to all orders that meet the criteria for the selected profiles and zones, subject to other cart value and weight criteria which you'll configure in the next step.

Step 3: Select the rates you want to include for this test.

Once you have selected the profiles and zones you want Intelligems rates to apply to, you will select which rates from those zones you want to test. These rates will be hidden during the test and replaced by an Intelligems rate, which will vary by test group.

In the example below, we are looking to test our store's shipping threshold at $50 and $100. We want to select both our Free Standard Shipping and our Standard Shipping because we are changing the condition on both rates.

Step 4: Create your test groups.

Fill in the Test Name and Test Description of the Experiment you wish to create.

Create between two and five groups to include in the test by clicking on the ‘+’ button. Name the groups for the experiment and use the slider to allocate what percentage of traffic will go to each group. 

Step 5: Configure the Intelligems rate(s).

In this step, you will configure the calculated carrier rate to be added to the previously selected profiles and zones.

For each group, follow these steps:

1. Choose the rate type that most closely aligns with what you're testing (i.e. what you want to change in each test group). See below for some tips on when to select each rate type and examples of combined rates:

Legend

• Flat Rate: You are testing rate(s) that are not set up with conditions in Shopify ("flat rates")
  Example 1: Testing a single flat rate
  
  
  Example 2: Testing multiple flat rates
  
  
  
• Flat Rate with Threshold: You are testing a flat rate and the threshold for free shipping
  
  
  
• Threshold Only: You are testing a free shipping threshold but want to keep your existing rates for those that do not reach the threshold
  Example 1: Testing a free threshold with rates under threshold provided by third party app
  
  
  Example 2: Testing a free threshold with weight-based rates under threshold configured in Shopify
  
  
  
• Tiered by Price or Weight:
  Example 1: Testing rates for each weight-based tier
  
  
  Example 2: Testing rates and conditions for each weight-based tier
  
• Custom*: Have something else you want to test? Let us know and we'll see what we can do!

*Diamond plan or above may be required.

2. Once you have selected your rate type, fill in the required information. This will vary by rate type, but typically includes things like rate name, the rate amount and the threshold the threshold.

3. Once you have filled in all required information, click 'Save' and complete the same steps for the remaining test groups. 

Step 6: Review your rates.

Use the drop-downs to select the shipping profile and zone, and test group to preview.

Step 7: Configure advanced settings.

There are two advanced settings steps, both of which are optional to set up. 

The first is Audience Targeting. This tool allows you to include, exclude or assign specific treatment to certain traffic for your test. Check out this article for more information on configuring this option.

The second is Find & Replace. This tool allows you to adjust site copy for site visitors based on their test group. Check out this article for more information on configuring this option.

The third is a Shipping Progress Bar. If you currently show a shipping progress bar on your site or if you would like to include one for your test and you are testing shipping thresholds, it is recommended to use the built-in option from Intelligems so that the bar will update with the test group. Adding the Intelligems progress bar typically requires adding a component to your theme's liquid code and customizing the bar's style in the Intelligems app.

Learn more about adding the Intelligems shipping progress here.

See below for a few examples of a configured shipping progress bar during a live test.

Step 8: Save your test.

Once you have completed all the above steps, you will be able to save your test with the button in the bottom right. Once the test is saved, you should be able to see Intelligems as a rate carrier in your Shopify shipping settings for each profile and zone you selected in step 2 - see an example below of what that would look like in Shopify. If you do not see it there automatically, you may need to click 'Add rate' and select it from the carrier options.

What happens next?

Now that you've created your shipping test, here are some things to do next:

• Start your shipping test
• Analyze your results