Running In-Store Experiments to Test Retail Marketing UX Strategies

It’s impossible to throw a stone in the online Realm and not hit someone advocating for A/B testing and experimentation in the eComm world. 

In itself, it’s great. 

Only through an experimentation-focused mindset can you uncover specific combinations that will bring your revenue up. 

But happens with traditional, offline retailers? How are they supposed to experiment? 

Well, to that end, we’ve put together a methodology for running experiments to test retail marketing and UX strategies. 

Ready? 

Let’s go!

Crafting Your Research Question and Hypothesis

The first step in running in-store experiments is to clearly define your research question and hypothesis. Your research question should be specific and relevant to your business goals and what your customers want.

For example, you might want to find out how changing the layout of a product category affects customer attention, engagement, and purchases. Your hypothesis should be a statement that you can test and should predict what you think will happen based on your assumptions and what you already know. For instance, you might think that putting the most profitable products at eye level will make more sales and make customers happier.

To do good research in traditional retail stores, you need to have good insights from data. 

As more traditional retailers learn about their customers, it’s really valuable to look at how customers behave when they buy things to come up with hypothesis that are based on evidence.

Here are some ways to do this:

• Segment your customers into groups using methods such as RFM or more advanced ways of segmenting.
• Hunt anomalies, like which products, brands, or stores have more frequent purchases or bring in more money from customers over time.
• Get ideas for tests from store managers or category managers—they might notice things that the data or retail teams don’t. 

Hint: we’re planning to use a game-like approach in our Omniconvert tools to encourage them to do this.

Deciding on Your Experimental Setup and Variables

The next step is to pick your experimental setup – which should match your research question and hypothesis. 

The setup is all about helping you figure out how your marketing idea affects your numbers without other variables tainting your results.

There are different ways to set up experiments, like randomized trials, split tests, and factorial designs. 

Each has its pros and cons. 

You also need to decide what variables you’ll change and measure in your experiment. 

The variable that changed is the hypothesis you’re testing, such as the price or where you place a product. 

The variable you measure is what you see happening, like how many customers show up, how long they stay, or how much they spend.

What variables could you test? 

Here are some ideas from our experience:

• Put Snickers bars near the checkout
• Give less shelf space to toilet paper and more to makeup
• Find and check out three new sellers of exotic products
• Adjust the lighting in the wine section
• Try buy-one-get-one deals for makeup
• Use signs in the store to promote Heineken
• Do a taste test for a new vegan brand
• Give discounts on cooked meals in the evening
• Give out special discount codes at 8 PM
• Put pictures of babies in the clothing area.

Choosing Your Sample and Location

The next step is to choose your sample group and location. 

Your sample group consists of the customers who will take part in your experiment, whether they know it or not. 

It’s crucial to make sure your sample represents your target customers accurately and is large enough to notice important differences between your test conditions. 

You should also think about how you’ll pick and assign your sample to different conditions, which could involve random selection, grouping, or letting people choose for themselves. 

Your location is simply where you’ll actually run your experiment, like a specific store, aisle, or shelf. 

It’s important to pick a place that fits what you’re testing and doesn’t have too many distractions like noise, crowds, or bad weather.

If you have regular customers who make a big difference to your sales, you’re in a good position to really see how your experiment affects things.

To put it plainly: if you’re hoping to boost sales by 15% by adding two new beer brands, how long should you wait if you’re only trying them out in four of your stores? 

The more stores you try it in, the faster you’ll get results because you’ll have more customers and orders.

Here are some things to think about:

• Which stores you’re using for the experiment and which ones you’re not
• What kinds of products you’re looking at
• What you’re hoping to achieve
• How much you think sales will go up
• How long you’re going to run the experiment
• How sure you want to be that your results are reliable (aim for over 95% confidence).

The length of your experiment depends on the minimum detectable effect (expected uplift).

Executing Your Experiment and Gathering Data

The next step is to put your experiment into action and start collecting data. 

You’ll need a clear plan that outlines exactly how you’ll carry out the experiment. 

This plan should cover factors such as:

• when and where you’ll make your changes,
• how you’ll keep track of what’s happening, 
• and what you’ll do if something unexpected comes up. 

It’s also important to use the right tools to collect data, whether it’s through surveys, observations, scanners, cameras, or sensors. 

And of course, you need to make sure your data collection is accurate, reliable, and follows all the rules and guidelines for your industry.

Before you kick things off, it’s essential to ensure that the changes you’re making are being consistently applied in all the stores involved in the experiment. 

Some store managers might not see the importance of these changes, so it’s crucial to get everyone on board. 

Our software, currently in beta, not only helps you run experiments but also tracks their impact through our customer analytics solution. 

This way, you can see how the changes are affecting both what customers are buying and what they’re saying about their experience (NPS).

Analyzing Your Data and Making Sense of Your Results

The last step is to analyze your data and make sense of your results. It’s crucial to use appropriate statistical methods to test your hypotheses and compare different conditions in your experiment.

Techniques like t-tests, ANOVA, or regression can help with this.

You should also double-check the accuracy and consistency of your data and results, and look out for any possible biases, errors, or factors that might affect your conclusions.

After that, you need to interpret your results in light of your original research question and hypotheses, explaining what they mean for your retail marketing strategy.

It’s also important to point out any limitations, implications, or suggestions that arise from your experiment, and propose any further research or actions that might be necessary.

We firmly believe in Customer Lifetime Value as a guiding metric in retail.

Sometimes, experiments may initially seem beneficial in terms of revenue, but they can have a slow, negative impact over time.

For example, you might create more space on your shelf by removing two beer brands.

At first, customers who used to buy those brands may continue shopping at your store.

However, gradually, they might become dissatisfied if they were loyal to those brands and start buying from competitors that still carry them.

That’s why it’s important to measure more than just immediate transactional metrics like orders, customers, and revenue.

Long-term metrics such as NPS (Net Promoter Score), CSAT (Customer Satisfaction Score), and CLV (Customer Lifetime Value) are also important, when deploying a customer-centric approach.

Additional Points to Consider

Before signing off, we’d like to make a final rundown of crucial aspects of experimentation, which don’t necessarily fit into a specific step. 

As you’ll see, these points apply holistically to the entire experiment, influencing both your results and your overall enjoyment of the process. 

Let’s start with data privacy.

Make sure you’re handling customer data with care and following all the rules about privacy. It’s important to keep their information safe and secure – both legally, and morally.

Then, you need to understand that running any type of experiment will require time, people, and money. 

Make sure you’ve got enough of each to do it right.

Be mindful of feedback.

Pay attention to what both customers and store staff have to say during and after the experiments. Their insights can be really valuable in figuring out what’s working and what’s not.

Remember that the effects of your experiments might not show up right away. 

Keep an eye on how things change over time, like customer habits and what people think of your brand.

Finally, don’t forget to keep good records of what you’re doing and what you’re learning. It’ll help you gain knowledge from your experiences and share your insights with others.

Good luck!