Amazon A/B testing: Lessons from a 7-figure seller

I was first introduced to Amazon A/B testing back in 2019. A friend showed me a listing where he’d tested a pricing increase from 70 USD to 90 USD. The results showed the same conversion rate and amount of clicks for both prices. This resulted in a 30 % increase in revenue – from one small change.

I was amazed, and in the coming years, I tweaked and tested different listing attributes to increase my Amazon conversion rate. This ultimately resulted in a significant increase in both conversion rate and sales.

In this article, I will go through the lessons I have learned from A/B testing on Amazon, and my best advice on how you can get started.

What is A/B testing?

An A/B test (sometimes called split test) is a method of testing variants of a page next to each other, to see which one converts the best.

For instance, an A/B test could be done on pricing, product images, or the product title. The goal is to see if a change on the site would lead to a higher conversion rate.

A/B testing is not something that is unique for Amazon. Many websites, including other e-commerce sites, run A/B tests continuously to increase their conversion rate.

Amazon, however, have been famous for doing a lot of A/B testing on their website. It could be anything from changing how the search result is presented to changing the design of the buy box. Every little detail is tested to see if it can yield a better conversion rate.

What you can A/B test on Amazon

You can A/B test the following attributes on your Amazon listings;

• Product title. A good product title is essential for your listing. It affects your click-through rate, keyword rankings, and conversion rate.
• Product images. Your main image is the main driver for your click-through rate in Amazon’s search results. The rest of your images are also very important for your conversion rate.
• Pricing. One of the absolute most important elements of your listing. Unfortunately, pricing is not possible to A/B test via Amazon’s own A/B test tool. Because of this, you will have to do it manually.
• Bullet points. The main text your customers shopping via desktop will see. Not always top-priority to test according to me, but still an important part of your listing.
• A+ content. I think the A+ content was one of the first A/B test functions Amazon rolled out. I have seen better results with A/B testing the attributes above, but depending on the product, your A+ content could be worth testing as well.
• Product description. Not a top priority to test as it will almost never have the same impact as the main image, title, and pricing. But once you have optimized the rest of your listing, testing your product description can still be worth it.

Why is A/B testing important for Amazon sellers?

If you want to become part of the top 1% of Amazon sellers, you will need to run A/B tests. Below I explain why.

• You optimize your listing based on real data, instead of guessing. Your Amazon listings are basically your whole business on Amazon. The details of your listing (title, images, A+ content, etc.) are huge factors that determine how well your business will be doing. By doing A/B tests you make sure that all these attributes are optimized based on what actually works, and that any changes you make are backed by real numbers.
• You will continuously increase your conversion rate. The goal of doing A/B tests is to increase your listings’ conversion rates. Higher conversion rate = more sales and higher revenue. By continuously optimizing your listings with A/B tests you will over time
• You will create a positive cycle. As mentioned above, an increased conversion rate equals more sales. More sales lead to; higher organic ranking and lower ACOS. This means that you will get more visitors to your listing, resulting in even more sales. You will basically create a positive cycle for your Amazon listing.
• You will get an edge over the competition. My experience is that very, very few sellers are working with Amazon A/B testing in a strategic way. Most sellers are not doing it at all. The ones that do it, often test things randomly without much thought behind them. If you master the art of A/B testing and optimizing your Amazon listings you will be way ahead of 99 % of your competition.

How to create an Amazon A/B test

When I first started doing A/B tests on Amazon, it was not possible to do it via Seller Central. Instead, I had to use a third-party tool.

However, in 2019, Amazon finally launched their own A/B test tool for Sellers in Seller Central, called “Manage your experiments“. This tool lets you as a seller run different A/B tests on your listings.

An important note; you must be registered for Amazon Brand Registry and your ASIN must have received enough traffic in the past couple of weeks to be eligible for the A/B test tool in Seller Central.

To create an A/B test in Seller Central, you will need to do the following;

1. In Seller Central, Hover over the “Brands” tab and click on “Manage experiments”.
2. Click on “Create new experiment”.
3. Select which product and what kind of listing attribute you want to test.
4. Enter your Experiment name and your hypothesis.
5. Add the variation that you want to test.

What results can you expect from your A/B tests?

The first time I conducted an Amazon A/B test I was dreaming of doubling my sales from some minor change in a few images. This was of course, very naive of me.

Don’t expect your conversion rate or sales to increase with high double digits because of minor listing changes. It is not going to happen.

Amazon A/B testing is a continuous, long-term game. For some A/B tests, you might get the opposite result from what you expected; the original variant performed better. For some, you might see that there is no difference. And for some, you will see that your new variant performs better.

So even if one small tweak to your listings won’t skyrocket your sales overnight, many small tweaks over time will definitely add up.

If you run 5 A/B tests in a year, and they on average each result in a 5 % increase in conversion rate, that’s a 25 % increase. This will then create a positive cycle in which you also boost your organic ranking, lower your ACOS and get more visitors to your listing.

My tips to succeed with Amazon A/B testing

Randomly running an A/B test on Amazon and hoping for the best is one thing. Actually working with A/B testing strategically to make continuous gains over time is something else.

Previously to starting my Amazon business, I had been working with A/B testing on other E-commerce sites. I used several of the strategies I learned and implemented for my Amazon business.

Below I have listed my main lessons on how to succeed with Amazon A/B testing.

Make hypotheses.

The main pillar of an A/B test is a solid hypothesis.

Why?

Because this is what you will base your test on. Without a good hypothesis, you are just testing changes randomly.

So what is a hypothesis then?

A hypothesis is basically an educated guess. You are guessing that change X will lead to outcome Y.

For instance; “I think that replacing our main image with a more clear photo, that includes the packaging, will result in a higher conversion rate”.

Preferably, your hypothesis should be backed by some kind of argument. For example; “Because we have seen that other listings in the same category that has made this change increased the conversion rate”. The stronger your argument is, the better your hypothesis will be.

Another example could be; “I think that showing what sizes are product is suitable for will increase the conversion rate”. Argument; “Because a lot of customers are sending emails and writing questions on our listing’s Q&A about this”.

When creating your hypothesis it’s good to speak with other colleagues or sellers to get their input as well.

Make a priority list

You have a lot of different things that you can test on your listing (pricing, title, images, etc.). So how do you know where to start?

Of course, you would want to begin with tests that have the highest potential to increase your conversion rate.

In my experience, generally, the most impactful changes of Amazon listings are; product title, main image, and pricing. But it all depends on the listing.

It might be that your title and main image already look good, but your bullet points don’t promote the product in a good way.

List all changes you can come up with that might increase your listing’s conversion rate.

When I make a priority list, I include the following;

• The issue to fix
• Priority (possible impact on CR, CTR, and sales)
• Implementation (how difficult it is to implement)

I give all the items on the list a score from 1 – 5. Number 5 on priority means high priority. Number 5 on implementation means it takes a lot of work to implement.

I start with testing changes that have a high score on priority, but a low implementation score. For example; changing the title might have a huge impact on the listing, and is easy to implement.

Adding a video might have a high impact, but also a high implementation.

Start with testing number 1 on the list, and work your way down.

Test one thing at a time

Once you have made your priority list, you might be eager to start testing everything at once. Please do not do this. This will only result in inconclusive results. Since the tests might interfere with each other. Even if some tests lead to an increased conversion rate, this might be affected by another you are running.

Don’t stop your tests too early

A very common mistake that I have seen among Amazon sellers is to draw conclusions from A/B tests too early. It might be very tempting to end the test after a week or two if your new variant is performing much better. But this would not be wise. The risk is then that you draw conclusions that are not correct.

I would suggest always running your tests for a minimum of four weeks. Amazon even recommends running your tests for eight weeks. Basically, the longer you run your tests, the more data you will have, meaning you can have more confidence in your results.

Manage your expectations

As mentioned earlier, it is not realistic that one minor change will double your sales overnight. Having unreal expectations will kill your motivation to keep running A/B tests.

I really want to emphasize this because I have seen this issue several times with sellers. Basically, the seller decides that he or she wants to try A/B testing for a listing. They test something randomly (like another main image without much difference). When they don’t get any significant results, they give up and conclude that “A/B testing on Amazon is not worth it”.

Do not make this mistake.

Remember that getting significant results might take time, but long term it can have a huge impact on your Amazon business.

Test your pricing

Today, testing your pricing on Amazon is a bit tricky. Unfortunately, the function is not available in Seller Central’s Experiments tool.

And since Spltily discontinued its services, there is not really any good alternative to A/B test pricing on Amazon.

However, since pricing is such an important factor for your listing you should test it anyway. But you will have to do it manually.

You will do this by changing your price to amount X for two weeks. And then change it to amount Y for two weeks. Afterward, you can compare the sales and conversion rates for the different periods.

Doing manual testing over different time periods is not optimal. Because there might be other factors that affect the results. For example; it could be that sales, in general, are higher for one of the two-week periods. Or it might be that your competitor is running a very aggressive promotion for one of the periods, meaning that your sales will be lower.

Even though manual price testing is not optimal, it is the best option we have at the moment and is still worth doing.

Test things that are actually significantly different

This is related to having a good hypothesis for your tests. Changing two words at the end of a long title probably won’t make any difference. Neither will use the same main image shown at a slightly different angle.

But cutting down your title from 175 to 75 characters might, or using a completely new main image with better resolution and format.

Again; remember that you should have some kind of argument about why you think the new version will perform better when making your hypotheses.

How to test listings that are not eligible in Amazon Experiments

If possible, you should always run your A/B tests in Amazon’s Experiments tool.

But what if your ASIN is not getting enough traffic and is not eligible – should you still test them?

The answer is, it depends.

For listings that don’t have traffic enough to run A/B tests, I usually put potential listing changes that I come up with in two different categories; Just change it and test it.

The “Just change it” category is for changes that are obvious improvements. Such as adding more images to the listing, or removing weird wording in the description.

The “Test it” category is for changes that I have a hypothesis for, but where I am not sure if the new version actually will perform better.

To test these hypotheses there are two main options that I use;

• A third-party tool with user testers. My favorites are Pickfu and Intellivy. With these tools, you can A/B test different listing attributes with real people in your target audience. As you pay to use these tools, make sure to only use them for tests that you think could have a big impact on your listing.
• Manual testing in Seller Central. Here, you would manually set your listing attribute to Variant A for 2 weeks, and then change it to variant B for the same period of time. After, you can compare and see if there is any difference in sales and conversion rate. Again, you have to be careful here as seasonality, competitors’ promotions, etc. might affect the results.

Summary and final thoughts

A/B testing on Amazon is definitely something that you should do if you want to be part of the top 1 % of sellers. My main advice would be to carry out your A/B tests in a strategic way.

Make your hypotheses. Create a priority list. Test one thing at a time. Manage your expectations.

And remember that optimizing your Amazon listing with A/B tests is a long-term game.

If you have any questions about A/B testing on Amazon, feel free to reach out to me at martin@onepercentsellers.com.

Good luck!