A/B Testing Limitations: Why Your Tests Miss Fast Shifts in Shopper Interest

For a week, the numbers looked steady. Then, within a few hours, the revenue lift we had been measuring disappeared. We hadn’t changed anything on the site.

That happened at Betsey Johnson while we were building Product Genius. Shopper interests, ad campaigns, competitor pricing and new product launches were all moving, and they moved abruptly. By the time a change showed up as statistically significant, it was often too late to do much about it.

If you test changes on your store, the timing matters this month. A standard two-week test started on November 10 finishes on November 24, two days before Thanksgiving. Its result describes shoppers from early November. The people who arrive on Black Friday and Cyber Monday come through different channels, on different devices, looking for different deals.

A/B tests are the right tool for proving that a change worked. They are a slow tool for reacting when your shoppers change. Below are four places where that gap shows up, and what you can do about each.

What an A/B test is built to do

A controlled experiment answers one question well: did this change cause that outcome? Shoppers are split at random, one group sees the change, and the comparison removes most of the noise that makes before-and-after reports unreliable.

Testing is also humbling. Not every hypothesis of what you want to test turns into a winning idea. Without testing, most of those ideas would have shipped on intuition.

The method rests on an assumption that is easy to miss: the effect you are measuring holds still long enough for you to measure it. That is why standard practice is to fix the sample size in advance and run in whole weeks, usually at least two, so weekday and weekend shoppers are both represented.

Two to four weeks is a long time in e-commerce.

1. Shopper interest moves faster than a test collects evidence

A test assumes that the shoppers in week one behave like the shoppers in week three. Several ordinary events break that assumption:

• A new ad campaign changes who arrives and what they expect to find.
• A competitor cuts prices, and the same product page converts differently.
• A product launches or sells out, and attention moves across the catalog.
• A sale event pulls demand forward by weeks.

The holiday season makes this concrete. Adobe’s 2026 forecast says the October sale event has “changed the shape of the holiday season,” creating a spike in shopping before the season starts. Adobe expects online sales of personal hygiene products to rise 150% and clothing basics 210% during Cyber Week, compared with September. It also forecasts that traffic to retail sites from AI chat and browser tools will grow 130% over the season and 159% on Thanksgiving alone.

A test that started in early November measures a different mix of shoppers from the one arriving at the end of the month. The result is still valid for the period it covered. It answers a question about a period that has already passed.

That was our experience at Betsey Johnson. The lift held for a week and then disappeared within a few hours, with nothing changed on our side. Aggregate statistics could only confirm the change after it had happened.

2. Checking early makes the problem worse

When results feel urgent, the natural response is to watch the dashboard every day and stop the test as soon as it crosses significance. This is called peeking, and it undermines the statistics.

A standard significance test assumes you look once, at the end. Each extra look gives random noise another chance to cross the threshold. Researchers at Stanford who studied continuous monitoring found that, even with 10,000 samples, “Type I error can easily increase fivefold.” In other words, a test designed to produce false winners 5% of the time can produce them much more often.

The base rate adds to the problem. If only about one in ten ideas truly improves the metric, a significant result is wrong far more often than 5% of the time. Kohavi and colleagues estimate that at a 10% success rate, the chance that a significant win is a false positive is about 22%. Statistical methods built for monitoring results as data arrives do fix the false-positive problem. They do not make the data arrive any faster.

3. The effect changes while you watch

Even a well-run test can measure an effect that is still moving. Researchers at Microsoft, which runs more than 1,000 online experiments a month, note that “observed treatment effects are not always permanently stable.” Shoppers may react to something new with curiosity that fades (a novelty effect), or warm to it slowly as they get used to it (a primacy effect).

Google’s ads researchers found the same pattern over longer periods: a short-term result does not always predict what happens once users adapt.

A winning variant is a snapshot. If it won during the two weeks you measured, that is useful evidence. It is not a guarantee about the next two weeks, and it says little about a week unlike the test period.

4. Segments multiply the wait

When an overall result looks flat, teams split it. Social traffic on mobile responds one way; shoppers from search respond another. At Betsey Johnson, the lift at any given moment differed sharply between segments like these, and tracking each one separately meant fewer sessions behind every estimate, which made tracking slower still.

Detecting a difference between segments costs more again. Columbia statistician Andrew Gelman shows that, under common assumptions, estimating a difference between two groups takes about 16 times the sample size needed for the overall effect; under other assumptions it is closer to four times. Either way, a segment-level answer takes many times longer than an overall one, and the mix between segments keeps moving. Adobe expects mobile devices to account for 57.4% of online holiday spending, but 63% on Thanksgiving.

Measure with statistics, act with beliefs

The answer is not to stop testing. It is to separate two jobs that A/B tests are usually asked to do at once: proving what works, and deciding what to show the shopper in front of you right now.

Proving what works still belongs to controlled experiments with a fixed plan. Deciding what to show can use a different approach. Adaptive methods, often called multi-armed bandits, move traffic toward what is working as evidence arrives instead of waiting for a final verdict. When Google built this approach into its analytics product, it described bandit experiments as “typically much more efficient than ‘classical’ A-B experiments.” In its simulated example, the bandit finished 175 days sooner on average than a classical test and picked the right winner 96.4% of the time.

When conditions keep changing, you need a Seller’s Agent to be able to learn as quick as those changing conditions.

This is the principle we built Product Genius on. Instead of waiting for statistical significance, the system updates what it believes about each shopper and context after every interaction, acts on that belief right away, and uses its remaining uncertainty to decide when to try something new and when to stay with what is working. The engine behind it is a Large Interaction Model, or LIM, which learns from each shopper’s scrolls and clicks in real time.

Measurement still runs on standard statistics. Across strict A/B tests with 25 brands, Product Genius produced an average 36% increase in revenue. We averaged the lift over every day of each test, not only the strongest days at the end.

A checklist for your next test

Before you start a test this season:

• Set the sample size and end date before launch, and write them down.
• If you need to watch results daily, use a sequential method designed for it. Otherwise, wait for the end date.
• Check that each segment you plan to report has enough traffic on its own.
• Note what was unusual about the test period, such as promotions, launches or holiday traffic, and treat the result as a description of that period.
• Decide which questions need proof, and keep testing those. Decide which need a fast response, and use a system that adapts as shoppers change.

Frequently asked questions

What are the main limitations of A/B testing in e-commerce?

A test needs weeks of stable traffic to reach a reliable result, while shopper interest can shift within hours. Checking results early inflates false positives, effects can fade or grow during the test, and splitting results by segment multiplies the traffic you need.

How long should an A/B test run?

Long enough to reach the sample size you set before launch, in whole weeks, and usually at least two weeks so weekday and weekend shoppers are both included. A typical conversion test needs tens of thousands of users per variant, and often more.

Why is peeking at A/B test results a problem?

Standard significance tests assume one look at the end. Checking repeatedly and stopping when a result looks significant gives random noise many chances to cross the threshold, which can raise the false-positive rate several times over.

What is the novelty effect in A/B testing?

Shoppers sometimes respond to a change simply because it is new. That early lift fades as they get used to it, so a short test can overstate the long-term effect.

Is a multi-armed bandit better than an A/B test?

They do different jobs. A fixed A/B test is better for proving whether a change caused an outcome. A bandit is better for moving traffic toward what works while conditions are changing.

Want to see how Product Genius adapts between tests? Read the full methodology in our technical whitepaper, or request a demo and bring us your least predictable category.