The Challenge
Betsey Johnson turned out to have very fast moving changes in shopper interests, ad campaigns, marketplace dynamics (e.g. competitor pricing), and new product launches. The shifts were also highly abrupt. The AI would provide consistent significant revenue lift for a week, and then suddenly within a few hours without us making any change, the revenue lift would completely disappear.
These abrupt changes seemed to happen faster than we could collect statistics. By the time the change was statistically significant, it was often too late to do much about it. To make matters worse, the lift at any given time was radically different within different traffic segments (e.g. Facebook mobile, traffic from Google search, etc.). Trying to track each of these separate lift silos separately, slimmed down the number of sessions we could average together in order to estimate the lift, making tracking of the lift even more sluggish.
The Breakthrough
The "ah-ha" was that rather than track statistics and pull levers based on those signals, we needed to trust the AI to optimize its interactions with each shopper and reinforce its policies, before there was any statistically significant feedback. This solved the problem.
We essentially had to modify our learning rate to adapt "faster than the speed of statistics". The more we have increased the learning rate, the more revenue lift we observed across all Product Genius powered websites.
The Results
To plot a single curve for revenue lift, we take the total revenue from Product Genius (PG) sessions and subtract the total revenue from Original Website (OG) sessions. Then we divide by the OG revenue.
