Research on the Optimization of A/B Testing System Based on Dynamic Strategy Distribution
Round 1
Reviewer 1 Report
This article describes a dynamic traffic management to address the AB testing problems met on e-commerce platforms.
Unfortunately, this article is very difficult to read due to the poor quality of the writing and the lack of structure.
The used terminology must be correctly formulated.
The state of the art needs to be reviewed, especially on the corrective aspect of multiple tests (Bonferroni correction) and dynamic allocation with multi-armed bandit methods which are almost not discussed. The illustrative drawings are appreciated but unfortunately it's not enough to understand the article.
Author Response
Please see the attachment.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
A/B testing, also known as A/B/n testing, is a method to compare the effects of different strategies through objective indicators to select the best. A/B testing is a process of randomly assigning two (A/B) or multiple (A/B/n) different versions of strategies to some users in real scenarios at the same time, collecting experience data and business data of each group of users, analyzing and evaluating the best version through statistical methods, and finally adopting it.
This paper proposes an idea of dynamic strategy distribution, and based on this idea, a configuration-driven traffic multiplexing A/B testing system is designed. Through the asynchronous distribution of experimental strategy configuration information and the dynamic traffic management method realized by the method of traffic layering and multiplexing, the problem of high concurrency pressure can be solved, and the A/B testing system can be driven in a lightweight, flexible, and efficient manner.
In the scenario where multiple business systems share the same A/B testing service and the rapidly growing business volume brings a large number of experiment requests, the A/B testing service must be able to run independently in the form of a third-party system, and the independent A/B testing service needs to be highly reusable, scalable, and lightweight. These characteristics are embodied in the following aspects. First, the system can conduct multiple experiments at the same time, and can promptly and quickly respond to a large number of users' experiment strategy requests, analyze and issue these strategies. That is, it can withstand high concurrency pressure. Second, the system can realize traffic mutual exclusion, traffic multiplexing, and traffic orthogonality. Under the premise of ensuring reliable experimental results, it can dynamically manage traffic and achieve flexible start and stop of experiments.
If the scale of the experiment expands, and the number of users and experiments increases, such a communication method will put huge concurrency pressure on the experiment management module which stores the experiment strategy. The core of dynamic strategy distribution is to implement asynchronous message delivery and system decoupling by setting up modules such as message middleware, so that each business system can undertake the access of its own users, share access pressure, and solve high concurrency problems.
Compared with the method used in most A/B testing services currently, the biggest advantage of the "dynamic" method is that the A/B testing system does not communicate every time it receives a request for an experiment strategy. There is no need to occupy a high bandwidth to send a complete experiment strategy, but only distribute a small amount of configuration information at a few special time nodes such as the time when strategies are updated, which allows each module of the A/B testing system and business systems to keep abreast of the latest experimental strategies.
Authors claim that the proposed configuration-driven traffic multiplexing A/B testing system has advantages compared with the A/B testing systems currently used in the market but there are no experiments provided that proof it.
But, it is not enough only to write “The entire A/B testing system is driven by the dynamic distribution of strategy configuration information. In this way, when the A/B testing system we propose is applied to business systems with a large number of experiments and users, it can avoid frequent communication with a large amount of data when a large number of experimental strategy requests occur, solving the high concurrency problem faced by the traditional A/B testing system when docking with multi-service systems.”
It would be good to outline some experimental data and corresponded analysis to show possibilities of the proposed system.
Author Response
Please see the attachment.
Author Response File: Author Response.pdf
