Kuaishou Handed Its A/B Tests to an AI Agent

Kuaishou's A/B Agent, an AI system that studied 310 historical recommendation strategies and iteratively proposed production configurations, delivered a 4.829% gross merchandise value increase over the production comparison in its final of five successive strategies on the company's short-video ecommerce platform, according to a Kuaishou-authored paper published on 5 August 2026. The first two strategies raised the target metric while damaging wider platform measures, and a human engineer reviewed each proposed configuration before it entered the online testing platform. The agent used GLM-5.1 to generate configurations and organized historical experiments into a hierarchical tree by business domain, scenario, recommendation stage and objective.

7 min read Kuaishou’s recommendation engineers had a familiar growth problem. Improving the shopping feed required repeated experiments, but every round still depended on an expert to retrieve old tests, design the next strategy, read the results and decide what to change. Kuaishou tested whether an AI agent could take over more of that loop. It did not ask the system to write ad copy or recommend products directly. It asked the agent to study 310 previous recommendation strategies, propose a production configuration, learn from each A/B result and choose the next move. In a Kuaishou-authored paper published on 5 August 2026 https://arxiv.org/abs/2608.04625 , the team documents five successive strategies on its short-video ecommerce platform. The final configuration produced the strongest reported business result, with gross merchandise value up 4.829% against the production comparison. The experiment is more useful for what happened before that number: the first two strategies raised the target metric while damaging wider platform measures. Recommendation work was stuck in a manual loop Kuaishou https://www.kuaishou.com/en operates short-video, livestreaming and ecommerce services. Its shopping recommender has to decide which commercial videos users see while protecting the broader experience that keeps them watching, browsing and buying. A recommendation strategy can span retrieval, ranking, blending, scoring rules and parameter settings. Changing one part may improve ecommerce GMV while reducing watch time or revenue elsewhere on the platform. The paper says engineers traditionally spend hours analysing a typical experiment and selecting the next configuration. The constraint was therefore not a shortage of tests. It was the time and specialist judgement needed to connect them. Historical knowledge sat across documents, code, configurations and metric logs. Similar experiments could still differ in user intent, available features, pipeline stages and guardrails. The documented idea behind the Kuaishou A/B Agent was to turn those scattered records into reusable operating knowledge. The system would retrieve relevant precedents, generate an executable strategy and then use live experiment results to revise its next move. The agent organised 310 old strategies The team assembled an industrial benchmark from 310 historical strategies across three Kuaishou ecommerce scenarios. These covered retrieval, ranking, blending and other recommendation stages. Kuaishou is framing this as automation of the experimentation process itself, a claim worth weighing against how other vendors sell automation. Our deeper analysis of UiPath and a 15-person rival selling two different ideas of automation https://industrycontents.com/uipath-and-warmwind-agentic-automation/ lays out how those vendor claims tend to diverge from what actually ships. Instead of storing each experiment as an isolated document, the agent broke it into a motivation, mechanism, parameters, applicable context, outcome and risk. It then arranged those pieces in a hierarchical tree by business domain, scenario, recommendation stage and objective. When the team supplied a new optimization request, the system identified the scenario, target metric, guardrails and deployment constraints. It retrieved several possible precedents and used GLM-5.1 to generate a proposed configuration. A human engineer reviewed the strategy before it entered the online testing platform. This distinction matters. The AI recommendation strategy was not deployed without oversight. The paper describes an approval step before launch, although it does not say how often engineers rejected or modified the agent’s proposals. Once a test ran, the agent collected the core and guardrail metrics, compared the new version with its parent and sibling experiments, and recommended another parameter change. Promising branches triggered local tuning. A saturated branch or a guardrail failure could send the system back to retrieve a different mechanism. The first gains came with hidden costs The five-round sequence is the clearest evidence in the report. Strategy 1 increased GMV by 1.123%, but its four reported guardrail metrics declined by 0.405% on average. Watch time, live GMV and platform GMV all moved in the wrong direction. Strategy 2 pushed GMV to 2.984% and produced the largest click gain in the sequence, at 1.167%. Every guardrail remained negative. A team looking only at ecommerce conversion optimization could have mistaken the larger target lift for a clean win. The agent then changed the objective weights and decision thresholds. Strategy 3 raised GMV by 3.299% while the average guardrail change became positive for the first time, at 0.223%. Live GMV still declined slightly by 0.067%. Strategy 4 held GMV near the same level at 3.253% and kept the guardrail average positive. The authors describe this round as a stability check rather than a meaningful regression. Strategy 5 became the selected operating point, with every disclosed measure positive. This progression makes the test more informative than a single result. The early variants show why automated A/B testing still needs business guardrails. The system found configurations that improved the local target before it found one that avoided visible damage elsewhere. The final strategy raised ecommerce GMV Kuaishou deployed the final strategy in its ecommerce short-video recommendation system. Against the production comparison, the company reported GMV up 4.829%, gross profit measure up 4.677%, order profit measure up 0.841%, conversion rate up 0.578%, clicks up 0.370% and orders up 1.053%. The paper also compares Strategy 5 with an expert-designed strategy. The agent’s final configuration improved GMV by another 2.150 percentage points and the gross profit measure by 2.941 points. Watch time, live GMV and platform orders also moved more favourably than under the expert version. These are company-reported relative changes. The paper does not disclose the number of users, traffic allocation, experiment duration, absolute baselines, confidence intervals or p-values for the production sequence. It says the system extracts statistical confidence from experiment records, but it does not publish that uncertainty for the final test. The comparison also evaluates the complete agent-generated strategy. It does not isolate the contribution of the experience tree, retrieval method, language model, human review or individual parameter changes. The observed lift therefore belongs to the combined intervention. The growth loop became the product The transferable part of the experiment is not that every growth team needs an autonomous experimentation agent. Kuaishou had hundreds of previous strategies, a large production platform, mature metric infrastructure and enough traffic to run successive tests. Those conditions are difficult to reproduce. The practical idea is smaller. Historical tests become more useful when their context, mechanism, constraints and failure modes are stored alongside the result. That structure allowed the Kuaishou A/B Agent to distinguish a superficially similar tactic from one that fit the current stage and guardrails. The sequence also shows why a primary metric cannot make the decision alone. Strategies 1 and 2 improved shopping performance while weakening the surrounding platform. The next round was not simply a larger bet on the same setting. It changed the balance between the target and the constraints. A comparable business could test this operating model before automating it. Record each experiment as a decision with context, intervention, primary outcome, guardrails and next step. Retrieve the closest precedents when designing the next test. Let an AI experimentation agent propose configurations, but require human approval and a predeclared set of measures that can stop the rollout. Kuaishou’s paper ends with another unresolved constraint. Future work includes better requirement alignment, more accurate attribution and automated checks that a proposed strategy can be engineered safely. The experiment moved strategy selection closer to a closed loop, but the missing disclosures and retained human review show where that loop still opens. Other platforms are testing similar ground. Taobao Live trained its AI shopping hosts to keep working when operators changed the rules around them https://industrycontents.com/taobao-live-a-b-test-ai-shopping-hosts/ , and ShareChat rebuilt its own testing pipeline to reach significance on revenue metrics faster https://industrycontents.com/sharechat-revenue-a-b-tests/ . Both point to the same shift as the Kuaishou A/B Agent: the experimentation process itself is becoming something companies test and optimise, not just a tool they use.