Spotify Engineering explores the viability of using Large Language Model (LLM) predictions to replace human outcomes in A/B tests, weighing the benefits of speed against the loss of causal identification.
The article "When Can LLMs Replace Humans in A/B Tests?" from Spotify Engineering critically examines whether Large Language Models (LLM) can effectively substitute human outcomes in A/B tests. While …
The article "When Can LLMs Replace Humans in A/B Tests?" from Spotify Engineering critically examines whether Large Language Models (LLM) can effectively substitute human outcomes in A/B tests. While acknowledging the promise of faster results by circumventing the need for user traffic, the authors emphasize that this approach inherently sacrifices the causal identification of treatment effects, which is a cornerstone of randomized experiments.
To address this, the article introduces a robust framework derived from surrogate endpoint theory in biostatistics. This framework outlines two crucial conditions for LLM outputs to serve as valid surrogates for human outcomes: first, "Surrogacy," meaning the LLM output must fully mediate the treatment effect on the human outcome; and second, "Comparability," which dictates that the relationship between LLM predictions and human outcomes must remain consistent across different experiments.
Empirical evaluation utilizing the Upworthy Research Archive revealed significant findings. Raw LLM predictions were found to be systematically biased, consistently underestimating treatment effects. While machine learning models (such as random forest and gradient-boosted trees) could be employed for calibration to recover human treatment effects when both the surrogacy and comparability conditions were met, the authors highlight a critical caveat: these conditions cannot be guaranteed, especially when dealing with novel treatments or significantly different interventions from past experiments.
The article concludes emphatically that human experiments remain indispensable for driving genuine product innovation. The upfront investment required to collect actual user responses is deemed crucial for the trustworthiness and reliability of any LLM-based A/B testing approach, particularly when exploring new ideas. Although LLMs can enhance human experiments by filtering out weaker ideas or reducing result variance, they should not fully replace human outcomes due to the inherent and often unverifiable assumptions involved in their use as surrogates.
This document explains how to calibrate Marketing Mix Models (MMMs) using Return on Ad Spend (ROAS) estimates derived from external experiments, treating these estimates as an additional observed likelihood to improve model accuracy and address biases.
This document outlines a method for calibrating Marketing Mix Models (MMMs) using Return on Ad Spend (ROAS) estimates from experiments, as demonstrated in a PyMC-Marketing notebook. Unlike traditional…
This document outlines a method for calibrating Marketing Mix Models (MMMs) using Return on Ad Spend (ROAS) estimates from experiments, as demonstrated in a PyMC-Marketing notebook. Unlike traditional prior placement, this approach treats external ROAS information as an additional "observed likelihood." This means the experiment's ROAS is viewed as a noisy observation of the model's implied ROAS, with the degree of noise and influence controlled by a `sigma` parameter. This method helps to address under-determined MMMs and mitigate bias.
A key distinction is that ROAS is a derived quantity, not a direct model parameter, thus justifying its incorporation as a likelihood term rather than a prior. This calibration acts as a soft constraint, allowing the model to balance the experimental ROAS with the primary sales likelihood. A smaller `sigma` signifies greater trust in the experimental data, pulling the model's estimates closer to the observed ROAS. In PyMC-Marketing, this is implemented via the `add_cost_per_target_calibration` method, which requires the channel, the measured ROAS value, and `sigma`. It's crucial that the model exposes channel contributions in their original units using `add_original_scale_contribution_variable` before applying calibration.
The tutorial utilizes a simulated dataset to illustrate the method's effectiveness. This dataset includes an unobserved confounder that significantly biases the ROAS estimate for one channel (`x1`). The demonstration highlights how `sigma` directly controls the calibration's influence: a "tight" calibration (small `sigma`) effectively corrects biased ROAS estimates and rectifies channel rankings, bringing them much closer to the true values. Conversely, a "loose" calibration (large `sigma`) has a gentler, less corrective effect. A crucial insight from the notebook is that in-sample predictive fit (e.g., CRPS) does not necessarily correlate with causal correctness. Models with good predictive accuracy might still provide inaccurate causal insights and lead to suboptimal budget allocation.
Beyond ROAS, this likelihood calibration method can also be adapted for Cost Per Acquisition (CPA) by setting `target_per_cost=False`. It's important to note that the uncertainty (`sigma`) should be calibrated on the scale the experiment reported its standard error, as ROAS and CPA uncertainties are not simple reciprocals. In summary, while powerful for enhancing MMMs, the success of this calibration technique hinges on appropriately setting `sigma` to reflect the trustworthiness of the experimental data, and recognizing the critical difference between achieving predictive accuracy and deriving causally correct insights.
This article explores how Bayesian borrowing can significantly reduce the size of control arms in clinical trials by reusing historical data. It provides a worked example in PyMC, demonstrating the design, validation, and read-out of such a trial, aligning with new FDA guidance.
Bayesian borrowing offers a powerful method to enhance the efficiency of clinical trials by leveraging historical control data, thereby allowing for significantly smaller control arms in new studies. …
Bayesian borrowing offers a powerful method to enhance the efficiency of clinical trials by leveraging historical control data, thereby allowing for significantly smaller control arms in new studies. Traditionally confined to methodological discussions, this approach is now gaining traction for pivotal trials, especially following the FDA's January 2026 draft guidance on Bayesian methodology. A key challenge, however, is to ensure the design's robustness, particularly when historical data might not perfectly align with current trial conditions.
The effectiveness and regulatory acceptance of Bayesian borrowing hinge critically on the choice of prior distribution for the control rate. The article strongly advocates for a "robust mixture prior" over simpler alternatives like vague, fully pooling, or fixed power priors. This adaptive prior combines an informative component derived from historical evidence with a vague, heavy-tailed "guard" component. This unique structure allows the prior to borrow heavily when new trial data corroborate history, but gracefully yield and follow the new data if a conflict arises. This adaptability is paramount for managing type I error rates under prior-data conflict, a crucial regulatory safeguard.
A defensible Bayesian borrowing design necessitates a rigorous, multi-step process. This includes establishing the exchangeability of historical and current populations, quantifying the effective sample size borrowed, justifying the mixing weight of the prior, and most importantly, stress-testing the design through extensive simulations. These simulations evaluate operating characteristics like type I error under conflict and power under agreement. The article also introduces "assurance" (or Bayesian power) as a more comprehensive metric for success probability, reflecting planning beliefs across the full distribution of possible effects rather than a single optimistic point.
The practical demonstration using PyMC vividly illustrates the benefits: a lean 40-patient control arm, when augmented with a robust mixture prior, achieves a level of precision and a probability of success (assurance of ~0.86) comparable to a full 150-patient control arm without borrowing. This not only translates into substantial savings in patient enrollment (e.g., 110 fewer patients) and associated costs but also ensures the viability of trials that might otherwise be underpowered. The code examples emphasize the prior's automatic adjustment when historical data conflict with new observations, satisfying the FDA's requirement for prior-data-conflict discipline and validating Bayesian borrowing as a credible and efficient tool in clinical development.
Decision Hub is an open-source registry and package manager created by PyMC Labs, specializing in modular skills for AI coding agents across data science, statistics, and machine learning workflows. It facilitates the publication and secure utilization of prompts and code packages, supporting over 40 agents with robust security and automated evaluation.
# Decision Hub: Empowering AI Agents with Verified Skills
Decision Hub, developed by PyMC Labs, emerges as a pivotal open-source registry and package manager tailored specifically for AI agent skills…
# Decision Hub: Empowering AI Agents with Verified Skills
Decision Hub, developed by PyMC Labs, emerges as a pivotal open-source registry and package manager tailored specifically for AI agent skills in the domains of data science, statistics, and machine learning. This innovative platform allows users to publish modular packages of prompts and code, known as "skills," which can then be seamlessly integrated and utilized by a diverse array of AI coding agents, including prominent names like Claude Code, Cursor, and GitHub Copilot. Its design enables cross-environment installation with a simple command, streamlining the process of deploying specialized AI functionalities.
A cornerstone of Decision Hub's architecture is its comprehensive security framework. Every published skill undergoes a rigorous security scanning process, meticulously designed to detect potential vulnerabilities such as shell injection, credential exfiltration, and other dangerous coding patterns. This proactive security measure culminates in the assignment of a trust grade (A/B/C/F) to each skill, providing users with a clear indicator of its reliability and safety. This emphasis on security is crucial for fostering trust in AI agent workflows.
Beyond security, Decision Hub integrates an advanced automated evaluation system. Upon publication, skills are subjected to evaluation cases executed within isolated sandboxes. An LLM judge then scores the output, ensuring the functionality and efficacy of the skill are thoroughly verified before it becomes available for use. This feature directly addresses common challenges, such as AI agents hallucinating library APIs; by installing specific skills, agents can be taught to interact correctly with designated tools. Furthermore, the platform offers an effective solution for sharing best practices, allowing them to be packaged, versioned, and distributed as readily installable components.
The entire Decision Hub ecosystem, encompassing the CLI, server, web UI, evaluation infrastructure, and security scanner, is openly accessible under the MIT license. Its open-source nature not only promotes transparency and collaborative development but also empowers users with the flexibility to self-host the platform. This capability grants organizations complete sovereignty over their AI skill supply chain, ensuring unparalleled control and customization opportunities for their AI-driven initiatives.