qbrixqbrix
Library/Analysis

The Hidden Costs of A/B Testing at Scale

Mustafa Eskin·March 2026·8 min read
The Hidden Costs of A/B Testing at Scale

The Hidden Costs of A/B Testing at Scale

A/B testing is the gold standard for causal inference in product development. Split your users, measure the difference, ship the winner. The logic is clean. The statistics are well-understood. And for a single, well-controlled experiment, it works.

But most companies aren't running a single experiment.

They're running ten. Or fifty. Or a hundred — simultaneously, across overlapping user segments, on pages that influence each other. And at that scale, the clean statistical guarantees that make A/B testing trustworthy begin to quietly collapse.

The contamination problem

When two A/B tests run concurrently on the same user population, their results are no longer independent. A user who sees a redesigned homepage (Experiment A) and a different checkout flow (Experiment B) is not the same user who sees only one change.

This is cross-test contamination — and it's not a theoretical concern. It's the default state of any organization running multiple experiments.

The standard mitigation is traffic isolation: partition your user base so each experiment gets its own exclusive slice. But this creates a direct tradeoff:

Most teams choose speed. They run experiments on overlapping populations and hope the interaction effects are small enough to ignore. Sometimes they are. Sometimes a checkout experiment silently invalidates a pricing experiment running on the same users, and nobody notices because the p-value still looks significant.

The cost isn't visible in any dashboard. It's embedded in every decision made on contaminated data.

The expiration problem

A/B tests produce a point-in-time answer: "Variant B was better than Variant A during the period of March 1–15, for the users who participated."

That answer has a shelf life.

Customer behavior is non-stationary. It shifts with seasons, marketing campaigns, competitive moves, product changes elsewhere in the funnel, and macroeconomic conditions. An experiment that showed a 3% lift in March may show no effect — or a negative effect — in July.

Yet the decision persists. The winning variant ships permanently, and the experiment is archived. Nobody re-runs it, because re-running experiments is expensive, and the organization has already moved on to the next test.

This creates a compounding problem: the older your shipped decisions, the less likely they reflect current reality. But you have no signal to tell you which decisions have expired, because you stopped measuring the moment the experiment concluded.

The parallel execution burden

Consider a mid-size e-commerce company running 30 concurrent A/B tests. To maintain statistical validity, each test needs:

  1. Sufficient sample size — typically thousands of users per variant, determined by the minimum detectable effect size
  2. Traffic isolation — exclusive user segments to avoid contamination
  3. Duration controls — enough time to capture weekly/daily behavioral cycles
  4. Multiple testing correction — adjusting significance thresholds to account for the increased false positive rate across 30 simultaneous tests

The math gets uncomfortable quickly. If each test requires 10,000 users per variant (a common requirement for detecting a 2% relative lift at 80% power), and you're running 30 tests with two variants each, you need 600,000 isolated users — simultaneously.

Most companies don't have that kind of traffic. So they compromise:

Each compromise is rational in isolation. Together, they produce an experimentation program that generates confident-looking but unreliable results.

The higher the statistical rigor, the higher the decision cost. This isn't a failure of execution — it's a structural limitation of the A/B testing model when applied at scale.

The engineering cost nobody talks about

A significant amount of engineering resources goes into experimentation infrastructure that has nothing to do with the actual product decision:

For a 30-experiment program, this infrastructure work often requires a dedicated experimentation team — data scientists, platform engineers, and analysts whose primary job is not building the product but operating the decision-making system around it.

This is engineering cost that compounds with the number of experiments, not with the complexity of the decisions being made.

Why multi-armed bandits change the equation

Multi-armed bandit algorithms don't eliminate the need for statistical thinking. But they fundamentally restructure the tradeoffs.

No traffic splitting required

A bandit algorithm doesn't split users into fixed groups. Instead, it dynamically allocates traffic based on observed performance. Every user contributes to every variant's estimate, weighted by the algorithm's current belief about which variant is best.

This means parallel decisions don't require parallel user segments. You can run 30 optimization problems on the same user population without contamination, because each decision is made independently at request time based on the current posterior estimates.

No expiration date

A bandit doesn't produce a point-in-time answer. It continuously adapts. If customer behavior shifts in July, the algorithm detects the change through its ongoing reward signal and rebalances allocation accordingly.

There is no "experiment conclusion" that freezes a decision in time. The system is always learning, always adjusting. Seasonal drift, competitive changes, and behavioral shifts are absorbed naturally.

Regret minimization, not hypothesis testing

A/B testing asks: "Is B better than A?" — a binary question that requires a definitive answer before any action is taken.

Bandits ask a different question: "Given what I know so far, which option should I show this user right now to minimize my total loss?"

This reframing has a profound practical consequence. During an A/B test, 50% of your traffic is being shown the worse variant by design, for the entire duration of the experiment. That's the opportunity cost of the information you're gathering.

A bandit algorithm minimizes this cost. It starts exploring broadly, then rapidly shifts traffic toward better-performing variants as evidence accumulates. The worse a variant performs, the less traffic it receives — automatically, without waiting for anyone to call the experiment.

Reduced engineering overhead

Because bandits operate as a continuous optimization layer rather than a discrete experiment lifecycle, the engineering overhead looks different:

When A/B testing is still the right choice

Bandits aren't universally better. A/B testing remains the right tool when:

But for the vast majority of product optimization — button colors, copy variants, pricing tiers, recommendation strategies, layout experiments — the question isn't "is B statistically significantly better than A at p < 0.05?" The question is "which option performs best right now, and how do I serve it to the next user?"

That's the question bandits were designed to answer.

qbrix implements production-grade bandit algorithms — Thompson Sampling, UCB1-Tuned, KL-UCB, and more — as a managed optimization layer. Define your arms, connect your reward signal, and let the system continuously converge on the best decision.