When to use a multi-armed bandit instead of an A/B test

The framing you usually see is a fight: bandits versus A/B tests, one of them modern and one of them obsolete. That framing is wrong, and it produces bad decisions in both directions — teams who put a bandit on a pricing change they needed to defend in a board meeting, and teams who spend six weeks and a hundred thousand users proving that one of five button labels is marginally better.

The two methods have different objectives. An A/B test is built to produce a reliable estimate of a difference. A bandit is built to maximize reward while it figures the difference out. Both are answering a question; they are not answering the same one.

So the useful question isn't which is better. It's which question am I actually asking — and there are four checks that settle almost every real case. This article walks through them, then through the five situations teams actually run into, including one where the honest answer is neither.

If you want the short version of the comparison itself — the mechanics, a side-by-side table, the measured gap — that lives on the multi-armed bandit vs. A/B testing page. This is the longer, more practical companion: how to decide.

Learning versus earning#

Start with the smallest possible example. You have four labels for a signup button — "Sign up", "Start free", "Get started", "Try it now" — and you want the one that converts best.

A conventional A/B test splits traffic evenly, 25% each, and holds that split until you have enough observations to call it. That design is not an accident or a limitation. It's the point. Fixed allocation is what makes the resulting estimate clean: every variant gets the same kind of traffic, at the same times, in the same proportions, so the difference you measure at the end is attributable to the variant rather than to how you allocated.

The cost of that cleanliness is that you keep sending a quarter of your traffic to the worst label for the entire run, including the part of the run where you already have a fair idea that it's the worst.

A bandit asks a different question on every request: given everything observed so far, where should this visitor go? Early on, the answer is close to an even split, because it doesn't know anything yet. As evidence accumulates, allocation moves — toward the labels performing well, away from the ones that aren't, while still sending enough traffic to the laggards to catch the case where an early bad streak was just noise.

The compressed version:

A/B tests optimize for information. Bandits optimize for reward.

The quantity that makes this concrete is regret — the gap between what you actually earned and what you would have earned if you had known the best variant from the first user. You can't observe regret while running, because you don't know which variant is best. That's the whole problem. But you can measure it in simulation, and the shape of the result is more interesting than the size.

We benchmarked this across sixty runs of 13,944 users each, three arms, a genuine 20% relative lift buried under a 10% baseline. The bandit paid 34% less regret than the fixed split. But the number that taught me something wasn't the average — it was the variance.

The A/B test paid 232.4 regret on every single run, identical to three decimal places, across twenty independent seeds. The bandit's regret varied: 114 on a good run, 195 on a bad one.

Four questions that settle it#

In rough order of how often they disqualify one method outright.

1. Does this decision repeat?#

This is the one that decides most cases, and it's the one people skip.

Some decisions happen once. You redesign onboarding, you ship it, and the old onboarding no longer exists. There is no stream of future users to allocate — there's one change, and a question about whether it worked. That's an experiment. Run an A/B test.

Other decisions happen on every request, forever. Which headline. Which recommendation. Which promo. Which model to route this prompt to. Which of six onboarding paths for this particular signup. Nothing about these is ever "finished" — the next user arrives and the decision is made again from scratch.

A repeated decision is a bandit problem almost by construction, because the thing a bandit does — improve a choice you keep making — is only valuable if you keep making it. Run an A/B test on a repeated decision and you get a winner, freeze it, and stop learning at exactly the moment you started having enough traffic to learn well.

A useful test: after the experiment ends, does the code still contain a branch? If the answer is no, we ship the winner and delete the other variants, that's an A/B test. If the answer is yes, we keep choosing per user, that's a bandit, and the A/B test was just a slow, expensive way to hard-code the first thing you learned.

2. How fast does the reward arrive?#

A bandit improves by closing a loop: select, observe, learn, select better. The loop can only turn as fast as its slowest leg, and that leg is almost always your feedback.

User sees the CTA → user clicks closes in seconds. User sees the pricing page → user subscribes → user is still subscribed in six months closes in six months. Same architecture, wildly different problem.

Two separate things go wrong when the reward is slow, and they're worth keeping apart:

The obvious one is that learning is slow. If rewards land weeks after the selection, the algorithm is steering on data that describes a version of the product from a month ago. Late feedback is not lost — any competent implementation correlates a reward back to the selection that earned it, however long it takes to arrive. It just doesn't help you allocate today's traffic, which is the thing you were buying.

The less obvious one is that you can't validate the reward you chose. Slow-reward problems tempt you into a proxy — optimize clicks because subscriptions take too long. That is a real technique and often correct, but it means the bandit will faithfully maximize the proxy, including in the cases where the proxy diverges from the thing you actually wanted. A bandit is a very effective way to over-optimize a badly chosen metric, and it will do it faster than an A/B test would.

Worth separating infrastructure latency from funnel latency here, because they get conflated. A production bandit typically reflects new feedback in its allocation within seconds to a minute. That number is irrelevant either way: if your reward horizon is minutes, it's already fast enough, and if your reward horizon is months, the bottleneck is your funnel and no platform's latency figure touches it.

3. Do you have the traffic?#

The question nobody asks, and the one that most often means the answer is neither method, not yet.

There is no universal number here, and you should distrust anyone who gives you one — including the round figures that happen to match a pricing tier. What you need is set by two things: your baseline conversion rate and the size of the effect you're trying to detect. Both matter enormously, and the second one matters quadratically.

Here's what a standard two-proportion z-test asks for at α = 0.05 and 80% power, per variant:

Multiply by the number of variants. Four button labels on a 5% baseline, chasing a 10% lift, is roughly 125,000 users. The same four labels on a 2% checkout funnel, chasing a 5% lift, is 1.26 million.

Two things fall out of that table, and they're the whole point of it.

The effect size dominates. Sample size scales with the inverse square of the difference you're chasing, so halving the effect roughly quadruples the traffic you need. Moving from a 20% lift to a 5% lift on the same funnel is a 15× change in cost. This is why "we'll just test everything" quietly fails: the interesting remaining changes are usually the small ones, and small is expensive.

Low-converting funnels are brutally expensive to test. Dropping the baseline from 10% to 2% multiplies the requirement roughly fivefold at every effect size, which is exactly why checkout and signup funnels — the places teams most want to optimize — are the hardest places to get a clean read.

Now hold your row against the traffic you actually have. If the surface sees 3,000 users a month and you need 125,000, that is a three-year experiment, and the product will have changed a dozen times before it finishes.

A bandit does not rescue you here, and it's worth being direct about that, because it's the most common way people get sold one. Bandits reduce regret — the traffic wasted on the losing variant. They do not manufacture statistical power. If you can't distinguish 10% from 12% with the traffic you have, no allocation strategy changes that, and a bandit will spend most of that traffic exploring, because exploring is the correct behavior when you know almost nothing.

What a bandit genuinely does better at low volume is fail more gracefully. It doesn't need you to pre-commit to a sample size or a stopping rule, it doesn't invalidate itself when you peek at it, and it keeps allocating sensibly under uncertainty rather than sitting at an even split until a threshold clears. But sensible-under-uncertainty means close to even. The honest framing is that it degrades gently toward random rather than failing loudly — not that it works at volumes where nothing works.

4. Do you need a defensible causal number?#

Sometimes the deliverable is not a better conversion rate. It's a number that has to survive scrutiny — from a regulator, a board, a customer contract, or an analyst who will pull on it for a week.

An A/B test gives you that: a pre-registered hypothesis, a fixed design, a known error rate, an estimated effect size with an interval around it. You can write it down, and it means the specific thing everyone agrees it means.

A bandit gives you something weaker on this axis, for a structural reason worth understanding rather than just accepting. Because allocation depends on outcomes observed so far, the data a bandit produces is not independently sampled — the amount of traffic a variant received is itself a function of how well it was doing. Naive statistics on that data are biased. There is a real literature on inference from adaptively collected data, and there are estimators that correct for it, but "run a t-test on the bandit's logs" is not one of them, and the resulting number will be wrong in a direction that flatters the winner.

So: if the number is the deliverable, run the test. If the improved outcome is the deliverable and the number is just how you check your work, the bandit's own regret and allocation history tell you what you need.

There's an honest counterpoint in the benchmark, too. The fixed split identified the truly-best arm in 100% of runs; the bandit managed 95%. One run in twenty, it finished slightly favoring the wrong variant. That's a real reliability edge for A/B testing — and it comes bundled with paying the full 232.4 regret on every run, including all the ones where it correctly picked the winner.

The decision table#

* This is specifically a contextual bandit — see below.

The last row is not an oversight. Below that volume, neither method will resolve a small effect, and the right move is to change the experiment rather than the method.

Five scenarios#

Checkout CTA#

Three button labels, maximizing completed purchases, outcome observed within seconds.

Bandit, comfortably. Repeated decision, fast reward, clear business metric, and the losing variant costs real money on every impression it gets. This is the shape the whole method was designed for.

A pricing restructure#

You're changing how the product is packaged, and you need to understand the effect on conversion, revenue mix, segment behavior and probably retention.

A/B test. Multiple outcome metrics, some of them slow, a decision that gets made once, and a result that has to be defensible to people who weren't in the room. Every one of the four checks points the same way. A bandit here would optimize one metric you picked in advance and tell you nothing about the other four.

Recommendation ranking#

Several ranking strategies, and the best one plausibly differs per user.

Contextual bandit. The key detail is that a global winner may not exist — averaging over your whole user base can produce a "winner" that is worse than the alternative for every individual segment. A non-contextual method cannot see this, and will confidently report the average.

Homepage headline#

Five headlines, maximizing signups, changeable without a deploy, feedback immediate.

Bandit. Same shape as the CTA, with a specific extra: headlines go stale. Copy that wins in January is not obviously winning in June, and a bandit that keeps running keeps tracking that, where a test hands you a winner and a slow decay you won't notice. Drift also changes which algorithm is right: methods that assume a fixed underlying rate degrade when it moves, and the ones built to discount old evidence start winning. If you don't want to re-diagnose that by hand every few months, it's an argument for a method that adapts its own strategy rather than a single fixed policy.

Rolling out a new feature#

You want a new feature to reach 5% of users, then 25%, then everyone, with the ability to stop instantly.

Neither. This is a rollout, not an experiment. There is no reward being maximized and no hypothesis being tested — you want controlled exposure and a kill switch. That's a feature gate, and dressing it up as an experiment adds machinery without adding an answer.

Worth stating plainly because it's the most common category error in this space: not everything with variants is an experiment. Ask what the mechanism is for. Optimization maximizes a reward. Experimentation estimates an effect. Rollout controls exposure. Three different jobs, and only two of them are on the bandit-versus-test axis at all.

When the answer depends on who's asking#

A standard bandit learns which variant is best overall. A contextual bandit learns which variant is best for this request, given what you know about it — device, geography, traffic source, plan tier, session depth, time of day.

This is a different problem, not a tuning option, and the difference shows up when there's no global winner. Suppose one onboarding flow works well for users arriving from search and badly for referrals, while a second is the reverse. A non-contextual method finds the one with the better average, and ships it to everyone, including the half of your traffic for whom it's the worse choice. It cannot see the split, because the split isn't in the data it's looking at.

The cost is that context isn't free. Every dimension you add divides your data — you're now learning a separate answer per region of the context space, and the traffic requirement from question three applies to each region rather than to the whole. Adding a context feature with eight values can mean needing eight times the traffic to reach the same confidence.

The practical rule: add a context dimension when you have a specific reason to believe it changes the answer, not because the field happens to be available in the request. Every feature you add is a bet that the split is real, paid for in traffic.

Most teams need both#

The choice is per-decision, not per-organisation, and a mature setup runs all of these simultaneously:

• A/B tests for the handful of foundational questions a year where you need a defensible number — pricing, positioning, a major flow.
• Bandits on the high-volume repeated surfaces where the reward is fast and the traffic is real — CTAs, headlines, ordering, offers.
• Contextual bandits where the best answer genuinely varies by user, and you have the volume to support it.
• Feature gates where the requirement is controlled exposure and the ability to stop.

They interact usefully, too. An A/B test is a reasonable way to decide whether a whole family of variants is worth optimizing at all before you point a bandit at it. And a bandit's allocation history is a decent early-warning signal for when a settled question has come unsettled and deserves a proper test.

What happens after the experiment ends#

Here's the reframe that I think matters most.

An A/B test has a shape:

experiment → winner → rollout → next experiment

Adaptive optimization has a different one:

decide → observe → learn → decide better → ...

The second loop doesn't terminate. That's not a missing feature — it's the whole proposition. For a genuinely one-off decision, termination is exactly what you want: you learned the thing, you shipped it, done. For a decision your product makes ten thousand times a day, termination is the bug. You froze an answer that was correct for a fortnight in spring, and now it's a constant in your codebase that nobody will revisit.

This is already how the systems that operate at scale work. Recommendation engines don't run a two-week test and hard-code the winner. Ad auctions, search ranking, dynamic pricing — all of them treat every request as both a decision and an observation. There has never been a good reason for product experimentation to be the one place that stays a static traffic split; the reason it did is that the infrastructure was hard to build, not that the model was right.

So: which one?#

Run an A/B test when the deliverable is knowing something reliably about the effect of a change.

Run a bandit when the deliverable is a better outcome on a decision you'll make again tomorrow.

Run a contextual bandit when the right answer depends on who's asking, and you have the traffic to learn it per segment.

Use a feature gate when what you want is controlled exposure, not an answer.

And before any of that, check the traffic against the effect you're chasing. If the surface can't produce it in a window you're willing to wait, fix the experiment first — neither method is what's limiting you.

The version of this question worth asking isn't bandit or A/B test. It's: am I trying to optimize the experiment, or optimize the outcome?

qbrix runs this loop as a managed service — selection on the request path, training off it, and allocation that moves on its own without a winner to declare or a rollout to schedule.