The regret gap: benchmarking qbrix against A/B testing

Before making a claim like this in public, I wanted to be sure it held up: that a multi-armed bandit running in real production — with all the mess that implies — can beat a traditional A/B test by a margin big enough to matter.

The theory says it should. But qbrix is a hard place to prove it, because the system pays for its own architecture. Selection happens on one set of servers; training happens on another. The numbers that connect them live in a Redis cache that refreshes on a schedule and is sometimes a few seconds stale. Feedback doesn't arrive instantly either — it travels through a queue and gets processed in batches. Every one of those choices is good for scaling and, in theory, bad for regret — the reward you give up by not always serving the best option. I've spent enough time with the math to know roughly what each one is supposed to cost me.

There's a second tax on top. By default, qbrix doesn't run one textbook algorithm. It runs auto, which is a meta-bandit: think of it as a bandit whose only job is to choose between other bandits, continuously shifting traffic toward whichever underlying learner is doing best right now. That's deliberate. You shouldn't have to know in advance whether your problem is stationary, whether your rewards are well-behaved, or whether you need protection against drift — the meta-bandit figures it out for you. But choosing between learners is its own kind of exploration, and exploration costs regret too.

So here is the claim I actually wanted to test: qbrix in its default setup, running through the real distributed stack, paying the meta-level tax, and never once told in advance which algorithm was right — still beats a clean A/B test by enough to change what you'd do. No shortcuts. No in-process bandit. No cheating on the algorithm choice.

I built a simulator to find out. The result surprised me, and it surprised me in the direction I was hoping for — which is not usually how these things go.

The setup is deliberately boring, and that's the point. Three options (call them arms), each with a coin-flip outcome: a user either converts or doesn't. One arm is genuinely the best at 12% conversion, one is genuinely the worst at 9.5%, and one sits at the 10% baseline. This is a common and genuinely hard shape of problem — a 20% relative lift buried under 90% noise. The better option really is better, just not by enough to be obvious.

How many users do you need to tell them apart? A standard two-proportion z-test at α = 0.05 and 80% power says 4,648 per arm, or 13,944 users for one full run. I ran twenty independent runs per strategy, each with its own random seed, each fully reproducible.

Three strategies went head to head. First, a uniform random baseline — no learning at all, traffic split evenly and left there. It's the control, there to show what pure noise looks like on this metric. Second, a fixed-split A/B test, the industry default: allocate evenly across all three arms for the entire run, then read off the winner at the end. Third, qbrix, configured with the auto policy and nothing else, running through a real deployment — every selection an actual HTTP call to the proxy service, routed over gRPC to the motor service, answered from the Redis parameter cache, with every outcome published to a stream and consumed by the cortex training service in batches. This is not a bandit running in a notebook. It's the production stack under simulated load.

Here is what came back.

The confidence intervals on reward don't touch — and that matters. A confidence interval is just the range the true value is very likely to fall in; when two intervals don't overlap, the difference between them is almost certainly real rather than luck. A Welch's t-test on the qbrix-versus-A/B gap comes back at t = 5.82 with 35 degrees of freedom, which puts the odds of seeing a gap this big by chance at somewhere around one in ten million. In plain terms: qbrix delivered 75 more conversions per run, a 5.21% relative lift in total reward, with enough separation that re-running the benchmark isn't going to change the answer.

The number I find more interesting, though, is regret.

Regret is the running total of what a strategy left on the table — the gap between the conversions it actually earned and the conversions it would have earned if it had somehow known the best arm from the very first user and committed to it immediately. It's the quantity bandit theory is built around, because it's the one that ties how an algorithm behaves to what it actually costs you to run.

The A/B test paid 232.4 regret per run. Not on average — exactly 232.4, run after run. The spread across all twenty runs was on the order of 10⁻¹⁴, which is essentially the smallest number a computer can represent. Every run paid the same amount, down to three decimal places.

An A/B test has no variance in its regret because it never learns from itself while it runs. It executes the plan to the end, and the bill is the bill.

qbrix, over the same twenty runs, paid 153.4 regret on average, 95% CI [143.7, 163.0]. That's a 34% reduction, with a t-statistic of -16.0 on the difference. And qbrix's regret does vary from run to run, which is the informative part: some runs cost 114, others 195, depending on whether the early draws happened to be kind. That variance is the algorithm doing its job — occasionally it catches a bad early streak and spends a few extra trials before committing, then recovers, because the meta-bandit keeps pulling traffic away from whichever learner is lagging in the moment. On most runs it moves fast. On a few it moves slowly. On every run it moves, and that's the part that matters.

The 34% gap is the one I care about, because it's what survived every tax: the distributed-system overhead, the meta-level exploration, and the decision never to tell the system which algorithm was right for the problem. It's what's left after the theory's worst-case costs have already been subtracted.

It's worth turning that regret number into money, because a percentage is easy to wave away and a dollar figure is not.

Seventy-five extra conversions per 13,944 users, at a modest $50 average order value, is $3,767 of extra revenue per experiment cycle — and that's a single 13,944-user run. But nobody runs one experiment and stops. Optimization is continuous, so the frame that actually matters is monthly traffic.

Here is the same 0.54-point conversion lift at different scales, holding the $50 order value fixed:

These aren't projections or models — they're arithmetic on the measured gap. A company making a million decisions a month picks up $270,000 a year, pulled out of decisions the A/B test was already making, just worse. At ten million decisions a month — consumer scale — it's $2.7M a month, or $32.4M a year. Same users, same funnel, same team, same product. The only thing that changed is how traffic got allocated while the system was still learning.

And none of this assumes a heroic deployment. The benchmark carried every production cost I could think to include: network latency between services, cache staleness capped by the TTL, feedback processed asynchronously in batches, and the meta-level exploration stacked on top of whichever learner was being tried. The 34% regret reduction is the number after all of it.

There's one honest counter-argument, and the benchmark shows it too: the fixed A/B split found the correct best arm in 100% of its runs, while qbrix found it in 95%. One run in twenty, qbrix finished slightly favoring the wrong variant. The A/B test never did.

People usually read that as a reliability win for A/B testing, and narrowly, it is. But I think it's the wrong lens. The A/B test's perfect identification comes bundled with that guaranteed 232.4 regret, paid in full on every single run — including all the runs where it "correctly" picked the winner. What you're buying with a fixed split isn't reliable outcomes; it's uniform outcomes, at a price that's uniformly high. qbrix's 95% comes bundled with a third less wasted traffic on every run. That's a very different trade.

The way I'd put it: an A/B test buys you certainty about the label it prints at the end, and pays for it with certainty about how much the experiment costs. qbrix accepts a little uncertainty about the label in exchange for a real cut in waste. If you're making a one-time regulatory decision that has to hold up in court, take the A/B test — that's the honest answer. If you're allocating traffic through a product funnel that serves users continuously, the trade isn't close.

And 95% identification, on a signal this faint, is an accuracy most real experiments would be thrilled to hit at any regret cost.

The thing that genuinely taught me something, separate from the headline numbers, was sitting with qbrix's regret distribution and realizing how much it says. A/B testing's zero-variance regret isn't a flaw — it's the logical end of a method built for a world where experiments were rare, expensive, run one at a time, and meant to produce a single clean answer. In that world, paying a fixed cost for a crisp p-value was a sensible deal.

That's not the world qbrix runs in. Here, decisions arrive one user at a time, and every request is at once a chance to learn and a chance to earn. A method that treats user number 13,000 exactly like user number 1 — ignoring everything it has seen in between — is throwing away signal it already paid for. That's the cost. It shows up, identical, every single time. And it compounds every month the system keeps running.

qbrix was built on a bet: that continuous learning, under all the real constraints of a distributed system, would still beat discrete testing under idealized ones. The numbers here — 75 more conversions per run, a 34% reduction in regret, a 5.21% lift in total reward, confidence intervals that don't touch — are that bet, measured. The theory said the gap should be there. The data says it's larger than I expected, and it holds up under every overhead the system carries.

If you're running a product experiment right now on a fixed 50/50 or 33/33/33 split, do the quick multiplication: your monthly volume, times half a percentage point of conversion, times your average order value. Look at the number. It's usually big enough to change what you do next quarter.

The gap isn't a claim. It's a measurement.