Comparing dispatching policies with A/B tests
WareMax runs A/B tests between dispatching policies on identical seeded scenarios using Welch's t-test, so teams can tell whether a policy change is a real improvement or noise.
The question “did that policy change actually help?” is usually unanswerable on a live warehouse — too many confounds, too few trials. On a seeded WareMax scenario it is a one-command A/B test.
Run the comparison
waremax ab-test scenario.yaml \
--policy-a nearest_robot \
--policy-b routed \
--seeds 20WareMax runs both policies across the same 20 seeds and reports the difference with a Welch’s t-test. Because each run is byte-identical under its seed, the only variation is the scenario, not the simulator.
The five baselines
nearest_robot, least_busy, round_robin, auction, and workload-balanced ship
in-box. They are honest baselines: often, state-blind round_robin is hard to beat,
because the simulated system is capacity- and contention-bound.
Where policy has leverage
WareMax lets you tune load, congestion, replicas, and SKU count to find the regimes where dispatching choice matters. The delay attribution tells you whether your bottleneck is even something dispatching can move.
Related: reproducible RL research · fleet sizing. Built by Skelf Research.
Frequently Asked Questions
Which baselines are included?
Five ship in-box and are selectable by policy name: nearest_robot, least_busy, round_robin, auction, and workload-balanced. A learned policy can be plugged in via the rl_agent option.
How do I know a difference is real?
WareMax's ab-test command applies Welch's t-test across multi-seed runs, and its benchmark command detects regressions, so a reported difference is backed by statistics rather than a single lucky seed.