On June 5, 2026, Shopify shipped Rollouts — native functionality to schedule, gradually roll out, and split-test themes, checkout configurations, and customer-account configurations directly from the admin, with no third-party app. It's part of the Summer '26 cycle, and for anyone spending money to drive traffic to a store, it's one of the more useful releases of the year.
The short version: Shopify now does its own A/B testing of the surfaces that decide whether your paid clicks convert. Themes and checkout, the two things that most directly turn a visitor into a customer, can now be tested natively. Here's what that changes for CRO and how to run a test that actually tells you something.
What Rollouts actually does
Three capabilities in one feature:
- Schedule changes to go live at a set time instead of pushing them manually.
- Gradually roll out a change to a slice of traffic before going to 100%, so a bad change can't take down the whole store at once.
- Split-test two configurations against each other and measure which performs better.
And it spans three surfaces: themes, checkout configurations, and customer-account configurations. Theme testing existed in various forms before, but native checkout testing from the admin is the headline. Checkout was historically the hardest, highest-stakes thing to experiment on. Now it's built in.
Why native checkout testing is a big deal
Checkout is where the money is and where the conversion leaks are. We've written about those leaks in the Shopify checkout conversion leak audit, and the recurring problem was always the same: you could suspect a checkout change would help, but proving it required external tooling, developer time, or Plus-level access.
Rollouts lowers that barrier. You can now compare checkout variants — field order, messaging, trust elements, payment presentation — and let real shopper behavior decide, all from the admin. For paid traffic that's significant, because the checkout is the last and most expensive place a conversion can die.
How to structure a clean test
Native tooling makes testing easy. Easy testing makes bad testing easy too. A clean experiment needs discipline:
1. One change, one hypothesis
Test a single variable. If you change the theme layout and the checkout copy in the same variant and it wins, you don't know which change did it. Write the hypothesis down first: "Moving the express payment buttons above the form will increase checkout completion because it reduces perceived effort."
2. Decide sample size and duration before you start
Pick how long you'll run it — at least one full business cycle, typically one to two weeks — and roughly what sample you need, before launch. Then hold to it. The most common A/B testing mistake is stopping early because a variant "looks like it's winning." Early leads reverse constantly.
3. Use gradual rollout as a safety net, not as the test
Rolling a change to 10% of traffic first protects you from a broken variant tanking revenue. But a 10% safety canary is not the same as a controlled split test. Use the gradual rollout to de-risk, then run the actual A/B split for your measurement.
4. Segment paid traffic where it matters
Cold paid traffic behaves differently from returning customers and email clicks. A checkout change that helps first-time ad visitors might not move the needle for loyal repeat buyers. Keep that in mind when you read results, especially if a campaign sends a specific audience to a specific landing experience.
Where Rollouts fits with your ad strategy
The reason this matters for AdsX clients is simple math. Paid clicks have a fixed cost. If Rollouts helps you lift checkout conversion from, say, 2.4% to 2.9%, your effective cost per acquisition drops without spending another dollar on ads. That's the cheapest performance gain available.
Pair landing and checkout tests with the upstream work. A rebuilt Meta ads account structure sends better-qualified traffic; Rollouts makes sure that traffic converts once it arrives. And as AI shopping traffic grows, converting AI shopping traffic becomes another segment worth its own checkout test.
Where you'd still want dedicated tooling
Rollouts is genuinely good for most stores, but be honest about its edges. You'll still reach for a dedicated experimentation platform if you need:
- Server-side experiments or feature flags across a custom or headless stack.
- Advanced statistical reporting — confidence intervals, sequential testing, multi-variant analysis beyond what's native.
- Cross-surface tests that span beyond Shopify, like a test that touches your ad landing page, app, and store together.
For the vast majority of merchants, that's overkill. Start native, and graduate to a dedicated tool only when a real limitation forces it.
What to do this week
- Identify your highest-value test. Usually it's the checkout for stores running paid ads, or the landing theme for a campaign you're scaling.
- Write one hypothesis and one variable. Resist bundling changes.
- Set duration and sample size up front — at least one full business week.
- Use gradual rollout to de-risk, then run the split test for measurement.
- Roll the winner to 100% and log the result. Build a record so you're not retesting the same ideas next quarter.
If you're not on Shopify yet and want native experimentation built in, you can spin up a Shopify store and test from day one. Rollouts is part of a broader Summer '26 push you can read about in our Shopify Summer '26 AI merchandising preview — and it lands the same month Shopify Scripts sunset, covered in our Scripts to Functions migration guide.
Native A/B testing was the missing piece for a lot of Shopify CRO programs. Now that it's here, the merchants who win are the ones who test with discipline, not the ones who test the most.