What does an A/B test measure?
In website development, the change should address a problem the visitor actually faces. Version A is the existing experience and version B is the proposed alternative. Showing the old page one week and the new page the following week is not a controlled A/B test: campaign activity, prices and seasonal demand may have changed too. Random assignment helps create a more credible comparison.
Write down the unit of assignment: a person, a session or a business account. A visitor repeatedly switching versions can blur the effect of the experience. Repeat purchases from one person should not casually become independent new participants. An experiment compares a defined behaviour within a defined audience. It does not independently explain why someone hesitated or how they perceive the brand; interviews and usability observation help answer those questions.
Build the hypothesis around a customer problem
As with interface research, start with friction rather than an arbitrary design preference. If support conversations repeatedly mention delivery uncertainty, bringing delivery information closer to the purchase decision is a concrete hypothesis. “Change the button” is not a sufficient rationale. Record the problem, proposed change, expected behaviour and possible harm in one experiment note.
The scenarios below are illustrative; they are not measured Prix client results. Changing the headline, price and form together can compare two complete experiences, but you cannot attribute the outcome to a single component. Record simultaneous experiments that might interact as well. An experiment on the navigation or pricing page can affect visitors who participate in another experiment further along the journey.
Ecommerce purchase
Place delivery information beside the product price. Use completed orders as the main outcome and watch cancellations or returns. More add-to-cart actions are not automatically more revenue.
B2B enquiry
Remove unnecessary form fields. Alongside submission volume, review the share of enquiries sales considers suitable and the follow-up work needed to recover missing information.
SaaS activation
Break initial setup into clearer tasks. Compare users completing the core action alongside registrations. Opening a new account does not establish that someone successfully used the product.
FROM READING TO A NEXT STEP
Turn your first test into a useful decision plan
Share the page, your current conversion definition and the customer problem you want to resolve. We can review measurement and traffic capacity before defining a feasible experiment.
Connect conversions to accepted enquiries and orders
Your measurement setup should distinguish pressing a form button from a successfully accepted enquiry. Record success after the system accepts the request. For orders, check whether repeated notifications create duplicate conversions. An experiment identifier, variant and event definition are enough to describe the interaction; do not send email addresses or phone numbers as analytics event properties.
Consent choices may mean the measurable audience differs from all site visitors. Document that limitation. If event definitions or consent behaviour differ between variants, the apparent change may come from measurement rather than design. In B2B journeys, sales qualification may take several days. Give enquiries received near the end the same evaluation window before comparing qualified outcomes, and document how unresolved enquiries are handled.
Choose the sample and stopping rule before launch
Traffic from performance campaigns influences a website's capacity to run experiments, but there is no universal visitor threshold. The baseline conversion rate, smallest worthwhile change, error tolerance and statistical method all matter. A positive dashboard label does not repair broken assignment or event collection. Before running the test, decide whether the expected learning justifies its implementation and operating cost.
With a fixed-sample approach, follow the planned stopping rule rather than stopping at the first favourable result. If you use a sequential method, choose it from the start instead of switching methods to obtain a preferred answer. Consider complete business cycles when planning duration. Record campaign outages, unavailable stock and payment failures. An interruption may require invalidating or restarting the experiment rather than selectively deleting inconvenient observations.
Starting record
Define the audience, assignment unit, baseline and exact experience change. Keep other releases during the same period visible in the experiment log.
Decision rule
Write down the main metric, acceptable harm, analysis method and stopping conditions before release. Do not replace the main metric after seeing which one improves.
Result note
Report the effect estimate and uncertainty together. If evidence is insufficient, say so. An inconclusive result does not establish that the experiences are exactly equivalent.
Check experiment integrity and search behaviour
Add variant checks to your technical SEO review. Mobile navigation, form validation, keyboard access, visual flicker and loading impact should work across both experiences. A substantial mismatch between expected and observed allocation can indicate assignment or missing-data problems. Investigate that issue before interpreting a conversion difference as a trustworthy winner. Keep a record of device and browser checks, not just a screenshot of the desktop page.
If variants use different URLs, review Google's website-testing guidance for canonical signals and temporary redirects. A temporary experiment should not accidentally become a permanent site migration. Do not give Googlebot a special experience solely because it is a search crawler. When the experiment ends, remove unnecessary variant URLs, scripts and temporary rules. Keep a release record and a rollback route for the experience you decide to retain.
Make progress when an experiment is impractical
Your technical delivery plan should not wait for an A/B test to fix a broken checkout, unreadable text or non-working mobile form. Task-based usability sessions, customer conversations and support records can offer more practical learning at low volume. Establish the problem first, then consider a meaningful change that your traffic could realistically evaluate. Label observations accurately rather than presenting every improvement as a statistically proven result.
A more frequent upstream event can provide earlier observations, but it does not replace the commercial outcome. Add-to-cart activity could improve while completed orders fall. Separate decisions supported by interviews, obvious technical failures and controlled experiments. The useful first step with Prix is to review your current traffic and measurement, identify a feasible question and define what would count as enough evidence to act.
BEFORE YOU DECIDE
Frequently asked questions
How long should an A/B test run?
There is no fixed number of days. Traffic, the baseline rate, a worthwhile effect and the analysis method influence duration. Plan enough observations and relevant business cycles, then follow a written stopping rule.
Is testing a button colour a bad idea?
Not necessarily, but the change should address an actual visibility or usability problem. First review the offer, working forms and major customer obstacles. Small differences can be difficult to distinguish with limited traffic.
Does a variant win if it generates more enquiries?
Check enquiry quality and progression through sales as well. More unsuitable requests may increase workload without helping the business. Define qualification before the experiment rather than adjusting the definition to favour a variant.
What if the result is inconclusive?
Report uncertainty and evaluate cost and risk. Insufficient evidence is not proof of equality. A better hypothesis, a more meaningful change or qualitative research may be the appropriate next step instead of repeatedly inspecting the same result.
Can website experiments affect SEO?
Incorrect URL or redirect management can affect search signals. Follow Google's guidance for alternate-URL experiments, avoid serving crawler-specific experiences and remove temporary experiment rules after the test is finished.
Can Prix help choose an experimentation tool?
We first review the scope, traffic capacity, consent behaviour and technical setup. Tool selection follows those requirements. Licence costs, implementation and result evaluation are separate items to clarify when agreeing the project scope.
LET’S DEFINE THE SCOPE
Turn your first test into a useful decision plan
Share the page, your current conversion definition and the customer problem you want to resolve. We can review measurement and traffic capacity before defining a feasible experiment.
Discuss an experiment plan