Both professor Ron Kenett and David Steinberg of KPA Group sat down to talk to more candidly about some of the topics discussed in this article.
Read the full transcript
If you run an A/A test and you quickly identify that some A’s look better than other A’s, and you know they are the same, then your confidence in the A/B testing methodology and technology is questionable. So the simplistic approach to A/A test is, just to give you some sense of confidence to debug the system, maybe there is a problem with the random assignment but nothing beyond that. You could think of more sophisticated approaches, where the A/A test is not just done as a precursor to the A/B test, but it can be done also in parallel with A/B tests for ongoing monitoring. So you could develop more sophisticated systems, which give you control over what’s going on, and that would give you the ability to manage changes in traffic composition and all sorts of things that can happen, that would impact your decision.
For example, it could happen that, there’s some breakdown in the system, you’re running a test and it has, your allocation is being handled by distributed servers, it’s no longer on a single server. One of the servers didn’t work, and in that geographic area, everyone was put into this bucket, and another one was put into that bucket and the people in that geographic area, happened to respond a bit differently than people in other geographic areas. Alright, well now you’ve biased your experiment, your experimental results because all of those people went into those visitors went into one bucket. And those are exactly the kinds of problems and bugs that you’d like to be able to filter out of the system, and the A/A test is a very good way to do that. So that when you do get to comparing different options and drawing conclusions, making business decisions on them, you have confidence that those are working properly.
Another way that it can easily happen and this was pretty common, is from over monitoring the experiment. It’s very natural if you’ve got data coming in, to not wait a week to look at them but to look at them, every day, maybe several times a day. So, someone who comes into the site, maybe they will respond, maybe they won’t respond, maybe they’ll buy something, maybe they won’t. There’s gonna be variability in the data that you see. One of the two A/As, one of your two buckets, it’s gonna show a slightly better KPI, Is it sufficiently better? But if you watch it over time, what’s gonna happen? It’s gonna drift, so, it might show that bucket, this bucket is better, it might show that this bucket is better, but you have to exercise some caution about trying to reach judgements too fast and that’s one thing that can again they can lead to a situation that may be, instead of a confidence builder, something that drives down your confidence. So, be a little bit careful about data peeking, because it’s gonna happen.
In clinical trials, there is something called intermediate analysis. You plan a trial and in some cases you plan but it has to be planned ahead of time, that you’re going to have, an analysis of the data halfway through. This has to be accounted for. You cannot just collect data and decide to look at it, if this was not planned ahead of time. So what happens in A/B testing or A/A testing is that people might do that without having a planned that ahead of time and that has an effect, people might not be aware of that effect and that might mislead them.
Almost everything that we’ve talked about, in terms of A/A testing is also relevant to A/B tests. You do expect to see some differences – that’s the reason that you’re running the test to begin with. So you’re not gonna be as surprised to see differences, in the case of the A/B test, but again, in order to make sure that you’re making the right business decisions, you do wanna make sure you observe the differences that you’re seeing are real ones, that if you now make a decision based on those, on those differences, this is not just something that’s an artifact in the data, it’s something that’s really there, and you can really build a sound, financial decision on it, business decision, that’s gonna, increase your business activity.
This becomes even more complex when you have several variants. So it’s not A/B, it’s A, B one, B two, B three, B four. So the complexity can be okay, we have A and we have five variants. So what do we want? We want to show, which is the best, we want to show that, they are better than A, we want to pick up the top two and if you add to this the option to design the alternatives using what I mentioned, in terms of designing experiments with combinations of the factors, then you get into even more complexity. So, the testing effort is not more difficult to do it’s the design and the analysis that requires a bit more.
Right, some of the early questions that were looked at, were things like, what is the question that Laplace study? What’s the probability that the sun will rise tomorrow? Alright, we’ve seen the sunrise for many days in a row, what’s the probability that it will rise tomorrow? And can you even, does it make sense to even talk about a question like that in a probabilistic sense?
Many of these early discussions had this very metaphysical nature to them. In terms of modern statistical data analysis, the use of Basie and technology and Basie and thinking, has become much much more established. Part of the reason why there was resistance and has been resistance, and some case is still some resistance is I think, sort of on the grounds of objective versus subjective science. And the idea that you’re bringing prior information to a bayesian analysis is it can be regarded and often has been regarded as expressing subjectivity. So I might have a different prior than Ron does, and we might come to different conclusions from the same data and data which problems and this is the standard environment of course for A/B testing, this is much less critical because again the data are so abundant that anything that you fit in through the prior is essentially going to be washed out by the data. And as a result, it doesn’t really make any difference. What I thought in advance, what Ron thought in advance we gonna agree with one another in the end, after we’ve seen the data from the experiment.
Summarize this articleHere’s what you need to know:
- A/A testing acts as a pre-flight check for A/B tests, identifying and fixing underlying system issues to ensure reliable A/B testing results.
- A/A testing can monitor ongoing A/B tests, verifying their proper execution and guarding against external influences.
- Data peeking, or frequently checking A/A test data, can lead to inaccurate conclusions and should be avoided.
- The sample size for an A/A test depends on data type, current system performance, and the desired significance level of improvement.
- A/A testing is a valuable tool for ensuring accurate and reliable A/B testing results.
The customer experience, which encapsulates the holistic relationship between a brand and customer, is made up of various interactions across channels and touchpoints. It’s no surprise, then, that companies think very carefully about how they design these experiences to attract more business.
How does a marketer know, though, if a proposed change to the customer experience will be a successful one?
The use of A/B testing is an effective, data-driven method of finding out. The basic idea is simple: compare two versions of a page, for example, and show each one to a certain portion of the individuals entering the site and evaluate their results.
Visitors are allocated to one of the two variations randomly. Following well-established scientific principles, this generates a “fair test” between the two versions of the experience. Then, visitor behavior is summarized via informative KPIs like click-through rate (CTR), conversion rate (CR), or revenue, and the average KPIs for the two experiences are compared. With A/B testing, you let customers’ behavior decide which version is the most successful, instead of relying on “intuition.” And, as the old adage has it, “the customer is always right.”
Although A/B tests are simple to describe, a number of important questions arise, which we’ll address below.
- How can I decide which variant (A or B) is better?
- If I declare a winner (B is better than A), how certain am I that this is correct?
- Can I extend the analysis to more than two options?
- How can I know that my testing engine is working properly?
- What sample size is needed for an A/B test?
- Can a test be terminated early if results are clear before the end of the planned testing period?
Decision Making
A/B tests serve the joint goals of learn and earn. The ultimate goal is to earn more from your site or digital property. To do so, you need to learn by collecting and analyzing the data on KPIs during the test. As you collect more data, you will gain confidence as to which experience is better.
An excellent summary of what you have learned is the Probability to Be Best (or P2BB). This is a simple statistic that states the probability that B is the better variant, i.e. has the larger KPI across the visitors to your site. For example, if P2BB is 0.8, then the probability that B is the better choice is 80%, and the probability that A is the better choice is 20%. P2BB is an ideal basis for guiding the business decisions that must be made following an A/B test.
How do we compute P2BB?
To answer that question, we need to give you some background on what is called Bayesian statistics, named after the British clergyman Thomas Bayes who lived in the 18th century. The Bayesian approach to statistics uses probability distributions to describe anything that we don’t know and want to learn from data.
Some early questions that were looked at were “what’s the probability that the sun will rise tomorrow?” Back then, many wondered whether it even made sense to talk about a question like that in a probabilistic sense. The idea that you’re bringing prior information to a bayesian analysis was often regarded as expressing subjectivity.
That’s very different from the classical Frequentists approach you might have studied in an introductory statistics course, where either A is better than B, B is better than A, or they are identical. The question “what is the probability that B is better” is not part of the lexicon – you can’t give an answer, and in fact, you’re not permitted to ask the question!
However, anything that you fit in through the prior is essentially going to be washed out by the abundance of data generated in an A/B test. So in the end, it doesn’t really matter what was thought in advance, as there will be an agreement in the end after the data from an experiment is seen.
The Bayesian view makes probability summaries the common currency. You will end the A/B test with probability distributions describing what you know about the KPI for A and the KPI for B. Combining those two distributions lets you compute P2BB.
Another advantage of P2BB is that it extends directly to tests with more than two variants. At the end of the experiment, you can compute the P2BB for each one of the variants. If there is a clear winner, it will have P2BB close to 1. Clear losers will have P2BB close to 0. You may have two variants that are essentially the same and are better than all the others. In that case, each of the two leading variants will have P2BB near 0.5.
Checking the System
You need a reliable test engine to run a valid A/B test so that when you make a decision, it reflects a real difference between the tested alternatives and is not due to chance, or worse, a bug in the engine.
There are many things that can go wrong:
- The advantage of one variant is momentary and due to unusual traffic fluctuations
- The randomization routine might be faulty, outcome data may not be recorded correctly
- The analysis routine might not be correct
For example, suppose the allocation was random for some visitors but gave variant A to all those who came from a particular geographic region. If the visitors from that region react differently than those in other regions, the allocation failure will bias your experiment.
You also need a representative sample of visitors. When you make a decision from an A/B test, you are always extrapolating from the site visitors involved in your experiment to all those who will visit your site in the future. For extrapolation to be valid, you need to make sure that all segments of those future visitors are covered in your test, and in roughly the same proportions. That’s what we mean by a representative sample.
As an example, web commerce is often subject to natural time trends in the outcome data. Diurnal effects or day-of-week effects may be present. If so, you want to make sure that (i) at least one full-time cycle is included in the experiment, and (ii) that randomization continues to work properly throughout the cycle. As another example, some of your traffic may come from laptops and some from smartphones. You want to know that both of these groups get adequate exposure in your test.
To check that the underlying test engine is running properly, you may want to begin with a small number of A/A tests. The idea of A/A testing is to compare an experience to itself. As both experiences are identical, you obviously don’t expect to see any differences between them – if you run an A/A test and quickly identify that some A’s look better than other A’s, and you know they are the same, then your confidence in the A/B testing methodology and technology is questionable.
A/A testing gives a basis for trusting the results of subsequent A/B tests; setting a benchmark for what happens when there are no differences between the two variants under the test helps you appreciate the difference you see when comparing two different variants. Moreover, you can use your standard experience in the A/A test, so there is no risk of lost activity from using a new version.
In a more sophisticated approach, the A/A test might not just be done as a precursor to the A/B test, but also in parallel for ongoing monitoring. This would give you more control and the ability to manage changes in traffic composition as well as all sorts of things that can impact decision-making.
The A/A test format sounds easy, but there are some subtle issues to think about. Data is variable, so even if both groups in your experiment got the same experience, you will still see some difference in outcome. You need some yardsticks to tell you whether those differences are big enough to indicate problems, or are just part of the natural variation in the data. You need to think about how long to run an A/A test and on how many A/A tests you want to run before moving ahead.
We have already suggested a good yardstick: Probability to Be Best (P2BB). This yardstick takes full account of the variability in the data. It is important to check the average KPIs as well. Since you are comparing an experience to itself, you expect these to be very similar to one another. You should also look at the fraction of traffic directed to each condition. The allocation should be close to 50% for each one – an allocation that deviates from 50% is a sign of trouble.
Although P2BB is an excellent summary of what is learned from the data, we need to add a few words of caution.
First, data has natural variability. P2BB depends on the data, so it is also variable. Consequently, there will be A/A tests that give a P2BB close to 0 or 1, even though you know there is no difference. No A/B testing strategy is immune to occasionally declaring a winner in this case.
Second, data volumes are huge in many online experiments. With huge sample sizes, even very small differences in average KPI can lead to an extreme P2BB. This reiterates the importance of checking the KPI averages as well. You might have a clear indication of a winner from P2BB but associated with a KPI difference that makes almost no difference to your bottom line.
Third, beware of over-monitoring. Testing engines let you track the results from your experiment. So it is natural to check regularly to see if the data accumulated thus far points to a winner. P2BB will vary during the course of the experiment, reflecting the variation in visitor outcomes. If you check the “current P2BB” every hour in a two-week A/A test, that adds up to more than 300 P2BB values. Some may be close to 1, others may be close to 0. A common, but risky, practice is to stop a test as soon as P2BB first exceeds a threshold (say dropping below 0.05 or rising above 0.95). Why is that risky? Because it is not so surprising that the most extreme hourly P2BB value will cross one of these boundaries. Getting a final P2BB below 0.05, for example, will happen in about 5% of your A/A tests. But getting just one P2BB result below 0.05 will happen far more often. Data peeking can seriously inflate your false winner rate if not preplanned. You should use only the final P2BB (meaning you’ve gathered the required data in relation to the sample size), together with the average KPIs, to evaluate your engine.
Further, if you want to thoroughly check the engine, run several A/A tests. We all know that it is dangerous to reach too many conclusions based on a sample of size 1. The resulting final P2BB’s should be “spread out” between 0 and 1.
When data volume leads to an extreme P2BB alongside a small difference in average KPIs, it may be helpful to carry out a more refined analysis that reflects the business impact of the comparison. Consider defining a minimal critical difference in KPIs that has an impact on your business. For example, you might decide that a difference of less than 0.5% between two competing variants would not justify making a change. In that case, you can use the Bayesian analysis to break down the summary of an A/B test into three distinct outcomes: B is better, B is worse, or B is the same as A (i.e. the difference is smaller than the critical margin). For A/A tests, you will often find that the third option has high probability, meaning you have strong evidence that there is not a meaningful difference between the two variants.
One should also be mindful of the complexity of adding several variants – when it’s not A/B, but A, B one, B two, B three, B four. So, if we have A and four variants, we want to show which is the best, that they are better than A, and pick up the top two. The testing effort is not more difficult to do, but the design and analysis of combinations requires a bit more when it comes to introducing these alternatives.
How large should an A/A test be?
The idea of the A/A test is to provide a realistic dry run of the system. You want to be sure that about half the visitors are allocated to each option to check that the KPIs stay close to one another and that P2BB does not declare a clear winner.
The sample size will depend on several factors:
- The type of data (binary, like CTR; continuous, like revenue)
- The current performance
- The performance level that would be a significant improvement
- For continuous KPIs, the extent of variation between visitors
Below we will expand on each of these and highlight with an example.
The type of data is important, as different sample size formulas are relevant for each. The current performance must be known or estimated. The sample size is always set with the goal of identifying some target change. For an A/B test, you can set the target improvement to match your expectations from the new test page. That approach doesn’t work for an A/A test, where both versions should have the same KPI.
Instead, we recommend setting the target to be the level of importance that you see as an improvement. For example, with a KPI of 0.010, a 10% increase to 0.011 might be your threshold for large. If your average revenue is $1 per visitor, an increase of just 3%, to $1.03, might be important.
A simple guide is to ask, “what is the size improvement that we would definitely want to identify?” The extent of variation is important for continuous KPIs. Typically, what you will want is the standard deviation (SD) of the result among those who are converters.
It is worth noting that the same inputs are relevant for an A/B test and they are used in essentially the same way. But there is one distinction – the goal of an A/B test is to find a difference, so it’s clearly important to provide a near guarantee that important differences will be identified. We achieve that sharper focus by increasing the sample sizes. In an A/A test, though, the goal is to check the system. There is no need for the tight comparison of an A/B test, so we use a less stringent criterion on focus, which leads to smaller sample sizes.
Further, we recommend running an A/B test for at least one complete time cycle. If weekday visitors prefer A but weekend visitors prefer B, we would want to have both of these groups in our data before making a decision. With an A/A test, there is no concern that preferences for one option versus the other will change over time, so there is no need to run a test for a full cycle. If you run several A/A tests on the same experience, it is a good idea to spread them out over time so that you do see the engine in operation across a full-time cycle.
We can illustrate these ideas with an example.
Suppose your current CTR is 0.0060. An improvement to 0.0065 would be important. The test should include 191,000 visitors to each of the two options.
The calculation should be tempered with practical wisdom. If you compute a sample size that will take weeks to achieve, then the target improvement, even if it is an important gain, is probably too difficult to achieve. In that case, you should reset the target so that your A/A test runs no more than two weeks.
Continue reading
<!DOCTYPE html> <!--[if IE 6]><html id="ie6" lang="en-US"> <![endif]--> <!--[if IE 7]><html id="ie7" lang="en-US"> <![endif]--> <!--[if IE 8]><html id="ie8" lang="en-US"> <![endif]--> <!--[if !(IE 6) | !(IE 7) | !(IE 8) ]><!--><html lang="en-US"> <!--<![endif]--><head><meta name="msvalidate.01" content="1033997EC383621B874EB0392F261984"/><link rel="preconnect" href="//cdn.dynamicyield.com"><link rel="preconnect" href="//st.dynamicyield.com"><link rel="preconnect" href="//rcom.dynamicyield.com"><link rel="dns-prefetch" href="//cdn.dynamicyield.com"><link rel="dns-prefetch" href="//st.dynamicyield.com"><link rel="dns-prefetch" href="//rcom.dynamicyield.com"> <script data-cfasync="false">window.dataLayer = window.dataLayer || [];
window.gtag = function() { dataLayer.push(arguments); }
window.gtag('consent', 'default', {
ad_storage: 'denied',
ad_user_data: 'denied',
ad_personalization: 'denied',
analytics_storage: 'denied',
functionality_storage: 'denied',
personalization_storage: 'denied',
security_storage: 'granted', // Only this is granted by default
});</script> <script data-cfasync="false">(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'//www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-MCQHF8');</script> <script data-cfasync="false" async>// Init the adobeDataLayer array
var adobeDataLayer = adobeDataLayer || [];
// Only when the API is ready
function adobeWaitForDY() {
if (typeof(window.DY) !== 'undefined' && typeof(window.DY.AdDetection) !== 'undefined' &&
typeof($dy) !== 'undefined' && typeof($dy(window)) !== 'undefined' && typeof($dy.renderSmartTagSlider) !== 'undefined') {
var zi_data = localStorage.getItem('ZoomInfoData'),
ada_page_info = {
"pageName": "A/A testing and decision making in experimentation",
"siteName": "Mastercard Dynamic Yield",
"pageTitle": "en_US :aa-testing-in-experimentation",
"pageURL": "https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/",
"country": "US",
"language": "en",
"dyID": DY.dyid,
"userStatus": "Guest"
};
// Detect the hierarchy and
// site sections based on the URL slugs
var current_pathname = window.location.pathname.replace(/^\/|\/$/g, '');
if(current_pathname) {
var current_sitesections = current_pathname.split('/'),
current_hierarchy = current_sitesections.map(function(segment) {
return segment.split(/[-_]/).map(function(word) {
return word.charAt(0).toUpperCase() + word.slice(1);
})
.join(' ');
});
// Site section 1
if(current_sitesections[0]) {
ada_page_info.siteSection1 = current_sitesections[0];
}
// Site section 2
if(current_sitesections[1]) {
ada_page_info.siteSection2 = current_sitesections[1];
}
// Hierarchy
ada_page_info.hierarchy = current_hierarchy.join('|');
}
// Push the ZI params as well
if(zi_data) {
zi_data = JSON.parse(zi_data);
if(zi_data.name) {
ada_page_info.ziCompany = zi_data.name;
}
if(zi_data.domain) {
ada_page_info.ziDomain = zi_data.domain;
}
if(zi_data.employeesrange) {
ada_page_info.ziEmployees = zi_data.employeesrange;
}
if(zi_data.industry) {
ada_page_info.ziIndustry = zi_data.industry;
}
}
// Decode HTML prior to dispatching the event
Object.keys(ada_page_info).forEach(function(key) {
ada_page_info[key] = $('<textarea/>').html(ada_page_info[key]).text();
});
adobeDataLayer.push({
"event": "pageLoad",
"pageInfo": ada_page_info
});
} else {
setTimeout(adobeWaitForDY, 150);
}
}
adobeWaitForDY();</script> <script type="text/javascript">function OptanonWrapper() {
console.log('OneTrust Loaded');
DY_API_update_consent_mode();
OneTrust.OnConsentChanged(DY_API_update_consent_mode);
}
function DY_API_update_consent_mode() {
var groups = OnetrustActiveGroups.split(',');
DY.userActiveConsent = {accepted: groups.includes('C0004')};
DYO.ActiveConsent.updateConsentAcceptedStatus(groups.includes('C0004'));
if(groups.includes('C0004')) {
console.log('OneTrust_Consent_Is_Granted');
DY.API('event',{ name: 'OneTrust_Consent_Is_Granted', properties: {}});
}
console.log('OneTrust - DY API is ' + (groups.includes('C0004') ? 'Enabled' : 'Disabled'));
}
window.addEventListener('OTConsentApplied', function() {
var groups = OnetrustActiveGroups.split(',');
if(groups.includes('C0004')) {
console.log('OneTrust_Initial_Consent_Granted');
DY.API('event',{ name: 'OneTrust_Initial_Consent_Granted', properties: {}});
}
});</script> <script type="text/javascript">/*WebFontConfig = {
google: { families: [ 'Shadows+Into+Light::latin', 'Handlee::latin', 'Roboto:400,100,300,500:latin' ] }
};
(function() {
var wf = document.createElement('script');
wf.src = ('https:' == document.location.protocol ? 'https' : 'http') +
'://ajax.googleapis.com/ajax/libs/webfont/1/webfont.js';
wf.type = 'text/javascript';
wf.async = 'true';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(wf, s);
})(); */
console.log(
'%cLet’s personalize experiences together.\n%cDynamic Yield is hiring smart engineers like you to help solve hard problems.\n%cSee our jobs page: https://www.dynamicyield.com/careers',
'font-size:2.2em;line-height:1.4em;margin-top:20px;color:rgb(93, 56, 185);',
'font-size:1.1em;line-height:1.6em;color:rgb(0, 0, 0);',
'font-size:1.1em;line-height:1.6em;margin-bottom:20px;color:rgb(0, 0, 0);'
);
function waitForElementToDisplay(selector, time, functionName) {
if (document.getElementById(selector) != null) {
functionName();
return;
} else {
setTimeout(function () {
waitForElementToDisplay(selector, time, functionName);
}, time);
}
}
function dcWaitForJQuery(time, functionName) {
if (window.jQuery || window.n2) {
functionName();
return;
} else {
setTimeout(function () {
dcWaitForJQuery(time, functionName);
}, time);
}
}</script> <meta name="msvalidate.01" content="451B75B70520F4C76D67F116C9EFFE5C"/><meta property="twitter:account_id" content="4503599629852947"/><meta name="p:domain_verify" content="f8112f57c09778e892850075a87dc1cf"/><meta name="google-site-verification" content="9FbmfdC4cM7JG8sgYh6w9HhFlxvEHewltexeUVzuSfE"/><meta charset="UTF-8"/><link rel="pingback" href="https://www.dynamicyield.com/xmlrpc.php"/> <!--[if lt IE 9]> <script src="https://www.dynamicyield.com/wp-content/themes/Divi/js/html5.js" type="text/javascript"></script> <![endif]--> <script type="text/javascript">document.documentElement.className = 'js';</script> <script type="text/javascript">var getLangCode = 'en';</script> <link rel="prefetch" href="/wp-content/themes/Divi/css/custom-style.css" as="style" crossorigin="anonymous"><meta name='robots' content='index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1' /><link rel="alternate" hreflang="en" href="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/" /><link rel="alternate" hreflang="x-default" href="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/" /> <script data-cfasync="false" data-pagespeed-no-defer>var gtm4wp_datalayer_name = "dataLayer";
var dataLayer = dataLayer || [];</script> <title>A/A testing and decision making in experimentation — Mastercard Dynamic Yield</title><link rel="preload" as="font" href="https://www.dynamicyield.com/wp-content/themes/Divi/fonts/CircularXXTT/CircularXXWeb-Bold.woff2" crossorigin/><link rel="preload" as="font" href="https://www.dynamicyield.com/wp-content/themes/Divi/fonts/CircularXXTT/CircularXXWeb-Medium.woff2" crossorigin/><link rel="preload" as="font" href="https://www.dynamicyield.com/wp-content/themes/Divi/fonts/CircularXXTT/CircularXXWeb-Book.woff2" crossorigin/><meta name="description" content="A/A testing is meant to check that the underlying system is running properly. In this post, we'll cover some of subtle issues to consider as you run them." /><link rel="canonical" href="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/" /><meta property="og:locale" content="en_US" /><meta property="og:type" content="article" /><meta property="og:title" content="A/A testing and decision making in experimentation — Mastercard Dynamic Yield" /><meta property="og:description" content="A/A testing is meant to check that the underlying system is running properly. In this post, we'll cover some of subtle issues to consider as you run them." /><meta property="og:url" content="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/" /><meta property="og:site_name" content="Mastercard Dynamic Yield" /><meta property="article:publisher" content="https://www.facebook.com/DynamicYield" /><meta property="article:modified_time" content="2024-01-31T06:27:05+00:00" /><meta property="og:image" content="https://www.dynamicyield.com/wp-content/uploads/2020/07/AA-testing-and-decision-making-in-experimentation-FB.jpg" /><meta property="og:image:width" content="1200" /><meta property="og:image:height" content="628" /><meta property="og:image:type" content="image/jpeg" /><meta name="twitter:card" content="summary_large_image" /><meta name="twitter:image" content="https://www.dynamicyield.com/wp-content/uploads/2020/07/AA-testing-and-decision-making-in-experimentation-FB.jpg" /><meta name="twitter:site" content="@DynamicYield" /><meta name="twitter:label1" content="Est. reading time" /><meta name="twitter:data1" content="18 minutes" /><meta name="twitter:label2" content="Written by" /><meta name="twitter:data2" content="David Steinberg, Ron Kenett" /> <script type="application/ld+json" class="yoast-schema-graph">{"@context":"https://schema.org","@graph":[{"@type":"WebPage","@id":"https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/","url":"https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/","name":"A/A testing and decision making in experimentation — Mastercard Dynamic Yield","isPartOf":{"@id":"https://www.dynamicyield.com/#website"},"primaryImageOfPage":{"@id":"https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/#primaryimage"},"image":{"@id":"https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/#primaryimage"},"thumbnailUrl":"https://www.dynamicyield.com/wp-content/uploads/2020/07/AA-testing-and-decisioning-making-in-experimentation-blog.jpg","datePublished":"2020-07-09T12:37:49+00:00","dateModified":"2024-01-31T06:27:05+00:00","description":"A/A testing is meant to check that the underlying system is running properly. In this post, we'll cover some of subtle issues to consider as you run them.","breadcrumb":{"@id":"https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/#primaryimage","url":"https://www.dynamicyield.com/wp-content/uploads/2020/07/AA-testing-and-decisioning-making-in-experimentation-blog.jpg","contentUrl":"https://www.dynamicyield.com/wp-content/uploads/2020/07/AA-testing-and-decisioning-making-in-experimentation-blog.jpg","width":795,"height":320},{"@type":"BreadcrumbList","@id":"https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.dynamicyield.com/"},{"@type":"ListItem","position":2,"name":"Lessons","item":"https://www.dynamicyield.com/lesson/"},{"@type":"ListItem","position":3,"name":"A/A testing and decision making in experimentation"}]},{"@type":"WebSite","@id":"https://www.dynamicyield.com/#website","url":"https://www.dynamicyield.com/","name":"Mastercard Dynamic Yield","description":"","publisher":{"@id":"https://www.dynamicyield.com/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://www.dynamicyield.com/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https://www.dynamicyield.com/#organization","name":"Mastercard Dynamic Yield","url":"https://www.dynamicyield.com/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https://www.dynamicyield.com/#/schema/logo/image/","url":"https://www.dynamicyield.com/wp-content/uploads/2019/10/DY-Hero_Logo-Black@x2.png","contentUrl":"https://www.dynamicyield.com/wp-content/uploads/2019/10/DY-Hero_Logo-Black@x2.png","width":458,"height":458,"caption":"Mastercard Dynamic Yield"},"image":{"@id":"https://www.dynamicyield.com/#/schema/logo/image/"},"sameAs":["https://www.facebook.com/DynamicYield","https://x.com/DynamicYield","https://www.instagram.com/dynamic.yield/","https://www.linkedin.com/company/dynamic-yield","http://www.pinterest.com/dynamicyield/","https://www.youtube.com/channel/UCPWEcR8lQRadVKYW8H4sndQ","https://en.wikipedia.org/wiki/Dynamic_Yield"]}]}</script> <link rel='dns-prefetch' href='//pages.dynamicyield.com' /><link rel="alternate" type="application/rss+xml" title="Mastercard Dynamic Yield » Feed" href="https://www.dynamicyield.com/feed/" /><link rel="alternate" type="application/rss+xml" title="Mastercard Dynamic Yield » Comments Feed" href="https://www.dynamicyield.com/comments/feed/" /><meta content="Divi v." name="generator"/><link rel='stylesheet' id='sbi_styles-css' href='https://www.dynamicyield.com/wp-content/plugins/instagram-feed/css/sbi-styles.min.css?ver=6.2.6' type='text/css' media='all' /><style id='wp-emoji-styles-inline-css' type='text/css'>img.wp-smiley, img.emoji {
display: inline !important;
border: none !important;
box-shadow: none !important;
height: 1em !important;
width: 1em !important;
margin: 0 0.07em !important;
vertical-align: -0.1em !important;
background: none !important;
padding: 0 !important;
}</style><link rel='stylesheet' id='wp-block-library-css' href='https://www.dynamicyield.com/wp-includes/css/dist/block-library/style.min.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><style id='classic-theme-styles-inline-css' type='text/css'>/*! This file is auto-generated */
.wp-block-button__link{color:#fff;background-color:#32373c;border-radius:9999px;box-shadow:none;text-decoration:none;padding:calc(.667em + 2px) calc(1.333em + 2px);font-size:1.125em}.wp-block-file__button{background:#32373c;color:#fff;text-decoration:none}</style><style id='global-styles-inline-css' type='text/css'>body{--wp--preset--color--black: #000000;--wp--preset--color--cyan-bluish-gray: #abb8c3;--wp--preset--color--white: #ffffff;--wp--preset--color--pale-pink: #f78da7;--wp--preset--color--vivid-red: #cf2e2e;--wp--preset--color--luminous-vivid-orange: #ff6900;--wp--preset--color--luminous-vivid-amber: #fcb900;--wp--preset--color--light-green-cyan: #7bdcb5;--wp--preset--color--vivid-green-cyan: #00d084;--wp--preset--color--pale-cyan-blue: #8ed1fc;--wp--preset--color--vivid-cyan-blue: #0693e3;--wp--preset--color--vivid-purple: #9b51e0;--wp--preset--gradient--vivid-cyan-blue-to-vivid-purple: linear-gradient(135deg,rgba(6,147,227,1) 0%,rgb(155,81,224) 100%);--wp--preset--gradient--light-green-cyan-to-vivid-green-cyan: linear-gradient(135deg,rgb(122,220,180) 0%,rgb(0,208,130) 100%);--wp--preset--gradient--luminous-vivid-amber-to-luminous-vivid-orange: linear-gradient(135deg,rgba(252,185,0,1) 0%,rgba(255,105,0,1) 100%);--wp--preset--gradient--luminous-vivid-orange-to-vivid-red: linear-gradient(135deg,rgba(255,105,0,1) 0%,rgb(207,46,46) 100%);--wp--preset--gradient--very-light-gray-to-cyan-bluish-gray: linear-gradient(135deg,rgb(238,238,238) 0%,rgb(169,184,195) 100%);--wp--preset--gradient--cool-to-warm-spectrum: linear-gradient(135deg,rgb(74,234,220) 0%,rgb(151,120,209) 20%,rgb(207,42,186) 40%,rgb(238,44,130) 60%,rgb(251,105,98) 80%,rgb(254,248,76) 100%);--wp--preset--gradient--blush-light-purple: linear-gradient(135deg,rgb(255,206,236) 0%,rgb(152,150,240) 100%);--wp--preset--gradient--blush-bordeaux: linear-gradient(135deg,rgb(254,205,165) 0%,rgb(254,45,45) 50%,rgb(107,0,62) 100%);--wp--preset--gradient--luminous-dusk: linear-gradient(135deg,rgb(255,203,112) 0%,rgb(199,81,192) 50%,rgb(65,88,208) 100%);--wp--preset--gradient--pale-ocean: linear-gradient(135deg,rgb(255,245,203) 0%,rgb(182,227,212) 50%,rgb(51,167,181) 100%);--wp--preset--gradient--electric-grass: linear-gradient(135deg,rgb(202,248,128) 0%,rgb(113,206,126) 100%);--wp--preset--gradient--midnight: linear-gradient(135deg,rgb(2,3,129) 0%,rgb(40,116,252) 100%);--wp--preset--font-size--small: 13px;--wp--preset--font-size--medium: 20px;--wp--preset--font-size--large: 36px;--wp--preset--font-size--x-large: 42px;--wp--preset--spacing--20: 0.44rem;--wp--preset--spacing--30: 0.67rem;--wp--preset--spacing--40: 1rem;--wp--preset--spacing--50: 1.5rem;--wp--preset--spacing--60: 2.25rem;--wp--preset--spacing--70: 3.38rem;--wp--preset--spacing--80: 5.06rem;--wp--preset--shadow--natural: 6px 6px 9px rgba(0, 0, 0, 0.2);--wp--preset--shadow--deep: 12px 12px 50px rgba(0, 0, 0, 0.4);--wp--preset--shadow--sharp: 6px 6px 0px rgba(0, 0, 0, 0.2);--wp--preset--shadow--outlined: 6px 6px 0px -3px rgba(255, 255, 255, 1), 6px 6px rgba(0, 0, 0, 1);--wp--preset--shadow--crisp: 6px 6px 0px rgba(0, 0, 0, 1);}:where(.is-layout-flex){gap: 0.5em;}:where(.is-layout-grid){gap: 0.5em;}body .is-layout-flow > .alignleft{float: left;margin-inline-start: 0;margin-inline-end: 2em;}body .is-layout-flow > .alignright{float: right;margin-inline-start: 2em;margin-inline-end: 0;}body .is-layout-flow > .aligncenter{margin-left: auto !important;margin-right: auto !important;}body .is-layout-constrained > .alignleft{float: left;margin-inline-start: 0;margin-inline-end: 2em;}body .is-layout-constrained > .alignright{float: right;margin-inline-start: 2em;margin-inline-end: 0;}body .is-layout-constrained > .aligncenter{margin-left: auto !important;margin-right: auto !important;}body .is-layout-constrained > :where(:not(.alignleft):not(.alignright):not(.alignfull)){max-width: var(--wp--style--global--content-size);margin-left: auto !important;margin-right: auto !important;}body .is-layout-constrained > .alignwide{max-width: var(--wp--style--global--wide-size);}body .is-layout-flex{display: flex;}body .is-layout-flex{flex-wrap: wrap;align-items: center;}body .is-layout-flex > *{margin: 0;}body .is-layout-grid{display: grid;}body .is-layout-grid > *{margin: 0;}:where(.wp-block-columns.is-layout-flex){gap: 2em;}:where(.wp-block-columns.is-layout-grid){gap: 2em;}:where(.wp-block-post-template.is-layout-flex){gap: 1.25em;}:where(.wp-block-post-template.is-layout-grid){gap: 1.25em;}.has-black-color{color: var(--wp--preset--color--black) !important;}.has-cyan-bluish-gray-color{color: var(--wp--preset--color--cyan-bluish-gray) !important;}.has-white-color{color: var(--wp--preset--color--white) !important;}.has-pale-pink-color{color: var(--wp--preset--color--pale-pink) !important;}.has-vivid-red-color{color: var(--wp--preset--color--vivid-red) !important;}.has-luminous-vivid-orange-color{color: var(--wp--preset--color--luminous-vivid-orange) !important;}.has-luminous-vivid-amber-color{color: var(--wp--preset--color--luminous-vivid-amber) !important;}.has-light-green-cyan-color{color: var(--wp--preset--color--light-green-cyan) !important;}.has-vivid-green-cyan-color{color: var(--wp--preset--color--vivid-green-cyan) !important;}.has-pale-cyan-blue-color{color: var(--wp--preset--color--pale-cyan-blue) !important;}.has-vivid-cyan-blue-color{color: var(--wp--preset--color--vivid-cyan-blue) !important;}.has-vivid-purple-color{color: var(--wp--preset--color--vivid-purple) !important;}.has-black-background-color{background-color: var(--wp--preset--color--black) !important;}.has-cyan-bluish-gray-background-color{background-color: var(--wp--preset--color--cyan-bluish-gray) !important;}.has-white-background-color{background-color: var(--wp--preset--color--white) !important;}.has-pale-pink-background-color{background-color: var(--wp--preset--color--pale-pink) !important;}.has-vivid-red-background-color{background-color: var(--wp--preset--color--vivid-red) !important;}.has-luminous-vivid-orange-background-color{background-color: var(--wp--preset--color--luminous-vivid-orange) !important;}.has-luminous-vivid-amber-background-color{background-color: var(--wp--preset--color--luminous-vivid-amber) !important;}.has-light-green-cyan-background-color{background-color: var(--wp--preset--color--light-green-cyan) !important;}.has-vivid-green-cyan-background-color{background-color: var(--wp--preset--color--vivid-green-cyan) !important;}.has-pale-cyan-blue-background-color{background-color: var(--wp--preset--color--pale-cyan-blue) !important;}.has-vivid-cyan-blue-background-color{background-color: var(--wp--preset--color--vivid-cyan-blue) !important;}.has-vivid-purple-background-color{background-color: var(--wp--preset--color--vivid-purple) !important;}.has-black-border-color{border-color: var(--wp--preset--color--black) !important;}.has-cyan-bluish-gray-border-color{border-color: var(--wp--preset--color--cyan-bluish-gray) !important;}.has-white-border-color{border-color: var(--wp--preset--color--white) !important;}.has-pale-pink-border-color{border-color: var(--wp--preset--color--pale-pink) !important;}.has-vivid-red-border-color{border-color: var(--wp--preset--color--vivid-red) !important;}.has-luminous-vivid-orange-border-color{border-color: var(--wp--preset--color--luminous-vivid-orange) !important;}.has-luminous-vivid-amber-border-color{border-color: var(--wp--preset--color--luminous-vivid-amber) !important;}.has-light-green-cyan-border-color{border-color: var(--wp--preset--color--light-green-cyan) !important;}.has-vivid-green-cyan-border-color{border-color: var(--wp--preset--color--vivid-green-cyan) !important;}.has-pale-cyan-blue-border-color{border-color: var(--wp--preset--color--pale-cyan-blue) !important;}.has-vivid-cyan-blue-border-color{border-color: var(--wp--preset--color--vivid-cyan-blue) !important;}.has-vivid-purple-border-color{border-color: var(--wp--preset--color--vivid-purple) !important;}.has-vivid-cyan-blue-to-vivid-purple-gradient-background{background: var(--wp--preset--gradient--vivid-cyan-blue-to-vivid-purple) !important;}.has-light-green-cyan-to-vivid-green-cyan-gradient-background{background: var(--wp--preset--gradient--light-green-cyan-to-vivid-green-cyan) !important;}.has-luminous-vivid-amber-to-luminous-vivid-orange-gradient-background{background: var(--wp--preset--gradient--luminous-vivid-amber-to-luminous-vivid-orange) !important;}.has-luminous-vivid-orange-to-vivid-red-gradient-background{background: var(--wp--preset--gradient--luminous-vivid-orange-to-vivid-red) !important;}.has-very-light-gray-to-cyan-bluish-gray-gradient-background{background: var(--wp--preset--gradient--very-light-gray-to-cyan-bluish-gray) !important;}.has-cool-to-warm-spectrum-gradient-background{background: var(--wp--preset--gradient--cool-to-warm-spectrum) !important;}.has-blush-light-purple-gradient-background{background: var(--wp--preset--gradient--blush-light-purple) !important;}.has-blush-bordeaux-gradient-background{background: var(--wp--preset--gradient--blush-bordeaux) !important;}.has-luminous-dusk-gradient-background{background: var(--wp--preset--gradient--luminous-dusk) !important;}.has-pale-ocean-gradient-background{background: var(--wp--preset--gradient--pale-ocean) !important;}.has-electric-grass-gradient-background{background: var(--wp--preset--gradient--electric-grass) !important;}.has-midnight-gradient-background{background: var(--wp--preset--gradient--midnight) !important;}.has-small-font-size{font-size: var(--wp--preset--font-size--small) !important;}.has-medium-font-size{font-size: var(--wp--preset--font-size--medium) !important;}.has-large-font-size{font-size: var(--wp--preset--font-size--large) !important;}.has-x-large-font-size{font-size: var(--wp--preset--font-size--x-large) !important;}
.wp-block-navigation a:where(:not(.wp-element-button)){color: inherit;}
:where(.wp-block-post-template.is-layout-flex){gap: 1.25em;}:where(.wp-block-post-template.is-layout-grid){gap: 1.25em;}
:where(.wp-block-columns.is-layout-flex){gap: 2em;}:where(.wp-block-columns.is-layout-grid){gap: 2em;}
.wp-block-pullquote{font-size: 1.5em;line-height: 1.6;}</style><link rel='stylesheet' id='rfw-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_7c036e0987ef80b703d7eb1abf922895.css?ver=2026020242' type='text/css' media='all' /><link rel='stylesheet' id='divi-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_ab3666a686d3dac03fd3032e5757637e.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='ctt-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_f1d1b2f761cf3768f427cdc5c4c2544b.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='inline-tweet-sharer-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_daa32ca85e114a0c6e5d1432c89237f3.css?ver=2.6.4' type='text/css' media='all' /><link rel='stylesheet' id='et_lb_modules-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_c0102af27938567cb82cc02e4da3a673.css?ver=2.4' type='text/css' media='all' /><link rel='stylesheet' id='et-shortcodes-css-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_40cb404848bc2e556105c8aea77f29aa.css?ver=3.0' type='text/css' media='all' /><link rel='stylesheet' id='et-shortcodes-responsive-css-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_3a30a77569bc4544b9465acbee8e5f97.css?ver=3.0' type='text/css' media='all' /><link rel='stylesheet' id='wp-featherlight-css' href='https://www.dynamicyield.com/wp-content/plugins/wp-featherlight/css/wp-featherlight.min.css?ver=1.3.4' type='text/css' media='all' /><link rel='stylesheet' id='baskerville-font-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_8c23179f030d02e0065d2054eab34279.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='markformc-font-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_0eb51906eca648ca4559d0a03b898d83.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='mackinac-font-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_9c8fec2609ebc9764a6e16471d9451ae.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='button-pulse-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_4cabc8dd29eaa9af7bf2df62f1c40af9.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='fontawesome-css' href='https://www.dynamicyield.com/wp-content/themes/Divi/custom/font-awesome/css/font-awesome.min.css' type='text/css' media='all' /><link rel='stylesheet' id='bootstrap-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_d375e60add46d8de26a78728b517660f.css' type='text/css' media='all' /><link rel='stylesheet' id='slick-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_13b1b6672b8cfb0d9ae7f899f1c42875.css' type='text/css' media='all' /><link rel='stylesheet' id='slick-theme-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_f9faba678c4d6dcfdde69e5b11b37a2e.css' type='text/css' media='all' /><link rel='stylesheet' id='custom-theme-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_57f41281576414370c141cd00d2a80a9.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='custom-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_417039b6d54561fe1e8c94ee67cd902c.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='header-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_d75ad7543415df427913a5fe91606386.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='footer-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_25400e1cfd7099cc557083769c928231.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='video_tabs_module-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_f1109a8bc8c46ec9f2205cc8603c132f.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='mkto-reset-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_d117c8ecde800f5dccd137c16bab71af.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='learning-center-general-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_73dd3b78cf6e21c28be39db6740774b1.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='article-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_e1d850b2d8d6fd6331ad74bd35ddde51.css?ver=1772203379' type='text/css' media='all' /><link rel='stylesheet' id='customers-shortcode-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_c6bf61e9c8beb560d64d119f58757b8b.css?ver=26.27' type='text/css' media='all' /> <script type="text/javascript" src="https://www.dynamicyield.com/wp-includes/js/jquery/jquery.min.js?ver=3.7.1" id="jquery-core-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=3.4.1" id="jquery-migrate-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_dbdab0f35217f6041eb7e2028dc94ba4.js?ver=2.6.4" id="inline-tweet-sharer-js-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/themes/Divi/js/blazy.min.js" id="blazy-js"></script> <script type="text/javascript" src="//pages.dynamicyield.com/js/forms2/js/forms2.min.js?ver=21.85" id="mkto-form-script-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_db6be3b57dba139e789fec1a78186c8e.js?ver=21.85" id="learning-center-js-js"></script> <link rel="https://api.w.org/" href="https://www.dynamicyield.com/wp-json/" /><link rel="EditURI" type="application/rsd+xml" title="RSD" href="https://www.dynamicyield.com/xmlrpc.php?rsd" /><link rel='shortlink' href='https://www.dynamicyield.com/?p=131419' /><link rel="alternate" type="application/json+oembed" href="https://www.dynamicyield.com/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fwww.dynamicyield.com%2Flesson%2Faa-testing-in-experimentation%2F" /><link rel="alternate" type="text/xml+oembed" href="https://www.dynamicyield.com/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fwww.dynamicyield.com%2Flesson%2Faa-testing-in-experimentation%2F&format=xml" /><meta name="generator" content="WPML ver:4.6.8 stt:1,4,3,28,2;" /> <script data-cfasync="false" data-pagespeed-no-defer>var dataLayer_content = {"visitorLoginState":"logged-out","visitorType":"visitor-logged-out","pagePostType":"lesson","pagePostType2":"single-lesson","pagePostAuthor":"David Steinberg","pagePostDate":"July 9, 2020","pagePostDateYear":"2020","pagePostDateMonth":"07","pagePostDateDay":"09","pagePostDateDayName":"Thursday","pagePostDateHour":"08","pagePostDateMinute":"37","pagePostDateIso":"2020-07-09T08:37:49-04:00","pagePostDateUnix":1594283869};
dataLayer.push( dataLayer_content );</script> <script type="text/javascript">console.warn && console.warn("[GTM4WP] Google Tag Manager container code placement set to OFF !!!");
console.warn && console.warn("[GTM4WP] Data layer codes are active but GTM container must be loaded using custom coding !!!");</script> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=3.0, user-scalable=1" /><link rel="icon" type="image/png" href="/favicon-16x16.png" sizes="16x16"><link rel="icon" type="image/png" href="/favicon-32x32.png" sizes="32x32"><link rel="icon" type="image/png" href="/favicon-96x96.png" sizes="96x96"><link rel="icon" type="image/png" href="/favicon-180x180.png" sizes="180x180"><link rel="icon" type="image/png" href="/favicon-192x192.png" sizes="192x192"><link rel="icon" type="image/png" href="/favicon-194x194.png" sizes="194x194"><link rel="shortcut icon" href="/favicon.ico"> <script type="text/javascript" data-noptimize>function dy_bind_adobe_mkto_events(form, form_id, form_name) {
// Init the adobeDataLayer object
window.adobeDataLayer = window.adobeDataLayer || [];
var focus_reported = false;
var form_completed = false;
var form_started_reported = false;
// Monitor input interactions - first focus (form start)
form.getFormElem().find('input, select').on('focus', function() {
// Only if it hasn't been reported yet
if(! focus_reported) {
focus_reported = true;
form_started_reported = true;
adobeDataLayer.push({
"event": "form_start",
"trackAction": {
"formName": form_name,
"formID": form_id,
"zoomInfo": localStorage.ZoomInfoData !== undefined,
}
});
}
});
// Detect dropoff if the form hasn't been completed
window.addEventListener("beforeunload", function(event) {
if (form_completed) return;
var filledLabels = [];
var count = 0;
form.getFormElem().find('input, select').each(function() {
var value = this.value && this.value.trim();
if (value) {
var label = form.getFormElem().find("label[for='" + this.id + "']").text().trim();
if (label) {
filledLabels.push(label);
count++;
}
}
});
if (count > 0) {
adobeDataLayer.push({
"event": "form_dropoff",
"formInfo": {
"formName": form_name,
"formID": form_id,
"zoomInfo": localStorage.ZoomInfoData !== undefined,
"completedFields": {
"labels": filledLabels.join("|"),
"count": count.toString()
}
}
});
}
});
// Hook into successful submission
form.onSuccess(function(values, followUpUrl) {
if(form_started_reported) {
// Report form completed
form_completed = true;
// Send an Adobe Analytics DataLayer event
adobeDataLayer.push({
"event": "form_complete",
"formInfo": {
"formName": form_name,
"formID": form_id,
"zoomInfo": localStorage.ZoomInfoData !== undefined,
"successAction": {
"action": "redirect",
"redirectUrl": followUpUrl
}
}
});
}
});
}
function dy_bind_adobe_cta_event(cta, cta_type, cta_section) {
var cta_name = cta.text().trim(),
cta_url,
page_url = window.location.href;
// const domainName = window.location.origin;
if (cta.attr('href') !== undefined) {
// cta_url = domainName + cta.attr('href');
cta_url = cta.attr('href');
} else {
cta_url = page_url;
}
adobeDataLayer.push({
"event": "ctaClick",
"clickInfo": {
"ctaName": cta_name,
"ctaType": cta_type,
"destinationURL": cta_url,
"ctaLocation": page_url,
"ctaSection": cta_section
}
});
}</script> <link rel="shortcut icon" href="https://www.dynamicyield.com/wp-content/uploads/2018/11/favicon-194x194.png" /><link rel="amphtml" href="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/amp/"></head><body class="lesson-template-default single single-lesson postid-131419 wp-featherlight-captions et_fixed_nav windows et_right_sidebar chrome et_includes_sidebar elementor-default default_header learning-center"><div id="page-container"><header id="main-header" class="et_nav_text_color_dark"
style="background: #fff;"><div class="container-fluid clearfix"><div class="logo-combined"> <a href="https://www.dynamicyield.com/learn"
title="XP Squared"><svg xmlns="http://www.w3.org/2000/svg" width="112.004" height="53.42" viewBox="0 0 112.004 53.42"> <path d="M8968.651,4606.571a49.057,49.057,0,0,0-6.159,6.344q-1.263,1.485-2.782,3.339t-3.079,3.71q-1.559,1.856-2.969,3.6t-2.522,3.079q.3.445,1.261,1.781t2.227,3.19q1.26,1.856,2.745,3.97t2.856,4.044q1.373,1.931,2.486,3.488t1.633,2.375q1.484,2.153,2.708,3.858a25.747,25.747,0,0,0,2.411,2.931,12.808,12.808,0,0,0,2.449,2,7.254,7.254,0,0,0,2.819,1v.742h-24.041v-.742q1.039-.147,2.226-.333a12.155,12.155,0,0,0,2.227-.557,4.522,4.522,0,0,0,1.706-1.039,2.31,2.31,0,0,0,.668-1.707,6.3,6.3,0,0,0-.965-2.783q-.964-1.743-2.226-3.673-1.708-2.672-3.9-5.825t-5.009-7.086q-1.188,1.411-2.6,3.116t-2.819,3.451q-1.411,1.744-2.708,3.376t-2.189,2.894a32.577,32.577,0,0,0-1.854,2.968,5.867,5.867,0,0,0-.816,2.82,2.706,2.706,0,0,0,.668,1.818,5.342,5.342,0,0,0,1.707,1.3,10.711,10.711,0,0,0,2.226.816,21.722,21.722,0,0,0,2.226.445v.742H8919.6v-.742a18.531,18.531,0,0,0,6.158-3.228,45.326,45.326,0,0,0,6.53-6.2q1.112-1.261,2.745-3.228t3.413-4.081q1.783-2.115,3.414-4.118t2.745-3.339q-.445-.593-2.04-2.856t-3.6-5.083q-2-2.819-3.97-5.6t-3.153-4.415q-1.559-2.152-2.782-3.784a26.076,26.076,0,0,0-2.375-2.782,12.747,12.747,0,0,0-2.449-1.93,13.428,13.428,0,0,0-3-1.3v-.742h24.189v.742q-.965.075-2.227.3a14.111,14.111,0,0,0-2.3.593,5.268,5.268,0,0,0-1.78,1.039,2.194,2.194,0,0,0-.743,1.707,5.789,5.789,0,0,0,1.039,2.746q1.039,1.708,2.374,3.636l3.785,5.454q1.928,2.782,4.675,6.715,2.449-2.818,5.009-6.159t4.488-5.787a31.259,31.259,0,0,0,1.93-2.968,6.278,6.278,0,0,0,1.039-2.968,2.5,2.5,0,0,0-.705-1.781,6.393,6.393,0,0,0-1.707-1.262,10.075,10.075,0,0,0-2.188-.816q-1.188-.3-2.152-.445v-.742h19.367v.742A15.309,15.309,0,0,0,8968.651,4606.571Z" transform="translate(-8919.604 -4602.453)"/> <path d="M9422.068,4621.77a14.179,14.179,0,0,1-2.338,3.932,14.748,14.748,0,0,1-3.376,2.931,20.786,20.786,0,0,1-3.934,1.967,24.367,24.367,0,0,1-4.451,1.113q-2.376.373-4.787.557t-4.711.186h-4.008v6.975q0,1.263.037,2.338t.038,2.337q.072,2.376.148,4.081a5.411,5.411,0,0,0,.815,2.82,6.25,6.25,0,0,0,2.3,1.929,10.813,10.813,0,0,0,4.75,1.113v.742h-23.745v-.742a20.734,20.734,0,0,0,3.561-.89,5.8,5.8,0,0,0,2.672-1.781,3.388,3.388,0,0,0,.594-1.336,14.222,14.222,0,0,0,.334-2q.111-1.113.186-2.375t.074-2.449c0-.692.011-1.632.037-2.82s.048-2.485.074-3.9.037-2.856.037-4.341v-18.773q0-1.334-.037-2.708c-.026-.915-.074-1.781-.148-2.6a13.947,13.947,0,0,0-.371-2.226,3.157,3.157,0,0,0-.705-1.447,5.554,5.554,0,0,0-2.856-1.6q-1.745-.407-3.748-.705v-.742h1.781q1.409,0,3.3-.037t4.117-.037q2.227,0,4.415-.037t4.119-.037h3.339a69.43,69.43,0,0,1,8.682.482,19.691,19.691,0,0,1,7.123,2.337,15.1,15.1,0,0,1,5.417,5.046,13.712,13.712,0,0,1,2.152,7.717A13.3,13.3,0,0,1,9422.068,4621.77Zm-9.312-12.986a13.487,13.487,0,0,0-9.535-6.159,28.427,28.427,0,0,0-4.155-.3,20,20,0,0,0-2.412.111q-.854.112-1.372.186a5.343,5.343,0,0,1-.186.779,4.946,4.946,0,0,0-.187.853q-.073.521-.147,1.447t-.149,2.634q-.075,2.6-.111,6.307t-.037,7.272v9.5q3.042,0,5.715-.186a18.593,18.593,0,0,0,4.971-1,13.885,13.885,0,0,0,7.271-5.379,14.741,14.741,0,0,0,2.523-8.422A13.63,13.63,0,0,0,9412.756,4608.784Z" transform="translate(-9329.39 -4601.215)"/> <g transform="translate(98.35)"> <path d="M9839.271,4616.067a10.2,10.2,0,0,1,3.648-3.89,26.827,26.827,0,0,0,3.232-2.56,3.789,3.789,0,0,0,1.127-2.635,2.928,2.928,0,0,0-.646-1.939,2.637,2.637,0,0,0-4.094.456,6.113,6.113,0,0,0-.4,2.256h-3.512a8.885,8.885,0,0,1,.85-3.789q1.447-2.75,5.139-2.75a6.52,6.52,0,0,1,4.643,1.616,5.584,5.584,0,0,1,1.727,4.277,5.783,5.783,0,0,1-1.221,3.624,12.091,12.091,0,0,1-2.627,2.345l-1.447,1.026q-1.359.964-1.859,1.395a4.076,4.076,0,0,0-.844,1h8.023v3.181h-12.584A8.616,8.616,0,0,1,9839.271,4616.067Z" transform="translate(-9838.422 -4601.215)"/> </g> </svg></a> <a href="https://www.dynamicyield.com/"
title="Dynamic Yield"><svg xmlns="http://www.w3.org/2000/svg" width="112" height="14.15" viewBox="0 0 112 14.15"> <g> <path d="M8919.6,5180.139v-10.982h1.729v4.505a2.662,2.662,0,0,1,2.321-1.107c2.214,0,3.412,1.684,3.412,3.853,0,2.23-1.32,3.914-3.473,3.914a2.538,2.538,0,0,1-2.276-1.168v.986Zm3.7-6.021a2.035,2.035,0,0,0-1.987,2.305,2.071,2.071,0,0,0,1.987,2.336c1.153,0,1.986-.88,1.986-2.336S8924.473,5174.118,8923.3,5174.118Z" transform="translate(-8919.604 -5168.976)"/> <path d="M8990.422,5213.118l1.774-3.868-3.154-6.507h1.987l2.124,4.672,1.986-4.672h1.866l-4.7,10.375Z" transform="translate(-8981.609 -5198.966)"/> <path d="M9105.4,5178.79c0,.5.031,1.046.061,1.35h-1.926a5.038,5.038,0,0,1-.076-.864,2.34,2.34,0,0,1-2.094,1.046,3.926,3.926,0,0,1-.061-7.827,2.238,2.238,0,0,1,2.109.925v-4.263h1.986Zm-3.793-.272a1.9,1.9,0,0,0,1.836-2.14,1.849,1.849,0,0,0-1.836-2.078,2.127,2.127,0,0,0,0,4.217Z" transform="translate(-9078.662 -5168.976)"/> <path d="M9178.639,5212.441l1.76-3.884-3.17-6.522h2.26l1.987,4.369,1.835-4.369h2.154l-4.687,10.406Z" transform="translate(-9149.652 -5198.333)"/> <path d="M9264.135,5207.852h-2.019v-7.462h1.958v.925a2.476,2.476,0,0,1,2.153-1.123c1.851,0,2.7,1.32,2.7,2.958v4.7h-2.019V5203.5a1.312,1.312,0,0,0-1.38-1.486,1.4,1.4,0,0,0-1.395,1.547Z" transform="translate(-9225.454 -5196.688)"/> <path d="M9339.31,5203.306l1.835-.272c.425-.061.562-.274.562-.531,0-.531-.409-.971-1.259-.971a1.325,1.325,0,0,0-1.427,1.213l-1.79-.379a2.969,2.969,0,0,1,3.2-2.458c2.366,0,3.247,1.335,3.247,2.836v3.671a8.454,8.454,0,0,0,.09,1.183h-1.851a5.105,5.105,0,0,1-.076-.9,2.494,2.494,0,0,1-2.2,1.107,2.357,2.357,0,0,1-2.563-2.246A2.3,2.3,0,0,1,9339.31,5203.306Zm2.4,1.274v-.334l-1.685.258a.942.942,0,0,0-.925.94.876.876,0,0,0,.971.864A1.516,1.516,0,0,0,9341.706,5204.581Z" transform="translate(-9292.393 -5196.435)"/> <path d="M9414.173,5207.726v-7.463h1.927v.91a2.565,2.565,0,0,1,2.185-1.123,2.308,2.308,0,0,1,2.214,1.244,2.76,2.76,0,0,1,5.066,1.592v4.839h-1.956V5203.3a1.245,1.245,0,0,0-1.32-1.411,1.372,1.372,0,0,0-1.381,1.472v4.369h-2V5203.3a1.252,1.252,0,0,0-1.319-1.411,1.366,1.366,0,0,0-1.4,1.472v4.369Z" transform="translate(-9361.234 -5196.561)"/> <path d="M9535.157,5167.457a1.244,1.244,0,1,1-1.243,1.259A1.243,1.243,0,0,1,9535.157,5167.457Zm-1,11.165v-7.463h2.018v7.463Z" transform="translate(-9468.158 -5167.457)"/> <path d="M9568.379,5203.868a1.921,1.921,0,0,0,1.91,2.108,1.707,1.707,0,0,0,1.715-1.259l1.775.592a3.466,3.466,0,0,1-3.49,2.518,3.861,3.861,0,0,1-3.928-3.959,3.821,3.821,0,0,1,3.867-3.959,3.42,3.42,0,0,1,3.506,2.519l-1.807.607a1.626,1.626,0,0,0-1.652-1.26A1.9,1.9,0,0,0,9568.379,5203.868Z" transform="translate(-9497.133 -5196.435)"/> <path d="M9669.8,5212.441l1.761-3.884-3.171-6.522h2.26l1.987,4.369,1.836-4.369h2.153l-4.687,10.406Z" transform="translate(-9588.244 -5198.333)"/> <path d="M9752.258,5167.457a1.244,1.244,0,1,1-1.244,1.259A1.244,1.244,0,0,1,9752.258,5167.457Zm-1,11.165v-7.463h2.018v7.463Z" transform="translate(-9662.02 -5167.457)"/> <path d="M9790.711,5205.46a3.36,3.36,0,0,1-3.4,2.366,3.966,3.966,0,0,1-.182-7.918c2.3,0,3.687,1.472,3.687,3.868,0,.289-.031.592-.031.622h-5.34a1.818,1.818,0,0,0,1.881,1.7,1.647,1.647,0,0,0,1.7-1.137Zm-1.881-2.442a1.5,1.5,0,0,0-1.668-1.472,1.615,1.615,0,0,0-1.669,1.472Z" transform="translate(-9690.994 -5196.435)"/> <path d="M9863.529,5180.139v-10.982h2.016v10.982Z" transform="translate(-9762.492 -5168.976)"/> <path d="M9901.386,5178.79c0,.5.03,1.046.061,1.35h-1.927a5.163,5.163,0,0,1-.075-.864,2.341,2.341,0,0,1-2.094,1.046,3.926,3.926,0,0,1-.061-7.827,2.234,2.234,0,0,1,2.108.925v-4.263h1.987Zm-3.793-.272a1.9,1.9,0,0,0,1.836-2.14,1.85,1.85,0,0,0-1.836-2.078,2.127,2.127,0,0,0,0,4.217Z" transform="translate(-9789.442 -5168.976)"/> </g> </svg></a></div><div id="et-top-navigation" class="cd-morph-dropdown"><nav id="top-menu-nav" class="main-nav"><ul id="top-menu" class="nav"></li></li></li></li></li></li></li></li></li></li></li></li></li><li class="top-link menu-item menu-item-type-post_type menu-item-object-page menu-item-home" data-content="discover-dynamic-yield" title=""> <a href="https://www.dynamicyield.com/">Discover Dynamic Yield</a></li><li class="has-dropdown learning_center menu-has-cta top-link-use-cases menu-item menu-item-type-custom menu-item-object-custom menu-item-has-children" data-content="learning-center" title=""> <a href="">Learning Center</a></li><li class="contact top-link walker-last mobile-only menu-item menu-item-type-post_type menu-item-object-page" data-content="xp2-newsletter" title=""> <a href="https://www.dynamicyield.com/newsletter/"><span class="button-arrow">XP<sup>2</sup> Newsletter</span></a></li></ul></nav><div class="morph-dropdown-wrapper"><div class="dropdown-list"><div class="dropdown-tooltip"></div><ul><li class="top-link menu-item menu-item-type-post_type menu-item-object-page menu-item-home" data-content="discover-dynamic-yield" title=""> <a href="https://www.dynamicyield.com/">Discover Dynamic Yield</a></li><li id="learning-center" class="dropdown links"><a href="" class="label">Learning Center</a><div class="content"><ul><li class="top"><ul class="links-list"><li class="resources-xp hidden-mobile menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/learn/">Take your knowledge to exponential levels</a></li><li class="hidden-desktop menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/learn/">XP<sup>2</sup> Hub<span>Take your knowledge to exponential levels</span></a></li><li class="resources-paths menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/learning-paths/">Learning Paths<span>Curated courses on key skill areas</span></a></li><li class="resources-webinars menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/talks/">Talks<span>Engaging discussions taking place in CX</span></a></li><li class="resources-articles menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/articles/">Articles<span>An expansive collection of in-depth playbooks</span></a></li><li class="resources-glossary menu-item menu-item-type-custom menu-item-object-custom"><a href="/glossary-terms/">Encyclopedia<span>A glossary of experience optimization terms</span></a></li><li class="resources-use-cases menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/personalization-examples/">Inspiration Library<span>Personalization examples from real brands</span></a></li><li class="resources-maturity menu-item menu-item-type-custom menu-item-object-custom"><a href="/personalization-maturity/"><strong class="badge-new">Personalization Maturity</strong><span>How global businesses prioritize personalization</span></a></li><li class="resources-guides menu-item menu-item-type-custom menu-item-object-custom"><a href="/guides/">Guides & reports<span>Comprehensive topic-specific materials</span></a></li><li class="resources-benchmarks menu-item menu-item-type-custom menu-item-object-custom"><a href="https://marketing.dynamicyield.com/benchmarks/">Benchmarks<span>Industry performance metrics and insights</span></a></li></ul></li><li class="bottom"><a href="/rooted-personalization/" target="_self"><div class="menu-banner"><div class="menu-cta--left"> <img src="/wp-content/uploads/2023/03/rooted-icon.svg" width="60" height="60" /></div><div class="menu-cta--right"> <strong class="menu-cta-title">Rooted Personalization<span>new</span></strong> <span class="menu-cta-subtitle">Learn a scalable methodology for building an audience-based personalization program.</span> <span class="menu-cta-button">Find out more →</span></div></div></a></li></ul></div></li><li class="contact top-link walker-last mobile-only menu-item menu-item-type-post_type menu-item-object-page" data-content="xp2-newsletter" title=""> <a href="https://www.dynamicyield.com/newsletter/"><span class="button-arrow">XP<sup>2</sup> Newsletter</span></a></li></ul><div class="bg-layer" aria-hidden="true"></div></div></div></li></ul></nav></div><div class="topnav-buttons"><div class="language-switcher header-language-switcher"><ul><li>en<ul><li><a class="active" href="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/">English</a></li><li><a href="https://www.dynamicyield.com/es/">Español</a></li><li><a href="https://www.dynamicyield.com/de/">Deutsch</a></li><li><a href="https://www.dynamicyield.com/fr/">Français</a></li><li><a href="https://www.dynamicyield.com/ja/">日本語</a></li></ul></li></li></ul></div> <a href="#" data-toggle="modal" data-target="#lc-subscription"
class="btn-desktop-only pulse-button v5 pulsed topnav-btn"> <span class="button-inner"> XP² Newsletter </span> </a> <a href="/newsletter/" class="btn-mobile-only pulse-button v5 pulsed topnav-btn"> <span class="button-inner"> XP² Newsletter </span> </a> <a href="" class="nav-trigger"> Open Nav <span class="nav-trigger-box"><span class="nav-trigger-inner"></span></span> </a></div></div></header><div id="lc-subscription" class="modal lc-subscription-popup" tabindex="-1" role="dialog" data-backdrop="false" aria-labelledby="lcSubscriptionModal"><div class="lc-container"><div class="lc-subscription-block"> <span class="lc-close-subscription-popup" data-dismiss="modal" aria-label="close"><i class="fa fa-close"></i></span><div class="lc-subscription-form-block lc-subscription-inner"><div class="lc-subscription-title-wrap"><h4 class="lc-subscription-title">Sign up for the XP² newsletter</h4><div class="lc-subscription-text">Join thousands of readers from Target, Citi, Spotify, Hulu, Google, Sephora, and other innovative brands who read our bi-weekly XP² newsletter, delivering educational content, research, and insights straight to your inbox</div><div class="lc-privacy">You may unsubscribe at any time. Visit our <a href="/privacy-policy/">privacy policy</a> to learn more about how we process your data and your rights as a data subject.</div></div><form class="mktoForm" data-formId="3876" data-formInstance="sub-popup"></form></div><div class="lc-subscription-ty-block lc-subscription-inner"><h4 class="lc-subscription-title text-center">Thanks for<br> signing up!</h4></div></div></div></div><div id="top-nav-backdrop"></div><div id="et-main-area"><div id="main-content"><div id="content-area"><header class="article-sticky-header"><div class="lc-header-container lc-flex"><div class="lc-header-logo"> <a href="https://www.dynamicyield.com/learn" title="XP Squared" class="xp-logo-container"> <svg xmlns="http://www.w3.org/2000/svg" width="112.004" height="53.42" viewBox="0 0 112.004 53.42"> <path d="M8968.651,4606.571a49.057,49.057,0,0,0-6.159,6.344q-1.263,1.485-2.782,3.339t-3.079,3.71q-1.559,1.856-2.969,3.6t-2.522,3.079q.3.445,1.261,1.781t2.227,3.19q1.26,1.856,2.745,3.97t2.856,4.044q1.373,1.931,2.486,3.488t1.633,2.375q1.484,2.153,2.708,3.858a25.747,25.747,0,0,0,2.411,2.931,12.808,12.808,0,0,0,2.449,2,7.254,7.254,0,0,0,2.819,1v.742h-24.041v-.742q1.039-.147,2.226-.333a12.155,12.155,0,0,0,2.227-.557,4.522,4.522,0,0,0,1.706-1.039,2.31,2.31,0,0,0,.668-1.707,6.3,6.3,0,0,0-.965-2.783q-.964-1.743-2.226-3.673-1.708-2.672-3.9-5.825t-5.009-7.086q-1.188,1.411-2.6,3.116t-2.819,3.451q-1.411,1.744-2.708,3.376t-2.189,2.894a32.577,32.577,0,0,0-1.854,2.968,5.867,5.867,0,0,0-.816,2.82,2.706,2.706,0,0,0,.668,1.818,5.342,5.342,0,0,0,1.707,1.3,10.711,10.711,0,0,0,2.226.816,21.722,21.722,0,0,0,2.226.445v.742H8919.6v-.742a18.531,18.531,0,0,0,6.158-3.228,45.326,45.326,0,0,0,6.53-6.2q1.112-1.261,2.745-3.228t3.413-4.081q1.783-2.115,3.414-4.118t2.745-3.339q-.445-.593-2.04-2.856t-3.6-5.083q-2-2.819-3.97-5.6t-3.153-4.415q-1.559-2.152-2.782-3.784a26.076,26.076,0,0,0-2.375-2.782,12.747,12.747,0,0,0-2.449-1.93,13.428,13.428,0,0,0-3-1.3v-.742h24.189v.742q-.965.075-2.227.3a14.111,14.111,0,0,0-2.3.593,5.268,5.268,0,0,0-1.78,1.039,2.194,2.194,0,0,0-.743,1.707,5.789,5.789,0,0,0,1.039,2.746q1.039,1.708,2.374,3.636l3.785,5.454q1.928,2.782,4.675,6.715,2.449-2.818,5.009-6.159t4.488-5.787a31.259,31.259,0,0,0,1.93-2.968,6.278,6.278,0,0,0,1.039-2.968,2.5,2.5,0,0,0-.705-1.781,6.393,6.393,0,0,0-1.707-1.262,10.075,10.075,0,0,0-2.188-.816q-1.188-.3-2.152-.445v-.742h19.367v.742A15.309,15.309,0,0,0,8968.651,4606.571Z" transform="translate(-8919.604 -4602.453)"/> <path d="M9422.068,4621.77a14.179,14.179,0,0,1-2.338,3.932,14.748,14.748,0,0,1-3.376,2.931,20.786,20.786,0,0,1-3.934,1.967,24.367,24.367,0,0,1-4.451,1.113q-2.376.373-4.787.557t-4.711.186h-4.008v6.975q0,1.263.037,2.338t.038,2.337q.072,2.376.148,4.081a5.411,5.411,0,0,0,.815,2.82,6.25,6.25,0,0,0,2.3,1.929,10.813,10.813,0,0,0,4.75,1.113v.742h-23.745v-.742a20.734,20.734,0,0,0,3.561-.89,5.8,5.8,0,0,0,2.672-1.781,3.388,3.388,0,0,0,.594-1.336,14.222,14.222,0,0,0,.334-2q.111-1.113.186-2.375t.074-2.449c0-.692.011-1.632.037-2.82s.048-2.485.074-3.9.037-2.856.037-4.341v-18.773q0-1.334-.037-2.708c-.026-.915-.074-1.781-.148-2.6a13.947,13.947,0,0,0-.371-2.226,3.157,3.157,0,0,0-.705-1.447,5.554,5.554,0,0,0-2.856-1.6q-1.745-.407-3.748-.705v-.742h1.781q1.409,0,3.3-.037t4.117-.037q2.227,0,4.415-.037t4.119-.037h3.339a69.43,69.43,0,0,1,8.682.482,19.691,19.691,0,0,1,7.123,2.337,15.1,15.1,0,0,1,5.417,5.046,13.712,13.712,0,0,1,2.152,7.717A13.3,13.3,0,0,1,9422.068,4621.77Zm-9.312-12.986a13.487,13.487,0,0,0-9.535-6.159,28.427,28.427,0,0,0-4.155-.3,20,20,0,0,0-2.412.111q-.854.112-1.372.186a5.343,5.343,0,0,1-.186.779,4.946,4.946,0,0,0-.187.853q-.073.521-.147,1.447t-.149,2.634q-.075,2.6-.111,6.307t-.037,7.272v9.5q3.042,0,5.715-.186a18.593,18.593,0,0,0,4.971-1,13.885,13.885,0,0,0,7.271-5.379,14.741,14.741,0,0,0,2.523-8.422A13.63,13.63,0,0,0,9412.756,4608.784Z" transform="translate(-9329.39 -4601.215)"/> <g transform="translate(98.35)"> <path d="M9839.271,4616.067a10.2,10.2,0,0,1,3.648-3.89,26.827,26.827,0,0,0,3.232-2.56,3.789,3.789,0,0,0,1.127-2.635,2.928,2.928,0,0,0-.646-1.939,2.637,2.637,0,0,0-4.094.456,6.113,6.113,0,0,0-.4,2.256h-3.512a8.885,8.885,0,0,1,.85-3.789q1.447-2.75,5.139-2.75a6.52,6.52,0,0,1,4.643,1.616,5.584,5.584,0,0,1,1.727,4.277,5.783,5.783,0,0,1-1.221,3.624,12.091,12.091,0,0,1-2.627,2.345l-1.447,1.026q-1.359.964-1.859,1.395a4.076,4.076,0,0,0-.844,1h8.023v3.181h-12.584A8.616,8.616,0,0,1,9839.271,4616.067Z" transform="translate(-9838.422 -4601.215)"/> </g> </svg> </a></div><div class="lc-header-title"> A/A testing and decision making in experimentation</div><div class="lc-header-right lc-flex"><ul class="lc-header-social lc-flex"><li class="et-social-icon"> <a href="#" title="Share on Facebook" class="icon" onclick="window.open('https://www.facebook.com/sharer/sharer.php?u=' + encodeURIComponent(document.URL) + '&t=' + encodeURIComponent(document.URL)); return false;"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 460.1990662 458.5138245"><path d="M460.1990662,230.0995331C460.1990662,103.0190277,357.1800232,0,230.0995331,0S0,103.0190277,0,230.0995331C0,338.011261,74.2903519,428.5578003,174.5126801,453.4230957V300.4146423H127.064209v-70.3151093h47.4484711v-30.2983704c0-78.3184433,35.4433594-114.619339,112.3326569-114.619339c14.5774536,0,39.730896,2.8583221,50.020874,5.7166443v63.7409363c-5.4308167-0.5716553-14.8633118-0.8574677-26.58255-0.8574677c-37.7300415,0-52.3075867,14.2916718-52.3075867,51.4501038v24.8674927h75.158783l-12.9122009,70.3151093h-62.246582v158.0991821C371.9082336,444.7527161,460.1990662,347.742981,460.1990662,230.0995331z"/></svg> </a></li><li class="et-social-icon"> <a href="#" title="Share on Linkedin" class="icon" onclick="window.open( 'https://www.linkedin.com/shareArticle?mini=true&url=' + encodeURIComponent(document.location.href)); return false;"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M100.28 448H7.4V148.9h92.88zM53.79 108.1C24.09 108.1 0 83.5 0 53.8a53.79 53.79 0 0 1 107.58 0c0 29.7-24.1 54.3-53.79 54.3zM447.9 448h-92.68V302.4c0-34.7-.7-79.2-48.29-79.2-48.29 0-55.69 37.7-55.69 76.7V448h-92.78V148.9h89.08v40.8h1.3c12.4-23.5 42.69-48.3 87.88-48.3 94 0 111.28 61.9 111.28 142.3V448z"/></svg> </a></li><li class="et-social-icon"> <a href="#" title="Tweet" class="icon" onclick="window.open('https://twitter.com/intent/tweet?text=' + encodeURIComponent(document.title) + ':%20' + encodeURIComponent(document.URL)); return false;"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M389.2 48h70.6L305.6 224.2 487 464H345L233.7 318.6 106.5 464H35.8L200.7 275.5 26.8 48H172.4L272.9 180.9 389.2 48zM364.4 421.8h39.1L151.1 88h-42L364.4 421.8z"/></svg> </a></li></ul><div class="topnav-buttons"> <a href="#" data-toggle="modal" data-target="#lc-subscription" class="btn-desktop-only pulse-button v5 topnav-btn"> <span class="button-inner"> XP² Newsletter </span> </a> <a href="/newsletter/" class="pulse-button v5 topnav-btn btn-mobile-only"> <span class="button-inner"> XP² Newsletter </span> </a></div></div></div><div class="lc-progress-wrap"><div class="lc-scroll-progress"></div></div></header><article id="post-131419" class="et_pb_post post-131419 lesson type-lesson status-publish has-post-thumbnail hentry article-category-testing-optimization article-industry-agnostic article-tags-personalization-and-testing-frameworks article-function-bi-analytics article-function-conversion-optimization article-function-marketing article-type-tactical-guides course-testing-and-optimization"><div class="lesson-content-sidebar"><div class="lesson-content-sidebar-inner" style="background-color: #f4c0c2"><div class="lesson-content-sidebar-header"><h4>A/B Testing & Optimization Course</h4></div><style>@media only screen and (max-width: 1199px) {
.lesson-content-sidebar-header {
background-color: #f4c0c2;
}
.lesson-content-sidebar-header:after {
background-color: rgb(242,178,182);
}
}</style><div class="lesson-content-sidebar-list"><ol><li class=""> <a href="https://www.dynamicyield.com/lesson/introduction-to-ab-testing/">An introduction to A/B testing and optimization</a> <span class="mark-as-read" data-id="97"></span> <span class="article-read-mark">Already read</span></li><li class="current"> <a href="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/">A/A testing and decision making in experimentation</a> <span class="mark-as-read" data-id="131419"></span> <span class="has-video"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/statistical-significance/">Why reaching and protecting statistical significance is so important in A/B tests</a> <span class="mark-as-read" data-id="1783"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/traffic-allocation/">Choosing the right traffic allocation in A/B testing</a> <span class="mark-as-read" data-id="115918"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/conversion-attribution-scoping/">Understanding conversion attribution scoping in A/B testing</a> <span class="mark-as-read" data-id="21258"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/optimization-objective/">Choosing the right conversion optimization objective</a> <span class="mark-as-read" data-id="7468"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/bayesian-testing/">Frequentist vs. Bayesian approach in A/B testing</a> <span class="mark-as-read" data-id="17150"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/">Guidelines for running effective Bayesian A/B tests</a> <span class="mark-as-read" data-id="131426"></span> <span class="has-video"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/contextual-bandit-optimization/">Beyond A/B testing: Multi-armed bandit experiments</a> <span class="mark-as-read" data-id="7414"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/client-side-vs-server-side/">Client-side vs server-side A/B testing and personalization</a> <span class="mark-as-read" data-id="130425"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/ab-testing-without-segmentation/">Segmented A/B tests: Avoiding average experiences</a> <span class="mark-as-read" data-id="32157"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/impact-of-ab-testing-on-seo/">The impact of A/B tests and personalization on SEO</a> <span class="mark-as-read" data-id="91"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/no-failed-ab-tests/">There are no failed A/B tests: How to ensure every experiment yields meaningful results</a> <span class="mark-as-read" data-id="135394"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/ab-testing-analysis/">How to analyze and interpret A/B testing results</a> <span class="mark-as-read" data-id="32159"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/kopari-beauty/">Building an experimentation growth plan: A Kopari Beauty case study</a> <span class="mark-as-read" data-id="22970"></span> <span class="article-read-mark">Already read</span></li></ol><div class="view-all-courses-wrap"> <a href="https://www.dynamicyield.com/learning-paths/" class="view-all-courses">Discover more courses<svg xmlns="http://www.w3.org/2000/svg" width="10.928" height="9.501" viewBox="0 0 10.928 9.501" style=" "><path d="M10.929,4.753a.859.859,0,0,0-.19-.491L6.938.224A.835.835,0,0,0,5.892.16.736.736,0,0,0,5.9,1.2L8.577,4.041H.746a.733.733,0,0,0-.734.586.711.711,0,0,0,.7.84H8.577L5.9,8.3a.784.784,0,0,0-.008,1.038.8.8,0,0,0,1.037-.063l3.8-4.039a.652.652,0,0,0,.2-.483Z" transform="translate(-0.001 -0.003)"/></svg></a></div></div></div></div><div class="article-content"><div class="lesson-content-inner"><div class="article-breadcrumbs"> <a href="https://www.dynamicyield.com/learning-paths/" title="Learning Paths">Learning Paths</a> <a href="https://www.dynamicyield.com/course/testing-and-optimization/">A/B Testing & Optimization</a></div><h1 class="article-title">A/A testing and decision making in experimentation</h1><h2 class="article-excerpt">A/A testing is meant to check that the underlying system is running properly. In this post, we'll cover some of subtle issues to consider as you run them.</h2><div class="article-authors lc-flex"><div class="article-author-box lc-col lc-flex"><div class="article-author-image lc-col"><div class="entry_author_image"><img src="https://www.dynamicyield.com/wp-content/authors/DavidSteinberg-106.jpeg" alt="David Steinberg" width="76" height="76" /></div></div><div class="article-author-title lc-col"><div class="article-author-name"> <a href="https://www.dynamicyield.com/blog/author/davidsteinberg/" title="Posts by David Steinberg" rel="author">David Steinberg</a></div><div class="article-author-title-text"> Professor David Steinberg of KPA Group</div></div></div><div class="article-author-box lc-col lc-flex"><div class="article-author-image lc-col"><div class="entry_author_image"><img src="https://www.dynamicyield.com/wp-content/authors/RonKenett-105.png" alt="Ron Kenett" width="76" height="76" /></div></div><div class="article-author-title lc-col"><div class="article-author-name"> <a href="https://www.dynamicyield.com/blog/author/ronkenett/" title="Posts by Ron Kenett" rel="author">Ron Kenett</a></div><div class="article-author-title-text"> Professor Ron Kenett of KPA Group</div></div></div></div><div class="entry-content article-entry-content"><p style="font-size: 12px;"><em>Both professor Ron Kenett and David Steinberg of KPA Group sat down to talk to more candidly about some of the topics discussed in this article.</em></p><div class="wistia_responsive_padding" style="margin-top: 30px; padding: 56.25% 0 28px 0; position: relative;"><div class="wistia_responsive_wrapper" style="height: 100%; left: 0; position: absolute; top: 0; width: 100%;"><iframe class="wistia_embed" src="//fast.wistia.net/embed/iframe/ypxftgkdgw?videoFoam=true" name="wistia_embed" width="100%" height="100%" frameborder="0" scrolling="no" allowfullscreen="allowfullscreen"></iframe></div></div><p><amp-wistia-player data-media-hashed-id="kiomiesn1r" width="320" height="180" ></amp-wistia-player><script src="//fast.wistia.net/assets/external/E-v1.js" async=""></script></p><div class="video-transcript" style="margin-top: -55px;"><div class='et-learn-more clearfix'><h3 class='heading-more'>Read the full transcript<span class='et_learnmore_arrow'><span></span></span></h3><div class='learn-more-content'>If you run an A/A test and you quickly identify that some A’s look better than other A’s, and you know they are the same, then your confidence in the A/B testing methodology and technology is questionable. So the simplistic approach to A/A test is, just to give you some sense of confidence to debug the system, maybe there is a problem with the random assignment but nothing beyond that. You could think of more sophisticated approaches, where the A/A test is not just done as a precursor to the A/B test, but it can be done also in parallel with A/B tests for ongoing monitoring. So you could develop more sophisticated systems, which give you control over what’s going on, and that would give you the ability to manage changes in traffic composition and all sorts of things that can happen, that would impact your decision.</p><p>For example, it could happen that, there’s some breakdown in the system, you’re running a test and it has, your allocation is being handled by distributed servers, it’s no longer on a single server. One of the servers didn’t work, and in that geographic area, everyone was put into this bucket, and another one was put into that bucket and the people in that geographic area, happened to respond a bit differently than people in other geographic areas. Alright, well now you’ve biased your experiment, your experimental results because all of those people went into those visitors went into one bucket. And those are exactly the kinds of problems and bugs that you’d like to be able to filter out of the system, and the A/A test is a very good way to do that. So that when you do get to comparing different options and drawing conclusions, making business decisions on them, you have confidence that those are working properly.</p><p>Another way that it can easily happen and this was pretty common, is from over monitoring the experiment. It’s very natural if you’ve got data coming in, to not wait a week to look at them but to look at them, every day, maybe several times a day. So, someone who comes into the site, maybe they will respond, maybe they won’t respond, maybe they’ll buy something, maybe they won’t. There’s gonna be variability in the data that you see. One of the two A/As, one of your two buckets, it’s gonna show a slightly better KPI, Is it sufficiently better? But if you watch it over time, what’s gonna happen? It’s gonna drift, so, it might show that bucket, this bucket is better, it might show that this bucket is better, but you have to exercise some caution about trying to reach judgements too fast and that’s one thing that can again they can lead to a situation that may be, instead of a confidence builder, something that drives down your confidence. So, be a little bit careful about data peeking, because it’s gonna happen.</p><p>In clinical trials, there is something called intermediate analysis. You plan a trial and in some cases you plan but it has to be planned ahead of time, that you’re going to have, an analysis of the data halfway through. This has to be accounted for. You cannot just collect data and decide to look at it, if this was not planned ahead of time. So what happens in A/B testing or A/A testing is that people might do that without having a planned that ahead of time and that has an effect, people might not be aware of that effect and that might mislead them.</p><p>Almost everything that we’ve talked about, in terms of A/A testing is also relevant to A/B tests. You do expect to see some differences – that’s the reason that you’re running the test to begin with. So you’re not gonna be as surprised to see differences, in the case of the A/B test, but again, in order to make sure that you’re making the right business decisions, you do wanna make sure you observe the differences that you’re seeing are real ones, that if you now make a decision based on those, on those differences, this is not just something that’s an artifact in the data, it’s something that’s really there, and you can really build a sound, financial decision on it, business decision, that’s gonna, increase your business activity.</p><p>This becomes even more complex when you have several variants. So it’s not A/B, it’s A, B one, B two, B three, B four. So the complexity can be okay, we have A and we have five variants. So what do we want? We want to show, which is the best, we want to show that, they are better than A, we want to pick up the top two and if you add to this the option to design the alternatives using what I mentioned, in terms of designing experiments with combinations of the factors, then you get into even more complexity. So, the testing effort is not more difficult to do it’s the design and the analysis that requires a bit more.</p><p>Right, some of the early questions that were looked at, were things like, what is the question that Laplace study? What’s the probability that the sun will rise tomorrow? Alright, we’ve seen the sunrise for many days in a row, what’s the probability that it will rise tomorrow? And can you even, does it make sense to even talk about a question like that in a probabilistic sense?</p><p>Many of these early discussions had this very metaphysical nature to them. In terms of modern statistical data analysis, the use of Basie and technology and Basie and thinking, has become much much more established. Part of the reason why there was resistance and has been resistance, and some case is still some resistance is I think, sort of on the grounds of objective versus subjective science. And the idea that you’re bringing prior information to a bayesian analysis is it can be regarded and often has been regarded as expressing subjectivity. So I might have a different prior than Ron does, and we might come to different conclusions from the same data and data which problems and this is the standard environment of course for A/B testing, this is much less critical because again the data are so abundant that anything that you fit in through the prior is essentially going to be washed out by the data. And as a result, it doesn’t really make any difference. What I thought in advance, what Ron thought in advance we gonna agree with one another in the end, after we’ve seen the data from the experiment.</div></div></p></div><p></p><div id="WYNTK-Sec" class="what-you-need-to-know-sec"><p><img decoding="async" src="/wp-content/uploads/2020/09/star-icon.svg" style="width: 20px; margin-right: 5px; vertical-align: -2px;"><strong class="first">Summarize this article</strong><strong class="hidden">Here’s what you need to know:</strong></p><ul style="margin-left: 4px; font-size: 16px;"><li style="margin: 10px 0;">A/A testing acts as a pre-flight check for A/B tests, identifying and fixing underlying system issues to ensure reliable A/B testing results.</li><li style="margin: 10px 0;">A/A testing can monitor ongoing A/B tests, verifying their proper execution and guarding against external influences.</li><li style="margin: 10px 0;">Data peeking, or frequently checking A/A test data, can lead to inaccurate conclusions and should be avoided.</li><li style="margin: 10px 0;">The sample size for an A/A test depends on data type, current system performance, and the desired significance level of improvement.</li><li style="margin: 10px 0;">A/A testing is a valuable tool for ensuring accurate and reliable A/B testing results.</li></ul></div><p></p><p>The customer experience, which encapsulates the holistic relationship between a brand and customer, is made up of various interactions across channels and touchpoints. It’s no surprise, then, that companies think very carefully about how they design these experiences to attract more business. </p><p>How does a marketer know, though, if a proposed change to the customer experience will be a successful one?</p><p>The use of <a href="https://www.dynamicyield.com/lesson/introduction-to-ab-testing/">A/B testing</a> is an effective, data-driven method of finding out. The basic idea is simple: compare two versions of a page, for example, and show each one to a certain portion of the individuals entering the site and evaluate their results. </p><p><a href="https://www.dynamicyield.com/lesson/traffic-allocation/">Visitors are allocated</a> to one of the two variations randomly. Following well-established scientific principles, this generates a “fair test” between the two versions of the experience. Then, visitor behavior is summarized via informative KPIs like <a href="https://www.dynamicyield.com/glossary/ctr/">click-through rate (CTR)</a>, <a href="https://www.dynamicyield.com/glossary/conversion-rate/">conversion rate (CR)</a>, or revenue, and the average KPIs for the two experiences are compared. With A/B testing, you let customers’ behavior decide which version is the most successful, instead of relying on “intuition.” And, as the old adage has it, “the customer is always right.”</p><p>Although A/B tests are simple to describe, a number of important questions arise, which we’ll address below.</p><ul><li>How can I decide which variant (A or B) is better?</li><li>If I declare a winner (B is better than A), how certain am I that this is correct?</li><li>Can I extend the analysis to more than two options?</li><li>How can I know that my testing engine is working properly?</li><li>What sample size is needed for an A/B test?</li><li>Can a test be terminated early if results are clear before the end of the planned testing period?</li></ul><h2>Decision Making</h2><p>A/B tests serve the joint goals of <em>learn and earn</em>. The ultimate goal is to <em>earn</em> more from your site or digital property. To do so, you need to <em>learn</em> by collecting and analyzing the data on KPIs during the test. As you collect more data, you will gain confidence as to which experience is better. </p><p>An excellent summary of what you have learned is the <a href="https://www.dynamicyield.com/glossary/probability-to-be-best/"><em>Probability to Be Best</em></a> (or P2BB). This is a simple statistic that states the probability that B is the better variant, i.e. has the larger KPI across the visitors to your site. For example, if P2BB is 0.8, then the probability that B is the better choice is 80%, and the probability that A is the better choice is 20%. P2BB is an ideal basis for guiding the business decisions that must be made following an A/B test.</p><p>How do we compute P2BB? </p><p>To answer that question, we need to give you some background on what is called Bayesian statistics, named after the British clergyman Thomas Bayes who lived in the 18th century. The <a href="https://www.dynamicyield.com/blog/bayesian-approach-to-ab-testing/">Bayesian approach</a> to statistics uses probability distributions to describe anything that we don’t know and want to learn from data. </p><p>Some early questions that were looked at were “what’s the probability that the sun will rise tomorrow?” Back then, many wondered whether it even made sense to talk about a question like that in a probabilistic sense. The idea that you’re bringing prior information to a bayesian analysis was often regarded as expressing subjectivity.</p><p>That’s very different from the classical <a href="https://www.dynamicyield.com/blog/frequentists-approach-to-ab-testing/">Frequentists approach</a> you might have studied in an introductory statistics course, where either A is better than B, B is better than A, or they are identical. The question “what is the probability that B is better” is not part of the lexicon – you can’t give an answer, and in fact, you’re not permitted to ask the question! </p><p>However, anything that you fit in through the prior is essentially going to be washed out by the abundance of data generated in an A/B test. So in the end, it doesn’t really matter what was thought in advance, as there will be an agreement in the end after the data from an experiment is seen.</p><p>The Bayesian view makes probability summaries the common currency. You will end the A/B test with probability distributions describing what you know about the KPI for A and the KPI for B. Combining those two distributions lets you compute P2BB.</p><p>Another advantage of P2BB is that it extends directly to tests with more than two variants. At the end of the experiment, you can compute the P2BB for each one of the variants. If there is a clear winner, it will have P2BB close to 1. Clear losers will have P2BB close to 0. You may have two variants that are essentially the same and are better than all the others. In that case, each of the two leading variants will have P2BB near 0.5.</p><h2>Checking the System</h2><p>You need a reliable test engine to run a valid A/B test so that when you make a decision, it reflects a real difference between the tested alternatives and is not due to chance, or worse, a bug in the engine. </p><p>There are many things that can go wrong: </p><ul><li>The advantage of one variant is momentary and due to unusual traffic fluctuations </li><li>The randomization routine might be faulty, outcome data may not be recorded correctly </li><li>The analysis routine might not be correct </li></ul><p>For example, suppose the allocation was random for some visitors but gave variant A to all those who came from a particular geographic region. If the visitors from that region react differently than those in other regions, the allocation failure will bias your experiment. </p><p>You also need a <em>representative sample</em> of visitors. When you make a decision from an A/B test, you are always extrapolating from the site visitors involved in your experiment to all those who will visit your site in the future. For extrapolation to be valid, you need to make sure that all segments of those future visitors are covered in your test, and in roughly the same proportions. That’s what we mean by a <em>representative sample</em>. </p><p>As an example, web commerce is often subject to natural time trends in the outcome data. Diurnal effects or day-of-week effects may be present. If so, you want to make sure that (i) at least one full-time cycle is included in the experiment, and (ii) that randomization continues to work properly throughout the cycle. As another example, some of your traffic may come from laptops and some from smartphones. You want to know that both of these groups get adequate exposure in your test. </p><p>To check that the underlying test engine is running properly, you may want to begin with a small number of <em>A/A tests</em>. The idea of <a href="https://www.dynamicyield.com/glossary/aa-testing/">A/A testing</a> is to compare an experience <em>to itself</em>. As both experiences are identical, you obviously don’t expect to see any differences between them – if you run an A/A test and quickly identify that some A’s look better than other A’s, and you know they are the same, then your confidence in the A/B testing methodology and technology is questionable.</p><p>A/A testing gives a basis for trusting the results of subsequent A/B tests; setting a benchmark for what happens when there are no differences between the two variants under the test helps you appreciate the difference you see when comparing two different variants. Moreover, you can use your standard experience in the A/A test, so there is no risk of lost activity from using a new version.</p><p>In a more sophisticated approach, the A/A test might not just be done as a precursor to the A/B test, but also in parallel for ongoing monitoring. This would give you more control and the ability to manage changes in traffic composition as well as all sorts of things that can impact decision-making.</p><p>The A/A test format sounds easy, but there are some subtle issues to think about. Data is variable, so even if both groups in your experiment got the same experience, you will still see some difference in outcome. You need some yardsticks to tell you whether those differences are big enough to indicate problems, or are just part of the natural variation in the data. You need to think about how long to run an A/A test and on how many A/A tests you want to run before moving ahead.</p><p>We have already suggested a good yardstick: Probability to Be Best (P2BB). This yardstick takes full account of the variability in the data. It is important to check the average KPIs as well. Since you are comparing an experience to itself, you expect these to be very similar to one another. You should also look at the fraction of traffic directed to each condition. The allocation should be close to 50% for each one – an allocation that deviates from 50% is a sign of trouble.</p><p>Although P2BB is an excellent summary of what is learned from the data, we need to add a few words of caution. </p><p><strong>First</strong>, data has natural variability. P2BB depends on the data, so it is also variable. Consequently, there will be A/A tests that give a P2BB close to 0 or 1, even though you know there is no difference. No A/B testing strategy is immune to occasionally declaring a winner in this case. </p><p><strong>Second</strong>, data volumes are huge in many online experiments. With huge sample sizes, even very small differences in average KPI can lead to an extreme P2BB. <em>This reiterates the importance of checking the KPI averages as well.</em> You might have a clear indication of a winner from P2BB but associated with a KPI difference that makes almost no difference to your bottom line. </p><p><strong>Third</strong>, beware of over-monitoring. Testing engines let you track the results from your experiment. So it is natural to check regularly to see if the data accumulated thus far points to a winner. P2BB will vary during the course of the experiment, reflecting the variation in visitor outcomes. If you check the “current P2BB” every hour in a two-week A/A test, that adds up to more than 300 P2BB values. Some may be close to 1, others may be close to 0. A common, but risky, practice is to stop a test as soon as P2BB first exceeds a threshold (say dropping below 0.05 or rising above 0.95). Why is that risky? Because it is not so surprising that the <em>most extreme </em>hourly P2BB value will cross one of these boundaries. Getting a final P2BB below 0.05, for example, will happen in about 5% of your A/A tests. But getting just one P2BB result below 0.05 will happen far more often. <em>Data peeking can seriously inflate your false winner rate if not preplanned. </em>You should use only the final P2BB (meaning you’ve gathered the required data in relation to the sample size), together with the average KPIs, to evaluate your engine.</p><p><strong>Further</strong>, if you want to thoroughly check the engine, run several A/A tests. We all know that it is dangerous to reach too many conclusions based on a sample of size 1. The resulting final P2BB’s should be “spread out” between 0 and 1.</p><p>When data volume leads to an extreme P2BB alongside a small difference in average KPIs, it may be helpful to carry out a more refined analysis that reflects the business impact of the comparison. Consider defining a minimal critical difference in KPIs that has an impact on your business. For example, you might decide that a difference of less than 0.5% between two competing variants would not justify making a change. In that case, you can use the Bayesian analysis to break down the summary of an A/B test into three distinct outcomes: B is better, B is worse, or B is the same as A (i.e. the difference is smaller than the critical margin). For A/A tests, you will often find that the third option has high probability, meaning you have strong evidence that there is not a meaningful difference between the two variants.</p><p>One should also be mindful of the complexity of adding several variants – when it’s not A/B, but A, B one, B two, B three, B four. So, if we have A and four variants, we want to show which is the best, that they are better than A, and pick up the top two. The testing effort is not more difficult to do, but the design and analysis of combinations requires a bit more when it comes to introducing these alternatives.</p><h2>How large should an A/A test be?</h2><p>The idea of the A/A test is to provide a realistic dry run of the system. You want to be sure that about half the visitors are allocated to each option to check that the KPIs stay close to one another and that P2BB does not declare a clear winner. </p><p><strong>The sample size will depend on several factors</strong>:</p><ul><li>The type of data (binary, like CTR; continuous, like revenue)</li><li>The current performance</li><li>The performance level that would be a significant improvement</li><li>For continuous KPIs, the extent of variation between visitors</li></ul><p>Below we will expand on each of these and highlight with an example. </p><p>The type of data is important, as different sample size formulas are relevant for each. The current performance must be known or estimated. The sample size is always set with the goal of identifying some target change. For an A/B test, you can set the target improvement to match your expectations from the new test page. That approach doesn’t work for an A/A test, where both versions should have the same KPI. </p><p>Instead, we recommend setting the target to be the level of importance that you see as an improvement. For example, with a KPI of 0.010, a 10% increase to 0.011 might be your threshold for large. If your average revenue is $1 per visitor, an increase of just 3%, to $1.03, might be important. </p><p>A simple guide is to ask, “what is the size improvement that we would definitely want to identify?” The extent of variation is important for continuous KPIs. Typically, what you will want is the standard deviation (SD) of the result among those who are converters. </p><p>It is worth noting that the same inputs are relevant for an A/B test and they are used in essentially the same way. But there is one distinction – the goal of an A/B test is to find a difference, so it’s clearly important to provide a near guarantee that important differences will be identified. We achieve that sharper focus by increasing the sample sizes. In an A/A test, though, the goal is to check the system. There is no need for the tight comparison of an A/B test, so we use a less stringent criterion on focus, which leads to smaller sample sizes.</p><p>Further, we recommend running an A/B test for at least one complete time cycle. If weekday visitors prefer A but weekend visitors prefer B, we would want to have both of these groups in our data before making a decision. With an A/A test, there is no concern that preferences for one option versus the other will change over time, so there is no need to run a test for a full cycle. If you run several A/A tests on the same experience, it is a good idea to spread them out over time so that you do see the engine in operation across a full-time cycle.</p><p>We can illustrate these ideas with an example. </p><p>Suppose your current CTR is 0.0060. An improvement to 0.0065 would be important. The test should include 191,000 visitors to each of the two options.</p><p>The calculation should be tempered with practical wisdom. If you compute a sample size that will take weeks to achieve, then the target improvement, even if it is an important gain, is probably too difficult to achieve. In that case, you should reset the target so that your A/A test runs no more than two weeks.</p><div class="article-separator"><svg width="54" height="26" viewBox="0 0 54 26" fill="none" xmlns="http://www.w3.org/2000/svg"> <path d="M53.0088 25.6367C52.7884 21.9564 51.8025 18.3625 50.1141 15.0849C48.4256 11.8073 46.0718 8.9181 43.2031 6.60203L43.2031 25.6367L53.0088 25.6367Z" fill="black"/> <path d="M14.4395 3.59038C10.3237 5.69174 6.82743 8.83082 4.29638 12.6973C1.76533 16.5638 0.28711 21.0239 0.0078125 25.6367L14.4395 25.6367L14.4395 3.59038Z" fill="black"/> <path d="M17.6563 2.20809L17.6562 25.6367L39.9829 25.6367L39.9829 4.36701C39.9829 4.36701 30.7303 -2.31342 17.6563 2.20809Z" fill="black"/> </svg></div><div class="article-tags-block"> <strong>Categories:</strong> <a href="https://www.dynamicyield.com/course/testing-and-optimization/">A/B Testing & Optimization</a></div><div class="article-tags-block"> <strong>Tags:</strong> <a href="https://www.dynamicyield.com/tags/personalization-and-testing-frameworks/">Personalization and Testing Frameworks</a></div><div class="article-pagination"><h5>Continue reading</h5><div class="article-pagination-row lc-flex"> <a rel="next" style="background-color: #f4c0c2" class="article-next-link pagination-link" href="https://www.dynamicyield.com/lesson/statistical-significance/" title="Why reaching and protecting statistical significance is so important in A/B tests">Why reaching and protecting statistical significance is so important in A/B tests</a><a rel="prev" style="background-color: #f4c0c2" class="article-previous-link pagination-link" href="https://www.dynamicyield.com/lesson/introduction-to-ab-testing/" title="An introduction to A/B testing and optimization">An introduction to A/B testing and optimization</a></div></div></div></div></div></article></div><section class="lc-subscription-section"><div class="lc-subscription-form-block lc-subscription-inner"><h4 class="lc-subscription-title">Sign up for the XP² newsletter</h4><div class="lc-subscription-text">Join thousands of readers from Target, Citi, Spotify, Hulu, Google, Sephora, and other innovative brands who read our bi-weekly XP² newsletter, delivering educational content, research, and insights straight to your inbox</div><form class="mktoForm" data-formId="3876" data-formInstance="sub-inline"></form><div class="lc-privacy-wrapper"><div class="lc-privacy"> You may unsubscribe at any time. Visit our <a href="/privacy-policy/">privacy policy</a> to learn more about how we process your data and your rights as a data subject.</div></div></div><div class="lc-subscription-ty-block lc-subscription-inner"><h4 class="lc-subscription-title text-center">Thanks for<br> signing up!</h4></div></section></div> <script type='text/javascript'>//DY.API('siteVar', {id: 128, value: 'David Steinberg'});
//DY.API('siteVar', {id: 129, value: ''});</script> <footer id="main-footer" class=""><div class="footer-top"><div class="footer-container"> <a href="https://www.dynamicyield.com/" class="dy-footer-logo"> <img src="/wp-content/uploads/2025/09/ma_symbol.svg" alt="Dynamic Yield footer logo" width="100" height="66"> </a><h4>Make hyper-personalization a reality</h4><p>Trusted by 450+ forward-thinking enterprise businesses:</p><div class="footer-clients retina_parent"> <img src="/wp-content/uploads/2023/08/customer-logos-footer.png" alt="DY Clients" width="498" height="123"></div><div class="footer-btns-wrap"> <a href="https://www.dynamicyield.com/request-demo/" class="btn-teal">Contact Sales</a> <a href="https://www.dynamicyield.com/product-demo/" class="btn-transparent">Watch a product demo<svg xmlns="http://www.w3.org/2000/svg" width="15.625" height="15.625" viewBox="0 0 15.625 15.625"> <path id="Icon_ionic-ios-play-circle" data-name="Icon ionic-ios-play-circle" d="M10.688,3.375A7.313,7.313,0,1,0,18,10.688,7.311,7.311,0,0,0,10.688,3.375Zm2.946,7.45L8.81,13.743a.156.156,0,0,1-.236-.137V7.77a.156.156,0,0,1,.236-.137l4.823,2.918A.161.161,0,0,1,13.634,10.825Z" transform="translate(-2.875 -2.875)" fill="none" stroke="rgba(0,0,0,0.8)" stroke-width="1"/> </svg> </a></div><ul class="footer-certificates"><li class="retina_parent"> <a href="https://www.dynamicyield.com/compliance/"> <img src="/wp-content/uploads/2022/05/gdpr-icon.png" alt="gdpr icon" width="139" height="70"> </a></li><li class="retina_parent"> <a href="https://www.dynamicyield.com/compliance/"> <img src="/wp-content/uploads/2023/04/SOCII-Badge.png" alt="socii icon" width="70" height="70"> </a></li><li> <a href="https://www.dynamicyield.com/compliance/"> <img src="/wp-content/uploads/2022/05/ccpa-icon.svg" alt="ccpa icon" width="117" height="40"> </a></li><li class="retina_parent"> <a href="https://www.dynamicyield.com/compliance/"> <img src="/wp-content/uploads/2022/05/iso-27001.png" alt="iso icon" width="106" height="68"> </a></li></ul></div></div><div id="et-footer-nav"><div class="footer-container"><ul id="menu-footer-menu" class="bottom-nav"><li id="menu-item-19183" class="footer_menu_title menu-item menu-item-type-custom menu-item-object-custom menu-item-19183"><a>Recommended Guides:</a></li><li id="menu-item-211968" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-211968"><a href="https://www.dynamicyield.com/article/personalization-guide/">Personalization</a></li><li id="menu-item-138198" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-138198"><a href="https://www.dynamicyield.com/lesson/introduction-to-ab-testing/">A/B Testing</a></li><li id="menu-item-147897" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-147897"><a href="https://www.dynamicyield.com/lesson/cro-plan/">Conversion Rate Optimization</a></li><li id="menu-item-19184" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-19184"><a href="/lesson/shopping-cart-abandonment-strategy/">Shopping Cart Abandonment</a></li><li id="menu-item-19185" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-19185"><a href="/lesson/product-recommendations-guide/">Product Recommendations</a></li><li id="menu-item-22805" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-22805"><a href="/lesson/promise-and-pitfalls-omnichannel-retailing/">Omnichannel Retailing</a></li><li id="menu-item-22885" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-22885"><a href="/lesson/the-economics-of-ecommerce-conversion-optimization/">eCommerce Conversion Rate Optimization</a></li><li id="menu-item-28240" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-28240"><a href="/article/50-most-important-dynamicyield-personalization-stats/">Personalization Statistics</a></li><li id="menu-item-33877" class="menu-item menu-item-type-post_type menu-item-object-post menu-item-33877"><a href="https://www.dynamicyield.com/article/tj-maxx-spired-checkout-optimization/">Checkout Optimization</a></li><li id="menu-item-123711" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-123711"><a href="/article/ecommerce-personalization-must-haves/">eCommerce Personalization</a></li><li id="menu-item-144090" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-144090"><a href="https://www.dynamicyield.com/lesson/strategizing-omnichannel-personalization/">Omnichannel Personalization</a></li><li id="menu-item-148146" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-148146"><a href="https://marketing.dynamicyield.com/benchmarks/cart-abandonment-rate/">Shopping Cart Abandonment Rate</a></li></ul><div class="footer-mach-logo retina_parent"> <img src="/wp-content/uploads/2022/05/mach-cert-22-wh.png" alt="Mach logo" width="63" height="70"></div><div class="language-switcher footer-language-switcher"><ul><li>en<ul><li><a class="active" href="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/">English</a></li><li><a href="https://www.dynamicyield.com/es/">Español</a></li><li><a href="https://www.dynamicyield.com/de/">Deutsch</a></li><li><a href="https://www.dynamicyield.com/fr/">Français</a></li><li><a href="https://www.dynamicyield.com/ja/">日本語</a></li></ul></li></li></ul></div></div></div><div id="footer-bottom"><div class="footer-container"><div class="footer_info_wrapper"><div id="footer-info"> <button id="ot-sdk-btn" class="ot-sdk-show-settings">Manage Cookies</button> <a href="/privacy-notice/" target="_blank" class="animlink">Privacy Notice</a> <a href="/tos/" target="_blank" class="animlink">Terms of use</a> <span>© 2026 Mastercard Dynamic Yield</span></div></div><div class="footer_social_icons"><style>.et-social-icons .et-social-icon img {
max-width: 16px;
transition: 0.2s all;
}
.et-social-icons .et-social-icon img:not(:hover) {
opacity: 0.5;
}</style><span itemscope itemtype="http://schema.org/Organization"><link itemprop="url" href="https://www.dynamicyield.com"><ul class="et-social-icons"><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.linkedin.com/company/dynamic-yield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-5.png" alt="LinkedIn Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.instagram.com/dynamicyield/" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-4.png" alt="Instagram Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.youtube.com/c/Dynamicyield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-3.png" alt="Youtube Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.tiktok.com/@dynamic.yield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-2.png" alt="TikTok Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.facebook.com/DynamicYield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-1.png" alt="Facebook Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://twitter.com/DynamicYield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame.png" alt="X Logo" /> </a></li></ul> </span></div></div></div></footer></div></div><div id="mobile_menu_overlay"></div> <script type='text/javascript' src='//cdn.dynamicyield.com/api/8765281/api_dynamic.js'></script> <script type='text/javascript' src='//cdn.dynamicyield.com/api/8765281/api_static.js'></script> <script type="text/javascript">var sbiajaxurl = "https://www.dynamicyield.com/wp-admin/admin-ajax.php";</script> <script type="text/javascript">!function(e,n){var r={"selectors":{"block":"pre.EnlighterJSRAW","inline":"code.EnlighterJSRAW"},"options":{"indent":2,"ampersandCleanup":true,"linehover":true,"rawcodeDbclick":false,"textOverflow":"break","linenumbers":true,"theme":"beyond","language":"generic","retainCssClasses":false,"collapse":false,"toolbarOuter":"","toolbarTop":"{BTN_RAW}{BTN_COPY}{BTN_WINDOW}{BTN_WEBSITE}","toolbarBottom":""},"resources":["https:\/\/www.dynamicyield.com\/wp-content\/plugins\/enlighter\/cache\/enlighterjs.min.css?tvrakvcLArVio6C","https:\/\/www.dynamicyield.com\/wp-content\/plugins\/enlighter\/resources\/enlighterjs\/enlighterjs.min.js"]},o=document.getElementsByTagName("head")[0],t=n&&(n.error||n.log)||function(){};e.EnlighterJSINIT=function(){!function(e,n){var r=0,l=null;function c(o){l=o,++r==e.length&&(!0,n(l))}e.forEach(function(e){switch(e.match(/\.([a-z]+)(?:[#?].*)?$/)[1]){case"js":var n=document.createElement("script");n.onload=function(){c(null)},n.onerror=c,n.src=e,n.async=!0,o.appendChild(n);break;case"css":var r=document.createElement("link");r.onload=function(){c(null)},r.onerror=c,r.rel="stylesheet",r.type="text/css",r.href=e,r.media="all",o.appendChild(r);break;default:t("Error: invalid file extension",e)}})}(r.resources,function(e){e?t("Error: failed to dynamically load EnlighterJS resources!",e):"undefined"!=typeof EnlighterJS?EnlighterJS.init(r.selectors.block,r.selectors.inline,r.options):t("Error: EnlighterJS resources not loaded yet!")})},(document.querySelector(r.selectors.block)||document.querySelector(r.selectors.inline))&&e.EnlighterJSINIT()}(window,console);</script><script type="text/javascript" id="rfw-script-js-extra">var rfw = {"speed":""};</script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_a5ec58aa1f222bedaf60cdced3b8a004.js?ver=2026020242" id="rfw-script-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_c1b7fbe6b1a3b777fddfe187094deb97.js?ver=2026020242" id="rfw-script-fitvid-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_ea3fad8947382ebf8709507a11a805db.js" id="divi-fitvids-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/themes/Divi/js/waypoints.min.js" id="waypoints-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_845779ea879c70773af95b2f6fc394d2.js" id="magnific-popup-js"></script> <script type="text/javascript" id="divi-custom-script-js-extra">var et_custom = {"ajaxurl":"https:\/\/www.dynamicyield.com\/wp-admin\/admin-ajax.php","images_uri":"https:\/\/www.dynamicyield.com\/wp-content\/themes\/Divi\/images","et_load_nonce":"6133b77f5f","subscription_failed":"Please, check the fields below to make sure you entered the correct information.","fill":"Fill","field":"field","invalid":"Invalid email","captcha":"Captcha"};</script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_43b4a88b2828bc5c023dc913689e8da5.js?ver=20.4" id="divi-custom-script-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/themes/Divi/custom/bootstrap/js/bootstrap.min.js" id="bootstrap-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/themes/Divi/custom/slick/slick.min.js" id="slick-js"></script> <script type="text/javascript" data-ot-ignore src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_12efd5a7d441d8b7f817fc6fc685dca9.js?ver=21.85" id="customscript-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_33fa37a5effe9361e40b8525e36e3eaf.js" id="typed-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_eb8981568f49ac148e96af3241eb95c7.js?ver=21.85" id="form_validation-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_be33bce741fd746dee1a6cf31c0b5bde.js" id="retina_js-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_9a50928b26148d1c35a95957e5ef2b9d.js?ver=21.85" id="mkto-scripts-js"></script> <script type="module" src="https://www.dynamicyield.com/wp-content/themes/Divi/js/cid-validation.js?ver=21.85" id="cid-validation"></script><script type="text/javascript" src="https://www.dynamicyield.com/wp-content/plugins/wp-featherlight/js/wpFeatherlight.pkgd.min.js?ver=1.3.4" id="wp-featherlight-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_17babe2fc54117766e24bd50889c6577.js?ver=1772203379" id="article-script-js"></script> <script type="text/javascript" id="dc_conversion_api-js-extra">var ajax_obj = {"nonce":"3cd30fb921","ajax_url":"https:\/\/www.dynamicyield.com\/wp-admin\/admin-ajax.php","ajax_action":"conversion_api_call"};</script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_34a6a3100a2ecbc7706af2d4c798f856.js?ver=935c7f8ce03960b02800bc8ae3231df8" id="dc_conversion_api-js"></script> <script type="text/javascript" id="et-shortcodes-js-js-extra">var et_shortcodes_strings = {"previous":"Previous","next":"Next"};</script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_bd5013e08f82109a2379477415b9b3af.js?ver=3.0" id="et-shortcodes-js-js"></script> <script type="text/javascript" data-noptimize>var dy_search_params = new URLSearchParams(window.location.search);
if(dy_search_params.has('utm_source')) {
localStorage.setItem('utm_source', dy_search_params.get('utm_source'));
}
if(dy_search_params.has('utm_medium')) {
localStorage.setItem('utm_medium', dy_search_params.get('utm_medium'));
}
if(dy_search_params.has('utm_content')) {
localStorage.setItem('utm_content', dy_search_params.get('utm_content'));
}
if(dy_search_params.has('utm_campaign')) {
localStorage.setItem('utm_campaign', dy_search_params.get('utm_campaign'));
}
if(dy_search_params.has('utm_term')) {
localStorage.setItem('utm_term', dy_search_params.get('utm_term'));
}
if(dy_search_params.has('mkt_tok')) {
localStorage.setItem('mkt_tok', dy_search_params.get('mkt_tok'));
}
// Also, fill in the data inside the Marketo form once its available
if(typeof window.MktoForms2 === 'object') {
MktoForms2.whenReady(function(_form) {
var form = _form.getFormElem();
console.log('MKTO FORM READY');
// Append the mkt_tok param if it exists in the URL
var ls_mkto_tok = localStorage.getItem('mkt_tok');
if(ls_mkto_tok) {
_form.addHiddenFields({
mkt_tok: ls_mkto_tok
});
_form.vals({ mkt_tok: ls_mkto_tok });
}
// Fill in the UTM data from localStorage
var dy_utms = {
'utm_source': 'txt_source__c',
'utm_medium': 'txt_medium__c',
'utm_content': 'txt_content__c',
'utm_campaign': 'txt_campaign_name__c',
'utm_term': 'txt_term__c',
'gclid': ['GCLID__c', 'GCLID_c__c']
};
for(var i = 0; i < Object.keys(dy_utms).length; i++) {
var ls_key = Object.keys(dy_utms)[i],
ls_value = localStorage.getItem(ls_key),
form_key = dy_utms[ls_key],
form_field;
if(ls_value) {
if(ls_key === 'gclid') {
ls_value = JSON.parse(ls_value).value;
}
if(typeof(form_key) === 'string') {
form_field = form[0].querySelector('input[type="hidden"][name="' + form_key + '"]');
if(form_field) {
form_field.value = ls_value;
}
} else {
for(var j = 0; j < form_key.length; j++) {
form_field = form[0].querySelector('input[type="hidden"][name="' + form_key[j] + '"]');
if(form_field) {
form_field.value = ls_value;
}
}
}
}
}
// IP Address
fetch('https://www.cloudflare.com/cdn-cgi/trace').then(function(result) {
result.text().then(function(values) {
values.split("\n").map(function(item) {
var pair = item.split('=');
if(['ip'].includes(pair[0])) {
form[0].querySelector('input[type="hidden"][name="IP__c"]').value = pair[1];
}
});
});
});
// GA ID and Referrer
var dy_utms_cookie = {};
document.cookie.split(';').forEach(function(el) {
var splitCookie = el.split('=');
var key = splitCookie[0].trim();
var value = splitCookie[1];
dy_utms_cookie[key] = value;
});
if(dy_utms_cookie["_ga"]) {
form[0].querySelector('input[type="hidden"][name="Visitor_ID__c"]').value = dy_utms_cookie["_ga"].substring(6);
}
if(dy_utms_cookie["personalize_referrer_string"]) {
form[0].querySelector('input[type="hidden"][name="Referral__c"]').value = dy_utms_cookie["personalize_referrer_string"].substring(6);
}
// ZoomInfo
if(localStorage.getItem('ZoomInfoData')) {
var zoominfo_data = JSON.parse(localStorage.getItem('ZoomInfoData'));
if(zoominfo_data) {
form[0].querySelector('input[type="hidden"][name="Alexa_Rank__c"]').value = zoominfo_data.alexaglobalrank;
form[0].querySelector('input[type="hidden"][name="Employees__c"]').value = zoominfo_data.employeesrange;
form[0].querySelector('input[type="hidden"][name="Company_Revenue_Range__c"]').value = zoominfo_data.estimatedannualrevenue;
}
}
// Also, dispatch a dataLayer event on success
_form.onSuccess(function(values, follow_up_url) {
window.dataLayer = window.dataLayer || [];
if([3739, 4802, 5094, 5110, 4946, 5076, 5116, 5171, 5172, 5173].includes(_form.getId())) {
window.dataLayer.push({
'event': 'demo_request_complete',
'eventCallback': function() {
if(follow_up_url) {
window.location.href = follow_up_url;
}
}
});
return false;
} else if([3876].includes(_form.getId())) {
window.dataLayer.push({
'event': 'newsletter_signup_complete',
'eventCallback': function() {
if(follow_up_url) {
window.location.href = follow_up_url;
}
}
});
return false;
}
});
});
}</script> <script>if(typeof(jQuery) !== 'undefined') {
$ = jQuery.noConflict();
}
function getUrlParameter(sParam) {
var sPageURL = decodeURIComponent(window.location.search.substring(1)),
sURLVariables = sPageURL.split('&'),
sParameterName,
i;
for (i = 0; i < sURLVariables.length; i++) {
sParameterName = sURLVariables[i].split('=');
if (sParameterName[0] === sParam) {
return sParameterName[1] === undefined ? true : sParameterName[1];
}
}
}
$(window).load(function(){
var videoID = getUrlParameter('video');
if(videoID !== undefined){
if($('#wistia-' + videoID + '-1').length){
var videoName = $('#wistia-' + videoID + '-1').parents('.tab-pane');
videoName = videoName[0].id
}
if($('a[href="#' + videoName +'"]').length){
$('a[href="#' + videoName +'"]').each(function(){
$(this).click();
});
}
addScript('https://fast.wistia.com/embed/medias/'+ videoID +'.jsonp');
addScript('https://fast.wistia.com/assets/external/E-v1.js');
$('body').append('<span class="wistia_embed wistia_async_'+videoID+' popover=true popoverContent=link dy-wistia-popup" style="display:inline"></span>');
window._wq = window._wq || [];
_wq.push({ id: videoID, onReady: function(video) {
var video = Wistia.api(videoID);
video.popover.show(); // open the popover
video.play(); // play the video
setTimeout(function(){video.unmute();}, 500);
}});
}
});</script> <style>#adBanner {
background-color: transparent;
height: 0;
width: 1px;
}</style><div id="wrapfabtest"><div id="adBanner"></div></div></body></html>
<!-- Dynamic page generated in 0.544 seconds. -->
<!-- Cached page generated by WP-Super-Cache on 2026-02-27 09:42:59 -->
<!-- super cache -->