Both professor Ron Kenett and David Steinberg of KPA Group sat down to talk to more candidly about some of the topics discussed in this article.
Read the full transcript
The internet today is one of the primary sources for bringing in customers, generating revenue. And often we have discussions about what might be the best ways to present ourselves, present our companies. The AB test lets you gain not just revenue and action from your website, it lets you gain information, and lets you gain information by making comparisons among those different alternatives that you might be considering. You can present different options to your customers, to your visitors to your website. That’s the A and the B, see how they respond, and when you see their responses, that lets you make a comparison, not based on what you thought might be the best, but on what you see works with your customers.
So the straightforward one is comparing two options, A and B. The more complex ones can compare combinations. So you could have A, which is a combination of two or three things, and B, which is a different combination, and C, which is still another combination. And if you apply methodology which is based on a statistical approach called design of experiments, in one test, you can learn about several factors, the effect of several factors.
So when you’re thinking about setting up an AB test, invariably, you have some alternatives that you wanna compare, and the tests usually follow methodology that’s been developed over many years in the statistical literature. What you’d like is to make sure that the people who see option A are, in some sense, as similar as possible to those who see option B. And it’s become well-established in the scientific literature, going back to work of the statistician Ronald Fisher, about 100 years ago, that the best way to do that is to make sure that you randomly allocate who gets which of the options that you want to compare.
So if you’re gonna start an experiment, you have to think about what are the options that are on the table. What do you want to compare? You have to have an engine at your disposal, framework that’s gonna let you make those random allocations. So if someone comes into your website, they’re not just gonna get a landing page, you have to decide what landing page they’re gonna get. You control that, and you want to control that by allocating it at random between the groups. That’s what’s gonna guarantee having a fair comparison.
Now, in order to validate the data that you’re capturing, you must ensure, for example, that this random allocation occurs. Because if you give the young people version A and the older people version B, and you see a difference, you will never know if it’s due to the age or to the web page design. That’s called confounding. So the random allocation, in a sense, establishes some causality on what is really impacting user behavior.
There are a number of ways that you can go about analyzing the data that you get from an AB test. Classical statistics takes a view that there is some true value for that KPI, and for example, it might be common if you’re comparing two options, to start with a framework that says both options are identical unless we get evidence that shows that they’re different. The Bayesian analyses take a somewhat different viewpoint, and rather than saying there’s some value, we just don’t know what it is, describe that value by putting a probability distribution on it. So you get more and more data. The probability distribution gets tighter and tighter about focusing on what we’ve now learned so that we have better information. It often gives a much more intuitive way, in particular in terms of business decisions, for looking at how to characterize what we know, rather than saying, “Well, it could be between this and this, and either it is, or it isn’t.” That’s sort of the classical way. And we give a more gradated answer by saying, “Well, this is the most likely.” And then there’s a distribution that describes how unlikely values become as we move away from what we think is the most likely value.
Bayesian statistics give you a, I think, a much more intuitive and natural way to express those understandings that you get at the end of an experiment in terms of say, what’s the probability that A is better, what’s the probability that B is better. So you bring in the data, use the data, and now Bayesian inference provides standard rules, and very strictly mathematical rules for taking the data, using that to update your prior to get what’s called a posterior distribution.
The posterior is, this is your current view. This is what you think after having seen the data. What describes your uncertainty about the KPIs after having seen the data, and you combine those two together in order to get your final statement about what it is you think. Can you really state a prior distribution? Is there a defensible way to state what you think in advance? As again, this has certainly been the source of friction between people in the different statistical camps. In the case of most AB tests, you’re getting lots and lots of data, and the data is going to wash out anything that you might’ve thought in the prior. So that for typical AB tests, again, because you have very large data volumes, it no longer becomes, in my mind, very controversial because whatever you put in to get the Bayesian engine going is essentially gonna be washed out by the data. And the data is really going to be the component that dominates what you get in the end.
So sample size is an important question in any study that’s going to be run. Whenever you go out to get the data. How long do I have to run my test? How many visitors to the website do I have to see before I can make conclusions? In order to get a test that gives a complete and honest picture, representative picture, of all the people who are long term gonna be exposed to whatever decisions you make, you wanna make sure that you include everyone within a cycle like that. You don’t wanna stop the test after having seen only visitors on Tuesday and Wednesday. And it may turn out that the decision you made is really catastrophic for those who come in on the weekends. So you always want to make sure that you pick up any potential time-cycles like this. And in terms of the sort of Bayesian descriptions that we were talking about later, I think this really gets to the heart of, in the end, you’re gonna get these posterior distributions that describe your KPIs, and how narrow do you want them to be? The tighter the interval, the more you know.
You can have a tight interval, but if the people who looked at the website and were part of the randomization subgroup, then whatever you’re going to get will not apply to the general group. So you really should work on both fronts. One is, how much information do you have? The tighter the interval, the more informative it is. But also how generalizable is the information that you provide?
Both of these aspects are important, and both of them can be the ones that really are the driver in dictating how large an experiment has to be. A website that has very heavy volume may quickly get to the needed sample size but without being able to see the full representative picture of all the visitors. And of course, it can happen just the other way, that by the time you get enough people in, enough visitors into the site, in order to get to the sample size you need, you’ve already gone through several of those time cycles. And so you have to see how both of these work out and balance against one another.
Summarize this articleHere’s what you need to know:
- A/B testing involves showing different versions of a webpage to different visitors and measuring which version performs better.
- Randomization is key to ensuring that the results of your A/B test are accurate and unbiased.
- It’s important to define your goals and metrics before running an A/B test so that you can measure success.
- Make sure your sample size is large enough to draw statistically significant conclusions from your A/B test results.
- Pay attention to the statistical significance of your results to avoid making decisions based on random fluctuations.
- Iterate on your A/B tests based on the results you learn to continuously improve your website.
A/B testing is a great way to compare alternative experiences or possible modifications to your website or digital properties. By running the two experiences in parallel and deciding at random which visitors land on which version, you get an honest comparison of which one delivers better performance.
Once you get the data from your test, you need reliable and informative ways to summarize them and reach conclusions. The Bayesian approach to doing statistics is a great way to accomplish this.
Here we describe the basic ideas behind Bayesian statistics and how they feed into the business decisions that you will need to make at the end of a test. We first present the approach to data analysis.
Then, we illustrate how data analysis leads to natural solutions for questions such as:
- Which variant drives better results?
- How much better are those results?
- Can we be confident in our conclusion?
- What if we want to test more than two variants?
- What sample size is needed for a Bayesian A/B test?
- Can a test be terminated early if results look clear?
What is Bayesian Statistics?
Bayesian statistics is named after the Reverend Thomas Bayes who lived in Britain in the 18th century and was responsible for proving Bayes Theorem. The guiding principle in Bayesian statistics is that you can use the language of probability to describe anything that we don’t know and want to learn from data. As we will see, this is an ideal language for discussing many of the questions that are most important to business decisions.
When you run an A/B test, the primary goal is to discover which variant is most effective. To do so, you compare the results with the two methods. The Bayesian analysis begins by looking at the KPIs for each of the variants. The data gives us information about their value but always leaves some uncertainty. We describe uncertainty using a probability distribution. We can simulate results from the distribution to understand how it looks.
The histogram in Figure 1. is typical of what you might get from your analysis. It shows the statistical distribution of difference in click-through rate (CTR) between A and B. Most of the curve is to the right of 0, providing evidence that A has a higher CTR. The fraction of the curve to the right of 0 is a great summary of that evidence – it is called the Probability to Be Best (P2BB). In Figure 1., P2BB is 0.699, in favor of version A.
Figure 1: Probability function for the CTR for A minus the CTR for B
If you have three or more variants, it is still easy to compute the probability of being best (P2BB), with probability split into three pieces, one for each variant.
How does Bayesian analysis work?
You need to start the Bayesian engine running with a prior probability distribution that reflects what you think about the KPI before seeing any data. The prior is then combined with the test data to obtain a posterior distribution for each variant. Figure 1. would be the posterior distribution for the difference between the A and B variants – this reflects the current view after having seen the data.
You need the prior to make things work, but when the volume of data is large – and that is almost always the case in A/B tests – the posterior is quickly dominated by the data and virtually “forgets” the prior; so you don’t need to invest too much thought about how to choose the prior. It is possible to work with some standard choices for priors without affecting the analysis.
But can you really state a prior distribution and what you think in advance?
This has certainly been the source of friction between people in the different statistical camps, so it is helpful to contrast the Bayesian summary with the “classical,” Frequentist approach you might have studied in an introductory statistics course. In the classical approach, either A is better than B, B is better than A, or they are identical. The question “what is the probability that A is better” is not part of the lexicon – you can’t give an answer, and in fact, you’re not even permitted to ask the question!
The classical paradigm does use probability, but only to describe the experimental data, not to summarize what you know about a KPI. That’s why you can’t make a summary like “the probability that B gives a positive uplift over A is 98%.
In most A/B tests, you’re getting lots and lots of data. Therefore, it no longer becomes very controversial, because whatever you put in to get the Bayesian engine going is essentially gonna be washed out by that information, which is really going to be the component that dominates what you get in the end. And that’s why we think the Bayesian summaries are so much better suited for business decisions.
For more on the different viewpoints when it comes to the Bayesian vs. Frequentist approach to A/B testing, check out this article.
How should traffic be distributed between variations?
What you’d like is to make sure that the people who see option A are, in some sense, as similar as possible to those who see option B. And it’s become well-established in the scientific literature, going back to work of the statistician Ronald Fisher, about 100 years ago, that the best way to do that is to make sure that you choose the right traffic allocation, or who gets which of the options that you want to compare.
So, you have to think about what are the options that are on the table. What do you want to compare? You have to have an engine at your disposal – a framework that’s gonna let you make those random allocations. If someone comes to your website, they’re not just gonna get a landing page, you have to decide what landing page they’re gonna get. You control that, and you want to control that by allocating it at random between the groups. That’s what’s gonna guarantee having a fair comparison.
In order to validate the data that you’re capturing, you must ensure, for example, that this random allocation occurs. Because if you give the young people version A and the older people version B, and you see a difference, you will never know if it’s due to the age or to the web page design. That’s called confounding. So the random allocation, in a sense, establishes some causality on what is really impacting user behavior.
What length of time is needed to run an a/b test?
In making these decisions, it is important to think about “what might go wrong” with the testing engine. You want your test to be sensitive to possible bugs. Often bugs are related to failures of the random allocation over time so that natural time trends in your KPIs bias the results. If there are weekly trends in your data, then you need to run an A/B test for two weeks to make sure that the engine is successfully handling them. If there are diurnal trends (but not weekly), you can afford to run shorter experiments. There may also be special events that suddenly increase or decrease the number of visitors to your site, or change the composition of visitors, affecting the performance of an experience. You may want to hold off on making any decisions until you see whether such issues have an impact on your results.
The considerations noted above will suggest a minimum time frame for running your experiment that ensures representative coverage in the A/B test of typical future site visitors. Often, these will dictate the length of time for the test.
How large a sample size is needed for a Bayesian test?
The sample size paradigm for Bayesian testing asks how narrow you want your final probability functions to be. You will probably want to focus on the probability function for the difference in KPIs between the two variants, as shown in Figure 1. And you will already have an idea what sort of difference could be important for your business; that gives a guideline for how narrow you want the function to be – the tighter the interval, the more you know.
It’s important to note that you can have a tight interval, but if the people who looked at the website and were part of the randomization subgroup, then whatever you’re going to get will not apply to the general group. So you really should work on both fronts. One is, how much information do you have? The tighter the interval, the more informative it is. But also how generalizable is the information that you provide?
Both of these aspects are important and can be the driver in dictating how large an experiment has to be. A website that has very heavy volume may quickly get to the needed sample size but without being able to see the full representative picture of all the visitors. And this can happen the other way around too, so that by the time you get the sample size you need, you’ve already gone through several time cycles – you have to see how both of these work out and balance against one another.
You will need to provide information on likely values for the KPIs. Often one of the versions is already running, so you can use historical data for this purpose. For new versions, you will need to think about what effect they might have. If the test includes multiple versions, the same ideas apply and the same inputs are needed.
The basic inputs needed to compute a sample size depend on the nature of the KPI:
- For binary KPIs like CTR or installation rate, all you need are the expected rates for each version
- If you are looking at purely continuous KPIs like revenue per conversion, you will need to supply an estimate of both the typical conversion size and the variation in size among those who do convert
- For a “mixed” KPI like revenue per visitor, you will need the average, variation, and an estimate of the fraction who will convert.
There are straightforward formulas you can use to determine the sample size, and also helpful A/B test sample size calculators.
What if there are more than two versions in the test?
The same sample size formulas are relevant, but here we distinguish between two goals: (1) finding the best and (2) showing that the standard version is not the best.
What is different about the second goal?
Suppose you have two new versions that you want to compare to the standard. Your best guess gives almost the same expected KPI to each of the new versions, with both of them improving over the standard. With very similar KPI’s, you will need a large sample to know which of the two new versions is better than the other. But if you expect both to be better than the standard, you can “rule out” the standard version with a much smaller test.
To further illustrate the idea:
The current experience is getting a CTR of 1%. A new version is proposed with the expectation that it will increase the CTR to 1.2%. The expected difference is 0.2%, so you need an A/B test that will tell you what is the true difference to a higher resolution. A good rule of thumb is to aim for a resolution that concentrates 95% of the Posterior Distribution in an interval that is the size of the expected difference.
In this example, that means narrowing down the difference so that you are 95% certain about an interval of length 0.002. To achieve this level of accuracy, you will need a sample of 83,575 visitors to both pages. What if even a 10% improvement in CTR, to 1.1%, is critical? Then you would need to know the difference up to, at most, 0.1%. That will require a sample size of 319,290 for each version.
Now let’s look at an example involving revenue per visitor:
A common scenario (often referred to as the Pareto principle)is that most visitors don’t provide any revenue and, among those who do, a small number of “big fish” pull up the average, and also the standard deviation.
The current landing page has a conversion rate of 1%; on average, converters spend $20; the standard deviation in the purchases of converters is $25. That means the average revenue per visitor is $0.20. Marketing thinks that the new version will bring in more purchasers, upping the conversion rate to 1.2%, but will decrease the proportion of “big fish,” so that the average conversion drops to $19 and the standard deviation to $22.
That translates to an average revenue of $0.228, an expected improvement of $0.028 per visitor. The A/B test will need to be large enough to narrow down the 95% credible interval for the difference in average revenue to no more than the expected difference, i.e. to $0.028. We will need 397,830 visitors to each experience to achieve that level of accuracy.
What if we want a stronger guarantee that we will find the important difference?
We advised above setting the sample size to control the width of the 95% credible interval. It is easy to reduce the chance of missing a valuable change by insisting on a higher level of probability, say 98% or 99%, but at the expense of increasing the sample size.
Can you stop a test early?
The probability that A is better (or worse) than B provides a natural metric to track as the test progresses. And a strong signal from the P2BB statistic can be used to guide early stopping. However, early stopping needs to be done with caution.
First, the Bayesian summary is not completely immune to data peeking. The P2BB will drift up and down during an experiment. This is especially true if there is no difference in performance between the variants, as in A/A testing. In that case, you will see that the P2BB increases and decreases as you collect more data. The fact that it crossed a threshold (like 95%) at some point during the test is not a guarantee that it would remain above the threshold if you wait until the planned termination time.
Second, it is always risky to stop before observing at least one full time cycle. Version B might be better for visitors at the start of the cycle, but worse for those later in the cycle. If you stop before the end of a full cycle, your results may be biased.
If you want to stop early, think about using stricter criteria for an early stopping point. For example, if you see that the probability becomes extreme (say above 0.999 or below 0.001), you can feel safe in stopping and making a decision. If there really is a difference between A and B, you can still expect to reach those thresholds relatively quickly.
Finally, we also reiterate the importance of waiting for time trends to reveal themselves in your test. You don’t want to decide on the basis of a short time window only to find that the results are very different later in the day or week – suppose weekday visitors respond better to A than B, but that trend flips on the weekend. Making a decision Wednesday from an experiment that began on Tuesday might lead to a bad decision.
Continue reading
<!DOCTYPE html> <!--[if IE 6]><html id="ie6" lang="en-US"> <![endif]--> <!--[if IE 7]><html id="ie7" lang="en-US"> <![endif]--> <!--[if IE 8]><html id="ie8" lang="en-US"> <![endif]--> <!--[if !(IE 6) | !(IE 7) | !(IE 8) ]><!--><html lang="en-US"> <!--<![endif]--><head><meta name="msvalidate.01" content="1033997EC383621B874EB0392F261984"/><link rel="preconnect" href="//cdn.dynamicyield.com"><link rel="preconnect" href="//st.dynamicyield.com"><link rel="preconnect" href="//rcom.dynamicyield.com"><link rel="dns-prefetch" href="//cdn.dynamicyield.com"><link rel="dns-prefetch" href="//st.dynamicyield.com"><link rel="dns-prefetch" href="//rcom.dynamicyield.com"> <script data-cfasync="false">window.dataLayer = window.dataLayer || [];
window.gtag = function() { dataLayer.push(arguments); }
window.gtag('consent', 'default', {
ad_storage: 'denied',
ad_user_data: 'denied',
ad_personalization: 'denied',
analytics_storage: 'denied',
functionality_storage: 'denied',
personalization_storage: 'denied',
security_storage: 'granted', // Only this is granted by default
});</script> <script data-cfasync="false">(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
'//www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-MCQHF8');</script> <script data-cfasync="false" async>// Init the adobeDataLayer array
var adobeDataLayer = adobeDataLayer || [];
// Only when the API is ready
function adobeWaitForDY() {
if (typeof(window.DY) !== 'undefined' && typeof(window.DY.AdDetection) !== 'undefined' &&
typeof($dy) !== 'undefined' && typeof($dy(window)) !== 'undefined' && typeof($dy.renderSmartTagSlider) !== 'undefined') {
var zi_data = localStorage.getItem('ZoomInfoData'),
ada_page_info = {
"pageName": "Guidelines for running effective Bayesian A/B tests",
"siteName": "Mastercard Dynamic Yield",
"pageTitle": "en_US :running-effective-bayesian-ab-tests",
"pageURL": "https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/",
"country": "US",
"language": "en",
"dyID": DY.dyid,
"userStatus": "Guest"
};
// Detect the hierarchy and
// site sections based on the URL slugs
var current_pathname = window.location.pathname.replace(/^\/|\/$/g, '');
if(current_pathname) {
var current_sitesections = current_pathname.split('/'),
current_hierarchy = current_sitesections.map(function(segment) {
return segment.split(/[-_]/).map(function(word) {
return word.charAt(0).toUpperCase() + word.slice(1);
})
.join(' ');
});
// Site section 1
if(current_sitesections[0]) {
ada_page_info.siteSection1 = current_sitesections[0];
}
// Site section 2
if(current_sitesections[1]) {
ada_page_info.siteSection2 = current_sitesections[1];
}
// Hierarchy
ada_page_info.hierarchy = current_hierarchy.join('|');
}
// Push the ZI params as well
if(zi_data) {
zi_data = JSON.parse(zi_data);
if(zi_data.name) {
ada_page_info.ziCompany = zi_data.name;
}
if(zi_data.domain) {
ada_page_info.ziDomain = zi_data.domain;
}
if(zi_data.employeesrange) {
ada_page_info.ziEmployees = zi_data.employeesrange;
}
if(zi_data.industry) {
ada_page_info.ziIndustry = zi_data.industry;
}
}
// Decode HTML prior to dispatching the event
Object.keys(ada_page_info).forEach(function(key) {
ada_page_info[key] = $('<textarea/>').html(ada_page_info[key]).text();
});
adobeDataLayer.push({
"event": "pageLoad",
"pageInfo": ada_page_info
});
} else {
setTimeout(adobeWaitForDY, 150);
}
}
adobeWaitForDY();</script> <script type="text/javascript">function OptanonWrapper() {
console.log('OneTrust Loaded');
DY_API_update_consent_mode();
OneTrust.OnConsentChanged(DY_API_update_consent_mode);
}
function DY_API_update_consent_mode() {
var groups = OnetrustActiveGroups.split(',');
DY.userActiveConsent = {accepted: groups.includes('C0004')};
DYO.ActiveConsent.updateConsentAcceptedStatus(groups.includes('C0004'));
if(groups.includes('C0004')) {
console.log('OneTrust_Consent_Is_Granted');
DY.API('event',{ name: 'OneTrust_Consent_Is_Granted', properties: {}});
}
console.log('OneTrust - DY API is ' + (groups.includes('C0004') ? 'Enabled' : 'Disabled'));
}
window.addEventListener('OTConsentApplied', function() {
var groups = OnetrustActiveGroups.split(',');
if(groups.includes('C0004')) {
console.log('OneTrust_Initial_Consent_Granted');
DY.API('event',{ name: 'OneTrust_Initial_Consent_Granted', properties: {}});
}
});</script> <script type="text/javascript">/*WebFontConfig = {
google: { families: [ 'Shadows+Into+Light::latin', 'Handlee::latin', 'Roboto:400,100,300,500:latin' ] }
};
(function() {
var wf = document.createElement('script');
wf.src = ('https:' == document.location.protocol ? 'https' : 'http') +
'://ajax.googleapis.com/ajax/libs/webfont/1/webfont.js';
wf.type = 'text/javascript';
wf.async = 'true';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(wf, s);
})(); */
console.log(
'%cLet’s personalize experiences together.\n%cDynamic Yield is hiring smart engineers like you to help solve hard problems.\n%cSee our jobs page: https://www.dynamicyield.com/careers',
'font-size:2.2em;line-height:1.4em;margin-top:20px;color:rgb(93, 56, 185);',
'font-size:1.1em;line-height:1.6em;color:rgb(0, 0, 0);',
'font-size:1.1em;line-height:1.6em;margin-bottom:20px;color:rgb(0, 0, 0);'
);
function waitForElementToDisplay(selector, time, functionName) {
if (document.getElementById(selector) != null) {
functionName();
return;
} else {
setTimeout(function () {
waitForElementToDisplay(selector, time, functionName);
}, time);
}
}
function dcWaitForJQuery(time, functionName) {
if (window.jQuery || window.n2) {
functionName();
return;
} else {
setTimeout(function () {
dcWaitForJQuery(time, functionName);
}, time);
}
}</script> <meta name="msvalidate.01" content="451B75B70520F4C76D67F116C9EFFE5C"/><meta property="twitter:account_id" content="4503599629852947"/><meta name="p:domain_verify" content="f8112f57c09778e892850075a87dc1cf"/><meta name="google-site-verification" content="9FbmfdC4cM7JG8sgYh6w9HhFlxvEHewltexeUVzuSfE"/><meta charset="UTF-8"/><link rel="pingback" href="https://www.dynamicyield.com/xmlrpc.php"/> <!--[if lt IE 9]> <script src="https://www.dynamicyield.com/wp-content/themes/Divi/js/html5.js" type="text/javascript"></script> <![endif]--> <script type="text/javascript">document.documentElement.className = 'js';</script> <script type="text/javascript">var getLangCode = 'en';</script> <link rel="prefetch" href="/wp-content/themes/Divi/css/custom-style.css" as="style" crossorigin="anonymous"><meta name='robots' content='index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1' /><link rel="alternate" hreflang="en" href="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/" /><link rel="alternate" hreflang="x-default" href="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/" /> <script data-cfasync="false" data-pagespeed-no-defer>var gtm4wp_datalayer_name = "dataLayer";
var dataLayer = dataLayer || [];</script> <title>Guidelines for running effective Bayesian A/B tests — Mastercard Dynamic Yield</title><link rel="preload" as="font" href="https://www.dynamicyield.com/wp-content/themes/Divi/fonts/CircularXXTT/CircularXXWeb-Bold.woff2" crossorigin/><link rel="preload" as="font" href="https://www.dynamicyield.com/wp-content/themes/Divi/fonts/CircularXXTT/CircularXXWeb-Medium.woff2" crossorigin/><link rel="preload" as="font" href="https://www.dynamicyield.com/wp-content/themes/Divi/fonts/CircularXXTT/CircularXXWeb-Book.woff2" crossorigin/><meta name="description" content="In this post, we describe the basic ideas behind Bayesian statistics and how they feed into business decisions you will need to make at the end of a test." /><link rel="canonical" href="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/" /><meta property="og:locale" content="en_US" /><meta property="og:type" content="article" /><meta property="og:title" content="Guidelines for running effective Bayesian A/B tests — Mastercard Dynamic Yield" /><meta property="og:description" content="In this post, we describe the basic ideas behind Bayesian statistics and how they feed into business decisions you will need to make at the end of a test." /><meta property="og:url" content="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/" /><meta property="og:site_name" content="Mastercard Dynamic Yield" /><meta property="article:publisher" content="https://www.facebook.com/DynamicYield" /><meta property="article:modified_time" content="2024-01-31T06:27:18+00:00" /><meta property="og:image" content="https://www.dynamicyield.com/wp-content/uploads/2020/07/Guidelines-for-running-effective-Bayesian-A-B-Tests-FB.jpg" /><meta property="og:image:width" content="1200" /><meta property="og:image:height" content="628" /><meta property="og:image:type" content="image/jpeg" /><meta name="twitter:card" content="summary_large_image" /><meta name="twitter:site" content="@DynamicYield" /><meta name="twitter:label1" content="Est. reading time" /><meta name="twitter:data1" content="19 minutes" /><meta name="twitter:label2" content="Written by" /><meta name="twitter:data2" content="Ron Kenett, David Steinberg" /> <script type="application/ld+json" class="yoast-schema-graph">{"@context":"https://schema.org","@graph":[{"@type":"WebPage","@id":"https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/","url":"https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/","name":"Guidelines for running effective Bayesian A/B tests — Mastercard Dynamic Yield","isPartOf":{"@id":"https://www.dynamicyield.com/#website"},"primaryImageOfPage":{"@id":"https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/#primaryimage"},"image":{"@id":"https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/#primaryimage"},"thumbnailUrl":"https://www.dynamicyield.com/wp-content/uploads/2020/07/Guidelines-for-running-effective-Bayesian-A-B-Tests-Blog.jpg","datePublished":"2020-07-09T12:45:26+00:00","dateModified":"2024-01-31T06:27:18+00:00","description":"In this post, we describe the basic ideas behind Bayesian statistics and how they feed into business decisions you will need to make at the end of a test.","breadcrumb":{"@id":"https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/#primaryimage","url":"https://www.dynamicyield.com/wp-content/uploads/2020/07/Guidelines-for-running-effective-Bayesian-A-B-Tests-Blog.jpg","contentUrl":"https://www.dynamicyield.com/wp-content/uploads/2020/07/Guidelines-for-running-effective-Bayesian-A-B-Tests-Blog.jpg","width":795,"height":320},{"@type":"BreadcrumbList","@id":"https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.dynamicyield.com/"},{"@type":"ListItem","position":2,"name":"Lessons","item":"https://www.dynamicyield.com/lesson/"},{"@type":"ListItem","position":3,"name":"Guidelines for running effective Bayesian A/B tests"}]},{"@type":"WebSite","@id":"https://www.dynamicyield.com/#website","url":"https://www.dynamicyield.com/","name":"Mastercard Dynamic Yield","description":"","publisher":{"@id":"https://www.dynamicyield.com/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://www.dynamicyield.com/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https://www.dynamicyield.com/#organization","name":"Mastercard Dynamic Yield","url":"https://www.dynamicyield.com/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https://www.dynamicyield.com/#/schema/logo/image/","url":"https://www.dynamicyield.com/wp-content/uploads/2019/10/DY-Hero_Logo-Black@x2.png","contentUrl":"https://www.dynamicyield.com/wp-content/uploads/2019/10/DY-Hero_Logo-Black@x2.png","width":458,"height":458,"caption":"Mastercard Dynamic Yield"},"image":{"@id":"https://www.dynamicyield.com/#/schema/logo/image/"},"sameAs":["https://www.facebook.com/DynamicYield","https://x.com/DynamicYield","https://www.instagram.com/dynamic.yield/","https://www.linkedin.com/company/dynamic-yield","http://www.pinterest.com/dynamicyield/","https://www.youtube.com/channel/UCPWEcR8lQRadVKYW8H4sndQ","https://en.wikipedia.org/wiki/Dynamic_Yield"]}]}</script> <link rel='dns-prefetch' href='//pages.dynamicyield.com' /><link rel="alternate" type="application/rss+xml" title="Mastercard Dynamic Yield » Feed" href="https://www.dynamicyield.com/feed/" /><link rel="alternate" type="application/rss+xml" title="Mastercard Dynamic Yield » Comments Feed" href="https://www.dynamicyield.com/comments/feed/" /><meta content="Divi v." name="generator"/><link rel='stylesheet' id='sbi_styles-css' href='https://www.dynamicyield.com/wp-content/plugins/instagram-feed/css/sbi-styles.min.css?ver=6.2.6' type='text/css' media='all' /><style id='wp-emoji-styles-inline-css' type='text/css'>img.wp-smiley, img.emoji {
display: inline !important;
border: none !important;
box-shadow: none !important;
height: 1em !important;
width: 1em !important;
margin: 0 0.07em !important;
vertical-align: -0.1em !important;
background: none !important;
padding: 0 !important;
}</style><link rel='stylesheet' id='wp-block-library-css' href='https://www.dynamicyield.com/wp-includes/css/dist/block-library/style.min.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><style id='classic-theme-styles-inline-css' type='text/css'>/*! This file is auto-generated */
.wp-block-button__link{color:#fff;background-color:#32373c;border-radius:9999px;box-shadow:none;text-decoration:none;padding:calc(.667em + 2px) calc(1.333em + 2px);font-size:1.125em}.wp-block-file__button{background:#32373c;color:#fff;text-decoration:none}</style><style id='global-styles-inline-css' type='text/css'>body{--wp--preset--color--black: #000000;--wp--preset--color--cyan-bluish-gray: #abb8c3;--wp--preset--color--white: #ffffff;--wp--preset--color--pale-pink: #f78da7;--wp--preset--color--vivid-red: #cf2e2e;--wp--preset--color--luminous-vivid-orange: #ff6900;--wp--preset--color--luminous-vivid-amber: #fcb900;--wp--preset--color--light-green-cyan: #7bdcb5;--wp--preset--color--vivid-green-cyan: #00d084;--wp--preset--color--pale-cyan-blue: #8ed1fc;--wp--preset--color--vivid-cyan-blue: #0693e3;--wp--preset--color--vivid-purple: #9b51e0;--wp--preset--gradient--vivid-cyan-blue-to-vivid-purple: linear-gradient(135deg,rgba(6,147,227,1) 0%,rgb(155,81,224) 100%);--wp--preset--gradient--light-green-cyan-to-vivid-green-cyan: linear-gradient(135deg,rgb(122,220,180) 0%,rgb(0,208,130) 100%);--wp--preset--gradient--luminous-vivid-amber-to-luminous-vivid-orange: linear-gradient(135deg,rgba(252,185,0,1) 0%,rgba(255,105,0,1) 100%);--wp--preset--gradient--luminous-vivid-orange-to-vivid-red: linear-gradient(135deg,rgba(255,105,0,1) 0%,rgb(207,46,46) 100%);--wp--preset--gradient--very-light-gray-to-cyan-bluish-gray: linear-gradient(135deg,rgb(238,238,238) 0%,rgb(169,184,195) 100%);--wp--preset--gradient--cool-to-warm-spectrum: linear-gradient(135deg,rgb(74,234,220) 0%,rgb(151,120,209) 20%,rgb(207,42,186) 40%,rgb(238,44,130) 60%,rgb(251,105,98) 80%,rgb(254,248,76) 100%);--wp--preset--gradient--blush-light-purple: linear-gradient(135deg,rgb(255,206,236) 0%,rgb(152,150,240) 100%);--wp--preset--gradient--blush-bordeaux: linear-gradient(135deg,rgb(254,205,165) 0%,rgb(254,45,45) 50%,rgb(107,0,62) 100%);--wp--preset--gradient--luminous-dusk: linear-gradient(135deg,rgb(255,203,112) 0%,rgb(199,81,192) 50%,rgb(65,88,208) 100%);--wp--preset--gradient--pale-ocean: linear-gradient(135deg,rgb(255,245,203) 0%,rgb(182,227,212) 50%,rgb(51,167,181) 100%);--wp--preset--gradient--electric-grass: linear-gradient(135deg,rgb(202,248,128) 0%,rgb(113,206,126) 100%);--wp--preset--gradient--midnight: linear-gradient(135deg,rgb(2,3,129) 0%,rgb(40,116,252) 100%);--wp--preset--font-size--small: 13px;--wp--preset--font-size--medium: 20px;--wp--preset--font-size--large: 36px;--wp--preset--font-size--x-large: 42px;--wp--preset--spacing--20: 0.44rem;--wp--preset--spacing--30: 0.67rem;--wp--preset--spacing--40: 1rem;--wp--preset--spacing--50: 1.5rem;--wp--preset--spacing--60: 2.25rem;--wp--preset--spacing--70: 3.38rem;--wp--preset--spacing--80: 5.06rem;--wp--preset--shadow--natural: 6px 6px 9px rgba(0, 0, 0, 0.2);--wp--preset--shadow--deep: 12px 12px 50px rgba(0, 0, 0, 0.4);--wp--preset--shadow--sharp: 6px 6px 0px rgba(0, 0, 0, 0.2);--wp--preset--shadow--outlined: 6px 6px 0px -3px rgba(255, 255, 255, 1), 6px 6px rgba(0, 0, 0, 1);--wp--preset--shadow--crisp: 6px 6px 0px rgba(0, 0, 0, 1);}:where(.is-layout-flex){gap: 0.5em;}:where(.is-layout-grid){gap: 0.5em;}body .is-layout-flow > .alignleft{float: left;margin-inline-start: 0;margin-inline-end: 2em;}body .is-layout-flow > .alignright{float: right;margin-inline-start: 2em;margin-inline-end: 0;}body .is-layout-flow > .aligncenter{margin-left: auto !important;margin-right: auto !important;}body .is-layout-constrained > .alignleft{float: left;margin-inline-start: 0;margin-inline-end: 2em;}body .is-layout-constrained > .alignright{float: right;margin-inline-start: 2em;margin-inline-end: 0;}body .is-layout-constrained > .aligncenter{margin-left: auto !important;margin-right: auto !important;}body .is-layout-constrained > :where(:not(.alignleft):not(.alignright):not(.alignfull)){max-width: var(--wp--style--global--content-size);margin-left: auto !important;margin-right: auto !important;}body .is-layout-constrained > .alignwide{max-width: var(--wp--style--global--wide-size);}body .is-layout-flex{display: flex;}body .is-layout-flex{flex-wrap: wrap;align-items: center;}body .is-layout-flex > *{margin: 0;}body .is-layout-grid{display: grid;}body .is-layout-grid > *{margin: 0;}:where(.wp-block-columns.is-layout-flex){gap: 2em;}:where(.wp-block-columns.is-layout-grid){gap: 2em;}:where(.wp-block-post-template.is-layout-flex){gap: 1.25em;}:where(.wp-block-post-template.is-layout-grid){gap: 1.25em;}.has-black-color{color: var(--wp--preset--color--black) !important;}.has-cyan-bluish-gray-color{color: var(--wp--preset--color--cyan-bluish-gray) !important;}.has-white-color{color: var(--wp--preset--color--white) !important;}.has-pale-pink-color{color: var(--wp--preset--color--pale-pink) !important;}.has-vivid-red-color{color: var(--wp--preset--color--vivid-red) !important;}.has-luminous-vivid-orange-color{color: var(--wp--preset--color--luminous-vivid-orange) !important;}.has-luminous-vivid-amber-color{color: var(--wp--preset--color--luminous-vivid-amber) !important;}.has-light-green-cyan-color{color: var(--wp--preset--color--light-green-cyan) !important;}.has-vivid-green-cyan-color{color: var(--wp--preset--color--vivid-green-cyan) !important;}.has-pale-cyan-blue-color{color: var(--wp--preset--color--pale-cyan-blue) !important;}.has-vivid-cyan-blue-color{color: var(--wp--preset--color--vivid-cyan-blue) !important;}.has-vivid-purple-color{color: var(--wp--preset--color--vivid-purple) !important;}.has-black-background-color{background-color: var(--wp--preset--color--black) !important;}.has-cyan-bluish-gray-background-color{background-color: var(--wp--preset--color--cyan-bluish-gray) !important;}.has-white-background-color{background-color: var(--wp--preset--color--white) !important;}.has-pale-pink-background-color{background-color: var(--wp--preset--color--pale-pink) !important;}.has-vivid-red-background-color{background-color: var(--wp--preset--color--vivid-red) !important;}.has-luminous-vivid-orange-background-color{background-color: var(--wp--preset--color--luminous-vivid-orange) !important;}.has-luminous-vivid-amber-background-color{background-color: var(--wp--preset--color--luminous-vivid-amber) !important;}.has-light-green-cyan-background-color{background-color: var(--wp--preset--color--light-green-cyan) !important;}.has-vivid-green-cyan-background-color{background-color: var(--wp--preset--color--vivid-green-cyan) !important;}.has-pale-cyan-blue-background-color{background-color: var(--wp--preset--color--pale-cyan-blue) !important;}.has-vivid-cyan-blue-background-color{background-color: var(--wp--preset--color--vivid-cyan-blue) !important;}.has-vivid-purple-background-color{background-color: var(--wp--preset--color--vivid-purple) !important;}.has-black-border-color{border-color: var(--wp--preset--color--black) !important;}.has-cyan-bluish-gray-border-color{border-color: var(--wp--preset--color--cyan-bluish-gray) !important;}.has-white-border-color{border-color: var(--wp--preset--color--white) !important;}.has-pale-pink-border-color{border-color: var(--wp--preset--color--pale-pink) !important;}.has-vivid-red-border-color{border-color: var(--wp--preset--color--vivid-red) !important;}.has-luminous-vivid-orange-border-color{border-color: var(--wp--preset--color--luminous-vivid-orange) !important;}.has-luminous-vivid-amber-border-color{border-color: var(--wp--preset--color--luminous-vivid-amber) !important;}.has-light-green-cyan-border-color{border-color: var(--wp--preset--color--light-green-cyan) !important;}.has-vivid-green-cyan-border-color{border-color: var(--wp--preset--color--vivid-green-cyan) !important;}.has-pale-cyan-blue-border-color{border-color: var(--wp--preset--color--pale-cyan-blue) !important;}.has-vivid-cyan-blue-border-color{border-color: var(--wp--preset--color--vivid-cyan-blue) !important;}.has-vivid-purple-border-color{border-color: var(--wp--preset--color--vivid-purple) !important;}.has-vivid-cyan-blue-to-vivid-purple-gradient-background{background: var(--wp--preset--gradient--vivid-cyan-blue-to-vivid-purple) !important;}.has-light-green-cyan-to-vivid-green-cyan-gradient-background{background: var(--wp--preset--gradient--light-green-cyan-to-vivid-green-cyan) !important;}.has-luminous-vivid-amber-to-luminous-vivid-orange-gradient-background{background: var(--wp--preset--gradient--luminous-vivid-amber-to-luminous-vivid-orange) !important;}.has-luminous-vivid-orange-to-vivid-red-gradient-background{background: var(--wp--preset--gradient--luminous-vivid-orange-to-vivid-red) !important;}.has-very-light-gray-to-cyan-bluish-gray-gradient-background{background: var(--wp--preset--gradient--very-light-gray-to-cyan-bluish-gray) !important;}.has-cool-to-warm-spectrum-gradient-background{background: var(--wp--preset--gradient--cool-to-warm-spectrum) !important;}.has-blush-light-purple-gradient-background{background: var(--wp--preset--gradient--blush-light-purple) !important;}.has-blush-bordeaux-gradient-background{background: var(--wp--preset--gradient--blush-bordeaux) !important;}.has-luminous-dusk-gradient-background{background: var(--wp--preset--gradient--luminous-dusk) !important;}.has-pale-ocean-gradient-background{background: var(--wp--preset--gradient--pale-ocean) !important;}.has-electric-grass-gradient-background{background: var(--wp--preset--gradient--electric-grass) !important;}.has-midnight-gradient-background{background: var(--wp--preset--gradient--midnight) !important;}.has-small-font-size{font-size: var(--wp--preset--font-size--small) !important;}.has-medium-font-size{font-size: var(--wp--preset--font-size--medium) !important;}.has-large-font-size{font-size: var(--wp--preset--font-size--large) !important;}.has-x-large-font-size{font-size: var(--wp--preset--font-size--x-large) !important;}
.wp-block-navigation a:where(:not(.wp-element-button)){color: inherit;}
:where(.wp-block-post-template.is-layout-flex){gap: 1.25em;}:where(.wp-block-post-template.is-layout-grid){gap: 1.25em;}
:where(.wp-block-columns.is-layout-flex){gap: 2em;}:where(.wp-block-columns.is-layout-grid){gap: 2em;}
.wp-block-pullquote{font-size: 1.5em;line-height: 1.6;}</style><link rel='stylesheet' id='rfw-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_7c036e0987ef80b703d7eb1abf922895.css?ver=2026020337' type='text/css' media='all' /><link rel='stylesheet' id='divi-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_ab3666a686d3dac03fd3032e5757637e.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='ctt-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_f1d1b2f761cf3768f427cdc5c4c2544b.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='inline-tweet-sharer-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_daa32ca85e114a0c6e5d1432c89237f3.css?ver=2.6.4' type='text/css' media='all' /><link rel='stylesheet' id='et_lb_modules-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_c0102af27938567cb82cc02e4da3a673.css?ver=2.4' type='text/css' media='all' /><link rel='stylesheet' id='et-shortcodes-css-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_40cb404848bc2e556105c8aea77f29aa.css?ver=3.0' type='text/css' media='all' /><link rel='stylesheet' id='et-shortcodes-responsive-css-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_3a30a77569bc4544b9465acbee8e5f97.css?ver=3.0' type='text/css' media='all' /><link rel='stylesheet' id='wp-featherlight-css' href='https://www.dynamicyield.com/wp-content/plugins/wp-featherlight/css/wp-featherlight.min.css?ver=1.3.4' type='text/css' media='all' /><link rel='stylesheet' id='baskerville-font-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_8c23179f030d02e0065d2054eab34279.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='markformc-font-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_0eb51906eca648ca4559d0a03b898d83.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='mackinac-font-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_9c8fec2609ebc9764a6e16471d9451ae.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='button-pulse-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_4cabc8dd29eaa9af7bf2df62f1c40af9.css?ver=935c7f8ce03960b02800bc8ae3231df8' type='text/css' media='all' /><link rel='stylesheet' id='fontawesome-css' href='https://www.dynamicyield.com/wp-content/themes/Divi/custom/font-awesome/css/font-awesome.min.css' type='text/css' media='all' /><link rel='stylesheet' id='bootstrap-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_d375e60add46d8de26a78728b517660f.css' type='text/css' media='all' /><link rel='stylesheet' id='slick-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_13b1b6672b8cfb0d9ae7f899f1c42875.css' type='text/css' media='all' /><link rel='stylesheet' id='slick-theme-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_f9faba678c4d6dcfdde69e5b11b37a2e.css' type='text/css' media='all' /><link rel='stylesheet' id='custom-theme-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_57f41281576414370c141cd00d2a80a9.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='custom-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_417039b6d54561fe1e8c94ee67cd902c.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='header-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_d75ad7543415df427913a5fe91606386.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='footer-style-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_25400e1cfd7099cc557083769c928231.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='video_tabs_module-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_f1109a8bc8c46ec9f2205cc8603c132f.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='mkto-reset-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_d117c8ecde800f5dccd137c16bab71af.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='learning-center-general-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_73dd3b78cf6e21c28be39db6740774b1.css?ver=26.27' type='text/css' media='all' /><link rel='stylesheet' id='article-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_e1d850b2d8d6fd6331ad74bd35ddde51.css?ver=1772206652' type='text/css' media='all' /><link rel='stylesheet' id='customers-shortcode-css' href='https://www.dynamicyield.com/wp-content/cache/autoptimize/css/autoptimize_single_c6bf61e9c8beb560d64d119f58757b8b.css?ver=26.27' type='text/css' media='all' /> <script type="text/javascript" src="https://www.dynamicyield.com/wp-includes/js/jquery/jquery.min.js?ver=3.7.1" id="jquery-core-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=3.4.1" id="jquery-migrate-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_dbdab0f35217f6041eb7e2028dc94ba4.js?ver=2.6.4" id="inline-tweet-sharer-js-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/themes/Divi/js/blazy.min.js" id="blazy-js"></script> <script type="text/javascript" src="//pages.dynamicyield.com/js/forms2/js/forms2.min.js?ver=21.85" id="mkto-form-script-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_db6be3b57dba139e789fec1a78186c8e.js?ver=21.85" id="learning-center-js-js"></script> <link rel="https://api.w.org/" href="https://www.dynamicyield.com/wp-json/" /><link rel="EditURI" type="application/rsd+xml" title="RSD" href="https://www.dynamicyield.com/xmlrpc.php?rsd" /><link rel='shortlink' href='https://www.dynamicyield.com/?p=131426' /><link rel="alternate" type="application/json+oembed" href="https://www.dynamicyield.com/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fwww.dynamicyield.com%2Flesson%2Frunning-effective-bayesian-ab-tests%2F" /><link rel="alternate" type="text/xml+oembed" href="https://www.dynamicyield.com/wp-json/oembed/1.0/embed?url=https%3A%2F%2Fwww.dynamicyield.com%2Flesson%2Frunning-effective-bayesian-ab-tests%2F&format=xml" /><meta name="generator" content="WPML ver:4.6.8 stt:1,4,3,28,2;" /> <script type="text/javascript">(function(url){
if(/(?:Chrome\/26\.0\.1410\.63 Safari\/537\.31|WordfenceTestMonBot)/.test(navigator.userAgent)){ return; }
var addEvent = function(evt, handler) {
if (window.addEventListener) {
document.addEventListener(evt, handler, false);
} else if (window.attachEvent) {
document.attachEvent('on' + evt, handler);
}
};
var removeEvent = function(evt, handler) {
if (window.removeEventListener) {
document.removeEventListener(evt, handler, false);
} else if (window.detachEvent) {
document.detachEvent('on' + evt, handler);
}
};
var evts = 'contextmenu dblclick drag dragend dragenter dragleave dragover dragstart drop keydown keypress keyup mousedown mousemove mouseout mouseover mouseup mousewheel scroll'.split(' ');
var logHuman = function() {
if (window.wfLogHumanRan) { return; }
window.wfLogHumanRan = true;
var wfscr = document.createElement('script');
wfscr.type = 'text/javascript';
wfscr.async = true;
wfscr.src = url + '&r=' + Math.random();
(document.getElementsByTagName('head')[0]||document.getElementsByTagName('body')[0]).appendChild(wfscr);
for (var i = 0; i < evts.length; i++) {
removeEvent(evts[i], logHuman);
}
};
for (var i = 0; i < evts.length; i++) {
addEvent(evts[i], logHuman);
}
})('//www.dynamicyield.com/?wordfence_lh=1&hid=DB214349C293124CDCB2BFE8939CA553');</script>
<script data-cfasync="false" data-pagespeed-no-defer>var dataLayer_content = {"visitorLoginState":"logged-out","visitorType":"visitor-logged-out","pagePostType":"lesson","pagePostType2":"single-lesson","pagePostAuthor":"Ron Kenett","pagePostDate":"July 9, 2020","pagePostDateYear":"2020","pagePostDateMonth":"07","pagePostDateDay":"09","pagePostDateDayName":"Thursday","pagePostDateHour":"08","pagePostDateMinute":"45","pagePostDateIso":"2020-07-09T08:45:26-04:00","pagePostDateUnix":1594284326};
dataLayer.push( dataLayer_content );</script> <script type="text/javascript">console.warn && console.warn("[GTM4WP] Google Tag Manager container code placement set to OFF !!!");
console.warn && console.warn("[GTM4WP] Data layer codes are active but GTM container must be loaded using custom coding !!!");</script> <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=3.0, user-scalable=1" /><link rel="icon" type="image/png" href="/favicon-16x16.png" sizes="16x16"><link rel="icon" type="image/png" href="/favicon-32x32.png" sizes="32x32"><link rel="icon" type="image/png" href="/favicon-96x96.png" sizes="96x96"><link rel="icon" type="image/png" href="/favicon-180x180.png" sizes="180x180"><link rel="icon" type="image/png" href="/favicon-192x192.png" sizes="192x192"><link rel="icon" type="image/png" href="/favicon-194x194.png" sizes="194x194"><link rel="shortcut icon" href="/favicon.ico"> <script type="text/javascript" data-noptimize>function dy_bind_adobe_mkto_events(form, form_id, form_name) {
// Init the adobeDataLayer object
window.adobeDataLayer = window.adobeDataLayer || [];
var focus_reported = false;
var form_completed = false;
var form_started_reported = false;
// Monitor input interactions - first focus (form start)
form.getFormElem().find('input, select').on('focus', function() {
// Only if it hasn't been reported yet
if(! focus_reported) {
focus_reported = true;
form_started_reported = true;
adobeDataLayer.push({
"event": "form_start",
"trackAction": {
"formName": form_name,
"formID": form_id,
"zoomInfo": localStorage.ZoomInfoData !== undefined,
}
});
}
});
// Detect dropoff if the form hasn't been completed
window.addEventListener("beforeunload", function(event) {
if (form_completed) return;
var filledLabels = [];
var count = 0;
form.getFormElem().find('input, select').each(function() {
var value = this.value && this.value.trim();
if (value) {
var label = form.getFormElem().find("label[for='" + this.id + "']").text().trim();
if (label) {
filledLabels.push(label);
count++;
}
}
});
if (count > 0) {
adobeDataLayer.push({
"event": "form_dropoff",
"formInfo": {
"formName": form_name,
"formID": form_id,
"zoomInfo": localStorage.ZoomInfoData !== undefined,
"completedFields": {
"labels": filledLabels.join("|"),
"count": count.toString()
}
}
});
}
});
// Hook into successful submission
form.onSuccess(function(values, followUpUrl) {
if(form_started_reported) {
// Report form completed
form_completed = true;
// Send an Adobe Analytics DataLayer event
adobeDataLayer.push({
"event": "form_complete",
"formInfo": {
"formName": form_name,
"formID": form_id,
"zoomInfo": localStorage.ZoomInfoData !== undefined,
"successAction": {
"action": "redirect",
"redirectUrl": followUpUrl
}
}
});
}
});
}
function dy_bind_adobe_cta_event(cta, cta_type, cta_section) {
var cta_name = cta.text().trim(),
cta_url,
page_url = window.location.href;
// const domainName = window.location.origin;
if (cta.attr('href') !== undefined) {
// cta_url = domainName + cta.attr('href');
cta_url = cta.attr('href');
} else {
cta_url = page_url;
}
adobeDataLayer.push({
"event": "ctaClick",
"clickInfo": {
"ctaName": cta_name,
"ctaType": cta_type,
"destinationURL": cta_url,
"ctaLocation": page_url,
"ctaSection": cta_section
}
});
}</script> <link rel="shortcut icon" href="https://www.dynamicyield.com/wp-content/uploads/2018/11/favicon-194x194.png" /><link rel="amphtml" href="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/amp/"></head><body class="lesson-template-default single single-lesson postid-131426 wp-featherlight-captions et_fixed_nav windows et_right_sidebar chrome et_includes_sidebar elementor-default default_header learning-center"><div id="page-container"><header id="main-header" class="et_nav_text_color_dark"
style="background: #fff;"><div class="container-fluid clearfix"><div class="logo-combined"> <a href="https://www.dynamicyield.com/learn"
title="XP Squared"><svg xmlns="http://www.w3.org/2000/svg" width="112.004" height="53.42" viewBox="0 0 112.004 53.42"> <path d="M8968.651,4606.571a49.057,49.057,0,0,0-6.159,6.344q-1.263,1.485-2.782,3.339t-3.079,3.71q-1.559,1.856-2.969,3.6t-2.522,3.079q.3.445,1.261,1.781t2.227,3.19q1.26,1.856,2.745,3.97t2.856,4.044q1.373,1.931,2.486,3.488t1.633,2.375q1.484,2.153,2.708,3.858a25.747,25.747,0,0,0,2.411,2.931,12.808,12.808,0,0,0,2.449,2,7.254,7.254,0,0,0,2.819,1v.742h-24.041v-.742q1.039-.147,2.226-.333a12.155,12.155,0,0,0,2.227-.557,4.522,4.522,0,0,0,1.706-1.039,2.31,2.31,0,0,0,.668-1.707,6.3,6.3,0,0,0-.965-2.783q-.964-1.743-2.226-3.673-1.708-2.672-3.9-5.825t-5.009-7.086q-1.188,1.411-2.6,3.116t-2.819,3.451q-1.411,1.744-2.708,3.376t-2.189,2.894a32.577,32.577,0,0,0-1.854,2.968,5.867,5.867,0,0,0-.816,2.82,2.706,2.706,0,0,0,.668,1.818,5.342,5.342,0,0,0,1.707,1.3,10.711,10.711,0,0,0,2.226.816,21.722,21.722,0,0,0,2.226.445v.742H8919.6v-.742a18.531,18.531,0,0,0,6.158-3.228,45.326,45.326,0,0,0,6.53-6.2q1.112-1.261,2.745-3.228t3.413-4.081q1.783-2.115,3.414-4.118t2.745-3.339q-.445-.593-2.04-2.856t-3.6-5.083q-2-2.819-3.97-5.6t-3.153-4.415q-1.559-2.152-2.782-3.784a26.076,26.076,0,0,0-2.375-2.782,12.747,12.747,0,0,0-2.449-1.93,13.428,13.428,0,0,0-3-1.3v-.742h24.189v.742q-.965.075-2.227.3a14.111,14.111,0,0,0-2.3.593,5.268,5.268,0,0,0-1.78,1.039,2.194,2.194,0,0,0-.743,1.707,5.789,5.789,0,0,0,1.039,2.746q1.039,1.708,2.374,3.636l3.785,5.454q1.928,2.782,4.675,6.715,2.449-2.818,5.009-6.159t4.488-5.787a31.259,31.259,0,0,0,1.93-2.968,6.278,6.278,0,0,0,1.039-2.968,2.5,2.5,0,0,0-.705-1.781,6.393,6.393,0,0,0-1.707-1.262,10.075,10.075,0,0,0-2.188-.816q-1.188-.3-2.152-.445v-.742h19.367v.742A15.309,15.309,0,0,0,8968.651,4606.571Z" transform="translate(-8919.604 -4602.453)"/> <path d="M9422.068,4621.77a14.179,14.179,0,0,1-2.338,3.932,14.748,14.748,0,0,1-3.376,2.931,20.786,20.786,0,0,1-3.934,1.967,24.367,24.367,0,0,1-4.451,1.113q-2.376.373-4.787.557t-4.711.186h-4.008v6.975q0,1.263.037,2.338t.038,2.337q.072,2.376.148,4.081a5.411,5.411,0,0,0,.815,2.82,6.25,6.25,0,0,0,2.3,1.929,10.813,10.813,0,0,0,4.75,1.113v.742h-23.745v-.742a20.734,20.734,0,0,0,3.561-.89,5.8,5.8,0,0,0,2.672-1.781,3.388,3.388,0,0,0,.594-1.336,14.222,14.222,0,0,0,.334-2q.111-1.113.186-2.375t.074-2.449c0-.692.011-1.632.037-2.82s.048-2.485.074-3.9.037-2.856.037-4.341v-18.773q0-1.334-.037-2.708c-.026-.915-.074-1.781-.148-2.6a13.947,13.947,0,0,0-.371-2.226,3.157,3.157,0,0,0-.705-1.447,5.554,5.554,0,0,0-2.856-1.6q-1.745-.407-3.748-.705v-.742h1.781q1.409,0,3.3-.037t4.117-.037q2.227,0,4.415-.037t4.119-.037h3.339a69.43,69.43,0,0,1,8.682.482,19.691,19.691,0,0,1,7.123,2.337,15.1,15.1,0,0,1,5.417,5.046,13.712,13.712,0,0,1,2.152,7.717A13.3,13.3,0,0,1,9422.068,4621.77Zm-9.312-12.986a13.487,13.487,0,0,0-9.535-6.159,28.427,28.427,0,0,0-4.155-.3,20,20,0,0,0-2.412.111q-.854.112-1.372.186a5.343,5.343,0,0,1-.186.779,4.946,4.946,0,0,0-.187.853q-.073.521-.147,1.447t-.149,2.634q-.075,2.6-.111,6.307t-.037,7.272v9.5q3.042,0,5.715-.186a18.593,18.593,0,0,0,4.971-1,13.885,13.885,0,0,0,7.271-5.379,14.741,14.741,0,0,0,2.523-8.422A13.63,13.63,0,0,0,9412.756,4608.784Z" transform="translate(-9329.39 -4601.215)"/> <g transform="translate(98.35)"> <path d="M9839.271,4616.067a10.2,10.2,0,0,1,3.648-3.89,26.827,26.827,0,0,0,3.232-2.56,3.789,3.789,0,0,0,1.127-2.635,2.928,2.928,0,0,0-.646-1.939,2.637,2.637,0,0,0-4.094.456,6.113,6.113,0,0,0-.4,2.256h-3.512a8.885,8.885,0,0,1,.85-3.789q1.447-2.75,5.139-2.75a6.52,6.52,0,0,1,4.643,1.616,5.584,5.584,0,0,1,1.727,4.277,5.783,5.783,0,0,1-1.221,3.624,12.091,12.091,0,0,1-2.627,2.345l-1.447,1.026q-1.359.964-1.859,1.395a4.076,4.076,0,0,0-.844,1h8.023v3.181h-12.584A8.616,8.616,0,0,1,9839.271,4616.067Z" transform="translate(-9838.422 -4601.215)"/> </g> </svg></a> <a href="https://www.dynamicyield.com/"
title="Dynamic Yield"><svg xmlns="http://www.w3.org/2000/svg" width="112" height="14.15" viewBox="0 0 112 14.15"> <g> <path d="M8919.6,5180.139v-10.982h1.729v4.505a2.662,2.662,0,0,1,2.321-1.107c2.214,0,3.412,1.684,3.412,3.853,0,2.23-1.32,3.914-3.473,3.914a2.538,2.538,0,0,1-2.276-1.168v.986Zm3.7-6.021a2.035,2.035,0,0,0-1.987,2.305,2.071,2.071,0,0,0,1.987,2.336c1.153,0,1.986-.88,1.986-2.336S8924.473,5174.118,8923.3,5174.118Z" transform="translate(-8919.604 -5168.976)"/> <path d="M8990.422,5213.118l1.774-3.868-3.154-6.507h1.987l2.124,4.672,1.986-4.672h1.866l-4.7,10.375Z" transform="translate(-8981.609 -5198.966)"/> <path d="M9105.4,5178.79c0,.5.031,1.046.061,1.35h-1.926a5.038,5.038,0,0,1-.076-.864,2.34,2.34,0,0,1-2.094,1.046,3.926,3.926,0,0,1-.061-7.827,2.238,2.238,0,0,1,2.109.925v-4.263h1.986Zm-3.793-.272a1.9,1.9,0,0,0,1.836-2.14,1.849,1.849,0,0,0-1.836-2.078,2.127,2.127,0,0,0,0,4.217Z" transform="translate(-9078.662 -5168.976)"/> <path d="M9178.639,5212.441l1.76-3.884-3.17-6.522h2.26l1.987,4.369,1.835-4.369h2.154l-4.687,10.406Z" transform="translate(-9149.652 -5198.333)"/> <path d="M9264.135,5207.852h-2.019v-7.462h1.958v.925a2.476,2.476,0,0,1,2.153-1.123c1.851,0,2.7,1.32,2.7,2.958v4.7h-2.019V5203.5a1.312,1.312,0,0,0-1.38-1.486,1.4,1.4,0,0,0-1.395,1.547Z" transform="translate(-9225.454 -5196.688)"/> <path d="M9339.31,5203.306l1.835-.272c.425-.061.562-.274.562-.531,0-.531-.409-.971-1.259-.971a1.325,1.325,0,0,0-1.427,1.213l-1.79-.379a2.969,2.969,0,0,1,3.2-2.458c2.366,0,3.247,1.335,3.247,2.836v3.671a8.454,8.454,0,0,0,.09,1.183h-1.851a5.105,5.105,0,0,1-.076-.9,2.494,2.494,0,0,1-2.2,1.107,2.357,2.357,0,0,1-2.563-2.246A2.3,2.3,0,0,1,9339.31,5203.306Zm2.4,1.274v-.334l-1.685.258a.942.942,0,0,0-.925.94.876.876,0,0,0,.971.864A1.516,1.516,0,0,0,9341.706,5204.581Z" transform="translate(-9292.393 -5196.435)"/> <path d="M9414.173,5207.726v-7.463h1.927v.91a2.565,2.565,0,0,1,2.185-1.123,2.308,2.308,0,0,1,2.214,1.244,2.76,2.76,0,0,1,5.066,1.592v4.839h-1.956V5203.3a1.245,1.245,0,0,0-1.32-1.411,1.372,1.372,0,0,0-1.381,1.472v4.369h-2V5203.3a1.252,1.252,0,0,0-1.319-1.411,1.366,1.366,0,0,0-1.4,1.472v4.369Z" transform="translate(-9361.234 -5196.561)"/> <path d="M9535.157,5167.457a1.244,1.244,0,1,1-1.243,1.259A1.243,1.243,0,0,1,9535.157,5167.457Zm-1,11.165v-7.463h2.018v7.463Z" transform="translate(-9468.158 -5167.457)"/> <path d="M9568.379,5203.868a1.921,1.921,0,0,0,1.91,2.108,1.707,1.707,0,0,0,1.715-1.259l1.775.592a3.466,3.466,0,0,1-3.49,2.518,3.861,3.861,0,0,1-3.928-3.959,3.821,3.821,0,0,1,3.867-3.959,3.42,3.42,0,0,1,3.506,2.519l-1.807.607a1.626,1.626,0,0,0-1.652-1.26A1.9,1.9,0,0,0,9568.379,5203.868Z" transform="translate(-9497.133 -5196.435)"/> <path d="M9669.8,5212.441l1.761-3.884-3.171-6.522h2.26l1.987,4.369,1.836-4.369h2.153l-4.687,10.406Z" transform="translate(-9588.244 -5198.333)"/> <path d="M9752.258,5167.457a1.244,1.244,0,1,1-1.244,1.259A1.244,1.244,0,0,1,9752.258,5167.457Zm-1,11.165v-7.463h2.018v7.463Z" transform="translate(-9662.02 -5167.457)"/> <path d="M9790.711,5205.46a3.36,3.36,0,0,1-3.4,2.366,3.966,3.966,0,0,1-.182-7.918c2.3,0,3.687,1.472,3.687,3.868,0,.289-.031.592-.031.622h-5.34a1.818,1.818,0,0,0,1.881,1.7,1.647,1.647,0,0,0,1.7-1.137Zm-1.881-2.442a1.5,1.5,0,0,0-1.668-1.472,1.615,1.615,0,0,0-1.669,1.472Z" transform="translate(-9690.994 -5196.435)"/> <path d="M9863.529,5180.139v-10.982h2.016v10.982Z" transform="translate(-9762.492 -5168.976)"/> <path d="M9901.386,5178.79c0,.5.03,1.046.061,1.35h-1.927a5.163,5.163,0,0,1-.075-.864,2.341,2.341,0,0,1-2.094,1.046,3.926,3.926,0,0,1-.061-7.827,2.234,2.234,0,0,1,2.108.925v-4.263h1.987Zm-3.793-.272a1.9,1.9,0,0,0,1.836-2.14,1.85,1.85,0,0,0-1.836-2.078,2.127,2.127,0,0,0,0,4.217Z" transform="translate(-9789.442 -5168.976)"/> </g> </svg></a></div><div id="et-top-navigation" class="cd-morph-dropdown"><nav id="top-menu-nav" class="main-nav"><ul id="top-menu" class="nav"></li></li></li></li></li></li></li></li></li></li></li></li></li><li class="top-link menu-item menu-item-type-post_type menu-item-object-page menu-item-home" data-content="discover-dynamic-yield" title=""> <a href="https://www.dynamicyield.com/">Discover Dynamic Yield</a></li><li class="has-dropdown learning_center menu-has-cta top-link-use-cases menu-item menu-item-type-custom menu-item-object-custom menu-item-has-children" data-content="learning-center" title=""> <a href="">Learning Center</a></li><li class="contact top-link walker-last mobile-only menu-item menu-item-type-post_type menu-item-object-page" data-content="xp2-newsletter" title=""> <a href="https://www.dynamicyield.com/newsletter/"><span class="button-arrow">XP<sup>2</sup> Newsletter</span></a></li></ul></nav><div class="morph-dropdown-wrapper"><div class="dropdown-list"><div class="dropdown-tooltip"></div><ul><li class="top-link menu-item menu-item-type-post_type menu-item-object-page menu-item-home" data-content="discover-dynamic-yield" title=""> <a href="https://www.dynamicyield.com/">Discover Dynamic Yield</a></li><li id="learning-center" class="dropdown links"><a href="" class="label">Learning Center</a><div class="content"><ul><li class="top"><ul class="links-list"><li class="resources-xp hidden-mobile menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/learn/">Take your knowledge to exponential levels</a></li><li class="hidden-desktop menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/learn/">XP<sup>2</sup> Hub<span>Take your knowledge to exponential levels</span></a></li><li class="resources-paths menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/learning-paths/">Learning Paths<span>Curated courses on key skill areas</span></a></li><li class="resources-webinars menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/talks/">Talks<span>Engaging discussions taking place in CX</span></a></li><li class="resources-articles menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/articles/">Articles<span>An expansive collection of in-depth playbooks</span></a></li><li class="resources-glossary menu-item menu-item-type-custom menu-item-object-custom"><a href="/glossary-terms/">Encyclopedia<span>A glossary of experience optimization terms</span></a></li><li class="resources-use-cases menu-item menu-item-type-post_type menu-item-object-page"><a href="https://www.dynamicyield.com/personalization-examples/">Inspiration Library<span>Personalization examples from real brands</span></a></li><li class="resources-maturity menu-item menu-item-type-custom menu-item-object-custom"><a href="/personalization-maturity/"><strong class="badge-new">Personalization Maturity</strong><span>How global businesses prioritize personalization</span></a></li><li class="resources-guides menu-item menu-item-type-custom menu-item-object-custom"><a href="/guides/">Guides & reports<span>Comprehensive topic-specific materials</span></a></li><li class="resources-benchmarks menu-item menu-item-type-custom menu-item-object-custom"><a href="https://marketing.dynamicyield.com/benchmarks/">Benchmarks<span>Industry performance metrics and insights</span></a></li></ul></li><li class="bottom"><a href="/rooted-personalization/" target="_self"><div class="menu-banner"><div class="menu-cta--left"> <img src="/wp-content/uploads/2023/03/rooted-icon.svg" width="60" height="60" /></div><div class="menu-cta--right"> <strong class="menu-cta-title">Rooted Personalization<span>new</span></strong> <span class="menu-cta-subtitle">Learn a scalable methodology for building an audience-based personalization program.</span> <span class="menu-cta-button">Find out more →</span></div></div></a></li></ul></div></li><li class="contact top-link walker-last mobile-only menu-item menu-item-type-post_type menu-item-object-page" data-content="xp2-newsletter" title=""> <a href="https://www.dynamicyield.com/newsletter/"><span class="button-arrow">XP<sup>2</sup> Newsletter</span></a></li></ul><div class="bg-layer" aria-hidden="true"></div></div></div></li></ul></nav></div><div class="topnav-buttons"><div class="language-switcher header-language-switcher"><ul><li>en<ul><li><a class="active" href="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/">English</a></li><li><a href="https://www.dynamicyield.com/es/">Español</a></li><li><a href="https://www.dynamicyield.com/de/">Deutsch</a></li><li><a href="https://www.dynamicyield.com/fr/">Français</a></li><li><a href="https://www.dynamicyield.com/ja/">日本語</a></li></ul></li></li></ul></div> <a href="#" data-toggle="modal" data-target="#lc-subscription"
class="btn-desktop-only pulse-button v5 pulsed topnav-btn"> <span class="button-inner"> XP² Newsletter </span> </a> <a href="/newsletter/" class="btn-mobile-only pulse-button v5 pulsed topnav-btn"> <span class="button-inner"> XP² Newsletter </span> </a> <a href="" class="nav-trigger"> Open Nav <span class="nav-trigger-box"><span class="nav-trigger-inner"></span></span> </a></div></div></header><div id="lc-subscription" class="modal lc-subscription-popup" tabindex="-1" role="dialog" data-backdrop="false" aria-labelledby="lcSubscriptionModal"><div class="lc-container"><div class="lc-subscription-block"> <span class="lc-close-subscription-popup" data-dismiss="modal" aria-label="close"><i class="fa fa-close"></i></span><div class="lc-subscription-form-block lc-subscription-inner"><div class="lc-subscription-title-wrap"><h4 class="lc-subscription-title">Sign up for the XP² newsletter</h4><div class="lc-subscription-text">Join thousands of readers from Target, Citi, Spotify, Hulu, Google, Sephora, and other innovative brands who read our bi-weekly XP² newsletter, delivering educational content, research, and insights straight to your inbox</div><div class="lc-privacy">You may unsubscribe at any time. Visit our <a href="/privacy-policy/">privacy policy</a> to learn more about how we process your data and your rights as a data subject.</div></div><form class="mktoForm" data-formId="3876" data-formInstance="sub-popup"></form></div><div class="lc-subscription-ty-block lc-subscription-inner"><h4 class="lc-subscription-title text-center">Thanks for<br> signing up!</h4></div></div></div></div><div id="top-nav-backdrop"></div><div id="et-main-area"><div id="main-content"><div id="content-area"><header class="article-sticky-header"><div class="lc-header-container lc-flex"><div class="lc-header-logo"> <a href="https://www.dynamicyield.com/learn" title="XP Squared" class="xp-logo-container"> <svg xmlns="http://www.w3.org/2000/svg" width="112.004" height="53.42" viewBox="0 0 112.004 53.42"> <path d="M8968.651,4606.571a49.057,49.057,0,0,0-6.159,6.344q-1.263,1.485-2.782,3.339t-3.079,3.71q-1.559,1.856-2.969,3.6t-2.522,3.079q.3.445,1.261,1.781t2.227,3.19q1.26,1.856,2.745,3.97t2.856,4.044q1.373,1.931,2.486,3.488t1.633,2.375q1.484,2.153,2.708,3.858a25.747,25.747,0,0,0,2.411,2.931,12.808,12.808,0,0,0,2.449,2,7.254,7.254,0,0,0,2.819,1v.742h-24.041v-.742q1.039-.147,2.226-.333a12.155,12.155,0,0,0,2.227-.557,4.522,4.522,0,0,0,1.706-1.039,2.31,2.31,0,0,0,.668-1.707,6.3,6.3,0,0,0-.965-2.783q-.964-1.743-2.226-3.673-1.708-2.672-3.9-5.825t-5.009-7.086q-1.188,1.411-2.6,3.116t-2.819,3.451q-1.411,1.744-2.708,3.376t-2.189,2.894a32.577,32.577,0,0,0-1.854,2.968,5.867,5.867,0,0,0-.816,2.82,2.706,2.706,0,0,0,.668,1.818,5.342,5.342,0,0,0,1.707,1.3,10.711,10.711,0,0,0,2.226.816,21.722,21.722,0,0,0,2.226.445v.742H8919.6v-.742a18.531,18.531,0,0,0,6.158-3.228,45.326,45.326,0,0,0,6.53-6.2q1.112-1.261,2.745-3.228t3.413-4.081q1.783-2.115,3.414-4.118t2.745-3.339q-.445-.593-2.04-2.856t-3.6-5.083q-2-2.819-3.97-5.6t-3.153-4.415q-1.559-2.152-2.782-3.784a26.076,26.076,0,0,0-2.375-2.782,12.747,12.747,0,0,0-2.449-1.93,13.428,13.428,0,0,0-3-1.3v-.742h24.189v.742q-.965.075-2.227.3a14.111,14.111,0,0,0-2.3.593,5.268,5.268,0,0,0-1.78,1.039,2.194,2.194,0,0,0-.743,1.707,5.789,5.789,0,0,0,1.039,2.746q1.039,1.708,2.374,3.636l3.785,5.454q1.928,2.782,4.675,6.715,2.449-2.818,5.009-6.159t4.488-5.787a31.259,31.259,0,0,0,1.93-2.968,6.278,6.278,0,0,0,1.039-2.968,2.5,2.5,0,0,0-.705-1.781,6.393,6.393,0,0,0-1.707-1.262,10.075,10.075,0,0,0-2.188-.816q-1.188-.3-2.152-.445v-.742h19.367v.742A15.309,15.309,0,0,0,8968.651,4606.571Z" transform="translate(-8919.604 -4602.453)"/> <path d="M9422.068,4621.77a14.179,14.179,0,0,1-2.338,3.932,14.748,14.748,0,0,1-3.376,2.931,20.786,20.786,0,0,1-3.934,1.967,24.367,24.367,0,0,1-4.451,1.113q-2.376.373-4.787.557t-4.711.186h-4.008v6.975q0,1.263.037,2.338t.038,2.337q.072,2.376.148,4.081a5.411,5.411,0,0,0,.815,2.82,6.25,6.25,0,0,0,2.3,1.929,10.813,10.813,0,0,0,4.75,1.113v.742h-23.745v-.742a20.734,20.734,0,0,0,3.561-.89,5.8,5.8,0,0,0,2.672-1.781,3.388,3.388,0,0,0,.594-1.336,14.222,14.222,0,0,0,.334-2q.111-1.113.186-2.375t.074-2.449c0-.692.011-1.632.037-2.82s.048-2.485.074-3.9.037-2.856.037-4.341v-18.773q0-1.334-.037-2.708c-.026-.915-.074-1.781-.148-2.6a13.947,13.947,0,0,0-.371-2.226,3.157,3.157,0,0,0-.705-1.447,5.554,5.554,0,0,0-2.856-1.6q-1.745-.407-3.748-.705v-.742h1.781q1.409,0,3.3-.037t4.117-.037q2.227,0,4.415-.037t4.119-.037h3.339a69.43,69.43,0,0,1,8.682.482,19.691,19.691,0,0,1,7.123,2.337,15.1,15.1,0,0,1,5.417,5.046,13.712,13.712,0,0,1,2.152,7.717A13.3,13.3,0,0,1,9422.068,4621.77Zm-9.312-12.986a13.487,13.487,0,0,0-9.535-6.159,28.427,28.427,0,0,0-4.155-.3,20,20,0,0,0-2.412.111q-.854.112-1.372.186a5.343,5.343,0,0,1-.186.779,4.946,4.946,0,0,0-.187.853q-.073.521-.147,1.447t-.149,2.634q-.075,2.6-.111,6.307t-.037,7.272v9.5q3.042,0,5.715-.186a18.593,18.593,0,0,0,4.971-1,13.885,13.885,0,0,0,7.271-5.379,14.741,14.741,0,0,0,2.523-8.422A13.63,13.63,0,0,0,9412.756,4608.784Z" transform="translate(-9329.39 -4601.215)"/> <g transform="translate(98.35)"> <path d="M9839.271,4616.067a10.2,10.2,0,0,1,3.648-3.89,26.827,26.827,0,0,0,3.232-2.56,3.789,3.789,0,0,0,1.127-2.635,2.928,2.928,0,0,0-.646-1.939,2.637,2.637,0,0,0-4.094.456,6.113,6.113,0,0,0-.4,2.256h-3.512a8.885,8.885,0,0,1,.85-3.789q1.447-2.75,5.139-2.75a6.52,6.52,0,0,1,4.643,1.616,5.584,5.584,0,0,1,1.727,4.277,5.783,5.783,0,0,1-1.221,3.624,12.091,12.091,0,0,1-2.627,2.345l-1.447,1.026q-1.359.964-1.859,1.395a4.076,4.076,0,0,0-.844,1h8.023v3.181h-12.584A8.616,8.616,0,0,1,9839.271,4616.067Z" transform="translate(-9838.422 -4601.215)"/> </g> </svg> </a></div><div class="lc-header-title"> Guidelines for running effective Bayesian A/B tests</div><div class="lc-header-right lc-flex"><ul class="lc-header-social lc-flex"><li class="et-social-icon"> <a href="#" title="Share on Facebook" class="icon" onclick="window.open('https://www.facebook.com/sharer/sharer.php?u=' + encodeURIComponent(document.URL) + '&t=' + encodeURIComponent(document.URL)); return false;"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 460.1990662 458.5138245"><path d="M460.1990662,230.0995331C460.1990662,103.0190277,357.1800232,0,230.0995331,0S0,103.0190277,0,230.0995331C0,338.011261,74.2903519,428.5578003,174.5126801,453.4230957V300.4146423H127.064209v-70.3151093h47.4484711v-30.2983704c0-78.3184433,35.4433594-114.619339,112.3326569-114.619339c14.5774536,0,39.730896,2.8583221,50.020874,5.7166443v63.7409363c-5.4308167-0.5716553-14.8633118-0.8574677-26.58255-0.8574677c-37.7300415,0-52.3075867,14.2916718-52.3075867,51.4501038v24.8674927h75.158783l-12.9122009,70.3151093h-62.246582v158.0991821C371.9082336,444.7527161,460.1990662,347.742981,460.1990662,230.0995331z"/></svg> </a></li><li class="et-social-icon"> <a href="#" title="Share on Linkedin" class="icon" onclick="window.open( 'https://www.linkedin.com/shareArticle?mini=true&url=' + encodeURIComponent(document.location.href)); return false;"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><path d="M100.28 448H7.4V148.9h92.88zM53.79 108.1C24.09 108.1 0 83.5 0 53.8a53.79 53.79 0 0 1 107.58 0c0 29.7-24.1 54.3-53.79 54.3zM447.9 448h-92.68V302.4c0-34.7-.7-79.2-48.29-79.2-48.29 0-55.69 37.7-55.69 76.7V448h-92.78V148.9h89.08v40.8h1.3c12.4-23.5 42.69-48.3 87.88-48.3 94 0 111.28 61.9 111.28 142.3V448z"/></svg> </a></li><li class="et-social-icon"> <a href="#" title="Tweet" class="icon" onclick="window.open('https://twitter.com/intent/tweet?text=' + encodeURIComponent(document.title) + ':%20' + encodeURIComponent(document.URL)); return false;"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><path d="M389.2 48h70.6L305.6 224.2 487 464H345L233.7 318.6 106.5 464H35.8L200.7 275.5 26.8 48H172.4L272.9 180.9 389.2 48zM364.4 421.8h39.1L151.1 88h-42L364.4 421.8z"/></svg> </a></li></ul><div class="topnav-buttons"> <a href="#" data-toggle="modal" data-target="#lc-subscription" class="btn-desktop-only pulse-button v5 topnav-btn"> <span class="button-inner"> XP² Newsletter </span> </a> <a href="/newsletter/" class="pulse-button v5 topnav-btn btn-mobile-only"> <span class="button-inner"> XP² Newsletter </span> </a></div></div></div><div class="lc-progress-wrap"><div class="lc-scroll-progress"></div></div></header><article id="post-131426" class="et_pb_post post-131426 lesson type-lesson status-publish has-post-thumbnail hentry article-category-testing-optimization article-industry-agnostic article-tags-personalization-and-testing-frameworks article-function-bi-analytics article-function-conversion-optimization article-function-marketing article-type-tactical-guides course-testing-and-optimization"><div class="lesson-content-sidebar"><div class="lesson-content-sidebar-inner" style="background-color: #f4c0c2"><div class="lesson-content-sidebar-header"><h4>A/B Testing & Optimization Course</h4></div><style>@media only screen and (max-width: 1199px) {
.lesson-content-sidebar-header {
background-color: #f4c0c2;
}
.lesson-content-sidebar-header:after {
background-color: rgb(242,178,182);
}
}</style><div class="lesson-content-sidebar-list"><ol><li class=""> <a href="https://www.dynamicyield.com/lesson/introduction-to-ab-testing/">An introduction to A/B testing and optimization</a> <span class="mark-as-read" data-id="97"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/aa-testing-in-experimentation/">A/A testing and decision making in experimentation</a> <span class="mark-as-read" data-id="131419"></span> <span class="has-video"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/statistical-significance/">Why reaching and protecting statistical significance is so important in A/B tests</a> <span class="mark-as-read" data-id="1783"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/traffic-allocation/">Choosing the right traffic allocation in A/B testing</a> <span class="mark-as-read" data-id="115918"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/conversion-attribution-scoping/">Understanding conversion attribution scoping in A/B testing</a> <span class="mark-as-read" data-id="21258"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/optimization-objective/">Choosing the right conversion optimization objective</a> <span class="mark-as-read" data-id="7468"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/bayesian-testing/">Frequentist vs. Bayesian approach in A/B testing</a> <span class="mark-as-read" data-id="17150"></span> <span class="article-read-mark">Already read</span></li><li class="current"> <a href="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/">Guidelines for running effective Bayesian A/B tests</a> <span class="mark-as-read" data-id="131426"></span> <span class="has-video"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/contextual-bandit-optimization/">Beyond A/B testing: Multi-armed bandit experiments</a> <span class="mark-as-read" data-id="7414"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/client-side-vs-server-side/">Client-side vs server-side A/B testing and personalization</a> <span class="mark-as-read" data-id="130425"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/ab-testing-without-segmentation/">Segmented A/B tests: Avoiding average experiences</a> <span class="mark-as-read" data-id="32157"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/impact-of-ab-testing-on-seo/">The impact of A/B tests and personalization on SEO</a> <span class="mark-as-read" data-id="91"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/no-failed-ab-tests/">There are no failed A/B tests: How to ensure every experiment yields meaningful results</a> <span class="mark-as-read" data-id="135394"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/ab-testing-analysis/">How to analyze and interpret A/B testing results</a> <span class="mark-as-read" data-id="32159"></span> <span class="article-read-mark">Already read</span></li><li class=""> <a href="https://www.dynamicyield.com/lesson/kopari-beauty/">Building an experimentation growth plan: A Kopari Beauty case study</a> <span class="mark-as-read" data-id="22970"></span> <span class="article-read-mark">Already read</span></li></ol><div class="view-all-courses-wrap"> <a href="https://www.dynamicyield.com/learning-paths/" class="view-all-courses">Discover more courses<svg xmlns="http://www.w3.org/2000/svg" width="10.928" height="9.501" viewBox="0 0 10.928 9.501" style=" "><path d="M10.929,4.753a.859.859,0,0,0-.19-.491L6.938.224A.835.835,0,0,0,5.892.16.736.736,0,0,0,5.9,1.2L8.577,4.041H.746a.733.733,0,0,0-.734.586.711.711,0,0,0,.7.84H8.577L5.9,8.3a.784.784,0,0,0-.008,1.038.8.8,0,0,0,1.037-.063l3.8-4.039a.652.652,0,0,0,.2-.483Z" transform="translate(-0.001 -0.003)"/></svg></a></div></div></div></div><div class="article-content"><div class="lesson-content-inner"><div class="article-breadcrumbs"> <a href="https://www.dynamicyield.com/learning-paths/" title="Learning Paths">Learning Paths</a> <a href="https://www.dynamicyield.com/course/testing-and-optimization/">A/B Testing & Optimization</a></div><h1 class="article-title">Guidelines for running effective Bayesian A/B tests</h1><h2 class="article-excerpt">In this post, we describe the basic ideas behind Bayesian statistics and how they feed into business decisions you will need to make at the end of a test.</h2><div class="article-authors lc-flex"><div class="article-author-box lc-col lc-flex"><div class="article-author-image lc-col"><div class="entry_author_image"><img src="https://www.dynamicyield.com/wp-content/authors/RonKenett-105.png" alt="Ron Kenett" width="76" height="76" /></div></div><div class="article-author-title lc-col"><div class="article-author-name"> <a href="https://www.dynamicyield.com/blog/author/ronkenett/" title="Posts by Ron Kenett" rel="author">Ron Kenett</a></div><div class="article-author-title-text"> Professor Ron Kenett of KPA Group</div></div></div><div class="article-author-box lc-col lc-flex"><div class="article-author-image lc-col"><div class="entry_author_image"><img src="https://www.dynamicyield.com/wp-content/authors/DavidSteinberg-106.jpeg" alt="David Steinberg" width="76" height="76" /></div></div><div class="article-author-title lc-col"><div class="article-author-name"> <a href="https://www.dynamicyield.com/blog/author/davidsteinberg/" title="Posts by David Steinberg" rel="author">David Steinberg</a></div><div class="article-author-title-text"> Professor David Steinberg of KPA Group</div></div></div></div><div class="entry-content article-entry-content"><p style="font-size: 12px;"><em>Both professor Ron Kenett and David Steinberg of KPA Group sat down to talk to more candidly about some of the topics discussed in this article.</em></p><div class="wistia_responsive_padding" style="margin-top: 30px; padding: 56.25% 0 28px 0; position: relative;"><div class="wistia_responsive_wrapper" style="height: 100%; left: 0; position: absolute; top: 0; width: 100%;"><iframe class="wistia_embed" src="//fast.wistia.net/embed/iframe/9q3a9gps1j?videoFoam=true" name="wistia_embed" width="100%" height="100%" frameborder="0" scrolling="no" allowfullscreen="allowfullscreen"></iframe></div></div><p><amp-wistia-player data-media-hashed-id="kiomiesn1r" width="320" height="180" ></amp-wistia-player><script src="//fast.wistia.net/assets/external/E-v1.js" async=""></script></p><div class="video-transcript" style="margin-top: -55px;"><div class='et-learn-more clearfix'><h3 class='heading-more'>Read the full transcript<span class='et_learnmore_arrow'><span></span></span></h3><div class='learn-more-content'>The internet today is one of the primary sources for bringing in customers, generating revenue. And often we have discussions about what might be the best ways to present ourselves, present our companies. The AB test lets you gain not just revenue and action from your website, it lets you gain information, and lets you gain information by making comparisons among those different alternatives that you might be considering. You can present different options to your customers, to your visitors to your website. That’s the A and the B, see how they respond, and when you see their responses, that lets you make a comparison, not based on what you thought might be the best, but on what you see works with your customers.</p><p>So the straightforward one is comparing two options, A and B. The more complex ones can compare combinations. So you could have A, which is a combination of two or three things, and B, which is a different combination, and C, which is still another combination. And if you apply methodology which is based on a statistical approach called design of experiments, in one test, you can learn about several factors, the effect of several factors.</p><p>So when you’re thinking about setting up an AB test, invariably, you have some alternatives that you wanna compare, and the tests usually follow methodology that’s been developed over many years in the statistical literature. What you’d like is to make sure that the people who see option A are, in some sense, as similar as possible to those who see option B. And it’s become well-established in the scientific literature, going back to work of the statistician Ronald Fisher, about 100 years ago, that the best way to do that is to make sure that you randomly allocate who gets which of the options that you want to compare.</p><p>So if you’re gonna start an experiment, you have to think about what are the options that are on the table. What do you want to compare? You have to have an engine at your disposal, framework that’s gonna let you make those random allocations. So if someone comes into your website, they’re not just gonna get a landing page, you have to decide what landing page they’re gonna get. You control that, and you want to control that by allocating it at random between the groups. That’s what’s gonna guarantee having a fair comparison.</p><p>Now, in order to validate the data that you’re capturing, you must ensure, for example, that this random allocation occurs. Because if you give the young people version A and the older people version B, and you see a difference, you will never know if it’s due to the age or to the web page design. That’s called confounding. So the random allocation, in a sense, establishes some causality on what is really impacting user behavior.</p><p>There are a number of ways that you can go about analyzing the data that you get from an AB test. Classical statistics takes a view that there is some true value for that KPI, and for example, it might be common if you’re comparing two options, to start with a framework that says both options are identical unless we get evidence that shows that they’re different. The Bayesian analyses take a somewhat different viewpoint, and rather than saying there’s some value, we just don’t know what it is, describe that value by putting a probability distribution on it. So you get more and more data. The probability distribution gets tighter and tighter about focusing on what we’ve now learned so that we have better information. It often gives a much more intuitive way, in particular in terms of business decisions, for looking at how to characterize what we know, rather than saying, “Well, it could be between this and this, and either it is, or it isn’t.” That’s sort of the classical way. And we give a more gradated answer by saying, “Well, this is the most likely.” And then there’s a distribution that describes how unlikely values become as we move away from what we think is the most likely value.</p><p>Bayesian statistics give you a, I think, a much more intuitive and natural way to express those understandings that you get at the end of an experiment in terms of say, what’s the probability that A is better, what’s the probability that B is better. So you bring in the data, use the data, and now Bayesian inference provides standard rules, and very strictly mathematical rules for taking the data, using that to update your prior to get what’s called a posterior distribution.</p><p>The posterior is, this is your current view. This is what you think after having seen the data. What describes your uncertainty about the KPIs after having seen the data, and you combine those two together in order to get your final statement about what it is you think. Can you really state a prior distribution? Is there a defensible way to state what you think in advance? As again, this has certainly been the source of friction between people in the different statistical camps. In the case of most AB tests, you’re getting lots and lots of data, and the data is going to wash out anything that you might’ve thought in the prior. So that for typical AB tests, again, because you have very large data volumes, it no longer becomes, in my mind, very controversial because whatever you put in to get the Bayesian engine going is essentially gonna be washed out by the data. And the data is really going to be the component that dominates what you get in the end.</p><p>So sample size is an important question in any study that’s going to be run. Whenever you go out to get the data. How long do I have to run my test? How many visitors to the website do I have to see before I can make conclusions? In order to get a test that gives a complete and honest picture, representative picture, of all the people who are long term gonna be exposed to whatever decisions you make, you wanna make sure that you include everyone within a cycle like that. You don’t wanna stop the test after having seen only visitors on Tuesday and Wednesday. And it may turn out that the decision you made is really catastrophic for those who come in on the weekends. So you always want to make sure that you pick up any potential time-cycles like this. And in terms of the sort of Bayesian descriptions that we were talking about later, I think this really gets to the heart of, in the end, you’re gonna get these posterior distributions that describe your KPIs, and how narrow do you want them to be? The tighter the interval, the more you know.</p><p>You can have a tight interval, but if the people who looked at the website and were part of the randomization subgroup, then whatever you’re going to get will not apply to the general group. So you really should work on both fronts. One is, how much information do you have? The tighter the interval, the more informative it is. But also how generalizable is the information that you provide?</p><p>Both of these aspects are important, and both of them can be the ones that really are the driver in dictating how large an experiment has to be. A website that has very heavy volume may quickly get to the needed sample size but without being able to see the full representative picture of all the visitors. And of course, it can happen just the other way, that by the time you get enough people in, enough visitors into the site, in order to get to the sample size you need, you’ve already gone through several of those time cycles. And so you have to see how both of these work out and balance against one another.</p></div></div></p></div><p></p><div id="WYNTK-Sec" class="what-you-need-to-know-sec"><p><img decoding="async" src="/wp-content/uploads/2020/09/star-icon.svg" style="width: 20px; margin-right: 5px; vertical-align: -2px;"><strong class="first">Summarize this article</strong><strong class="hidden">Here’s what you need to know:</strong></p><ul style="margin-left: 4px; font-size: 16px;"><li style="margin: 10px 0;">A/B testing involves showing different versions of a webpage to different visitors and measuring which version performs better.</li><li style="margin: 10px 0;">Randomization is key to ensuring that the results of your A/B test are accurate and unbiased.</li><li style="margin: 10px 0;">It’s important to define your goals and metrics before running an A/B test so that you can measure success.</li><li style="margin: 10px 0;">Make sure your sample size is large enough to draw statistically significant conclusions from your A/B test results.</li><li style="margin: 10px 0;">Pay attention to the statistical significance of your results to avoid making decisions based on random fluctuations.</li><li style="margin: 10px 0;">Iterate on your A/B tests based on the results you learn to continuously improve your website.</li></ul></div><p></p><p><a href="https://www.dynamicyield.com/lesson/introduction-to-ab-testing/">A/B testing</a> is a great way to compare alternative experiences or possible modifications to your website or digital properties. By running the two experiences in parallel and deciding at random which visitors land on which version, you get an honest comparison of which one delivers better performance.</p><p>Once you get the data from your test, you need reliable and informative ways to summarize them and reach conclusions. The <a href="https://www.dynamicyield.com/blog/bayesian-approach-to-ab-testing/">Bayesian approach</a> to doing statistics is a great way to accomplish this. </p><p>Here we describe the basic ideas behind Bayesian statistics and how they feed into the business decisions that you will need to make at the end of a test. We first present the approach to data analysis. </p><p>Then, we illustrate how data analysis leads to natural solutions for questions such as:</p><ul><li>Which variant drives better results? </li><li>How much better are those results?</li><li>Can we be confident in our conclusion?</li><li>What if we want to test more than two variants?</li><li>What sample size is needed for a Bayesian A/B test?</li><li>Can a test be terminated early if results look clear?</li></ul><h2>What is Bayesian Statistics?</h2><p>Bayesian statistics is named after the Reverend Thomas Bayes who lived in Britain in the 18th century and was responsible for proving <em>Bayes Theorem</em>. The guiding principle in Bayesian statistics is that you can use the language of probability to describe anything that we don’t know and want to learn from data. As we will see, this is an ideal language for discussing many of the questions that are most important to business decisions.</p><p>When you run an A/B test, the primary goal is to discover which variant is most effective. To do so, you compare the results with the two methods. The Bayesian analysis begins by looking at the KPIs for each of the variants. The data gives us information about their value but always leaves some uncertainty. We describe uncertainty using a probability distribution. We can simulate results from the distribution to understand how it looks. </p><p>The histogram in Figure 1. is typical of what you might get from your analysis. It shows the statistical distribution of <em>difference</em> in <a href="https://www.dynamicyield.com/glossary/ctr/">click-through rate (CTR)</a> between A and B. Most of the curve is to the right of 0, providing evidence that A has a higher CTR. The fraction of the curve to the right of 0 is a great summary of that evidence – it is called the <a href="https://www.dynamicyield.com/glossary/probability-to-be-best/">Probability to Be Best</a> (P2BB). In Figure 1., P2BB is 0.699, in favor of version A.</p><p><img fetchpriority="high" decoding="async" src="https://www.dynamicyield.com/wp-content/uploads/2020/07/Guidelines-for-running-effective-Bayesian-A-B-Tests-Inline-1.jpg" alt="" width="1330" height="550" class="aligncenter size-full wp-image-131831" /></p><p>Figure 1: Probability function for the CTR for A minus the CTR for B</p><p>If you have three or more variants, it is still easy to compute the probability of being best (P2BB), with probability split into three pieces, one for each variant. </p><h2>How does Bayesian analysis work? </h2><p>You need to start the Bayesian engine running with a <em>prior probability distribution</em> that reflects what you think about the KPI before seeing any data. The prior is then combined with the test data to obtain a <em>posterior distribution </em>for each variant. Figure 1. would be the posterior distribution for the difference between the A and B variants – this reflects the current view after having seen the data.</p><p>You need the prior to make things work, but when the volume of data is large – and that is almost always the case in A/B tests – the posterior is quickly dominated by the data and virtually “forgets” the prior; so you don’t need to invest too much thought about how to choose the prior. It is possible to work with some standard choices for priors without affecting the analysis. </p><p>But can you really state a prior distribution and what you think in advance?</p><p>This has certainly been the source of friction between people in the different statistical camps, so it is helpful to contrast the Bayesian summary with the “classical,” <a href="https://www.dynamicyield.com/lesson/frequentists-approach-to-ab-testing/">Frequentist approach</a> you might have studied in an introductory statistics course. In the classical approach, either A is better than B, B is better than A, or they are identical. The question “what is the probability that A is better” is not part of the lexicon – you can’t give an answer, and in fact, you’re not even permitted to ask the question! </p><p>The classical paradigm does use probability, but only to describe the experimental data, not to summarize what you know about a KPI. That’s why you can’t make a summary like “the probability that B gives a positive uplift over A is 98%.</p><p>In most A/B tests, you’re getting lots and lots of data. Therefore, it no longer becomes very controversial, because whatever you put in to get the Bayesian engine going is essentially gonna be washed out by that information, which is really going to be the component that dominates what you get in the end. And that’s why we think the Bayesian summaries are so much better suited for business decisions.</p><p><em>For more on the different viewpoints when it comes to the Bayesian vs. Frequentist approach to A/B testing, </em><a href="https://www.dynamicyield.com/lesson/bayesian-testing/"><em>check out this article</em></a><em>.</em></p><h2>How should traffic be distributed between variations?</h2><p>What you’d like is to make sure that the people who see option A are, in some sense, as similar as possible to those who see option B. And it’s become well-established in the scientific literature, going back to work of the statistician Ronald Fisher, about 100 years ago, that the best way to do that is to make sure that you <a href="https://www.dynamicyield.com/lesson/traffic-allocation/">choose the right traffic allocation</a>, or who gets which of the options that you want to compare. </p><p>So, you have to think about what are the options that are on the table. What do you want to compare? You have to have an engine at your disposal – a framework that’s gonna let you make those random allocations. If someone comes to your website, they’re not just gonna get a landing page, you have to decide what landing page they’re gonna get. You control that, and you want to control that by allocating it at random between the groups. That’s what’s gonna guarantee having a fair comparison.</p><p>In order to validate the data that you’re capturing, you must ensure, for example, that this random allocation occurs. Because if you give the young people version A and the older people version B, and you see a difference, you will never know if it’s due to the age or to the web page design. That’s called confounding. So the random allocation, in a sense, establishes some causality on what is really impacting user behavior.</p><h2>What length of time is needed to run an a/b test?</h2><p>In making these decisions, it is important to think about “what might go wrong” with the testing engine. You want your test to be sensitive to possible bugs. Often bugs are related to failures of the random allocation over time so that natural time trends in your KPIs bias the results. If there are weekly trends in your data, then you need to run an A/B test for two weeks to make sure that the engine is successfully handling them. If there are diurnal trends (but not weekly), you can afford to run shorter experiments. There may also be special events that suddenly increase or decrease the number of visitors to your site, or change the composition of visitors, affecting the performance of an experience. You may want to hold off on making any decisions until you see whether such issues have an impact on your results. </p><p>The considerations noted above will suggest a minimum time frame for running your experiment that ensures representative coverage in the A/B test of typical future site visitors. Often, these will dictate the length of time for the test.</p><h2>How large a sample size is needed for a Bayesian test? </h2><p>The sample size paradigm for Bayesian testing asks how narrow you want your final probability functions to be. You will probably want to focus on the probability function for the difference in KPIs between the two variants, as shown in Figure 1. And you will already have an idea what sort of difference could be important for your business; that gives a guideline for how narrow you want the function to be – the tighter the interval, the more you know.</p><p>It’s important to note that you can have a tight interval, but if the people who looked at the website and were part of the randomization subgroup, then whatever you’re going to get will not apply to the general group. So you really should work on both fronts. One is, how much information do you have? The tighter the interval, the more informative it is. But also how generalizable is the information that you provide?</p><p>Both of these aspects are important and can be the driver in dictating how large an experiment has to be. A website that has very heavy volume may quickly get to the needed sample size but without being able to see the full representative picture of all the visitors. And this can happen the other way around too, so that by the time you get the sample size you need, you’ve already gone through several time cycles – you have to see how both of these work out and balance against one another.</p><p>You will need to provide information on likely values for the KPIs. Often one of the versions is already running, so you can use historical data for this purpose. For new versions, you will need to think about what effect they might have. If the test includes multiple versions, the same ideas apply and the same inputs are needed.</p><p><strong>The basic inputs needed to compute a sample size depend on the nature of the KPI</strong>:</p><ul><li>For binary KPIs like CTR or installation rate, all you need are the expected rates for each version</li><li>If you are looking at purely continuous KPIs like revenue per conversion, you will need to supply an estimate of both the typical conversion size and the variation in size among those who do convert </li><li>For a “mixed” KPI like revenue per visitor, you will need the average, variation, and an estimate of the fraction who will convert.</li></ul><p>There are <a href="https://www.dynamicyield.com/blog/frequentists-approach-to-ab-testing/">straightforward formulas</a> you can use to determine the sample size, and also helpful <a href="https://marketing.dynamicyield.com/ab-test-duration-calculator/">A/B test sample size calculators</a>. </p><h2>What if there are more than two versions in the test? </h2><p>The same sample size formulas are relevant, but here we distinguish between two goals: (1) finding the best and (2) showing that the standard version is <em>not</em> the best. </p><p>What is different about the second goal? </p><p>Suppose you have two new versions that you want to compare to the standard. Your best guess gives almost the same expected KPI to each of the new versions, with both of them improving over the standard. With very similar KPI’s, you will need a large sample to know which of the two new versions is better than the other. But if you expect both to be better than the standard, you can “rule out” the standard version with a much smaller test.</p><p><strong>To further illustrate the idea</strong>: </p><p>The current experience is getting a CTR of 1%. A new version is proposed with the expectation that it will increase the CTR to 1.2%. The expected difference is 0.2%, so you need an A/B test that will tell you what is the true difference to a higher resolution. A good rule of thumb is to aim for a resolution that concentrates 95% of the Posterior Distribution in an interval that is the size of the expected difference. </p><p>In this example, that means narrowing down the difference so that you are 95% certain about an interval of length 0.002. To achieve this level of accuracy, you will need a sample of 83,575 visitors to both pages. What if even a 10% improvement in CTR, to 1.1%, is critical? Then you would need to know the difference up to, at most, 0.1%. That will require a sample size of 319,290 for each version.</p><p><strong>Now let’s look at an example involving revenue per visitor</strong>:</p><p>A common scenario (often referred to as <a href="https://www.dynamicyield.com/blog/targeting-high-value-customers/">the Pareto principle</a>)is that most visitors don’t provide any revenue and, among those who do, a small number of “big fish” pull up the average, and also the standard deviation. </p><p>The current landing page has a conversion rate of 1%; on average, converters spend $20; the standard deviation in the purchases of converters is $25. That means the average revenue per visitor is $0.20. Marketing thinks that the new version will bring in more purchasers, upping the conversion rate to 1.2%, but will decrease the proportion of “big fish,” so that the average conversion drops to $19 and the standard deviation to $22. </p><p>That translates to an average revenue of $0.228, an expected improvement of $0.028 per visitor. The A/B test will need to be large enough to narrow down the 95% credible interval for the difference in average revenue to no more than the expected difference, i.e. to $0.028. We will need 397,830 visitors to each experience to achieve that level of accuracy.</p><h2>What if we want a stronger guarantee that we will find the important difference? </h2><p>We advised above setting the sample size to control the width of the 95% credible interval. It is easy to reduce the chance of missing a valuable change by insisting on a higher level of probability, say 98% or 99%, but at the expense of increasing the sample size.</p><h2>Can you stop a test early? </h2><p>The probability that A is better (or worse) than B provides a natural metric to track as the test progresses. And a strong signal from the P2BB statistic can be used to guide early stopping. However, early stopping needs to be done with caution. </p><p>First, the Bayesian summary is <em>not</em> completely immune to data peeking. The P2BB will drift up and down during an experiment. This is especially true if there is no difference in performance between the variants, as in <a href="https://www.dynamicyield.com/glossary/aa-testing/">A/A testing</a>. In that case, you will see that the P2BB increases and decreases as you collect more data. The fact that it crossed a threshold (like 95%) at some point during the test is not a guarantee that it would remain above the threshold if you wait until the planned termination time. </p><p>Second, it is always risky to stop before observing at least one full time cycle. Version B might be better for visitors at the start of the cycle, but worse for those later in the cycle. If you stop before the end of a full cycle, your results may be biased.</p><p>If you want to stop early, think about using stricter criteria for an early stopping point. For example, if you see that the probability becomes extreme (say above 0.999 or below 0.001), you can feel safe in stopping and making a decision. If there really is a difference between A and B, you can still expect to reach those thresholds relatively quickly. </p><p>Finally, we also reiterate the importance of waiting for time trends to reveal themselves in your test. You don’t want to decide on the basis of a short time window only to find that the results are very different later in the day or week – suppose weekday visitors respond better to A than B, but that trend flips on the weekend. Making a decision Wednesday from an experiment that began on Tuesday might lead to a bad decision.</p><div class="article-separator"><svg width="54" height="26" viewBox="0 0 54 26" fill="none" xmlns="http://www.w3.org/2000/svg"> <path d="M53.0088 25.6367C52.7884 21.9564 51.8025 18.3625 50.1141 15.0849C48.4256 11.8073 46.0718 8.9181 43.2031 6.60203L43.2031 25.6367L53.0088 25.6367Z" fill="black"/> <path d="M14.4395 3.59038C10.3237 5.69174 6.82743 8.83082 4.29638 12.6973C1.76533 16.5638 0.28711 21.0239 0.0078125 25.6367L14.4395 25.6367L14.4395 3.59038Z" fill="black"/> <path d="M17.6563 2.20809L17.6562 25.6367L39.9829 25.6367L39.9829 4.36701C39.9829 4.36701 30.7303 -2.31342 17.6563 2.20809Z" fill="black"/> </svg></div><div class="article-tags-block"> <strong>Categories:</strong> <a href="https://www.dynamicyield.com/course/testing-and-optimization/">A/B Testing & Optimization</a></div><div class="article-tags-block"> <strong>Tags:</strong> <a href="https://www.dynamicyield.com/tags/personalization-and-testing-frameworks/">Personalization and Testing Frameworks</a></div><div class="article-pagination"><h5>Continue reading</h5><div class="article-pagination-row lc-flex"> <a rel="next" style="background-color: #f4c0c2" class="article-next-link pagination-link" href="https://www.dynamicyield.com/lesson/contextual-bandit-optimization/" title="Beyond A/B testing: Multi-armed bandit experiments">Beyond A/B testing: Multi-armed bandit experiments</a><a rel="prev" style="background-color: #f4c0c2" class="article-previous-link pagination-link" href="https://www.dynamicyield.com/lesson/bayesian-testing/" title="Frequentist vs. Bayesian approach in A/B testing">Frequentist vs. Bayesian approach in A/B testing</a></div></div></div></div></div></article></div><section class="lc-subscription-section"><div class="lc-subscription-form-block lc-subscription-inner"><h4 class="lc-subscription-title">Sign up for the XP² newsletter</h4><div class="lc-subscription-text">Join thousands of readers from Target, Citi, Spotify, Hulu, Google, Sephora, and other innovative brands who read our bi-weekly XP² newsletter, delivering educational content, research, and insights straight to your inbox</div><form class="mktoForm" data-formId="3876" data-formInstance="sub-inline"></form><div class="lc-privacy-wrapper"><div class="lc-privacy"> You may unsubscribe at any time. Visit our <a href="/privacy-policy/">privacy policy</a> to learn more about how we process your data and your rights as a data subject.</div></div></div><div class="lc-subscription-ty-block lc-subscription-inner"><h4 class="lc-subscription-title text-center">Thanks for<br> signing up!</h4></div></section></div> <script type='text/javascript'>//DY.API('siteVar', {id: 128, value: 'Ron Kenett'});
//DY.API('siteVar', {id: 129, value: ''});</script> <footer id="main-footer" class=""><div class="footer-top"><div class="footer-container"> <a href="https://www.dynamicyield.com/" class="dy-footer-logo"> <img src="/wp-content/uploads/2025/09/ma_symbol.svg" alt="Dynamic Yield footer logo" width="100" height="66"> </a><h4>Make hyper-personalization a reality</h4><p>Trusted by 450+ forward-thinking enterprise businesses:</p><div class="footer-clients retina_parent"> <img src="/wp-content/uploads/2023/08/customer-logos-footer.png" alt="DY Clients" width="498" height="123"></div><div class="footer-btns-wrap"> <a href="https://www.dynamicyield.com/request-demo/" class="btn-teal">Contact Sales</a> <a href="https://www.dynamicyield.com/product-demo/" class="btn-transparent">Watch a product demo<svg xmlns="http://www.w3.org/2000/svg" width="15.625" height="15.625" viewBox="0 0 15.625 15.625"> <path id="Icon_ionic-ios-play-circle" data-name="Icon ionic-ios-play-circle" d="M10.688,3.375A7.313,7.313,0,1,0,18,10.688,7.311,7.311,0,0,0,10.688,3.375Zm2.946,7.45L8.81,13.743a.156.156,0,0,1-.236-.137V7.77a.156.156,0,0,1,.236-.137l4.823,2.918A.161.161,0,0,1,13.634,10.825Z" transform="translate(-2.875 -2.875)" fill="none" stroke="rgba(0,0,0,0.8)" stroke-width="1"/> </svg> </a></div><ul class="footer-certificates"><li class="retina_parent"> <a href="https://www.dynamicyield.com/compliance/"> <img src="/wp-content/uploads/2022/05/gdpr-icon.png" alt="gdpr icon" width="139" height="70"> </a></li><li class="retina_parent"> <a href="https://www.dynamicyield.com/compliance/"> <img src="/wp-content/uploads/2023/04/SOCII-Badge.png" alt="socii icon" width="70" height="70"> </a></li><li> <a href="https://www.dynamicyield.com/compliance/"> <img src="/wp-content/uploads/2022/05/ccpa-icon.svg" alt="ccpa icon" width="117" height="40"> </a></li><li class="retina_parent"> <a href="https://www.dynamicyield.com/compliance/"> <img src="/wp-content/uploads/2022/05/iso-27001.png" alt="iso icon" width="106" height="68"> </a></li></ul></div></div><div id="et-footer-nav"><div class="footer-container"><ul id="menu-footer-menu" class="bottom-nav"><li id="menu-item-19183" class="footer_menu_title menu-item menu-item-type-custom menu-item-object-custom menu-item-19183"><a>Recommended Guides:</a></li><li id="menu-item-211968" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-211968"><a href="https://www.dynamicyield.com/article/personalization-guide/">Personalization</a></li><li id="menu-item-138198" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-138198"><a href="https://www.dynamicyield.com/lesson/introduction-to-ab-testing/">A/B Testing</a></li><li id="menu-item-147897" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-147897"><a href="https://www.dynamicyield.com/lesson/cro-plan/">Conversion Rate Optimization</a></li><li id="menu-item-19184" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-19184"><a href="/lesson/shopping-cart-abandonment-strategy/">Shopping Cart Abandonment</a></li><li id="menu-item-19185" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-19185"><a href="/lesson/product-recommendations-guide/">Product Recommendations</a></li><li id="menu-item-22805" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-22805"><a href="/lesson/promise-and-pitfalls-omnichannel-retailing/">Omnichannel Retailing</a></li><li id="menu-item-22885" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-22885"><a href="/lesson/the-economics-of-ecommerce-conversion-optimization/">eCommerce Conversion Rate Optimization</a></li><li id="menu-item-28240" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-28240"><a href="/article/50-most-important-dynamicyield-personalization-stats/">Personalization Statistics</a></li><li id="menu-item-33877" class="menu-item menu-item-type-post_type menu-item-object-post menu-item-33877"><a href="https://www.dynamicyield.com/article/tj-maxx-spired-checkout-optimization/">Checkout Optimization</a></li><li id="menu-item-123711" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-123711"><a href="/article/ecommerce-personalization-must-haves/">eCommerce Personalization</a></li><li id="menu-item-144090" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-144090"><a href="https://www.dynamicyield.com/lesson/strategizing-omnichannel-personalization/">Omnichannel Personalization</a></li><li id="menu-item-148146" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-148146"><a href="https://marketing.dynamicyield.com/benchmarks/cart-abandonment-rate/">Shopping Cart Abandonment Rate</a></li></ul><div class="footer-mach-logo retina_parent"> <img src="/wp-content/uploads/2022/05/mach-cert-22-wh.png" alt="Mach logo" width="63" height="70"></div><div class="language-switcher footer-language-switcher"><ul><li>en<ul><li><a class="active" href="https://www.dynamicyield.com/lesson/running-effective-bayesian-ab-tests/">English</a></li><li><a href="https://www.dynamicyield.com/es/">Español</a></li><li><a href="https://www.dynamicyield.com/de/">Deutsch</a></li><li><a href="https://www.dynamicyield.com/fr/">Français</a></li><li><a href="https://www.dynamicyield.com/ja/">日本語</a></li></ul></li></li></ul></div></div></div><div id="footer-bottom"><div class="footer-container"><div class="footer_info_wrapper"><div id="footer-info"> <button id="ot-sdk-btn" class="ot-sdk-show-settings">Manage Cookies</button> <a href="/privacy-notice/" target="_blank" class="animlink">Privacy Notice</a> <a href="/tos/" target="_blank" class="animlink">Terms of use</a> <span>© 2026 Mastercard Dynamic Yield</span></div></div><div class="footer_social_icons"><style>.et-social-icons .et-social-icon img {
max-width: 16px;
transition: 0.2s all;
}
.et-social-icons .et-social-icon img:not(:hover) {
opacity: 0.5;
}</style><span itemscope itemtype="http://schema.org/Organization"><link itemprop="url" href="https://www.dynamicyield.com"><ul class="et-social-icons"><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.linkedin.com/company/dynamic-yield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-5.png" alt="LinkedIn Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.instagram.com/dynamicyield/" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-4.png" alt="Instagram Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.youtube.com/c/Dynamicyield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-3.png" alt="Youtube Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.tiktok.com/@dynamic.yield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-2.png" alt="TikTok Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://www.facebook.com/DynamicYield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame-1.png" alt="Facebook Logo" /> </a></li><li class="et-social-icon"> <a itemprop="sameAs" href="https://twitter.com/DynamicYield" class="icon" target="_blank"> <img src="https://www.dynamicyield.com/wp-content/uploads/2025/09/Frame.png" alt="X Logo" /> </a></li></ul> </span></div></div></div></footer></div></div><div id="mobile_menu_overlay"></div> <script type='text/javascript' src='//cdn.dynamicyield.com/api/8765281/api_dynamic.js'></script> <script type='text/javascript' src='//cdn.dynamicyield.com/api/8765281/api_static.js'></script> <script type="text/javascript">var sbiajaxurl = "https://www.dynamicyield.com/wp-admin/admin-ajax.php";</script> <script type="text/javascript">!function(e,n){var r={"selectors":{"block":"pre.EnlighterJSRAW","inline":"code.EnlighterJSRAW"},"options":{"indent":2,"ampersandCleanup":true,"linehover":true,"rawcodeDbclick":false,"textOverflow":"break","linenumbers":true,"theme":"beyond","language":"generic","retainCssClasses":false,"collapse":false,"toolbarOuter":"","toolbarTop":"{BTN_RAW}{BTN_COPY}{BTN_WINDOW}{BTN_WEBSITE}","toolbarBottom":""},"resources":["https:\/\/www.dynamicyield.com\/wp-content\/plugins\/enlighter\/cache\/enlighterjs.min.css?tvrakvcLArVio6C","https:\/\/www.dynamicyield.com\/wp-content\/plugins\/enlighter\/resources\/enlighterjs\/enlighterjs.min.js"]},o=document.getElementsByTagName("head")[0],t=n&&(n.error||n.log)||function(){};e.EnlighterJSINIT=function(){!function(e,n){var r=0,l=null;function c(o){l=o,++r==e.length&&(!0,n(l))}e.forEach(function(e){switch(e.match(/\.([a-z]+)(?:[#?].*)?$/)[1]){case"js":var n=document.createElement("script");n.onload=function(){c(null)},n.onerror=c,n.src=e,n.async=!0,o.appendChild(n);break;case"css":var r=document.createElement("link");r.onload=function(){c(null)},r.onerror=c,r.rel="stylesheet",r.type="text/css",r.href=e,r.media="all",o.appendChild(r);break;default:t("Error: invalid file extension",e)}})}(r.resources,function(e){e?t("Error: failed to dynamically load EnlighterJS resources!",e):"undefined"!=typeof EnlighterJS?EnlighterJS.init(r.selectors.block,r.selectors.inline,r.options):t("Error: EnlighterJS resources not loaded yet!")})},(document.querySelector(r.selectors.block)||document.querySelector(r.selectors.inline))&&e.EnlighterJSINIT()}(window,console);</script><script type="text/javascript" id="rfw-script-js-extra">var rfw = {"speed":""};</script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_a5ec58aa1f222bedaf60cdced3b8a004.js?ver=2026020337" id="rfw-script-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_c1b7fbe6b1a3b777fddfe187094deb97.js?ver=2026020337" id="rfw-script-fitvid-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_ea3fad8947382ebf8709507a11a805db.js" id="divi-fitvids-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/themes/Divi/js/waypoints.min.js" id="waypoints-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_845779ea879c70773af95b2f6fc394d2.js" id="magnific-popup-js"></script> <script type="text/javascript" id="divi-custom-script-js-extra">var et_custom = {"ajaxurl":"https:\/\/www.dynamicyield.com\/wp-admin\/admin-ajax.php","images_uri":"https:\/\/www.dynamicyield.com\/wp-content\/themes\/Divi\/images","et_load_nonce":"6133b77f5f","subscription_failed":"Please, check the fields below to make sure you entered the correct information.","fill":"Fill","field":"field","invalid":"Invalid email","captcha":"Captcha"};</script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_43b4a88b2828bc5c023dc913689e8da5.js?ver=20.4" id="divi-custom-script-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/themes/Divi/custom/bootstrap/js/bootstrap.min.js" id="bootstrap-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/themes/Divi/custom/slick/slick.min.js" id="slick-js"></script> <script type="text/javascript" data-ot-ignore src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_12efd5a7d441d8b7f817fc6fc685dca9.js?ver=21.85" id="customscript-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_33fa37a5effe9361e40b8525e36e3eaf.js" id="typed-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_eb8981568f49ac148e96af3241eb95c7.js?ver=21.85" id="form_validation-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_be33bce741fd746dee1a6cf31c0b5bde.js" id="retina_js-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_9a50928b26148d1c35a95957e5ef2b9d.js?ver=21.85" id="mkto-scripts-js"></script> <script type="module" src="https://www.dynamicyield.com/wp-content/themes/Divi/js/cid-validation.js?ver=21.85" id="cid-validation"></script><script type="text/javascript" src="https://www.dynamicyield.com/wp-content/plugins/wp-featherlight/js/wpFeatherlight.pkgd.min.js?ver=1.3.4" id="wp-featherlight-js"></script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_17babe2fc54117766e24bd50889c6577.js?ver=1772206652" id="article-script-js"></script> <script type="text/javascript" id="dc_conversion_api-js-extra">var ajax_obj = {"nonce":"3cd30fb921","ajax_url":"https:\/\/www.dynamicyield.com\/wp-admin\/admin-ajax.php","ajax_action":"conversion_api_call"};</script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_34a6a3100a2ecbc7706af2d4c798f856.js?ver=935c7f8ce03960b02800bc8ae3231df8" id="dc_conversion_api-js"></script> <script type="text/javascript" id="et-shortcodes-js-js-extra">var et_shortcodes_strings = {"previous":"Previous","next":"Next"};</script> <script type="text/javascript" src="https://www.dynamicyield.com/wp-content/cache/autoptimize/js/autoptimize_single_bd5013e08f82109a2379477415b9b3af.js?ver=3.0" id="et-shortcodes-js-js"></script> <script type="text/javascript" data-noptimize>var dy_search_params = new URLSearchParams(window.location.search);
if(dy_search_params.has('utm_source')) {
localStorage.setItem('utm_source', dy_search_params.get('utm_source'));
}
if(dy_search_params.has('utm_medium')) {
localStorage.setItem('utm_medium', dy_search_params.get('utm_medium'));
}
if(dy_search_params.has('utm_content')) {
localStorage.setItem('utm_content', dy_search_params.get('utm_content'));
}
if(dy_search_params.has('utm_campaign')) {
localStorage.setItem('utm_campaign', dy_search_params.get('utm_campaign'));
}
if(dy_search_params.has('utm_term')) {
localStorage.setItem('utm_term', dy_search_params.get('utm_term'));
}
if(dy_search_params.has('mkt_tok')) {
localStorage.setItem('mkt_tok', dy_search_params.get('mkt_tok'));
}
// Also, fill in the data inside the Marketo form once its available
if(typeof window.MktoForms2 === 'object') {
MktoForms2.whenReady(function(_form) {
var form = _form.getFormElem();
console.log('MKTO FORM READY');
// Append the mkt_tok param if it exists in the URL
var ls_mkto_tok = localStorage.getItem('mkt_tok');
if(ls_mkto_tok) {
_form.addHiddenFields({
mkt_tok: ls_mkto_tok
});
_form.vals({ mkt_tok: ls_mkto_tok });
}
// Fill in the UTM data from localStorage
var dy_utms = {
'utm_source': 'txt_source__c',
'utm_medium': 'txt_medium__c',
'utm_content': 'txt_content__c',
'utm_campaign': 'txt_campaign_name__c',
'utm_term': 'txt_term__c',
'gclid': ['GCLID__c', 'GCLID_c__c']
};
for(var i = 0; i < Object.keys(dy_utms).length; i++) {
var ls_key = Object.keys(dy_utms)[i],
ls_value = localStorage.getItem(ls_key),
form_key = dy_utms[ls_key],
form_field;
if(ls_value) {
if(ls_key === 'gclid') {
ls_value = JSON.parse(ls_value).value;
}
if(typeof(form_key) === 'string') {
form_field = form[0].querySelector('input[type="hidden"][name="' + form_key + '"]');
if(form_field) {
form_field.value = ls_value;
}
} else {
for(var j = 0; j < form_key.length; j++) {
form_field = form[0].querySelector('input[type="hidden"][name="' + form_key[j] + '"]');
if(form_field) {
form_field.value = ls_value;
}
}
}
}
}
// IP Address
fetch('https://www.cloudflare.com/cdn-cgi/trace').then(function(result) {
result.text().then(function(values) {
values.split("\n").map(function(item) {
var pair = item.split('=');
if(['ip'].includes(pair[0])) {
form[0].querySelector('input[type="hidden"][name="IP__c"]').value = pair[1];
}
});
});
});
// GA ID and Referrer
var dy_utms_cookie = {};
document.cookie.split(';').forEach(function(el) {
var splitCookie = el.split('=');
var key = splitCookie[0].trim();
var value = splitCookie[1];
dy_utms_cookie[key] = value;
});
if(dy_utms_cookie["_ga"]) {
form[0].querySelector('input[type="hidden"][name="Visitor_ID__c"]').value = dy_utms_cookie["_ga"].substring(6);
}
if(dy_utms_cookie["personalize_referrer_string"]) {
form[0].querySelector('input[type="hidden"][name="Referral__c"]').value = dy_utms_cookie["personalize_referrer_string"].substring(6);
}
// ZoomInfo
if(localStorage.getItem('ZoomInfoData')) {
var zoominfo_data = JSON.parse(localStorage.getItem('ZoomInfoData'));
if(zoominfo_data) {
form[0].querySelector('input[type="hidden"][name="Alexa_Rank__c"]').value = zoominfo_data.alexaglobalrank;
form[0].querySelector('input[type="hidden"][name="Employees__c"]').value = zoominfo_data.employeesrange;
form[0].querySelector('input[type="hidden"][name="Company_Revenue_Range__c"]').value = zoominfo_data.estimatedannualrevenue;
}
}
// Also, dispatch a dataLayer event on success
_form.onSuccess(function(values, follow_up_url) {
window.dataLayer = window.dataLayer || [];
if([3739, 4802, 5094, 5110, 4946, 5076, 5116, 5171, 5172, 5173].includes(_form.getId())) {
window.dataLayer.push({
'event': 'demo_request_complete',
'eventCallback': function() {
if(follow_up_url) {
window.location.href = follow_up_url;
}
}
});
return false;
} else if([3876].includes(_form.getId())) {
window.dataLayer.push({
'event': 'newsletter_signup_complete',
'eventCallback': function() {
if(follow_up_url) {
window.location.href = follow_up_url;
}
}
});
return false;
}
});
});
}</script> <script>if(typeof(jQuery) !== 'undefined') {
$ = jQuery.noConflict();
}
function getUrlParameter(sParam) {
var sPageURL = decodeURIComponent(window.location.search.substring(1)),
sURLVariables = sPageURL.split('&'),
sParameterName,
i;
for (i = 0; i < sURLVariables.length; i++) {
sParameterName = sURLVariables[i].split('=');
if (sParameterName[0] === sParam) {
return sParameterName[1] === undefined ? true : sParameterName[1];
}
}
}
$(window).load(function(){
var videoID = getUrlParameter('video');
if(videoID !== undefined){
if($('#wistia-' + videoID + '-1').length){
var videoName = $('#wistia-' + videoID + '-1').parents('.tab-pane');
videoName = videoName[0].id
}
if($('a[href="#' + videoName +'"]').length){
$('a[href="#' + videoName +'"]').each(function(){
$(this).click();
});
}
addScript('https://fast.wistia.com/embed/medias/'+ videoID +'.jsonp');
addScript('https://fast.wistia.com/assets/external/E-v1.js');
$('body').append('<span class="wistia_embed wistia_async_'+videoID+' popover=true popoverContent=link dy-wistia-popup" style="display:inline"></span>');
window._wq = window._wq || [];
_wq.push({ id: videoID, onReady: function(video) {
var video = Wistia.api(videoID);
video.popover.show(); // open the popover
video.play(); // play the video
setTimeout(function(){video.unmute();}, 500);
}});
}
});</script> <style>#adBanner {
background-color: transparent;
height: 0;
width: 1px;
}</style><div id="wrapfabtest"><div id="adBanner"></div></div></body></html>
<!-- Dynamic page generated in 0.448 seconds. -->
<!-- Cached page generated by WP-Super-Cache on 2026-02-27 10:37:32 -->
<!-- super cache -->