Statistical Significance in Email AB Testing

AB testing, one of the simplest forms of a randomized controlled experiment, is basically a way to compare two versions of something to figure out which performs better. These days it’s most often associated with websites and apps. Digital marketing teams can use AB testing to answer questions like which subject line is more likely to get subscribers to open a promotional email, or which image on your website is more likely to get clicked. 

Null and Alternative Hypothesis 

Before testing, the first crucial step is to clarify your null and alternative hypotheses. The null hypothesis tends to be a statement that your change actually had no effect. For example, your average email open rate was previously 20%, and remains at 20% even after updating the email subject line. Any sample observations resulted purely from chance. The alternative hypothesis would be that your average email open rate is now greater than 20% after the change, influenced by some non-random factor.

Calculate the Required Sample Size

In short, the larger the sample size, the better. Larger sample sizes allow you to be more certain that your test results are truly representative of the overall population. 

There are many online tools available such as this one, to calculate your required sample size. You’ll need to prepare the following inputs.

• Significance Level: the probability that your result could have occurred by chance, usually set to 5% and denoted by the Greek letter α

• Power: the probability of correctly rejecting the null hypothesis when it is false, usually set to 80%.

• Baseline Conversion Rate: the current or expected percentage of recipients who will perform the desired action, such as opening an email or clicking on a link.

• Minimum Detectable Effect (MDE): The smallest difference between the two versions of the emails that you are willing to detect, expressed as a percentage of the baseline conversion rate.

MDE = (desired conversion rate lift / baseline conversion rate) x 100%

Replace “conversion rate” with another metric such as open rate or click thru rate, as needed. For example, if your current average CTR is 5% and the smallest difference that you want to detect is 0.5 percentage points, then your minimum detectable effect is 0.5/5 = 10%.

If you have more than 50,000 subscribers, Salesforce recommends sending to 5% per condition. For example, if your subscriber base is 100,000 and you plan to run a standard AB test to compare 2 different versions of content, you may send version A to 5,000 randomly selected subscribers and version B to 5,000 different subscribers. If you have less than 50,000 subscribers, Salesforce recommends using 10%. If you have a small audience (say, fewer than 500 subscribers) your results may not be statistically significant.

AB Testing in Salesforce Marketing Cloud

This section will briefly cover the best practices when rendering an AB test in Salesforce Marketing Cloud. Other CRM tools may have similar functionality. 

AB test for the following elements is available directly in Email Studio: subject line, email, content area, from name, send time, and preheaders. Here, you may also set the criteria to determine the winning email automatically, as either the version with highest unique open rate, or the version with the highest unique CTR.

Email Studio also allows users to define the test distribution, based on either the number of subscribers (i.e. 40 to receive version A, 60 to receive version B) or percentage of subscribers (i.e. 35% to receive version A, 65% to receive version B). The system then randomly splits contacts in the group, according to the user-defined parameters. Note that users cannot control placing a specific subscriber in one distribution or the other.

For more advanced AB tests with 2-10 paths, use the Path Optimizer Test activity in Journey Builder. This is a multi-variant testing tool for testing content, send frequency etc. Once a winner is chosen, either automatically by the system or manually by the user, the winning path will continue to receive new contacts while the losing path(s) will stop taking in new contacts, allowing for journey optimization. The Path Optimizer test activity also provides test summary analytics, including historical test context for decision making.

When using Journey Builder for AB tests, consider including a Validation activity as well. Validation confirms that the journey's components are working correctly, by checking for errors or configuration issues in the following activities: entry source, entry schedule, decision splits, wait activities, update contact activities, email engagement splits, journey settings, journey goals and exit criteria.

Don't forget that although you are AB "testing", the emails are still being sent to real subscribers. Promotional emails should always comply with CAN-SPAM requirements and Marketing Cloud's Anti-Spam policy.

Finally, note that if your AB test involves email open rates, note that Apple’s Mail Privacy Protection released in September, 2021 may cause your open rate data to be inflated and skew results. 

How to Measure Statistical Significance

Statistical significance measures how likely it would be to observe what was observed, assuming the null hypothesis is true. In other words, statistical significance means that if you were to run the test again, you should get the same result. There are actually two ways to determine whether your test results are statistically valid.

One-tailed tests (top image) allow for the possibility of an effect in one direction, and show evidence if the variation is better than the control. In a one-tailed test, the null hypothesis is that the variation is not better than the control, but could be worse. The two-tailed test (bottom image) allows for the possibility of either a positive or negative effect, and shows evidence if the variation and control are simply different. 

In the real world, I generally go with a two-tailed test for most cases of AB testing. I often don’t know the direction of difference in key metrics, and it’s totally possible that the new variation performs worse. One-tailed test may also be fine if you’re testing a new variation and only want to know whether it is better than the control or not. If you don’t plan to deploy the new variation if it doesn’t win, then there is no problem. However, there could be a problem if there is no statistically significant winner and you deploy the new variation by assuming that it didn’t perform worse.

Assuming that AB tests for digital marketing have large sample sizes, use z test to analyze the results. Under the Central Limit Theorem, the sample is expected to have a normal distribution, and you don't have to worry about normality. In my experience, most people don't actually check the sample distribution if they have reasonable confidence that the assumptions for using z test are true. To determine the statistical significance of your completed AB test, check the following values.

• Z score: how many standard deviations away from the mean your result lands, in a normally distributed sample.

Z score = (Data Value － Mean) / Standard Deviation

• Standard error: the standard deviation of a sample population, used to measure the accuracy with which the sample represents the whole population.

Standard Error = SQRT {P*(1-P) / Sample Size}

Standard Error = Standard Deviation / SQRT(Sample Size)

• P value: the probability of getting your result, assuming that the null hypothesis is true. Use the following Excel function to find the p value from z score.

＝ NORMDIST(z score, mean, standard deviation, cumulative)

* mean: the mean of the distribution, use 0 for standard normal distribution

* standard deviation: use 1 for the standard normal distribution

* cumulative: "FALSE" returns the probability density function, the probability distribution of a continuous random variable;  "TRUE" returns value of the cumulative distribution function, the probability distribution of both discrete and continuous random variable

If the p value is less than your significance level, reject the null hypothesis. It means that there’s sufficient evidence of the alternative hypothesis. Otherwise, fail to reject the null hypothesis.

Type I and Type II Errors

Type I error is a false positive, meaning that you falsely concluded that any variation of an AB outperformed the other(s) with statistical significance. The probability of a Type I error is the significance level of your AB test. So if your test result has 95% confidence level, your probability of a Type I error is 5%. 

Type II error on the other hand, implies a false negative. In other words, Type II error falsely concludes that there is no statistically significant difference between the variations in your AB test, when there actually is a difference. Type I and Type II errors have an inverse relationship - by increasing your confidence level, the probability of Type I error decreases, and the risk of a Type II error increases. 

Many organizations view AB test outcomes as trustworthy enough to influence their business decisions. It's generally not necessary to repeat a test, unless something went wrong or the environment has changed. So when running AB tests, always record not only the test results, but also the environment that you ran it in - what was the state of any relevant elements of your product and customers? 

Limitations of AB Tests for Email Marketing

The biggest limitation is probably that AB testing looks at dependent variable results without considering the interactions of independent variables, so you can only make one change at a time. If you don't have enough time to test this way, or you're changing multiple variables and want to know more about which change mattered, it's time to look at multivariate testing instead. Multivariate testing is a more complex version of traditional AB testing. With multivariate testing, you can investigate further into the interactions of multiple independent variables on the target dependent variable.

Finally, an abstract point to end this post... as long as humans are conducting data analysis, there will be some level of cognitive bias involved. In layman's terms, the Dunning Kruger Effect means that "the less competent you are, the more confident you tend to be." Basically, people who are unskilled suffer a dual burden: they tend to reach erroneous conclusions and miss key insights, and on top of that, their incompetence robs them of the metacognitive ability to realize it in the first place.

Google Sheets and Apps Script for Digital Marketing

Posted: June 23, 2021

Updated: July 22, 2022

1. Separate URLs into their Components

To separate URLs into their components, such as for analyzing a sitemap or site structure, go to the data menu in google sheets → split text to columns, and then choose "/" as the separator. Alternatively, use the split formula.

=SPLIT(A1,“/”,true,true)

2. Users' Browser Window Widths

This is useful when creating a new landing page. Get the browser size (both width and height) in Google Analytics, and then export that data to Google Sheets to see the distribution of browser window widths. I first learned about this in a blog post from 2018, and have used it regularly. It's a bit old but still works the same: How to make a chart of your users' window sizes.

3. Ad Spend Tracker

When managing PPC campaigns, I usually setup a quick Google Sheet to track the monthly ad spend and make sure that I'm spending the budget evenly.

=DAY(TODAY())/DAY(DATE(YEAR(NOW()),MONTH(NOW())+1,0)) 

4. Regression Analysis for SEO Forecasting

Regression analysis is a common technique for understanding how a dependent variable such as organic traffic, changes in relation to independent variables such as search volume. Start with the built-in forecasting tools such as XLMiner Analysis ToolPak for Google Sheets, and then experiment with other statistical models such as multiple linear regression to see which one fits best. If you have trouble formulating a statistically significant model, try adding other important factors such as the page's publish date or number of high-quality backlinks.

5. Google Ads and Google Analytics add-ons

Google Sheets has a built-in Google Ads add-on that can pull campaign data. It's a bit buggy but good enough for ad hoc analysis. Similarly, there is also a Google Analytics add-on that you can use to pull data, modify as needed (such as fixing historical data) and export to Data Studio.

6. Identify SEO Issues

Import the html of pages in your site and checking for things like missing h1, title, meta tags, image alts, etc. Use the IMPORTXML function to scrape structured data from webpages, then use XPath query to specify which part of the XML to extract.

Some basic XPath queries:

• //h1 - to scrape h1s

• //title - to scrape titles

• //@href - to scrape links

.

Other common XPath queries for SEO:

• //meta[@name='description']/@content - to scrape meta description

• //link[@rel=’canonical’]/@href - to scrape canonicals

.

An example for JavaScript elements:

• //script[contains(.,'gtm.js')] - to scrape (check for) GTM code

.

7. Find & Replace with RegEx

RegEx (regular expression) is a sequence of characters to define a search pattern, often used for cleaning and manipulating data. It is available in Google Sheets find and replace, as well as many programming languages.

For example, BigQuery uses SQL and also supports RegEx as well, allowing users to write simpler code and avoid complicated subqueries.

In standard SQL, the query below extracts the substring between two parentheses.

substr (column, instr (column, '(')+1,instr (column, ')') instr(column '(')-1)

In ReGex, it can be rewritten as follows.

regex_extract (column, r'/((.*)')---

－－－

The remaining examples use Apps Script, which mostly runs on JavaScript 1.6, with some additions from 1.7 and 1.8. So you’ll need some JavaScript knowledge to edit the script and link it to Google properties, but not to the level of app or web development.

8. Confirmation Emails

Although Google Forms already has an option to send a standard confirmation email, Google Apps Script allows for further customization - send different confirmation emails depending on the user's responses, add external links, etc. This post by Google Workspace for Developers offers an in-depth explanation.

9. Redirect Chains

Ensure all pages on your site are accounted for with the following script.

// Get destination urls including redirects

function getRedirects(url) {
  var params = {
    'followRedirects': false,
    'muteHttpExceptions': true
  };
  var followedUrls = [url];

  while (true) {
    var res = UrlFetchApp.fetch(url, params);
    if (res.getResponseCode() < 300 || res.getResponseCode() > 399) {
      return followedUrls;
    }
    var url = res.getHeaders()['Location'];
    followedUrls.push(url);
  }
}

function getDestination(url) {
  return getRedirects(url).pop();
}

10. http Server Status Code

Track the status code of all pages on your site with the following script.

// Get http server response status code
function getStatusCode(url){
  var options = {
     'muteHttpExceptions': true,
     'followRedirects': false
   };
  var statusCode ;
  try {
  statusCode = UrlFetchApp .fetch(url) .getResponseCode() .toString() ;
  }
  
  catch( error ) {
  statusCode = error .toString() .match( / returned code (\d\d\d)\./ )[1] ;
  }

  finally {
  return statusCode ;
  }
}

// Exceed importxml limit
function importRegex(url, regexInput) { 
  var output = ''; 
  var fetchedUrl = UrlFetchApp.fetch(url, {muteHttpExceptions: true}); 
  if (fetchedUrl) { 
    var html = fetchedUrl.getContentText(); 
    if (html.length && regexInput.length) { 
      output = html.match(new RegExp(regexInput, 'i'))[1]; 
    } 
  } 
  Utilities.sleep(1000); 
  return unescapeHTML(output); 
}  

For more resources, keywords in sheets has free scripts for SEO. There are different templates for tasks such as getting old webpages from wayback machine, SEO forecasting and clustering.

How to Audit a Google Search Ads Account

Updated: July 12, 2022

If you've recently taken new ownership of a Google Ads account, or simply haven't audited your existing account in a while, this could be a good time for a check up. Regular audits can ensure that your campaigns are still running as efficiently as they were when you launched them. Even if your ROI is positive, you can always find areas for improvement.

This post explains how to audit a Google search ads account, find areas for improvement, and implement changes.

Keyword Bids

Start by checking the overall health of the account, including keyword bids. There are several metrics to use as a guide. First, the Lin-Rodnitzky (L/N) ratio, a metric introduced by a 3Q Digital, is defined as follows: divide the overall account CPA by the CPA of all keywords with at least one conversion. A ratio between 1.5-2.0 is generally good. If it's below 1.5, you might be too conservative, potentially missing out on converting keywords. If the ratio is above 2.0, you might be wasteful, over spending on non-converting keywords.

Next, check the broad match ratio. What proportion of your budget is spent on broad match keywords? Broad match tends to be less efficient than phrase or exact match, so you generally want to keep the ratio around 20%.

Finally, check specific settings in the historical data using filters. For keywords with at least 3~5 conversions, you can pretty much rely on the CPC and CPA data that was accumulated. If there are less than 3 conversions, the data is still volatile - one additional conversion could greatly affect the CPA, so keep an eye on it and make adjustments as needed.

If there any keywords that had high spend but zero conversions over several months, consider cutting your losses and stopping them. I generally use 100$ as the cutoff but it depends what is "high" for your account and business.

Once the account has accumulated several months of data, 30+ conversions monthly, and proper conversion tracking setup, consider automated bidding for maximum conversions, if that fits your business goals.

Campaign Structure

It's important to check the high-level campaign structure when you conduct a Google Ads audit. Create a custom report and check that all eligible ad groups have at least one ad, and non-dynamic ad groups have 1~20 keywords. If an ad group has too many keywords, the ads might not be targeted to user search queries, and consider splitting it up into two or more ad groups.

For most campaigns, the best practice will be to use STAGs (single theme ad groups). However, if you have a large budget (over 1k) and high conversions (30+) you can try SKAGs - single keyword ad groups. SKAGs gives you control over showing unique ads for every single keyword, but it can get complicated to maintain the campaigns and make bulk changes when needed.

Even with the new dynamic ads, Google still recommends including 1 responsive and 2 expanded text ads per ad group. Finally, use negative keywords to prevent overlapping and inner competition within your account.

Attribution Model

The default attribution model is last click, and that works fine for many businesses. However, in some cases you could be over or under reporting a campaign's performance by using the wrong attribution model. If you're spending over 10k per month or getting traffic from different sources (display campaigns, retargeting, organic traffic, etc.) one of the other available models might be a better fit. There is a model comparison tool in Google Ads so you can check.

IP Exclusion

Avoid wasting budget on clicks from employees. You can access IP exclusion settings in the left sidebar. However, with many people working from home now, IP exclusion is more complicated. In some cases it might not be worth the hassle, but you can try using a VPN, Chrome plugin, or extension.

Targeting

Targeting depends heavily on your business goals, but some common oversights across all industries include not targeting specific areas, excluding the search partner network without having data to prove poor performance, and only checking English in the language targeting settings, which limits your reach to users with English in their browser settings. If you have a decent lead nurturing pool, it may also be worth experimenting with remarketing strategies such as Customer Match.

Average CPC, CPA, and CVR

The left table shows average performance metrics across all industries, according to WordStream. This is just a benchmark, since the metrics depend on many other factors, such as your brand strength and SEO to name a few. However, it's something to keep in mind and see how you compare.

Conversion Tracking

Assuming you are already tracking your main conversion, you could gain additional insights by tracking secondary conversions. might gain insights from tracking other events as well. For example, if your primary conversion is a contact form submission, you might add page as a secondary conversion. For CPA bidding and conversion data purposes, only the primary conversion should be included in conversions.

Visit the conversions tab to confirm that everything is being properly tracked. The status should ideally be recording conversions. No recent conversions and tag inactive status doesn't mean that your tracking is broken - it could be that you simply haven't gotten any conversions in the past 7 days. New conversion events will first have an unverified status that should change once Google Ads sees an initial conversion, and you can do a manual conversion test to speed up the process.

It can take up to 24 hours for conversion data to appear in Google Ads, even when everything is working perfectly. If after 24 hours conversions still are not showing up, you might start troubleshooting. Note that if you have some conversions reported, then conversion tracking is probably set up correctly (i.e. the tracking either works or it doesn’t, in terms of setup). However, it’s still possible to have missing conversions. Here are a few things

to consider.

App Tracking Transparency (ATT)

Apple released iOS 14.5 in April 2021, which included app tracking transparency (ATT), impacting conversion tracking on iOS across many platforms including Google. Even if Google is still able to track your conversion, you can now expect up to 5 day delay in reporting. Meanwhile, you might be able to check your missing conversion in Google Analytics to see it was from paid and on mobile, which will give you a clue as to whether the problem is related to ATT.

Ad Blockers

Sometimes if the user has a third-party tool such as an ad blocker installed, it can prevent conversion tracking. According to research by Moz, ad blocker usage can be in the 15–25% range depending on region, but many of these installs will be default setups of AdBlock Plus, which does not block tracking. While you may be missing some specific conversions due to ad blockers, it shouldn’t significantly impact campaign performance. And since all advertisers suffer this problem equally, it is a level playing field across the Google network.

Auto-Tagging

If you are collecting conversion data in Google Analytics and sending it to Google Ads, you must have auto-tagging enabled. This setting is disabled by default, so you’ll need to change it in your Google Ads account settings.

Auto-tagging works by adding the gclid (Google Click Identifier) parameter to the URLs people click through from your ads. For example, if someone clicks on your ad for www.example.com, the final URL will look like www.example.com/?gclid=123xyz. Depending on what service your website is hosted on, you may need to add a bit of code either directly or via Google Tag Manager, to successfully pass gclid to your landing pages.

By default, auto-tagging overrides and manual UTM parameters that you may have set up. If you would like to have UTMs take priority, you can do that by enabling "Allow manual tagging to override Auto-Tagging" in Google Analytics, under property settings in the admin section. Google Ads will still pass any values that are not overwritten to Google Analytics, so you won’t lose the tracking details from gclid.

So what can you do about missing conversions?

Import Offline Conversions

One solution is to manually import them back into Google Ads as “offline conversions.” You can simply enter the conversion details in an Excel or Google Sheet and upload it to Google Ads. This help page has step-by-step instructions and templates. Please note that you needed to have auto-tagging enabled at the time of the conversion, because the gclid needs to be included in your upload.

Ad Extensions

There's a lot to consider when optimizing Google Ads - bid strategy, campaign structure, conversion tracking, etc. Ad extensions are often an afterthought, but they actually have a ton of benefits. Not only do they increase your SERP real estate, but they can also get you more qualified leads.

Google has directly stated that ad extensions will increase ad rank, and that extensions are favored because they allow Google to offer a better variety of ad formats and include relevant information for users. All of these factors can increase your CTR, lower your CPC, and cut costs in your ad account overall.

Google Ad extensions do exactly as the name implies - they extend your ad. Ad extensions append additional information about your business to the main body of your ad.

Sitelink Extensions

Sitelink extensions add additional links under your text ad in the SERPs. These links direct users to different pages on your site, other than the landing page.

Examples of common sitelink extensions:

• Plans & Pricing

• Success Stories

• Contact Sales

• Download Brochure

.

Callout Extensions

Callout extensions are 25-character text snippets describing the strong qualities of your services. Examples of common callout extensions:

• Free Shipping & Delivery

• 3 Convenient Locations

• Free 30 Day Trial

.

Structured Snippet Extensions

Structured snippets are lists of specific services, amenities, locations, etc. Select a header from the drop down list of headers, and include at least 3 items for each one.

Lead Form Extensions

Note that your site requires a privacy policy page in order to use this extension type. Lead form extensions are a relatively new feature in Google Ads, as they were released in beta in 2019. They allow the user to fill out a form directly from the ad, without ever clicking into your website. This can be advantageous for BtoC companies looking to increase their lead pools. On the other hand, in my experience the user probably won't change their behavior just because the lead form is available. Depending on your business, it could be better to direct users to your website, where they can learn more about your offering before filling out a form.

Sitelink, callout, and structured snippet extensions are universal extensions, meaning that Google recommends every advertiser to use them. Other extension types such as call, app, lead form, etc. are more situational for each business.

Task Automation

Fortunately, Google offers many automation tools for managing your ads account efficiently. If you have custom needs that aren't covered, you can always use Google Ads Script and API to build your own tool.

Automated Rules

Google Ads has automated rules to save time and catch potential issues as soon as they occur. For example, you can receive an email notification when the CPC exceeds a certain amount, or automatically pause a campaign when the CPA exceeds a certain amount. You can also combine rules - for example, pause a campaign when the CPA exceeds $100 and the total cost is over $300. This way, the rule won't apply to new campaigns that might still be in the learning phase.

Google Ads Editor

For bulk editing, use Google Ads Editor to export your ads to Excel, make changes, and import again to publish to live. This is also useful if you require approval from a client or superior before publishing your ads. For more advanced campaign builds, you can automatically build a campaign from a spreadsheet.

Google Ads Scripts

Even if you aren't a developer, Google makes it easy to just copy and paste the whole source code to your Google Ads scripts. Some useful scripts to get started are the link checker, account anomaly detector, and ad customizer. There might be different source codes for single and manager accounts, so make sure you use the correct one for your account.

Google Data Studio

Google Data Studio is a great tool for automated reporting and sharing with your client. Obviously it's part of the G suite, so you'll get a smooth native connection to Google Ads.

If you're looking at third party tools beyond G suite, Tableau is an advanced tool similar to Data Studio, but with more powerful features for analytics.

Ads Disapproved

One of the most common reasons for disapproved ads is "destination not working," which means that the landing page can't be reached by Google's crawlers, due to some issue such as a broken link or redirect. Unfortunately ,Google does not provide the exact reason why your ad was disapproved, so you'll need to figure out the cause and solution. Here are some common possible reasons for ads disapproved due to "destination not working," and how to fix them.

Server Response Code

There are several ways to check the status code - developer tools in your browser, Google sheets, extensions, or SEO tools. Although Google does allow ad links to be redirected, too many redirects can cause slow page load speed and result in poor user experience. An http 200 response code works best. A common discrepancy is omitting www. from the destination URL, resulting in a 301 redirect to the www. subdomain. Also, note that the display URL does not need to exactly match the destination URL.

CNAME Record

Google ads can be disapproved if your landing page doesn't appear on your registered domain. This issue can arise if you are using different landing page building services. Parked domains are not allowed, and subdomains can get flagged although they are allowed. By creating a CNAME (canonical name) record, you can link your landing page to your root domain.

Accessibility

Even if your ads are only targeting a specific location, Google's policy is that your website should be universally accessible. You should have a robots.txt file granting googlebot explicit permission. A possible solution for this issue is to redirect foreign traffic to a splash page that says "this website is not available in your country."

Dynamic Tracking URLs

If you have enabled auto-tagging in Google Analytics, then Google will pass the gclid along with the URL. However, if your website can't handle URL parameters, then it can cause an error. In some cases, just enabling auto-tagging in Google Analytics was enough to get the ads approved. Although Google disapproves many ads, mistakes do happen - if you believe that your ad was mistakenly disapproved, you can try making a minor edit and resubmitting. If you still can't get the ads approved, go ahead with the appeal. Appeals usually go through within a few business days, and you can check the status in your policy manager. You can also try contacting Google Support.

Credits, Coupons, and Promo Codes

If you are running Google Ads campaigns, perhaps you were fortunate enough to receive free credits, coupons, or promo codes. The good news is that your account could be eligible for more.

Google offers free ad credits to incentive new or returning advertisers. In addition, website hosting services often offer Google Ads credits for new users of their premium plans -- Bluehost, Squarespace, and Wix, just to name a few. Also, check for any offers from marketing tools and software that you are already using.

During Covid, Google also gave existing SMB advertisers up to 1,000$ in free ad credits. There was no application process -- Google simply inserted the ad credits to qualifying accounts. This campaign had a few requirements, specifically, the account had to be active 10 out of 12 months in the past year to be eligible. Google also has a grant program for registered nonprofits.