Originally written by Katie Spencer on 8/19/2021 and updated by Rommel Alcobendas on 7/14/2022.
As digital marketing consultants and strategists, our job is to make recommendations to our clients, but we can’t base them on intuition alone. Besides making educated hypotheses on what could be harming webpage usability or causing friction on a site’s user journey, conversion rate optimization (CRO) strategists need a way to see whether our recommendations will positively impact conversions before implementing changes directly on the site.
Luckily for us, we have the opportunity to support our recommendations with data using powerful A/B testing platforms like Google Optimize. This guide will walk you through what Google Optimize is, what it can do, the basics of setting it up, and tips to keep in mind as you start testing.
What is Google Optimize & How Does It Work?
Google Optimize is a free website design testing tool that can help you create and monitor your experiments to help you accomplish your business goals. This tool empowers you to customize target audiences from your existing website traffic to experience your test.
Optimize can work as a basic web page editor, allowing you to make changes on your website that only your targeted testing audiences can see. It then provides insightful reports on how your variations performed over the duration of your test based on the conversions (or “objectives”) you assign to your experiment.
Google Optimize vs. Optimize 360 (Free vs. Paid)
Like all platforms available for free on the internet, there is a paid tier that you can access through Google Optimize itself, called “Optimize 360”, or via Analytics 360. Is the paid version worth it, though?
If your digital marketing strategy focuses on small and medium-sized businesses (or SME/SMB), the free version will set you up nicely. On the other hand, larger enterprises, e-commerce websites, or agencies with highly sophisticated testing and analysis programs that may already have Analytics 360 would absolutely benefit from the paid version.
The free version covers the most crucial CRO strategist needs, like running up to five simultaneous tests, or focusing on three experiment objectives/conversions. If you’re paying for Optimize 360, you can run anywhere between 25 to 100 simultaneous experiments, with up to 10 objectives/conversions to focus on. It’s quite the jump! Google provides a comparison table of Optimize features for both tiers, so you can ultimately decide which is best for you and your business (the Optimize Resource Hub updated their feature comparison for GA4 users as well).
Why Use Google Optimize?
Google Optimize can help you test your ideas, prove your points, and make your case. For instance, if you’re curious if users will be more inclined to purchase if a CTA button is styled differently, you can test that using Optimize. Or, let’s say you need to provide proof to your stakeholders on why you restructured an entire landing page, Optimize will have the information you need to back up your work. It’s a versatile tool with a lot of different capabilities.
At Portent, we love the integration between Google Optimize and Google Analytics (both Universal Analytics and GA4). For example, thanks to Google Analytics’ use of Experimentation ID/Name as a dimension, connecting Optimize to GA reporting provides a pathway that will earn more granular insights. And the more insights you earn in relation to your tests, the better user journeys or audiences you can build for future test iterations or digital strategies.
What Can You Do With Google Optimize?
Within Google Optimize, there are three types of tests and two site customizations you can use to improve user experience and conversion rates, depending on your needs. In this post, we’ll talk about testing and personalization (we won’t be talking through site banners).
A/B (Split) Tests
An A/B test, also known as a split test, is the most basic and most used test you can perform in Google Optimize. An A/B test allows you to test the risk and reward of the desired webpage change by splitting your audience and presenting two versions of your site: an original version to one group and one with a variable to test to the other group. Optimize will then measure which group had a more successful interaction between the original and the variant, so you can learn whether or not to implement the change directly on your site.
For instance, if you are considering updating a header on your page, you could test your original text against a new copy variation to see whether it can aid in boosting conversions. If you run this test and there is no clear winner, it indicates that a more meaningful change to optimize conversions is elsewhere, thus inspiring your next test.
A/B Tests are typically the best fit for most websites, but there are instances where you may want to create more complex test methods.
A redirect test uses the same framework as an A/B test. It still allows you to test two versions of web pages against each other; the only difference is that instead of creating a single change within the test page using Optimize’s visual editor, you simply add the URL of a variation experience’s page. Optimize will then redirect a predetermined percentage of your test audience from your control page to the variation page.
Redirect tests are a great option when variation experiences require more edits than are recommended within the A/B test editor (around 60 lines of code), like when you’re testing a new form format. Some CRO professionals will use redirect tests to compare the performances of two distinctly different pages. While this is great for reducing the risk of introducing a new page layout, we offer a word of caution that it can be difficult to identify why the winning page was more successful.
A multivariate test helps you determine the best performing combination of variants. Using a multivariate test is typically the best fit for sites with very high traffic or baseline conversion rates. For example, let’s say you wanted to test which headlines or images to use on a page, and you have two options for each. Instead of setting up an A/B test, waiting for the results, and then iterating on your findings, you can test which combination of headings and images performs the best all at once.
This testing format is great because it can speed up testing, but remember that it requires very large sample sizes in each experience, and exponentially increases the complexity of analysis. This is varsity-level testing.
A multi-page experience is similar to a classic A/B test, but it allows you to create a variant across multiple pages of the site to test a change throughout the user journey consistently.
For example, if you wanted to test whether referring to the checkout process as a “cart” or a “bag” helped nudge users to convert more, you could change the CTA on product pages (“Add to Cart” vs. “Add to Bag”), and then change the corresponding title on the checkout page to see which verbiage was more successful.
Personalizations are a bit different than the rest of the test options in Optimize. Think of this option more like a marketing technique, similar to targeted ad campaigns that aim to show the right content to the right visitors at the right time. Personalizations allow you to display a set of changes to your website in front of anyone meeting your targeting conditions. Unlike the other test types, personalizations do not use variants and can run forever.
Personalizations can be a stand-alone option if you solely want different users to have different experiences on your site (for example, show a unique phone number when a customer visits from outside your sales area or add customized context to a form if users arrive there from a specific referrer). Still, personalizations can also pair with the findings from other types of tests. If you find that a variant outperforms the original experience on a page, you can use personalization to deploy this updated version while the development team changes the site.
How to Use Google Optimize in Four Steps
Now that we’ve talked through what Optimize can do, let’s talk through the how. To run tests in Google Optimize, you must first create an Optimize account and install the Optimize script on your website. We recommend setting up Google Optimize via Google Tag Manager (GTM) for the fastest, most consistent deployment across your site. Once you are up and running, you can start building your tests with these four steps:
- Create an Account and Add a Container
- Set Up Your Test (Create Experience)
- Launch and Monitor Performance
- Analyze the Results
1. Create an Account and Add a Container
I’m going to hand you off to Google for the nitty-gritty of setting these up. Google Support has a great guide for setting up your Optimize Account for Google Analytics 4 (GA4) or Universal Analytics and connecting it to your Analytics Account (a crucially important step for measurement).
Once you get into your Google Optimize account, it will prompt you to add a container. This hierarchy is similar to Accounts and Properties within Google Analytics. Once you have your container, it’s time to install Optimize on your website. We recommend that you install Optimize with Google Tag Manager. This will allow flexibility in how and when Optimize is deployed. For example, you can easily pause the tag if you take a break from testing.
Once you have everything set up as described, you will want to go into the preview mode in GTM and make sure your tag works where and how you intended.
Similar to the screenshot provided above, use this Preview in GTM to confirm that the Optimize tag is firing correctly for the pages you need it on before publishing the tag to your live site. Once done, you’re set to create your first experiment in Google Optimize.
2. Set Up Your Test (Create Experience)
Once you are set up in Google Optimize, it is time to select the upper right button “Create Experience” to set up your test and choose parameters.
First, name your experience something that has not been previously used, and be sure to include information that will distinguish it from other tests you may run. Underneath the name, you will include the URL of the page you will be testing, called your editor page.
Next, you will select which type of test you would like to create between A/B, Multivariate, or Redirect.
Create Variant (or URL)
Once you are in your new experience, the first step is to set up the change you want to make on your site by selecting the blue “Add variant” button. Next, choose a name (we will call it “Variant 1”) and select “edit” next to your new variant.
This is the part where you can start making changes to your site that you want to test against the original. The editor allows you to change copy, element order, style colors, and more in a few simple steps. Other more complicated changes could require some help from a developer.
At this stage, you will also determine what proportion of test visitors will see the variation experience. We recommend a 50/50 split.
Select Your Audience
Once the variant is set that you would like to test, the next step is to determine who will see the experience. You can set it to appear to all users who visit your site or customize it so that only certain users encounter it. For example, you can specify that only people in certain locations or from certain devices will participate in the experiment.
Choose the Objective
When it comes to choosing an objective or what you want to measure, there are two options. You can either choose from a list of your current Google Analytics Goals or create a custom objective.
Custom objectives are useful when the goal of your test doesn’t correspond to a GA goal. For example, you can select a specific event on a webpage, like whether someone clicks on a “Next Step” button to measure whether a change you made on a page influenced a user to start a form on that page.
Determine Test Duration
When we set up tests, the duration will depend on the site’s typical traffic. If it is low, you may need to run a test longer to collect enough useful data. As a rule, we recommend you choose pages for tests that average at least 1,000 pageviews a month over the last three months. This will ensure that your experiment gets steady traffic and can complete in a matter of weeks, not months. If your site doesn’t get the traffic needed for a standard test duration and you don’t want to run a test for months, there are other A/B test strategies for low-traffic sites you can try.
For most sites, the sweet spot for running tests is between two and four weeks. A minimum of two weeks allows us to account for any cyclical nature within the week. A maximum of four weeks allows us to limit the introduction of too many other variables. If you plan to run your tests to statistical significance, we recommend using a tool like the Pre-Test Analysis on A/B Testguide to determine exactly how long your specific test needs to run. Make sure to do this before you start your test.
Semi-Pro Tip: If there’s a page that absolutely, positively must be tested quickly but has low traffic, consider whether you could use paid ads to drive the right kind of prospects to this page at a higher pace to glean actionable insights.
Before you launch your test, select your desired settings. First, check that Optimize is correctly installed by clicking the “Check Installation” button in the test’s Settings, towards the bottom of the Details page. If you’ve set Optimize up correctly for your domain, you’ll receive a little green indicator that shows you that you are good to go.
Next, we recommend turning on email notifications so you can receive updates about the experience without logging in to check in as often. Depending on your testing needs, select the percentage of visitors to your site who will see the experiment. For speed, we recommend that 100% of your traffic can see the experiment (they will be split based on the variant weight you assigned earlier).
Then, choose which event will trigger the test. The options are pageload, continuously, or a custom event. Pageload is the most common and the default option. Continuous activation is good for tests on dynamic pages as it will apply the variation’s changes to newly matching elements as they load. A custom event can be used when the event helps determine the test’s audience. For example, if you have a multi-step form and want to determine the influence of a change to a later step, you could trigger your experiment once a “Next Step” button from the form is clicked.
3. Launch and Monitor Performance
Once all of the variants, targeting, and settings are in place, it is time to push the shiny “Start” button you have been eyeing. But just because all you need to do now is wait does not mean your work here is done. It is imperative to monitor your test’s performance throughout the experiment to ensure that nothing is broken, that your variant is not creating a worse experience for users, and know when to pull or extend the test’s duration.
Google Optimize makes it very easy to monitor results within the tool. In the container under the “reporting” tab, you can view the data in various ways. At the top, you can view the sessions over time, broken down by each day. This view is important for ensuring data is being collected and that there are no alarming drops in numbers. We recommend clicking “View full chart” rather than just peeping at the thumbnail.
Fun story: We once had a test we let run for four weeks that we checked weekly and missed that it had broken ten days in. That was unlucky timing.
The “Collected Sessions” chart thumbnail:
The full “Collected Sessions” chart:
4. Analyze the Results
After your test has been running long enough, you can often (not always) start to see a pattern. One way to see how a test is performing is by looking at the “Probability to Be Best” section under “Reporting.” If the percentage is 80 or above, it indicates a strong likelihood of becoming the winner over time. In this example, the “Probability to Be Best” was 100% for our variant, giving us enough data to implement the variant on the live site.
In some cases, it is helpful to dive deeper than the “Probability to Be Best” to understand the data. Luckily, Optimize does some of this work for you by providing the data that directly compares the performance between the original and the variant. In order to do so, Optimize calculates the modeled conversion rate, which calculates how your variants have performed against a selected objective over time.
There are two ways that Optimize displays the modeled conversion rate. One is in a box and whiskers plot, while the other is in a chart.
There are a few indicators to consider when looking at performance. First, it is important to note whether the blue and black dotted lines diverge. Below is an example of a variant that outperformed the original page, where the lack of overlap shows a clear winner.
Once you find whether a variant helps or hurts your site, it is time to implement the successful change or go back to the drawing board to hypothesize other variants to test.
Tips to Consider When Using Google Optimize
Although you are probably ready to start testing, there are a few things to keep in mind when running experiments in Optimize that could save you in the long run.
One thing to be aware of is potential cross-contamination within your tests. If you run more than one test at a time, avoid cross-contaminating your data by making sure that your audiences and targeting do not overlap and that you run one variant at a time. Otherwise, you could end up with unhelpful data, wasted time, and an empty pocket, even if your variants would have been a hit.
When we needed to run two tests on one site simultaneously, we combatted this issue by displaying each test to two different markets to avoid overlap.
One thing you can expect from testing is that you will probably fail a few times. Sometimes you find that a test may malfunction or even receive disappointing data showing your variant was not successful. Just know that part of the job is to learn by trial and error. The best way to combat this is to expect and embrace failure so you can implement what you learned and avoid making the same mistake again.
Reach Out to Specialists for Technical Support
One of the ways to limit failure and avoid wasting time and money is to ask experts for help. The technical aspects of setting up a test or analyzing complex results can be beyond someone’s expertise.
We recommend working with a dedicated CRO or development team to lean on when it comes to creating changes on pages for variants, selecting audiences, and even managing strategies. Not only will they ensure your tests are running smoothly, but they can also help ideate future tests to identify areas of improvement on your site.
You’re now equipped with all the information you need to set up and start using Google Optimize! We hope you enjoy implementing these recommendations and developing these CRO skills for you and your team. Happy testing!