In September 2016 Optimizely launched their next generation platform “Optimizely X” – promising a better visual editor, faster performance and closer integration with their new personalisation features.
Having used Optimizely for over four years, running optimisation campaigns and hundreds of experiments for clients worldwide, I was keen to find out how the new platform stacks up. After six months using Optimizely X, I can now share some of the key differences, improvements and frustrations to help you to make the decision whether to make the switch.
So what’s changed in Optimizely X?
Almost everything! Optimizely X is a completely new platform that’s been built from the ground up. The principles are the same and you can still do everything you could in Optimizely Classic (the new name for the existing web testing tool), but the structure, approach and even naming of many common features have changed significantly.
The most exciting developments are areas such full stack testing, personalisation and recommendations. The changes have included a full rebuild of the web testing platform, however, so for typical Optimizely users, though, there’s a big change if you choose to switch. The only area where there hasn’t been significant change has been experiment results where Optimizely’s Stats Engine, already a key part of Optimizely Classic, is still used.
How do Optimizely X and Optimizely Classic compare?
Let’s take a look at some of the biggest changes and differences between the two platforms:
New approach to site and experiment setup with more upfront work required
One of the biggest changes in Optimizely X is the structure of experiments and the workflow for getting them live. This is primarily to fit with the new personalisation platform and to enable seamless integration between the two. What this means for users is a more structured approach to setting up and defining your site within the platform, with a bit more upfront work.
Traditionally Optimizely was focused entirely on experiments – you created an experiment, chose what pages and audience to target it to and defined your goals. That’s great for small scale testing but when you scale up to many experiments, it brings a lot of risk to the process. For example many experiments reuse the same targeting conditions over and over again e.g. example running an experiment on your Product Page involves targeting the same set of URLs every time. Every time you set up an experiment these need to be defined, though, which brings a risk of an error every time you set up a new experiment.
Optimizely X requires you to define the structure of your site first, then set up experiments (“Campaigns” if you’re using the new terminology) later. You use the platform to set up Pages (single pages or set of pages), which you can then use to target experiments. Each page can also have a set of Events (similar to goals in Optimizely Classic), used for tracking and targeting.
For an existing user of Optimizely, this seems like a time consuming process at first, but I was surprised at how much easier it is to get experiments live once you have your whole site defined within the platform. By going through the one-off task of setting up pages and events (including any custom coding which is needed), experiments are now trivial to set up, even for non-technical users. For anyone launching more than one or two experiments, the improvements in speed of creating tests and the reduction of risk from having pre-defined pages is clear.
New experiment workflow adding change management to reduce bugs
One of the biggest improvements in Optimizely X is the new workflow, which allows for draft changes to be made without publishing them live to all users. In Optimizely Classic it’s very simple, an experiment can either be live or paused. Although it is possible to preview a paused experiment, there’s no way to make changes to the code/layout without either stopping or cloning the experiment. Although it’s possible to use preview links set up QA cookies to fully test experiments before they go live, it’s a bit of a hack and means that its not always clear what the status of an experiment is on the dashboard.
Although changing a live experiment will invalidate your results, there are plenty of times when it is unavoidable, for example fixing bugs, making small changes to copy etc. due to changes outside your control or simply because most sites will serve winning experiments to 100% of traffic while they’re being implemented and changes are unavoidable.
In Optimizely X, the published version of an experiment is unaffected when you make changes. The changes are saved (and can be previewed using the new QA tool), but are kept in a staging environment until you choose to publish them. This far more closely matches a typical software development livecycle and again reduces risk of errors significantly.
Vastly improved visual editor with responsive support
When Optimizely was first launched, the editor was revolutionary, allowing point and click changes to sites without having to write code. Over time, however, it has become less able to cope with modern sites, with conversion optimisation agencies and advanced users preferring to make the changes in pure JavaScript.
The new editor brings full responsive support, allowing you to easily and quickly create responsive tests. The point and click functionality is fast, easy to use and integrates closely with the definition of events for tracking experiments, making many changes easier to do in the visual editor than by writing code. For anyone using the visual editor, this is a clear reason to upgrade.
Image Credit: Optimizely
Technical changes with a new coding style
For users who develop their tests in pure JavaScript code, there are huge improvements to the native code editor. Optimizely have clearly focused on this aspect. The code editor now includes syntax highlighting, simple error checking and is far faster to load.
Optimizely Classic uses a complicated jQuery-like syntax to try to reduce flicker on pages, which, although effective, required developers to learn a new style of coding. For anyone who didn’t, pages tended to suffer from a flickering effect, harming the user experience and results.
Optimizely X does away with this, instead giving the option to run the code synchronously (as part of the Optimizely snippet) or asynchronously (as the page loads), with a few useful utility functions (for example one which waits for a specific element to appear on the page). Although it still requires a good developer to take advantage of this, it allows for a lot more flexibility when coding changes and is a welcome improvement to the platform.
Powerful QA tool to test experiments for bugs
QA is one of the most vital parts of any testing program. Any bugs or errors in tests can cause massive damage to a site and undo a lot of the good work in testing. Running QA for a test in Optimizely Classic has always been a little painful, requiring either query string parameters or special cookies, or the use of a limited QA tool which takes over the bottom of the page, reducing its effectiveness.
Optimizely X brings a new approach. You can still use your old methods, however the new QA tool is impressive:
Loading as a “button” on top of the site, it offers the ability to enter into different variations of multiple tests, preview different audiences and, usefully, provides a preview of the events and actions the tool is taking, very helpful in testing tracking and evaluating why experiments don’t run.
For more technical users, the full log is now shown in the console, and has been improved from Optimizely Classic, further helping to diagnose any issues with tests.
“Always on” tracking
One of the biggest changes to the platform, and a very welcome one, is “always-on” tracking. Previously, results were tied to specific experiments, so a goal only tracked when it was added to an experiment. If you realised later that you’d forgotten to track a key page or metric, adding it to the experiment would only give you data from the moment you’d added it. Frustrating, especially if you have seen a trend and want to get more data on it.
Optimizely X splits tracking from experiments. Your events (goals) are set up independently, so always track users. That means you can add an event to an experiment half way through and you’ll have data right back to the beginning of the experiment, overcoming one of my own biggest frustrations with the old platform.
Results themselves haven’t changed a lot with Optimizely’s stats engine powering the largely similar results page. Stats engine is a great improvement on traditional statistics, using a Bayesian approach which means that the results are statistically valid whenever you look at them. Unfortunately you still need some common sense when looking at results you will still get results declared when clearly there isn’t enough traffic:
New software which still has some teething issues
When considering the switch, it’s worth remembering this is still a new tool. Although more than six months old, I’ve still come across some bugs in the platform. Some are relatively minor (occasional cryptic error messages), others are bigger and more frustrating (experiments not live when the platform says that they should be). Although Optimizely’s focus will be on Optimizely X for future development, you’re still choosing a 6-month old tool, rather than one that has been tried and tested by thousands of companies over many years.
Personalisation and testing beyond the web
When considering whether to make the switch, a key consideration is the integration of Optimizely X with Optimizely’s personalisation platform along with other new tools such as Optimizely Full Stack and recommendations. The personalisation tool is simple to use and powerful, using the same point and click editor as the experiment platform to set up personalised experiences for customers. This, combined with the recommendations tool means Optimizely X is now a fully featured platform for far more than just web optimisation.
The introduction of Optimizely Full Stack is an exciting development. Offering SDK integration for many programming languages, it allows companies to test the backend systems of their site, not just JavaScript changes on pages. Whether that means testing features and functionality that couldn’t be tested on the frontend site or testing changes to apps and software, this is a great opportunity for mature optimisers to go beyond the web.
If you’re considering using these features in the future, it makes sense to switch to Optimizely X to have an integrated conversion optimisation platform.
PCI DSS compliance to allow testing on payment pages
Another big change which may affect some businesses is that Optmizely X is now fully PCI DSS compliant. Many businesses and payment providers will require compliance for any tools used on payment pages on sites. This has always been a challenge for Optimizely Classic, leaving some companies unable to use the tool or test their payment pages and funnels. With full compliance for Optimizely X, these companies should now be able to benefit from A/B testing throughout the funnel.
How easy is it to switch?
Unfortunately the biggest challenge for many companies will come with the switch between the two platforms. Although Optimizely have published some useful guides, the reality is that you need to completely move your testing from one platform to another. There’s no change to the snippet of code on the site, but everything else will need to be set up from scratch.
That means creating the Pages, Events and Audiences in Optimizely X, which for a large or complex site could be time consuming. Because the platforms are separate, none of your tests will be transferred from Optimizely Classic either. That means any tests which are live (or being served to 100%) will either need to be recreated (and restarted) in Optimizely X, or you will need to run both platforms alongside each other (known as a “phased rollout”). This takes away some of the pain, but is only really a short term solution and brings a complexity of managing tests across two platforms.
The biggest challenge when switching is likely to be the learning curve, especially for developers. For advanced users and experienced developers the changes to the platform, although welcome, will take some time to get used to. This especially true of the new coding styles, and I’d strongly recommend spending time getting familiar with building tests in the new platform before attempting to switch.
Ultimately, the switch will be harder for more advanced and active users as the change impacts the advanced features more than basic functionality.
Will Optimizely force you to change platforms?
At the moment, Optimizely is supporting both platforms, and will continue to do so for some time, however previous experience has shown that they are willing to push through changes (such as new packages and pricing) without a long consultation or transition period. The large volume of sites using Optimizely Classic should mean that there is support for the near future, but with personalisation as the company’s focus, an enforced switch to Optimizely X in the future is almost a certainty. Planning and making the switch on your own terms is a sensible approach.
Should I switch from Optimizely Classic to Optimizely X?
If you’re keen to test backend features which you can’t do with JavaScript, you’re interested in trying personalisation or aren’t currently actively testing, I suggest making the move straight away.
If you’re currently happy with Classic, you need to weigh up how important the differences outlined above are to you. There’s certainly a cost (both in time and effort) to switch platforms, particularly in the setup of Pages and Events and for developers to change to the new coding style, however there are significant improvements and the likelihood is high that Optimizely will force the switch sometime in the future.
Ultimately, I’d recommend starting to plan for the switch and looking into the impact on your site over the next few months. There are huge benefits from Optimizely X and, despite some pain in changing, you will ultimately be using a far better platform for your testing.
What about other platforms?
When making the switch, it might get you considering other A/B testing platforms. With a typical Optimizely subscription now costing between $2,000/month and $4,000/month the big question is whether it’s still worth the money or whether other tools offer better value. In a future post I’ll be comparing the leading optimisation tools to find which is the best choice for your site.