This article will help you:
- Understand how mobile optimization can have an impact on your mobile business
- Create an Android or iOS experiment using the Optimizely Visual Editor
- Create variations, target the experiment, allocate traffic, set goals, and preview the experiment
Welcome to Optimizely -- in this article, we'll get you set up and walk you through how to create an experiment for your mobile app in Optimizely's mobile Editor.
What is mobile optimization?
Put simply, mobile optimization is the process of discovering how you can deliver an experience that achieves your conversion goals -- like increasing daily active users (DAUs), in-app purchases (IAPs), downloads, registrations, ratings/reviews, or revenue.
Optimizely enables you to do this through A/B testing (running multiple versions of your experience to compare which is most effective) and delivering personalized, targeted content. We have a more in-depth description in our Getting Started Guide for Web, but let's focus on how optimization can make a big difference for your app.
Mobile usage is quickly outpacing web traffic in many areas, which means that increasingly, your main touchpoint with your users and customers is their mobile device, not their desktop. But, unlike the web, your releases are dependent on the App Store, which could make it tougher for you to stay agile while still providing the best possible experience. This is where testing and optimization fit in:
- Acquisition is tough, so lifetime value (LTV) matters. By testing multiple experiences, you can discover what changes most affect engagement and retention through causal (not correlative) relationships.
- App store approval cycles mean that you need a "kill switch." Platforms like Optimizely make it easier for you to modify and remove content that impairs the user experience.
- Less code to maintain. When you make changes using a mobile optimization platform, you can modify content simply, without much (or any) code.
- Mobile is real-time and high-touch. Unlike on the web, mobile users usually have a constant connection to their device, which makes apps stickier than the web. Optimizely enables you to implement real-time feedback and personalize content to increase stickiness and value.
Optimizely provides three mechanisms for crafting A/B tests:
Modify your app without touching any code to change text, colors, and reposition existing elements. For example:
Modify the variables that govern UX and functionality including design, process flow, and when/how the app requests feedback. For example:
Introduce new elements and change the sequence of content loads. For example:
This guide will walk you through setting up an Android and iOS experiment using the Optimizely Visual Editor. If you are using Interface Builder or Storyboards, the Visual Editor will be ready to use. However, if you are creating your views programmatically, some additional setup is required.
Setup and implementation
To get started, you will create an Android or iOS project from the Home page and work with your developer to install the Optimizely SDK.
Installing the SDK enables you to connect your app (running on a device or simulator) with your Optimizely account. Once you make the connection, you can immediately use Optimizely's Visual Editor to make changes to elements in your app.
Here's what that process looks like:
- Go to the Home page, click the project menu icon () next to your project name, and click New Project.
- Select either an Android or iOS project, give it a name, and click Create project.
- From the Home page, look in the Settings tab to find your project ID and API key, which you'll use during implementation.
- Click on Install the SDK to begin, or email the instructions to your developer - they will be able to view the instructions whether or not they have an Optimizely account. You can either install the SDK using CocoaPods (recommended for iOS), Gradle (recommended for Android Studio users), Maven(recommended for IDEA/Eclipse users), or manually. If you're unfamiliar with how to install our SDK, send our iOS or Android guide to your developer.You will know if your SDK has been successfully installed when your Home page signals that Optimizely has detected your application and you can create a new experiment.
- (Optional) Optimizely provides rich analytics on our mobile Results Page, but you can also integrate with your mobile analytics framework (Android, iOS) to slice and dice your experiment data against the data you track in your analytics platform.
- (Optional) Set user roles and permissions to control levels of access to your mobile experiments. You can do this from the Collaborators tab in Optimizely.
Always remember the following when you edit an iOS experiment.
- When connecting to the Editor, make sure that you have the correct project code in your application.
- The device can have only one connection active at a time with the Optimizely Editor.
- Your app (with the Optimizely SDK and project code installed) must be running while you launch and edit any mobile experiment.
- If your app is live in the app store, you will not be able to connect to Optimizely's editor to create experiments. If you would like to enable this functionality, you can have your developer follow the instructions here. Note that if you enable this functionality, all your users will be able to put your app into edit mode if they draw the Optimizely 'O' in the app.
Congratulations! Now you have Optimizely configured to create a mobile experiment. In the rest of this article, we'll walk you through how to actually create an experiment. Read on to find out how!
Create an Experiment in Five Steps
Create VariationsOnce you have a project created and the SDK installed, you should be ready to start running experiments to generate more value from your mobile apps.
- In the Optimizely Application, select the project that you use for your iOS experiments and click Create Experiment.
- Once you have chosen a name and created the experiment, the Editor begins waiting for a device to enter Edit Mode; this can either be the mobile Simulator, like XCode iOS, or your physical device (that has a version of the app with the Optimizely mobile SDK installed).
- Once you have connected your device, it's time to create a variation.
Here's how to make changes in your variation:
- In your app, navigate to the screen on which you want to run an experiment.
- Now, in Optimizely's Visual Editor, make changes by clicking on elements from inside the browser.
- In the right drawer, you will see different visual changes you can make.
- The change will be immediately reflected in your app! Click Save to commit your change(s) to the variation.
Optimizely’s Targeting capability lets you define which users will be included in your experiment or new experience. You can select and define any number of conditions available in Optimizely from the Options > Targeting menu.
Once you have defined the criteria that users must meet, click Done to return to the Editor. Only users who meet this criteria will be included and measured in the experiment or new experience running through Optimizely.
The following conditions are available for targeting:
- App version: The version of your app, as defined by you.
- Custom tag: Functions identically to custom tags in Web experiments. For more information on how to add custom tags to your code, see our developer resources (Android, iOS).
- Device model (iOS): The hardware on which the app is currently running (iPhone, iPod, iPad, iPhone/iPad Simulator).
- Is an iPad / iPhone / retina device (iOS): Boolean value for if the current device is / is not an iPad, iPhone, Retina.
- Is a tablet/ phone (Android): Boolean value for if the current device is / is not a tablet, phone.
- NSLocaleIdentifier: The locale is formed from the settings for the current user’s chosen system locale overlaid with any custom settings the user has specified in System Preferences.
- Language: The language that the device is using within its System Preferences; this is also a subset of NSLocaleIdentifier.
- SDK version: The version of Optimizely's SDK.
- System version: The iOS or Android version on the device currently running the app.
By default, an equal percentage of your traffic will see each variation. If you want to change from the default, you can adjust the traffic allocation from the Options menu.
Traffic allocation applies only to users who meet the targeting conditions. Users who do not meet the targeting conditions will see the original experience and will not be included in the experiment.
Traffic allocation applies only to new visitors to the experiment. If someone has already seen the experiment, they will continue to see the same variation on repeat visits.
This does not need to add up to 100%
To get the most out of your experiment, you're going to want to add conversion goals that measure the most meaningful conversion events to your business. We automatically track all view controller transitions and screen taps so that they can easily be used as goals. In order to select a particular transition or screen tap as a goal, open the Goals dialogue and click Create a New Goal in the bottom right. The Create Goal dialogue will then appear.
You can select three types of goals for a mobile experiment:
- Taps (iOS only, coming soon for Android): Tracks whether a certain view is tapped
- Views(iOS only, coming soon for Android): Tracks whether a certain view appears
- Custom events: Tracks visitor events or actions distinct from taps or views, including revenue. For more information on setting up revenue goals, check out our revenue goal guide. To learn how to set up other custom goals, check out our custom goals guide.
You will need to ensure that any custom goals you wish to track are predefined in your app prior to releasing your app to the app store.
As you interact with your app, all the view transitions and taps that occur will instantaneously appear on the Create Goal screen. Select the one that you want to be your goal, give it a name, and click Save. Optimizely will now track the percentage of your users in each variation that complete that action and the results will appear in our dashboard.
Always preview your experiments to make sure that the variations show up as expected before they go live. Preview Mode simulates a particular variation just as the user will experience it in production.
To use Preview:
- Select the variation you want to see and click the Variation Settings icon.
- This will cause the app to shut down.
- When you reopen the app, it will have all the changes you specified in your variation.
- To exit Preview, tap on the Preview Status bar from your device or simulator, and you should see an option to exit Preview mode.
Elements not showing up as expected in the Visual Editor? Try navigating to the view controller you are trying to modify. If it's still not showing up, open the console and send us the data in a support ticket!
Debug Preview works the same way as Preview Mode, except that logs from the device are sent to the editor to help you debug your app. This can help you verify that goals you have set up are triggering. If they do trigger, they will show up in the debug log. The messages that appear here are a identical to those printed to the console in the mobile editor (Xcode) when
verboseLogging is turned on, less the messages sent before the device reconnects to the editor. To run in Debug Preview mode, select the variation you want to see and select Debug Preview from the dropdown menu.
A few final best practices as you set up more mobile experiments:
- Plan ahead and loop in your developers. Unlike the web, mobile optimization has a longer lead time, so work with your developers in advance to identify what you want to test. That way, you can tag views for the Visual Editor or identify Live Variables/Code Blocks that you want to change.
- Look at the developer build and approve changes mutually. When you preview and QA experiments, go beyond Optimizely's Preview mode and make sure to actually review your developer build, including test cases, to ensure that the experiment is working as planned.
- Wash, rinse, repeat. Once you're ready to launch your experiment, you can release to the app store and click Start Experiment in Optimizely. After that, start planning your future experiments in your development sprints!
After installing your SDK and following the five steps above, you’re now ready to collect results! You'll do this on Optimizely's Results Page, and our article on interpreting results gives you the full run-down on how you can view and interpret your results data. Here's a sneak peek of the results page:
Once Optimizely has enough data, it will declare a winner out of all the variations to show you, with real visitor data, which version of your page is best at driving the conversion goal. Aren’t you glad you tested?
That's the end of our Getting Started Guide for mobile Optimization, but as you create your first experiments and learn more about experience optimization, we encourage you to further check out the Knowledge Base for more detailed technical and strategic guidance.