This article will help you:

  • Understand how mobile optimization can have an impact on your mobile business
  • Create an Android or iOS experiment using the Optimizely Visual Editor
  • Create variations, target the experiment, allocate traffic, set goals, and preview the experiment

Welcome to Optimizely -- in this article, we'll get you set up and walk you through how to create an experiment for your mobile app in Optimizely's mobile Editor.

What is mobile optimization?

Put simply, mobile optimization is the process of discovering how you can deliver an experience that achieves your conversion goals -- like increasing daily active users (DAUs), in-app purchases (IAPs), downloads, registrations, ratings/reviews, or revenue.

Optimizely enables you to do this through A/B testing (running multiple versions of your experience to compare which is most effective) and delivering personalized, targeted content. We have a more in-depth description in our Getting Started Guide for Web, but let's focus on how optimization can make a big difference for your app.

Mobile usage is quickly outpacing web traffic in many areas, which means that increasingly, your main touchpoint with your users and customers is their mobile device, not their desktop. But, unlike the web, your releases are dependent on the App Store, which could make it tougher for you to stay agile while still providing the best possible experience. This is where testing and optimization fit in:

 

Tip:

Wondering what mobile app testing is all about? Check out our mobile optimization strategy guide and testing ideas for mobile apps!

Have more questions about Optimizely's mobile functionality? Visit our FAQ or check out our SDK install guide if you're a developer.

  • Acquisition is tough, so lifetime value (LTV) matters. By testing multiple experiences, you can discover what changes most affect engagement and retention through causal (not correlative) relationships.
  • App store approval cycles mean that you need a "kill switch." Platforms like Optimizely make it easier for you to modify and remove content that impairs the user experience.
  • Less code to maintain. When you make changes using a mobile optimization platform, you can modify content simply, without much (or any) code.
  • Mobile is real-time and high-touch. Unlike on the web, mobile users usually have a constant connection to their device, which makes apps stickier than the web. Optimizely enables you to implement real-time feedback and personalize content to increase stickiness and value.

Editing modes

Optimizely provides three mechanisms for crafting A/B tests:

Visual Editor

Modify your app without touching any code to change text, colors, and reposition existing elements. For example:

  • Make CTAs more prominent
  • Hide distractions or obstacles to your user flow

Live Variables

Modify the variables that govern UX and functionality including design, process flow, and when/how the app requests feedback. For example:

  • Change in-game physics
  • Display different number of articles
  • Request permissions at different times
  • Modify default search sorting

Code Blocks

Introduce new elements and change the sequence of content loads. For example:

  • Add floating modules
  • Change registration flow

This guide will walk you through setting up an Android and iOS experiment using the Optimizely Visual Editor. If you are using Interface Builder or Storyboards, the Visual Editor will be ready to use. However, if you are creating your views programmatically, some additional setup is required.

Setup and implementation

To get started, you will create an Android or iOS project from the Home page and work with your developer to install the Optimizely SDK.

 

Important:

Have you or your developer already installed the SDK and set up your mobile project? Then skip this section and start creating your experiment!

Installing the SDK enables you to connect your app (running on a device or simulator) with your Optimizely account. Once you make the connection, you can immediately use Optimizely's Visual Editor to make changes to elements in your app.

Here's what that process looks like:

  1. Go to the Home page, click the project menu icon () next to your project name, and click New Project.



  2. Select either an Android or iOS project, give it a name, and click Create project.
  3. From the Home page, look in the Settings tab to find your project ID and API key, which you'll use during implementation.



  4. Click on Install the SDK to begin, or email the instructions to your developer - they will be able to view the instructions whether or not they have an Optimizely account. You can either install the SDK using CocoaPods (recommended for iOS), Gradle (recommended for Android Studio users), Maven(recommended for IDEA/Eclipse users), or manually. If you're unfamiliar with how to install our SDK, send our iOS or Android guide to your developer.You will know if your SDK has been successfully installed when your Home page signals that Optimizely has detected your application and you can create a new experiment.
  5. (Optional) Optimizely provides rich analytics on our mobile Results Page, but you can also integrate with your mobile analytics framework (Android, iOS) to slice and dice your experiment data against the data you track in your analytics platform.
  6. (Optional) Set user roles and permissions to control levels of access to your mobile experiments. You can do this from the Collaborators tab in Optimizely.
 

Important:

Always remember the following when you edit an iOS experiment.

  • When connecting to the Editor, make sure that you have the correct project code in your application.
  • The device can have only one connection active at a time with the Optimizely Editor.
  • Your app (with the Optimizely SDK and project code installed) must be running while you launch and edit any mobile experiment.
  • If your app is live in the app store, you will not be able to connect to Optimizely's editor to create experiments. If you would like to enable this functionality, you can have your developer follow the instructions here. Note that if you enable this functionality, all your users will be able to put your app into edit mode if they draw the Optimizely 'O' in the app.


Congratulations!
Now you have Optimizely configured to create a mobile experiment. In the rest of this article, we'll walk you through how to actually create an experiment. Read on to find out how!

Create an Experiment in Five Steps

Create Variations

Once you have a project created and the SDK installed, you should be ready to start running experiments to generate more value from your mobile apps.
  1. In the Optimizely Application, select the project that you use for your iOS experiments and click Create Experiment.



  2. Once you have chosen a name and created the experiment, the Editor begins waiting for a device to enter Edit Mode; this can either be the mobile Simulator, like XCode iOS, or your physical device (that has a version of the app with the Optimizely mobile SDK installed).

     

    Tip:

    If you would like to connect to Optimizely from your device (with your app installed), please be sure to check with your developer that they implemented the steps listed in the iOS or Android guide.

    Draw the Optimizely "O" on your device to connect to the Optimizely Editor. If you are using an emulator, draw the "O" with your mouse.



  3. Once you have connected your device, it's time to create a variation.

Here's how to make changes in your variation:

  1. In your app, navigate to the screen on which you want to run an experiment.
  2. Now, in Optimizely's Visual Editor, make changes by clicking on elements from inside the browser.
  3. In the right drawer, you will see different visual changes you can make.
  4. The change will be immediately reflected in your app! Click Save to commit your change(s) to the variation.
 

Tip:

Did you follow the steps above? Congratulations, you've created an experiment using Optimizely's Visual Changes mode, represented by the arrow icon!
 
Continue exploring by selecting Live Variables or Code Blocks listed below Visual Changes. Don't see your variables and/or code blocks in the Editor? Make sure you've navigated through your app where those variables are being used.
 

 

Targeting

Optimizely’s Targeting capability lets you define which users will be included in your experiment or new experience. You can select and define any number of conditions available in Optimizely from the Options > Targeting menu.

Once you have defined the criteria that users must meet, click Done to return to the Editor. Only users who meet this criteria will be included and measured in the experiment or new experience running through Optimizely.

iOS:

 

Android:


The following conditions are available for targeting:

  • App version: The version of your app, as defined by you.
  • Custom tag: Functions identically to custom tags in Web experiments. For more information on how to add custom tags to your code, see our developer resources (Android, iOS).
  • Device model (iOS): The hardware on which the app is currently running (iPhone, iPod, iPad, iPhone/iPad Simulator).
  • Is an iPad / iPhone / retina device (iOS): Boolean value for if the current device is / is not an iPad, iPhone, Retina.
  • Is a tablet/ phone (Android): Boolean value for if the current device is / is not a tablet, phone.
  • NSLocaleIdentifier: The locale is formed from the settings for the current user’s chosen system locale overlaid with any custom settings the user has specified in System Preferences.
  • Language: The language that the device is using within its System Preferences; this is also a subset of NSLocaleIdentifier.
  • SDK version: The version of Optimizely's SDK.
  • System version: The iOS or Android version on the device currently running the app.

Traffic Allocation

By default, an equal percentage of your traffic will see each variation. If you want to change from the default, you can adjust the traffic allocation from the Options menu.

Traffic allocation applies only to users who meet the targeting conditions. Users who do not meet the targeting conditions will see the original experience and will not be included in the experiment.

 

Note:

Traffic allocation applies only to new visitors to the experiment. If someone has already seen the experiment, they will continue to see the same variation on repeat visits.

This does not need to add up to 100%

Goals

To get the most out of your experiment, you're going to want to add conversion goals that measure the most meaningful conversion events to your business. We automatically track all view controller transitions and screen taps so that they can easily be used as goals. In order to select a particular transition or screen tap as a goal, open the Goals dialogue and click Create a New Goal in the bottom right. The Create Goal dialogue will then appear.

You can select three types of goals for a mobile experiment:

  • Taps (iOS only, coming soon for Android): Tracks whether a certain view is tapped
  • Views(iOS only, coming soon for Android): Tracks whether a certain view appears
  • Custom events: Tracks visitor events or actions distinct from taps or views, including revenue. For more information on setting up revenue goals, check out our revenue goal guide. To learn how to set up other custom goals, check out our custom goals guide.
 

Note:

You will need to ensure that any custom goals you wish to track are predefined in your app prior to releasing your app to the app store.

As you interact with your app, all the view transitions and taps that occur will instantaneously appear on the Create Goal screen. Select the one that you want to be your goal, give it a name, and click Save. Optimizely will now track the percentage of your users in each variation that complete that action and the results will appear in our dashboard.

Preview

Always preview your experiments to make sure that the variations show up as expected before they go live. Preview Mode simulates a particular variation just as the user will experience it in production.

To use Preview:

  1. Select the variation you want to see and click the Variation Settings icon.
  2. This will cause the app to shut down.
  3. When you reopen the app, it will have all the changes you specified in your variation.
  4. To exit Preview, tap on the Preview Status bar from your device or simulator, and you should see an option to exit Preview mode.

 

Tip:

Elements not showing up as expected in the Visual Editor? Try navigating to the view controller you are trying to modify. If it's still not showing up, open the console and send us the data in a support ticket!

 

Debug Preview

Debug Preview works the same way as Preview Mode, except that logs from the device are sent to the editor to help you debug your app. This can help you verify that goals you have set up are triggering. If they do trigger, they will show up in the debug log. The messages that appear here are a identical to those printed to the console in the mobile editor (Xcode) when verboseLogging is turned on, less the messages sent before the device reconnects to the editor. To run in Debug Preview mode, select the variation you want to see and select Debug Preview from the dropdown menu.

A few final best practices as you set up more mobile experiments:

  • Plan ahead and loop in your developers. Unlike the web, mobile optimization has a longer lead time, so work with your developers in advance to identify what you want to test. That way, you can tag views for the Visual Editor or identify Live Variables/Code Blocks that you want to change.
  • Look at the developer build and approve changes mutually. When you preview and QA experiments, go beyond Optimizely's Preview mode and make sure to actually review your developer build, including test cases, to ensure that the experiment is working as planned.
  • Wash, rinse, repeat. Once you're ready to launch your experiment, you can release to the app store and click Start Experiment in Optimizely. After that, start planning your future experiments in your development sprints!

What's next?

After installing your SDK and following the five steps above, you’re now ready to collect results! You'll do this on Optimizely's Results Page, and our article on interpreting results gives you the full run-down on how you can view and interpret your results data. Here's a sneak peek of the results page:

Once Optimizely has enough data, it will declare a winner out of all the variations to show you, with real visitor data, which version of your page is best at driving the conversion goal. Aren’t you glad you tested?

That's the end of our Getting Started Guide for mobile Optimization, but as you create your first experiments and learn more about experience optimization, we encourage you to further check out the Knowledge Base for more detailed technical and strategic guidance.