THIS ARTICLE WILL HELP YOU:
- Understand the purpose of a testing charter
- Decide what is important to include in your charter
- Build your own testing plan using our starter template
Imagine the following scenario: You recently purchased a new A/B testing platform for your company. You have high hopes for showing a great return on your investment after having spent weeks (or maybe months) deciding on the platform, negotiating the contract’s duration and cost, and hiring a new employee (or transitioning a whole team) to use it. The platform’s code has been installed and it’s now possible to begin testing.
The question is, how ready are you? From the initial consideration phase through to purchase and installation, it’s the uncommon company that has fully considered the necessary details of how they’re going to make the most effective use of the platform. But if you immediately jump into testing, in the weeks and months that follow you may find yourself facing:
- Inefficient test planning and setup
- Tests launched with errors or mistakes
- Tests with missing goals or audiences
- Low testing velocity
- Poor communication between team members
- Tests that generally have no impact
- Missed opportunities to test
- An inability to effectively learn from test results
Each of these scenarios can be avoided or attenuated with a little planning before you get started. Broadly speaking, documenting a game plan will allow for efficient, consistent, and meaningful testing. A testing charter helps you track the mission, resources, and workflow that govern your optimization strategy.
WHAT YOUR TESTING CHARTER SHOULD INCLUDE
Your testing charter (which you might document in a company wiki, a Word document, a Google Doc, etc.) may include a brief mission statement, but should definitely include many practical, referenceable details. When you create it, imagine a new employee, senior company leader, or anyone at your company who doesn't know about the testing program. For him or her, the testing charter is a great place to start learning about the program. Your document doesn’t need to include everything from the list below, but the more thorough you are based on the needs of your team, the better off you'll be.
The testing charter should be a working document. It will never be finished. The testing Program Manager (or equivalent) should own the charter. One or two months into the testing program, the Program Manager should revisit the document to see how the actual process compares to this plan. Depending on what’s working and what’s not, this is a good time to modify the plan. From there, the Program Manager can ensure that the program is meeting its goals by reviewing and updating the charter on a quarterly or semi-annual basis.
To get started, download our testing charter template. You can copy and modify this template in the program of your choice. Below you’ll find more detailed explanations for each section of the document.
- Introduction: Set the context with a paragraph of introduction. Why did you purchase the Optimizely testing platform? What problems were you trying to solve? What are the long-term goals and expected outcomes? Define the purpose for the rest of the document.
- Team: Define the testing team including the executive sponsor, power user, developer, designer, QA tester, and approver. Also include a description of their roles as defined by the Optimizely platform privileges .
- Process: What is the workflow from conception, execution, results-sharing, through to (potentially) onsite implementation? Create a workflow diagram in Keynote or PowerPoint and insert the image here.
For each of the steps you outline, add a brief description and a list of key participants in each step.
- Meetings & Meeting Cadence: When do you have testing brainstorm meetings? When do you have status meetings to check on upcoming tests and review currently running and recently completed ones? How often does the testing team meet with a broader group?
- Test Velocity: What is the goal for testing velocity: One test always running, ten tests per week, twenty five per quarter? Whatever it is, set an initial goal and modify this as needed to ensure that it’s reasonable based on the actual pace of testing at your company.
- Test Plan & Results Templates: Testing programs that are efficient, run meaningful tests, and act on the lessons of each test create testing documentation. Create a link to the most-updated testing template the company uses to document test plans and results.
- Links Key: Your links key is where you link to all of other essential information including your current testing roadmap, templates (described above), and an archive of test results. If team members bookmark this page, they should easily be able to find any other relevant information they need. You may also want to link to pages that document definitions of the following:
- Test Goals, for example,“Purchase Confirmation - a pageview goal targeted to http://www.example.com/thank_you_page”
- Targeting Conditions, for example, “Product Pages - targeted with a substring match to example.com/productID”
- Audiences and Dimensions, for example, “Category Affinity defines users who are interested in a particular category and bases it off of a recent product or category pageview associated with that category”
- Other Guidelines & FAQs: What guidelines and rules will your team follow when it comes to testing? Do different types of tests have different processes for QA and approval? Are there standard goals included in every test? When will a winning variation be served through the Optimizely platform and when will it be hardcoded to the site?