We were lucky enough to have a group session last week with Vaithegi Vasanthakumar, from the global efficacy team of the Office of the Chief Education Officer at Pearson, in preparation for this week’s residential in London. Pearson is working towards delivering efficacy in its products and services. The company defines efficacy as having “a measurable impact on improving people’s lives through learning” – products that are based on robust research and shown to deliver meaningful, measureable outcomes. By 2018, the company will report publicly on the learner outcomes that their products will deliver. As Vaithegi put it, Pearson is “putting a stake in the ground” and holding themselves accountable.
For everything that the company builds, they ask themselves: How will this impact the learner at the end of the day? The main takeaway from this session is that educational products can be more effective if the people building them focus on outcomes instead of outputs. Vaithegi explained to us what the difference between outputs and outcomes is, how to define learning outcomes for a specific product, and then how to measure those outcomes and assess their impact.
Outputs vs. Outcomes
When you’re building a product, it’s easy to focus on outputs because that’s often what you offer the users – the features of your system. However, Vaithegi explained that it is important to consider the features that you design in relation to what they do for the learner. Sometimes the features that are popular and perhaps even sell well may not be truly helping your end users (learners or educators) to achieve the learner outcomes that would most help them. Outcomes, on the other hand, are what we impact through the product - like the acquisition of skills and knowledge, changed behaviour, or eve changed circumstance.
Vaithegi outlined the four outcomes categories that Pearson uses:
- The learner access and experience (i.e. increased confidence gained by the learner, level of learner satisfaction or the degree to which learners can accesss a product)
- The timelines of completion (i.e. how long it takes to complete a course)
- The standard of achievement or level of competence (i.e. acquisition of specific skills, qualification gained, performance on tests)
- Learner progression (i.e. the learner’s ability to progress to the next course of study, training, or employment)
Vaithegi pointed out that with the availability of learner data through the use of digital technology, “the time is ripe for a rigorous focus on outcomes across the education industry.” So, how can you integrate an outcome-focused approach to your product design?
How to define and measure outcomes for your product
1. Start with the learner – what is the vision/aspiration for the learner?
The first step is to truly understand your end user - the learner - and their aspirations. Identify the learner profile for your target user. Based on that, decide on the key features of your product or service. Remember, it’s not about the customer; it’s about the end learner.
2. Set measurable outcomes
Once you have a clear idea of who your user is and what their aspirations are, decide on a list of specific outcomes that embody those aspirations. They should be concrete and measurable. Here are some questions you can ask yourself to help come up with these outcomes:
- What could define the learner access and experience?
- How should timelines and completion be defined?
- What competency or standard of achievement can be attached to the learning?
- How should we capture learner progression?
An important tip is to prioritize your outcomes and select your top five or six to start with.
3. Identify metrics
The next step is to identify indicators that measure your outcomes. The metrics you choose should be data that you can accurately collect with your available tools (or can build tools to collect), and also reflect the outcomes that you chose.
4. Set targets
The last step is to set targets that you can then aim to achieve with your product. The example that Vaithegi gave is learning English as a second language. If your outcome is that learners become conversational in English, you could use test scores as one of your metrics and set a minimum average score as a target.
One of the questions that Tomás of Klass Data asked was about common learning outcomes that diverse education actors could all measure using standardized metrics. Vaithegi replied that there are no rigorous industry standards yet, but recommended several resources where we can find out more about current practices for measuring impact in the education sector.
- efficacy.pearson.com – learn more about Pearson´s approach and use their online interactive efficacy tool
- research.pearson.com – examples of excellent research and summaries of full research papers
- Nesta and Pearson’s report, “From From Good Intentions to Real Impact: Rethinking the role of evidence in education businesses” – page 16 includes a summary of the different levels of evidence and page 24 on how standards of evidence can be used by education start-ups and education accelerators