Innovation for impact: Results from Beyond the Stars

Over the past two years, the team here at S1T2 has worked tirelessly to conceptualise an innovative new approach. The result was Beyond the Stars (BTS), a world-first health education program using storytelling and technology to inspire children to adopt healthy eating habits.

While we’ve designed BTS to be part of a long-term solution to the NCD crisis, we first needed to prove that our approach would actually work. To do that, we needed a pilot.

The pilot

For those that may not know, a ‘pilot program’ is a small-scale, short-term experiment that helps determine whether an idea is likely to be successful, and how a larger-scale implementation might work. In our case, we’d be using the BTS pilot to achieve three main goals:

• Demonstrate how transmedia storytelling and innovative technologies are an effective method for educating children and fostering behaviour change.
• Increase children’s knowledge of and awareness about healthy and local food options.
• Inspire a shift in children’s dietary preferences towards healthy food.

Achieving these goals would not only demonstrate the potential impact of our approach to NCDs, but also give us invaluable insights into how this kind of program could work and scale in the future.

Run over the course of five weeks with 300+ children across nine Fijian primary schools, the BTS pilot involved six different components, all working together to create a cohesive educational experience:

• Engagement with schools and communities to co-design the program and foster local ownership
• Teacher training workshops and materials on using storytelling and technology in the classroom
• An animated short film shown at the beginning of the program to introduce children to the narrative and world of BTS
• A virtual reality experience inviting children to become the hero of their own story
• An educational storybook that children can learn from both at school and at home
• An interactive game that uses tangential learning on portable devices to inspire self-motivated learning

I won’t go into each of these elements individually, but if you’re interested in learning more about them, you can take a look at this blog post or watch the video below.

Monitoring + evaluation

Monitoring and evaluation (also known as M&E) is the process by which the results of a pilot program are measured and interpreted. The overall goal with M&E is to use data and evidence to improve performance, both of the project in question and other projects into the future.

While it might sound counterintuitive, this evaluation process actually starts at the very beginning – ideally at the same time that a project is being conceptualised. That’s important because a key part of the process is developing a Theory of Change (ToC). A ToC is a framework that outlines how an idea or project could achieve its desired impact. They’re especially important when you’re aiming to foster behaviour change (for example, inspiring children to eat healthier).

Here’s our ToC for Beyond the Stars:

If that looks pretty scary, don’t worry. All you need to know is that it lays out the path for BTS to create change, and gave us a clear way to measure that change at the end. If that picture looks super exciting, or you’d just like to know more, get in contact and we’ll send you a copy of our M&E Report.

Anyway, based on the ToC, we then came up with a series of evaluation questions that would help us figure out if we’d achieved the kind of change we intended. Those questions were centred around three main goals:

• Understand the reach and outcomes of the pilot (effectiveness).
• Understand what did and didn’t work around using transmedia storytelling and innovative technologies to address issues of healthy eating and nutrition (appropriateness).
• Understand what has been learnt about doing innovation in the Pacific Islands (learning).

To answer those questions, we designed a comprehensive measurement process that included child surveys, interviews with parents and teachers, data from the mobile game and reviews from our own team. The amount and diversity of data points here is crucial. For a small pilot like ours, it can be quite hard to determine impact. Having lots of different data sets would allow us to cross check our results and ultimately have more confidence in our findings.

While I’ll just be talking about the end results in this article, it’s important to note that we didn’t only collect data at the end of the pilot. We actually conducted seven field visits throughout the pilot’s development and implementation. These visits were super important in terms of ensuring our program had the best chance at success (but more on that later).

The results

Now the part you’re all here for: the results. At the end of the pilot, there were a number of notable outcomes relating to our aforementioned Theory of Change. In this section I’ll take you through some of them, though any data buffs out there who want to dig deeper should, as I said, get in touch.

Children had an increased ability to identify healthy and unhealthy foods.
Looking at the quantitative child surveys, it was found that participating children had, on average, an 11% increase in their ability to identify healthy foods, and a 10% increase in their ability to identify unhealthy foods. This was supported by qualitative interviews with teachers and parents: “Now they are able to identify what foods to eat and what not to eat” (participating parent).

Children demonstrated a shift in their attitudes towards healthy foods.Again looking at the quantitative child surveys, statistical analysis found that participating children were, on average, 14% more likely to want to eat healthy foods and 37-58% less likely to want to eat unhealthy foods. Teachers and parents were also confident about this change: “It is really working a lot, and it has changed our students’ attitude, especially eating good meals” (participating teacher).

Children appeared to experience a positive change in their eating behaviours.
Moving to the qualitative interviews now, 7 out of 9 (78%) of teachers reported observing a positive change in children’s eating behaviours as a result of the program: “Every lunchtime when we go and supervise them at lunch, we can see that most of them will be holding a banana in their hand” (participating teacher).

Teachers valued the use of transmedia storytelling and innovative technologies in teaching.
All of the teachers that we spoke to during qualitative interviews believed that the program’s approach of using storytelling and technology in the classroom was beneficial for child learning. Indeed, 6 out of 9 (67%) of teachers suggested that this approach was more effective than traditional teaching methods: “I believe that using technology and stories together, hand-in-hand, will really help the children learn and improve their learning” (participating teacher).

There is potential for parents and families to experience attitude change as a result of the program.
During the qualitative interviews, 38 out of 43 (88%) parents interviewed reported that their own attitudes around healthy eating had changed as a result of the program. This was an unexpected outcome, but demonstrates the potential of the program to have a wider sphere of influence: “My child has really taught me a lot about the choice of food we eat” (participating parent).

What’s next?

At the moment, we’re working on expanding Beyond the Stars to even more schools across Fiji and, hopefully, the wider Pacific Islands region. If you’re interested in helping Beyond the Stars expand, send us an email at creative@s1t2.com.