This Section is written by Paul Hogan with Debbie Marsden, both practising youth and community workers
Project evaluation: in search of the qualitative evidence
In the youth and community sector detailed evaluation has become an increasingly important expectation for both the statutory and voluntary organisations. Developing credible evaluation tools and techniques can often therefore be the difference between a project continuing or ending.
Such evaluations need to include both qualitative and quantitative evidence, with the balance between the two depending on the priorities of key stakeholders – most often those responsible for providing the money. Since the 1990s and the emergence of New Labour policies there has been an ever increasing demand for quantitative data for measuring a project’s success. Although these measurements can provide some credible information, many in the sector see them as failing to give a true account of a project’s ability to meet the expectations or requirements of the young people who access it.
This section focuses on a process for producing qualitative data which, supported by quantitative data, can be used to develop a more realistic evaluation of a project’s activity. By increasing the amount of this qualitative evidence to accompany the required quantitative data, it can provide a more detailed, rounded and multi dimensional view of the work undertaken. In doing this, it may also help educate those in positions of power – particularly funding bodies – about the influential role a project can play in the lives of its users.
My own use of this form of project evaluation emerged from a firsthand experience of the benefits of storytelling as a tool for personal reflection and development (See The experience of a first-time story-teller … and … and of a first-time facilitator). This led me to consider how, in a very organic way, this could be developed to enhance and highlight key features of the project I was running. Adopting a very similar structure to that used in the IDYW story-telling workshops (see Story-telling with experienced youth workers), it aimed to enable the young people – many of whom had been part of the project for some years – to recall and recite their journey through the project and produce their personal story of engagement.
Involving young people
The first stage was to ask the young people if they would be willing to share and discuss their involvement and journey through the project – something that would of course have to be completely voluntary. The request brought a positive response, with many of the young people apparently feeling that they had a positive story to tell. Some who were not yet feeling confident to share their own story still wanted to take part in the workshops to support the others – with some going on to offer their story once they felt comfortable within the workshop environment.
Once it was clear that we had a group who were willing to share their stories, the youth workers involved in the project identified a smaller group who, they felt, were capable of taking on a facilitator’s role. They were able to develop these skills in real scenarios sharing stories with each other with support from independent professionals from IDYW. This enabled us to minimise the involvement of adults from the project who might influence what the young people had to say, leaving them more comfortable to open up. The facilitators were made aware of the IDYW cornerstones (see Open Letter) to help them identify these in their stories and focus on certain key areas of the project’s practice. However, their priority was to explore the young people’s experiences and encourage them to ask questions about their journey as they would explain it rather than the journey that we, the youth workers, might recognise.
Generating the stories
The group of facilitators took responsibility for one story each, with another facilitator acting as note taker. Young people would narrate their story with other group members, supported by the facilitator, interjecting when they felt the teller could give more detail on a specific topic. After each workshop the story teller would sit with the note taker and ensure that the notes were a full and true reflection of the story as presented and ‘unpicked’. The stories were then compared and contrasted with the projects aims and objectives as well as with the IDYW cornerstones around which we had shaped the project ethos. This enabled us to produce a narrative which not only showed if and, if so, how we had met the project’s aims. It also sought to clarify how the work had impacted on the individuals involved.
For example:
I first heard about the Walton Youth Project through a cookery class in school via a youth worker. She encouraged me to go along as I wasn’t the most confident of people. It was also a chance to get out of the house. When I first arrived I only knew a few people there who were from my year group in school and I also knew one of the older young people. The project is the place to go to do things you wouldn’t normally do in your own house.
I think I would be lost without the project it gives me a purpose, something I enjoy being a part of.
The project gave a chance to socialise which I don’t feel school always offered… I really enjoyed the residentials and I have been on many in my time involved with the project for lots of different reasons from team building to consultation about the project.
Alongside these personal stories we also worked on project stories. As the project coordinator I identified the significant milestones within the project which were not necessarily the same as those identified by the young people. Having recognised this, we then produced stories for each year of the project using similar methods to those described above, though via group discussion rather than focused on individuals. We undertook this task with a number groups who had been engaged at different periods and whose milestones would therefore have been different or, depending on the individuals involved, occurred at different points in their involvement. These stories, supplemented by that of the staff team, gave a fuller and broader representation of the project’s overall development through a five year funding cycle.
Reflections
My aim in this Section has not been to present a definitive way of using storytelling as an evaluation tool. Rather, it has been to describe how it was used in one project to produce what I feel, and the funders also felt, was a body of evidence which gave a fuller and truer reflection of the work we have undertaken. Hopefully this will encourage others to consider producing detailed narratives within evaluations for funders who are interested in and receptive to the ethos of their organisation. Or, for those funders who put a greater emphasis on measurable outcomes, it could be an opportunity to broaden their views on what to be seeking about the relationship-based journey which young people take within a youth work project.
Story-telling in Youth Work 