Measuring Connection: Evaluating Land-Based Programs

Debbie DeLancey, Ioana Radu, Lawrence Enosse, & Stephen Ritchie

Debbie DeLancey
Welcome to the panel discussion on evaluation. I’m Debbie DeLancey and I’m moderating this panel, not because I work for the department but because it brings together my two passions, which are being on the land and evaluation.

I’ve lived across the NWT, so I’ve worked and lived in what’s known as Nunavut in Baker Lake and, like my minister, lived in Frobisher Bay, which doesn’t exist anymore [Editor’s note: Frobisher Bay is now called Iqaluit]. I spent several years living in Fort Good Hope, which is where my former partner was from, and had and raised my kids there. So, although I’ve been living back in Yellowknife for many years, I’m really tied to this land through my children’s Dene heritage and think of Fort Good Hope as my other home. Before I worked for government I worked for the Dene Nation and I worked for the Inuit Tapirisat, which is the precursor organization to N.T.I. [Nunavut Tunngavik Incorporated]. I spent a lot of time working with Good Hope and communities in the Sahtu doing traditional knowledge research, participatory action research, and trying to work on developing community-based Indigenous methodologies. But the most important thing, and the thing that I learned the most from, is that when I lived in Good Hope I was fortunate enough to spend many seasons on the land, fall trapping and spring hunting with my partner and my extended family, and I had first-hand experience of being on the land, of observing the healing power of the land on other people, and experiencing that healing power myself. And, so, in my time working for government I’ve always tried – not always successfully, I’m the first one to say – to bring a perspective of community knowledge, community wisdom, into policymaking.

So that’s a passion of mine, but my other passion is evaluation. My academic credentials are in program evaluation and, as a public servant, because I’ve worked for government for almost 30 years, I really feel it’s important. We’re spending your money – we’re spending the public’s money. We need to know if we’re having an impact on people’s lives and making a difference. When I saw the on-the-land conference evaluation stream, I just said to the guys, I really need to be part of this. So that’s me, and I’m very excited about the panel we have here today.

One of the reasons people haven’t trusted evaluation and evaluators is because, like in many areas of research, evaluators have tended to come out with academic methods that work in an urban context, that work in a university setting, and tried to impose them on communities.

I’m going to throw out three challenges to you that I would like people to think about as they listen to the evaluation presentations, and these are challenges that I’ve run into in my work trying to be an evaluator or work with evaluators in government, working in Indigenous communities. The first challenge is being really clear on who the evaluation is for – a lot of people don’t trust evaluators and they don’t trust evaluation, and historically governments have used evaluation as a tool of colonization. There’s been a power imbalance and it’s been: “We’re going to evaluate your programs and then, based on how we think you’re doing with our evaluation, will decide whether to give you some more money.” There is a new kind of awareness developing among evaluators of what one of my colleagues called “evaluation without dominion”, which is this idea that government and Indigenous governments have to co-create evaluation if it’s going to work. The second challenge I’ve run into is the perception that you can’t evaluate land-based programs. Last week I was making a presentation to our Health and Social Services Leadership Council, and when I talked about this conference one of the regional leaders said, “You can’t evaluate land-based programs. That’s ridiculous. We all know what it feels like when you get out of town and you get on the land. How do you measure that?” You hear, “Well this guy went on an on-the-land program and he’s still drinking so the program failed. Why should we fund it?” But obviously there are dimensions that we can evaluate, and I’m really looking forward to hearing what these folks have done and then hearing from you where we, in the Northwest Territories, should move forward. And then the last question is just one of methodology. One of the reasons people haven’t trusted evaluation and evaluators is because, like in many areas of research, evaluators have tended to come out with academic methods that work in an urban context, that work in a university setting, and tried to impose them on communities.

Why does government keep measuring the things that are important to government and not work with communities to measure what’s important to them? I think that lack of respect for traditional knowledge and community values has been at the root of the failure of a lot of evaluations. So those are things that I’m curious to know more about.

 

Debbie DeLancey (far right), former Deputy Minister of Health and Social Services for the Northwest Territories, introduces (from left) panelists on the topic of evaluation, Ioana Radu, Lawrence Enosse, and Stephen Ritchie. Photo Credit: Pat Kane

 

Ioana Radu
In a way, evaluation is not always about counting. We have to think of evaluation in terms of changing the mindset and translating the priorities of the community into the kind of language that can translate across governmental and funding contexts. A lot of the time when we think about evaluation, we think about passing a judgement. We think about judging the merit, worth, and significance of a program or a service. Often and a lot of the time, that judgement brings about this feeling that is punitive, that the standards set by others are imposed on communities, that we have to meet them, that there’s always this gap that we have to catch up with. But in many ways these standards are not necessarily something that we want to catch up with because each community, each program, and each service have their own priorities, their own indicators of measuring success, and indeed their own understanding of what success is.

Usually evaluation is used, first of all, to help make decisions, but also to make new knowledge, to understand what it is that we’re doing and understand it in a good way. So, I borrowed this phrase, “Coming to know,” to replace the word “evaluation.” By changing the language – John B. Zoe was talking about how we have to change the language, Phil Gatensby was talking about a language of connection – in thinking about evaluation, we can also think about this idea that we come to know something, that we try to learn and co-create something new from something that already exists, that is connected to each community’s history and cultural contexts and with the knowledge that exists and that was created from the interactions with the ancestral territories. If we apply this approach then, when I think about evaluation, I think about a group of people who come together. They all bring something that they need and that they want to see happen, and work together to make that happen. Evaluation should also be about that: a collective process that the community is asking for; that the community is leading; that the community has control over. In this way, Indigenous knowledge and knowledge keepers in our communities have a foundational part to play in any evaluation process because they can connect the language, the values, and the priorities of the community to the evaluation process.

Lawrence Enosse
Good morning. I’d like to thank you, Creator, for a beautiful day and beautiful week up here. I’d like to thank the community Elder for starting the summit off in a good way – that was very, very awesome to be a part of the ceremony. I’d also like to acknowledge the traditional territory of the Dene people and for welcoming us to network and share in their community and also the great hospitality. My name’s Lawrence. I’m a band member, an Ojibway of the Wikwemikong Unceded Territory. I’m a son, I’m a dad, I’m a brother, and grandson, and what I do is I manage youth programming. I’m also an elected member of my community; I’ve been there for four terms and I’ve held various portfolios. And I’m really excited to be here today to talk about our adventure leadership experience program with our co-presenter Stephen Ritchie.

What we’re going to do first before we talk about evaluation is we’re going to talk about the connection from a First Nation community to evaluation. As indicated in the introduction, there are some barriers to that. So Wikwemikong is located on Manitoulin Island in Northern Ontario. It’s the largest freshwater island in the world and we’re located about two hours from the greater city of Sudbury, which has Laurentian University, Collège Boréal, and Cambrian College, so those are the closest post-secondary institutes to our community, and it’s also the largest urban area to our community. We are kind of rural, we are quite remote, but not like the Northwest Territories. We do have our barriers and challenges in Ontario. As indicated, I’m the Brighter Futures manager for the Waasa Naabin Community Services Centre and my team runs programs and services for children aged 6 to 24. I have a team of 14 youth workers and we service about 1,500 children and youth in our community. Some of our programming is similar to a lot of the programming that was talked about this week. We do recreation and leisure, health and wellbeing, tradition and culture, youth mental health. We also have an active living studio – it’s a 24/7 fitness facility and a dance studio in our community. We also partner with Right to Play, which is an international organization for participation with children. And we’re here today for our Outdoor Adventure Leadership Experience program.

Our collaboration started back in 2001 with a youth needs assessment in our community. We had lots of challenges, we had suicides in our community, so we worked really hard to find out what was needed in our community. While we were doing this, our friends at Laurentian – a doctor who was a boxing coach – were trying to find out why elite Aboriginal athletes only got so far and fell off from going professional. We worked on this project for six years with athletes and coaches right across Canada, connecting with different provinces, and a lot of coaches were wondering, “How can these athletes be so good but fall off?” So, we started working on research and coming up with interviews and evaluations, and a lot of it was culture shock. The loss of family support. They just kind of returned back to their communities or fell off due to other barriers and challenges. From that project we started going into outdoor adventure leadership. And in our community we are blessed with being on an island. We’re surrounded by water, we have a lot of natural resources in our community. So, we started from there and we’ve grown to what we are today. We’ve been running our program since 2008, when it started as a pilot project. And today we’re a big player, I think, in regards to outdoor programming. We also work with the [Aboriginal] Children’s Health and Wellbeing Measure – it’s another form of evaluation designed by children for children. They came up with the questions themselves so that they could understand the questions and answer the questions.

So, for our outdoor adventure leadership program, we have a curriculum that we follow, and inside the curriculum there’s three goals. There’s leadership training modules, where we talk about implementation phases; principles, which are our seven grandfather teachings; and in the summer, we run five and 10-day canoe trips. Our five-day trips are geared toward children that are 12 and under, and they actually attend with their families, with their parents – and if they don’t have a parent, it could be a guardian, an uncle, an older brother. So, we gear it towards the families. And our 10-day expeditions are geared to 13 plus.

The goals that we have with our program are to prepare youth as leaders, to promote community and culture, and to protect youth through resiliency and wellbeing. Those are our three goals through the OALE program. Our program’s been very successful. We do monitor all of our kids and all of our participants who participated in the program.

Because we were able to monitor all of our participants through ID codes in years 2012, 2013, and 2014, one of the key things that we noticed is we had a 100% student success retention rate from every participant that participated in our program. They stayed in school.

French River Provincial Park is our starting point. Our traditional territory goes all the way along [Highway] 69, the water boundaries of Georgian Bay. That’s our travel route. There are significant points in our traditional territory that our ancestors followed. It’s a 10-day trip, a 10-day excursion, and then we finish right in our community, at the Wikwemikong Unceded Territory at our marina, with a homecoming ceremony.

So, the total distance is 137 kilometres that they take. A lot of the times the kids have never been in a canoe; it’s their first time so there’s lots of challenges, lots of barriers that they have. The kids, the participants, they pack their own food, they pack their own gear. One of the really unique things with our trip is we hire a lead guide for risk management, but, other than that, the youth are the lead guides on the trip. The youth lead the voyage, the expedition, and our participants follow suit, so it’s youth helping youth, youth leading youth, and youth sharing with youth.

 

Lawrence Enosse of the Wikwemikong Unceded Territory, located on Manitoulin Island in Northern Ontario, talks about the many ways in which the Outdoor Adventure Leadership Experience youth canoe tripping program is evaluated. Photo Credit: Pat Kane

 

We started with a trial project back in 2008 with the 10-day trips. Stephen was our lead guide and we’ve grown from there. What we noticed after year two is we had a lot of parents asking, “What about our younger children? We’re having these big homecoming celebrations where the community’s welcoming all of the canoers as they come in and all the little brothers and sisters are wanting to be part of it.” So, we said, “Okay, well, let’s do it. There’s nothing stopping us from doing this; let’s take a moment to create it and let’s do it.” So, we’ve been pretty successful at offering canoe excursions all summer for all the children in our community, and not only that, we’ve expanded and opened it up and had participants from as far away as Saskatchewan coming to participate in our OALE program. We encourage parents and guardians, family members, to come on.

We’ve been pretty successful, and when we talk about evaluation, there’s different forms of evaluation that we’ve done. Stephen and I participated at a youth justice conference last year and one of the things that we shared is, because we were able to monitor all of our participants through ID codes in years 2012, 2013, and 2014, one of the key things that we noticed is we had a 100% student success retention rate from every participant that participated in our program. They stayed in school.

Also, in 2013 we had a young man who had been in trouble with the law. He participated and changed his life. So, he participates in our program, he goes down to southern Ontario to participate in another outdoor program during the summer, and now he’s going to graduate from school. He participates in extracurricular activities, he participates in a dance program, and this was what people would identify as a “thug” in our community. This program changed his life. He’s moving on, he’s successful. And when we have our regular programming not focused on OALE, we don’t see him for other programs; but every time we offer an outdoor program, whether it’s tea boiling, cutting wood for Elders – anything to do with the outdoors – he’s there. So, when we’re talking about therapeutic and connection evaluations, there’s again different successes and ways of measuring it.

Into the future, when we talk about other populations, we like to work with kids, with youth justice, because we see there’s a way to do it – there’s a way to help them change their lives. We’ve seen it with some of our participants. And also, with looking at the mental health, we noticed at school we have a lot of problems across Canada, with the high dropout rates. On-the-land programming has been successful in our community and, just listening to everybody here talk this week, it is successful in others. I think it’s a very innovative and creative way to go. And if we can get into possibly helping students to get education credits, I think that’s something that we could do to help our kids. Receive these credits and graduate [from] school.

So again, we’re partnering with other communities in developing our programming and expanding it. As I indicated, we’ve had participants from Saskatchewan and we’re just enhancing staffing and training, capacity building. We hired Stephen in 2008, we hired a lead guide for risk management, and we’re at the point now where we don’t have to hire lead guides. We’ve been able to retain our staff. They look forward to this program. We’re trained in advanced wilderness first aid, our staff have all the training that we could almost be a business. But we don’t want to be a business because we want to be in the business of helping our kids and our communities.

When we talk about evaluation there’s so many different types of evaluating. I think the opportunity with our program is the uniqueness of it. When we go on our trips we have journals, so a lot of the kids write their experiences; a lot of the kids who can’t write or can’t express their words draw pictures. We’re able to keep these or they take them home. We’re also advanced with technology – kids relate to technology. So, we have tablets for evaluations. We also have digital recording so that when we have our talking circles on all of our trips, we’re able to record all that information. Just recently on our last trip this past summer we hired a graphic artist to come in for our two 10-day trips; she came in and she was there for the homecoming, so she was able to create [a piece using] all of the feedback from the kids on this trip.

Some of the comments are pretty funny and there are some that are pretty emotional. But one of the things that the participants learned from the trip is life skills. They have the seven grandfather teachings, so they are able to learn those on the trips. Some of the comments are about ups and downs – you have to ride the waves; sometimes things don’t go your way. They talk about resilience. The direction of the wind is important. In one trip you can learn things you can apply to your whole life. Nature and spirit guide us. The route might be the same, but the journey was our own – because we go down the same trail as our ancestors. They want to see cliff jumping, cliff diving, as an Olympic sport because we have lots of cliffs when we go through our traditional territory and we have some very, very brave people who are supporting each other to overcome fears of jumping. They’re amazed when they see the animals – bears, eagles, fish. They see rain as a gift; they see thunderstorms as peaceful when they’re out on the land. They talk about building a family, showing up as individuals and then coming home on the last day as a family, coming into a bigger community of a family. And one of the things I like is, you don’t look down; when you look up, you realize how beautiful it is.

Stephen Ritchie
Likely the most interesting part of our study or aspect of the evaluation that took the most time, in probably three years of work in transcribing over 200,000 words, was trying to understand what it was about the OALE experience that actually promoted resilience and wellbeing. What we found would be what you already know: that connecting with the land is what promotes resilience and well being.

And as a researcher it took me probably the longest time to get to that point where I realized that this is what it was. We were expecting, or I was expecting, to probably have confirmation of some of our programming elements that were contributing to resilience and wellbeing, but it was connecting to the land.

What we found would be what you already know: that connecting with the land is what promotes resilience and well being.

We noticed that the connecting was actually more external and experiential in the first few days of the trip. We collected data from journals and talking circles in the evening, and check-ins around the campfire, which we audio recorded. I also interviewed youth on the trip. We had focus groups after the trip to try to make sense of what occurred during the trip, so we collected data from numerous sources. And what seemed to occur is that the youth were experientially just engaging with the land and with the Creator, so it was sort of an external experiential process for the first few days. And then we started noticing in the journals that the youth were recording more reflective thoughts. So, there is some kind of connection process internally or a change process that was occurring over time internally. We actually found a day three phenomenon. There is a change that occurred on day three for the youth. In fact, there were some journals where I’d be reading and a youth on day two would say, “I hate it here. I wish I could go home. I’m dirty, I don’t like it.” And then you flip over the page and on day three the entry was all positive: “I love it out here. We’re canoeing. I like swimming. I have friends.” So, there was a change that occurred and it wasn’t just necessarily day three for each youth; it might have been day two, day three, day four, but there was definitely something impactful that happened early on in the trip. What that sort of led us to think about is that in planning programs, they should be long enough to allow the youth to adjust in the first few days and get to a point where they can have a change. And the other thing that sort of struck us is that maybe the change process is not linear; maybe there are real significant events that occur for each youth and they’re different for each youth that may occur during the experience at different times. ◉

This piece was edited for length and clarity and features exerpts from the panel discussion on evaluation.

Debbie DeLancey, MAE University of Melbourne, is a long-term Northerner with many years of experience working for communities, Indigenous organizations, and governments in the Northwest Territories. Her early professional career included working on community development and participatory action research projects with Indigenous groups in the Northwest Territories and Nunavut; after which she worked for the Government of the Northwest Territories for more than 25 years, including 15 years as Deputy Minister of Municipal and Community Affairs, Human Resources, and Health and Social Services. She recently retired from her career in the public service to pursue other interests, which include promoting the importance of land-based programming for Indigenous communities and advancing effective program delivery through performance measurement and evaluation. Debbie holds the designation of Credentialled Evaluator with the Canadian Evaluation Society and sits on the boards of the Canadian Foundation for Healthcare Improvement and the Canadian Frailty Network.
Ioana Radu, PhD Concordia University, is an interdisciplinary scholar, community-engaged researcher and educator based in Montreal. Her work focuses on Indigenous wellbeing, knowledge mobilization, and oral history. Ioana has been collaborating with the Cree Nation of Eeyou Istchee (Northern Quebec) for over a decade on community-led initiatives in various fields from education to public health. She is presently post-doctoral fellow at DIALOG (Aboriginal Peoples Research and Knowledge Network), Institut National de la recherche scientifique in Montreal. She continues to focus on community engagement in research, decolonization, and land-based healing.

Lawrence Enosse is an Ojibway from the Wikwemikong Unceded Territory located on Manitoulin Island in Northern Ontario. Lawrence is employed as the Brighter Futures Manager for the Waasa Naabin Community Youth Services Centre. Lawrence is responsible for the management and leadership of the Outdoor Adventure Leadership Experience (OALE) program that includes five and 10-day canoe excursions through their traditional territory during the summer and outdoor leadership boot camps during the winter months. Lawrence has been involved with research interests that are devoted to understanding outdoor adventure and experiential education programs in the context of achieving personal growth and holistic health outcomes. Lawrence is an elected member of the Wikwemikong Unceded Territory Chief and Council, serving four terms, and holds the portfolio of Economic and Employment Development/Department of Lands and Natural Resources.

Stephen Ritchie, PhD, Laurentian University, is an Associate Professor in the School of Human Kinetics at Laurentian University in Sudbury, Ontario. He has worked in the outdoor field on and off for over 30 years as a guide, teacher, facilitator, and more recently as professor and teacher. Over the past 15 years, he has taught a variety of senior level under-graduate courses at the university on topics such as outdoor facilitation and teambuilding; risk management; and adventure therapy. Stephen’s research interests are devoted to understanding outdoor adventure and experiential education programs in the context of achieving personal growth and holistic health outcomes. He is also very interested in the importance and impact of land-based programming, and he is currently active in a diverse portfolio of research, including several community-based participatory research collaborations with Indigenous communities in rural and remote locations in Canada.

Notes
1 http://achwm.ca/public/home


Download Article