This workshop, Evaluating System Change: An Inquiry Framework, explored a system change framework organised around three core results: strategic learning, system change, and mission level impact.

Speaker Mark Cabaj is the President of the consulting company From Here to There and an Associate of Tamarack - An Institute for Community Engagement.

Mark has first-hand knowledge of using evaluation as a policy maker, philanthropist, and activist, and has played a big role in promoting the merging practice of developmental evaluation in Canada.

Transcript

Laura Barnes

I'm not going to read Mark's bio I've got - I've got it here. It's extensive. And we certainly sent that out to all of you as well. But really, I would just like to hand over to Mark now and say thank you very much for coming to talk to us. And please everyone welcome Mark Cabaj to the stage.

 

Mark Cabaj

Hi, everyone. Let's just check the sound for the live streaming. Is that okay, great. So, thanks for the introductions. Thanks for having me back to your homeland in place. This is my second time here the first time I was in, I'm embarrassed to say this, the big tower.

 

I call it the big tower. What's the tower called? The Tower of Power. Okay, so I'm not so far.

 

And I thoroughly enjoyed it and thanks for spending time coming out what you think is terrible weather but I think you're wimps. This is nothing. I'm happy to be here for a couple reasons. One of which is this is what it’s like in my hometown right now in March in Edmonton - and we had a really rough winter. Plus 32 in the summer minus 34 in the winter.

 

So it's been a gruelling time period. And so this is vacation for me, quite frankly. I also like this topic a little bit. You're right, you know, I, I'm not good titles. It's a terrible title. I apologise for that. But I love it right. Some people when they hear the word evaluation, they have the same responses. They went to the word colonoscopy, a root canal. And I love it. And I love it. Because for me evaluation is reality testing.

 

My whole life I've been in complex change efforts and at multiple scales and multiple institutions and evaluation is a backdoor to strategy. Right? All innovators have hunches about the way the world works and how it could change and evaluation says how do you know what you think? You know, do you need some data for that? It asks us to slow down and consider other options. And in fact, I won't show you the diagram on the screen, but imagine two overlapping circles.

 

Innovation on complex issues is a creative act. Would you agree? We have to be creative to solve complex vexing issues. Evaluation is a critical act. It says how do you know what you think? You know, are there other ways of looking at other other perspectives on this? And when we're evaluating complex change efforts, we're interested in that sweet spot in between? How can we get the innovators to be more reality testing oriented?

 

And how can we get evaluators to slow down and make sure that they're doing their work in a way that helps not hinders the innovators. And this really is a big challenge for our time, because evaluation grew up in the projects in the United States for the most part, but I grew up here in Australia in the same way in the United Kingdom, Canada, my home country, where it was really good at assessing discrete interventions, and having high burdens of proof around whether or not these things are successful and just something like some kind of pharmaceutical drug, we would test that and then scale out a cookie cutter model and that tradition and evaluation still exists.

 

However, evaluation has not kept up to the fact that innovators don't think like that - we have intervention, some of which might be scaled often or in a recipe like format. But the change agenda is much bigger, broader and longer than that. And evaluation is desperately trying to keep up. So for all of you who feel like there's orthodoxy and kind of like a, in the evaluation field, don't worry, there is no worth that there's no heresy that Orthodoxy is, is only in some discrete places, the evaluation field is self-correcting.

 

In fact, my dad was a superintendent of schools and he had a hat that said, I'm their leader, Which way did they go? And I actually think evaluation needs a hat like that. The social innovators are off, where are they can we hurry up and catch up and be somewhat useful? So I'm really the valuation field is self-correcting. And one of the areas that has to self-correct is evaluating systems change and so I look at this as a very beautiful piece of work to say, I, if we're if we're going to be useful to evaluators, most innovators, most innovators talk about systems change.

 

By the way, the winter wasn't all bad. We had a lot of nice festivals this year. And Canadians are getting good at living in the winter. So here's what we're going to chat about why systems change? Number one? What is the system? And what does system change look like? How do we evaluate systems change? And so what are the implications for you? And I'm going to talk for about 45 minutes. And then have you chat for five or six minutes in small groups, and then we'll take questions for 15 minutes and some of us will stay on for the afternoon and go a little bit deeper than that. Yes.

 

Yes, thank you and and I changed it even up until last night based on some of my research, so there will be a copy sent out fairly shortly. It's already in the in the Dropbox. Thank you.

 

So here's a great quote - Karen Pittman forum for youth investment in the US in 2015. In Vancouver at the Tamarack Institute she came up with this phrase that sounds pretty good. She's a poet, programs and interventions help people beat the odds systems, change interventions, change their odds.

 

If we're in the game of moving population level needles, we are in the game of systems change. There's no getting out of it, programs can't tip systems. I was in New Hampshire a couple years ago, we were talking about early childhood development programs in New Hampshire versus Quebec. And they were talking about the New Hampshire programs and they were pretty cool.

 

Actually, if you ever been to different parts of the US, they have real good, great clusters of early learning and care and their programs were way more interesting than the ones in Quebec, Quebec had far better early learning and care outcomes. Why? Quebec, in Canada, there's 12 months maternity leave, and if depending on your employer, you can get topped up to 95% of your salary. Maybe it's going to be 18 months pretty soon.

 

Number two, child early learning and care centres $5 a day.

 

For early learning and care. $2500 for university, like, on and on, they live in an ecosystem that is so family and child friendly for a whole bunch of reasons. And there's a dark side to it. So I'm not going too overprivileged, I'm from the west - I can't compliment Quebec too much. But literally, public policy systems, ecosystems make a difference, right? When we're often looking at situations where someone's not doing well, we can think about what biologists do and when a when a frog is not healthy. They you could look at the frog or you could look at the marsh and saying what's wrong with the ecosystem that's making the frog so ill, so we are in the game of systems change.

 

Here's another phrase. Systems change is about shifting the conditions that hold the problem in place from Social Innovation Generation Canada, and this, this phrase has been adopted quite a bit and recently by FSG, and others, and I really think this is important because not only do systems hold the problem in place, sometimes they reproduce them.

 

Right problems get reproduced by systems they keep routinely generating the same kinds of problems. So we really have to get our systems literacy up. And I think almost everyone now would say systems change roles relatively easy off their tongue. But I don't know for that good at describing what the heck we mean. And that's why I love this quote from Donna Podems. Gonna read it to you.

 

“I was asked to work with innovators in the National Health programme of a smaller country. When I started working with group they said, ‘We aim to shift the health system’ -  after listening for a few hours. I said, honestly, I have no idea what you're doing, or what you're trying to achieve. And I haven't a clue how to measure it. I don't understand what it means to shift the health system. And they looked at each other and burst out laughing and said we've no idea either.”

 

So I think we're there folks. I think we all get it like we get it and then someone says, What's the system change? It's not that easy to make sense of so I feel this is an exemplar of our work. And the reason I was interested in coming to your work today is because I follow.

 

I'm kind of like a community development lawyer. I like watching big systems change. And I was quite preoccupied with Canada for a while. And I still am and do work in the States. But I'm watching you folks quite a bit. Both the emphasis on place-based change is more powerful in Australia right now than it is in Canada, stronger places at the federal government, Victoria has got a whole bunch of stuff going on. And you have something pretty special in Brisbane, here with the organization's hosting the work that you're doing.

 

So whether you like it or not, you're an exemplar. And we're watching what really well organised and committed group can do around some of these complex issues. And we're very happy that you're talking about systems change. So if we're all sort of on the cusp of this, here's what we wanted to deal with today. And I like how many people have read the book “Getting to Maybe”?

 

Nope. Oh, folks. I know you've got a big reading list. So I don't say this lightly, but it's getting to me.

 

How the world has changed. And it's all about social innovation and talks about the nature of social innovation, and has generated so many insights, at least that have helped us back home in Canada. The difference between simple complicated complex the adaptive cycle, what it's like to be innovating when you think the world is applauding you and you feel like you're in cold, dark heaven, like is this all there is. So there's really good insights in that book. And it was co-authored by a woman named Brenda Zimmerman. And when we were starting the Tamarack Institute, I said, Brenda, people are tackling complex issues. And when you talk about things like evaluating systems change, they're going to want tools and recipes. And she said she was one of the greatest teachers I've ever met. She said, Don't give people tools or techniques are just going to get to the wrong place faster.

 

What we need to do is frame issues so they are coherent, not you know, consistent, but can we make things coherent? Can we frame things so they're coherent, offer principles to guide their thinking?

 

in action, and then offer them illustrative practices and aids for action, so they can deal with them in their context. Now, this is important, right? What's the difference between a rule or a tool and a principle? If you come to a stop sign? What do you do?

 

You're supposed to stop.

 

What if you see a sign that says drive defensively? That's more of a principle. Because what a principal says is pay attention to your context. But I can't tell you what to do in that context, you have to use your noggin. If it's raining, maybe you slow down if it's a school zone, you look both ways, etc. It says pay attention to this and respond in a context appropriate way. So what we're about - there's a lot on going on with this right now around systems change. And we're just going to highlight the fact that there's so many good resources out there, aids for action, etc, that are context that have to be applied in a context specific manner.

 

So when you go out there, there's tonnes of tools. If you're tool junkie like me, you'll love it. Right? spent, grab a six pack of beer and a plate of nachos and just work through these things. There are a lot of fun for geeky people like me. However, a lot of them don't have framing. And that's what I wanted to deal with, kind of try and find a universal framing for systems change in a way that works for me, probably for you, too. I get lots of different I work in lots of different systems change initiatives. So what is very accurate and precise and useful for one group may not be for the other.

 

So I took this challenge up, took the challenge on and started to think what could we say that is universal about systems change when I get asked to evaluate? So here's a couple of things. Number one key features of a system - who's seen this diagram before?

 

Okay, not too many. Okay, well, it may have been may have been like a biology class or something. This is from john Kelly chase out of the UK. It's a pretty common framing of city - And just hold on to a bit it says the basic features of assistant are they have boundaries, geographic and domain boundaries, I'm dealing with the education system in Logan. Right? And it's got cross scale dynamics.

 

There's a school there's Logan there's the state or the school board, etc. It's got agents and actors who are all the players in the system and they are in some kind of relationship tight, loose, hierarchical not. And this is basic when people say map your system. Remember Donna's quote earlier about the healthcare system. They would have said start mapping it what scale what domain etc. Now, that's the starting point. Go to the next one. Oop.

 

Oh, I don't think I have my video. Who remembers the Wolves Change Rivers video? Okay, it's not on here. So now I'm in real time not able to show you a video but it was on earlier.

 

This could let's just recreate what happens they reintroduce wolves into the Yellowstone National Park. Right. And what happens after that it's called a trophic cascade, that input creates splatter of self-refuelling change processes. The wolves get rid of the deers and the bears start showing up and the berries start growing and then the orders come in. What happens is they unlock unlock a system that was an inertia by simply introducing a new species and the refuelling effects go on and on and on. This leads to this leads to this leads to that, so much so that at one point, the effects are so big and so cumulative, that the boundaries of the rivers begin to change. They meander less and the reason is because the soil because there's more trees erodes less, and the bank stabilise and they actually change the structure of the rivers. Now, this is interesting for a whole bunch of reasons, including if we would just let the planet heal itself help itself it probably could.

 

But here's some other dimensions to this. Is Yellowstone National Park, ie your system closed or is it open?

 

You're dealing with health - I mean, education in Logan, for example, you're dealing with overlapping system, the land use system, the transportation planning system, the employment system, etc. Number one, number two, it's emergent behaviour. When you're dealing with the system, the outcomes are the result of everyone interacting with each other. It's not just what the teachers are doing. It's what everyone is doing that cumulatively leads to those outcomes. So no one actor can impose a solution on their own. It's a cumulative effect.

 

Hey, we are we have a you have a food system in Brisbane. Right, the food system, how many actors are there in the food system in Brisbane? Well, at least two and a half million consumers. They're all actors. There's health inspectors, there's suppliers, there's transportation people, etc.

 

Look at all the players that interact with the food system. So you're worrying about the price of fruit, it's a cumulative effect of all those things interacting together. So that's important later on nonlinear dynamics. Anyone ever push hard on a systems change where you pushed and nothing happened?

 

Yeah, that's nonlinear effort does not equal outcome.

 

The Case of the Wolf shows just a little input, a pack of wolves creates a disproportionately large outcome. So the dynamics are nonlinear, you can't really do an easy logic model for systems change. Number three varying levels of predictability and they're ever evolving. It's really hard to estimate what happens when you intervene into a complex issue. It generates a splatter of outcomes, some of which are intended, some of which are not. And even if you do, eventually get to predict them, the system keeps moving on you. Now, if you ever have a chance of how wolves change rivers, take a look at that because it's a metaphor that I want you to hold on to as we get into the next pieces here.

 

This was big for me because - we're gonna oh, sorry, folks, one slides not showing up. Another one's not showing.

 

Who's ever seen the triangle here on the bottom, the upside down triangle.

 

This is from FSG foundation Strategy Group, the people who really popularised the phrase collective impact. And they actually took something from Donella Meadows, and other systems thinker and they said, when you look at systems change, it actually they say they've distilled into six conditions of change. And I'm gonna say this is inaccurate, but it's useful. And it's an accurate because all models are inaccurate. All frameworks are inaccurate, they reveal things and they distort things. But this one is really revealing of some basic things around conditions for change. And here's what they say, simplification but still useful.

 

The stuff on the surface around systems, things conditions, or drivers of or leverage points for change include policies, practices and resource flows. So we could push on those and say maybe we could see some different outcomes. They say go underneath the surface it's a bit semi explicit is some other dynamics that equally influence those conditions change if not more connections and relationships and power and authority.

 

If you can change the power and authority IE delegate responsibility for education to lower levels, or increased relationships, make sure service providers talk together better and coordinate services etc, you can get bigger outcomes. And finally, they say, All systems are held in place by mental models. So we are presenter back home of a European way of thinking about education. It's very Eurocentric, right. And if you're indigenous community, particularly around kids in care, boy, oh boy talked about a different worldview. In fact, I was in social services at one point and an indigenous elder says Mark, you know, how we keep getting dinged for being bad parents because our kids aren't at home, in a First Nations community, increasing In northern Alberta, when we were all living in teepees next to each other, the moment a kid got 12 or 13 years old, and we started, we started irritating each other.

 

Anyone who had 12 or 13 year old kids, they would just go live with their uncles and Auntie's two or three tents over. And it's expected that your kid is kind of got four sets of parents for four or five years. She goes, ‘but your worldview you can't see that and you punish us for ours, and we look diminished in your eyes.”

 

So you can understand the role of mental models. Now, what I love about that, that's a beautiful distillation of things. But here's the issue. Have you ever seen - moving a leverage point, a driver - and it not leading to any behaviour change?

 

The system actors keep doing what they're doing. Change a policy everyone keeps - this is the same behaviour.

 

If you've not seen that, yeah, in the case of the wolf, there was tonnes of behaviour change - the animals started interacting with each other. That’s what's so powerful about that video, but sometimes you can push and nothing happens. So we might change a policy, a set of service coordination, etc. but doesn't lead to behaviour change on one or more actors number.

 

That's the second one. The third one is, are there enough behaviour changes happening that we change the overall pattern of behaviour in a system? Now, in the Yellowstone National Park, we did the overall pattern of behaviour change, the dosage was big enough, it was nonlinear. There was a predatory species put in there, it created a cascading effect. There was behaviour change. But when we talk about systems change, we have to think about at least these three things. Did we change a driver? Is anyone changing behaviour and is the overall pattern of behaviour changing? Now I'm going to show you some case studies of that. Just feeling okay. Okay.

 

Oh, there we go. So change a policy does it change a behaviour? And, oh, here - here's my beautiful diagram that I'm not going to show you now. So what I'm going to do is I'm going to play a little bit with this by saying one day I was asked by Tamarack to hurry and produce a paper. And so I went - I don't smoke but I used to so whenever I get stressed I pretend I smoke again. I got a paper – what the heck am I gonna write on? I thought I'm gonna take Donna’s poems quote, the one about the healthcare system not knowing it's needs a kick in the butt and saying, how if I'm on stage in 10 minutes, how would I start thinking about change?

 

So there's three levels of change, saying, here's my wide angle lens, number one, systems change, we just talked about it. Is this initiative, changing drivers are there's behaviour change, and as the overall system behaving differently, number one - number two, what's the impact of that programmatic impact from a programme and services, targeted impact? Is that happening in a suburb, even though we can't get the whole population level?

 

And finally, population level outcome, three different scales? And the third one is, what are we learning about this issue that can inform the next iteration of strategy and the reason I want to talk about that is because that's the only thing we control in systems change. You don't understand the system until you start mucking around.

 

With it, interacting and your ability to be really thoughtful and strategic and get insight to inform the next iteration is the predictor of success or failure. When you're doing systems change, all you have is an opening move. You start, and then you act react, interact. Anyone ever play chess? Yet - would you say that that's a closed game? It's a finite game to players. Clear rules, clear endpoint.

 

We work in infinite games. When we're doing systems change, lots of rules, players come and go. And it's never clear when we won, if at all. We can just check progress. So if you were playing chess, even you wouldn't plan 30 moves in advance would you - no, so why would we in infinite games like systems change, you plan a couple moves in advance and you act react, adapt, and as you learn more about the games and the players and the shifting sands, you make bigger bets. So strategic learning is mission critical, and it's too often diminished and it's the only thing we control so I think it should be central.

 

Let me give you a couple examples of this case study ending homelessness in Medicine Hat, Alberta. I'm going to go through fairly quickly through these because all they want to do is to say, here's how the framework can organise our thinking about what constitutes a systems change. And this one's going to focus mostly on

 

this big idea of housing first, and I've done a bit of research. I know Housing First is in Brisbane. How many people are familiar with Housing First – maybe a third of the group. So what's the basic idea? There's a typical ladder in traditional housing, which is if you can manage some of your vulnerabilities over time, like excessive use of certain substances, etc, we will get you better housing, but it's like a salmon swimming upstream because you have to kind of win and earn the next level of housing.

 

So it's quite a journey. Housing First says, for two reasons, forget about it. For social justice reasons, you should be in a house number one and number two, if you're in a home that's relatively stable, you should be able you'll have a better chance of dealing with those balls.

 

So it's kind of makes sense. There's lots of limits with this strategy. And I'm not going to get them in into them today. So I'm not proposing this. But it's a useful way to think about systems change. So here's a quick example. I'm going to go through a couple of these conditions and talk about the evaluation implications of that because - here's the challenge. Again, there's a million tools and techniques out there out there, when to use them.

 

So policy change, anyone ever evaluate their advocacy efforts? There's tonnes of stuff out there. One of the most minor of the two dozen policy changes required in Alberta to enable place based approaches to homelessness include simple adjusting the way government services require ID, you must have that here. Low lying fruit relatively. So what happens this group in Medicine Hat Alberta had to track along with their coalition partners, how they were pressing the lead up on this, and then eventually it made it into the Alberta plan.

 

So number one, let's look at the policies. Method number one, Harvard Family Research Project advocacy toolkit. We use that we tracked the development of a policy over a continuum over time.

 

Not policy impact, but policy. Could we develop a proposal for it? Number one, ever try and develop a pro proposal with diverse partners. It's a workout. It's a workout, but you can do it, get it placed on the agenda, get it adopted, get it implemented, get it maintained, and get it upgraded. I work with a group in Calgary around fair fares for persons with social assistance, getting access to public transit. We went around that circle three times in 12 years. Defend, upgrade, defend upgrade, so we could track some kind of progress on that.

 

Number two, if you want to say ‘Yeah, well, that's pretty high level tracking a policy change’. Change, you can actually go even a little bit deeper and saying, are we creating the conditions for the policy change to happen. And if you want to look at another resource, the Centre for Evaluation, Innovation has this great resource ‘10 pathways to policy change’, each with their own theory of change, each with their own outcomes and each with their own indicators.

 

So imagine all you - how many of you are theory of change junkies. Right? Someone's done a little bit of heavy lifting for us. So you just go there and you adapt it. This gives you the intermediate progress. Number three, when you get really close to policy change, you can start asking deeper questions when you're tracking policymakers. And the US does this because the Americans are way better at it than the Aussies. And the Canadian - and the Canadians, because they've had to do advocacy work for much longer because they're dealing with more stuff than we are, I would argue, they've got these ways of rating policymakers on city council in the civil service, etc, about where they stand on policy issues, and they track them like crazy.

 

So I'm not proposing these techniques. There. I'm just saying when you zoom out on policy change, we have a field thick with techniques and policy change is but one driver. Now, if you take a look at relationships and connections that's even deeper, how many of you have used social network analysis before? Some of you have? It's very interesting. So you have to be really clear why you're doing it, because you can have so much fun just doing this connectivity stuff, but it's really important to know why you're doing it.

 

But in the case of Medicine Hat, they did a research study, and this isn't the one they did, but the findings were similar. So I used it. They did pre and post and they found just within the network of people who have clients and family and service providers to interesting findings, quantitative results, a reduction in network size over a year. So people get into homes and their network collapses. Would that be bad, maybe, however, increases in density and frequency of contact so it collapses but it gets - And then what happens, it starts moving away.

 

And they call it network decay, I think which was interesting, demonstrated a strengthening of the quality relationship to family and housing providers and a shedding of burdensome and abusive relationships. These suggests something called network decay. The old networks are decaying as the new ones are being created. Now this has been contested because I saw some other research people saying, even the old social capital is better than no social capital. So don't diminish those networks. So it's contested.

 

But they kept going with this analysis. And they found out that the clients were networking fine, but we were still having problems networking and coordinating among service providers. People were competing for clients. And they really had tension about doing collaborative relationships going forward. So transformational changes almost for clients, and evenly successful changes for service providers. So are we changing systems, not so much on the wrap round, not as big as we thought we might be having some interesting insights

 

Going to go one more mental models. Big one, how do you track the evolution of mental models? Well, here's one of my favourite techniques - Bellwether evaluation.

Bellwether evaluation is you spot someone who represents a constituency or an influential group not only represents but might influence that you're trying to change? Top 100? And will you track their actions, their behaviours and their comments over time as a Bellwether about whether mental models are shifting in key circles of influence? So many of you have seen this before. It's a great diagram, isn't it? If you're American, you know that Ohio is now the is a Bellwether for who's going to win the presidency, doesn't mean it's a predictor just means watch. They're probably going to tell you where the trend is going. Again, Bellwether can also not only track people who represent trends, but they can try and influence people who influence other groups.

 

So there's lots of geeky questions to ask here. And these ones are taken from Bellwether’s for policymakers. So kind of like these rating scales. How do you - what's the issue? How do you know about it? Where do you think it's on the agenda? How is important to you? These are sample questions. You could use any questions, but the Bellwether that worked the most in Medicine Hat was the mayor.

 

The mayor was pretty could Canadian prairie populist conservatism, pull yourself up by the bat, the bootstraps kind of fella and got involved in housing first, where he said, ‘I never thought it was possible with Housing First, it was. I'm converted’.

 

Here's what we did in the new format. Here's what we did. His statements over a five year period are nothing short of absolutely remarkable. He influenced the Chamber of Commerce, the Rural Economic Development Council, etc. The number of people that he then influenced because they worked him like crazy to get them involved with Housing First is an indicator systems change now.

 

So, we change those systems. We actually had behaviour changes when you look across all the evaluations and we tip they certainly tip the overall behaviour of the system including to the point where medicine now hat is now cross functionally. There's no more chronic homelessness if you have - if you're caught homeless in Medicine Hat, if you avail yourself to the services, within seven days you have a house with wraparound supports - pretty amazing.

 

Two years later - they got other problems, that the game is not over, but at least they achieve something there.

 

Okay, so that's his policy stuff that makes sense. Zoom in on specific drivers that are initiative specific, and there's lots of tools and techniques to track progress. And why are we tracking progress? Because we're trying to get population level outcomes.

 

The route is through systems change. So are we moving conditions or leverage points? Let's actually talk about increasing access to post-secondary education in Winnipeg’s North End, Winnipeg’s North End - pretty rough town. It's kind of like Chicago, Canada is little Chicago. In fact Chicago's north end of stuff so is Winnipeg’s North End.

 

Heavy indigenous First Nation and Winnipeg is very organised with First Nations communities, the biggest urban Aboriginal population in Canada 75,000, and very committed to reconciliation. So they actually have a six year strategy using human centred design collective impact and others to say neighbours are going to renew Point Douglas and we're going to do it from an indigenous lens.

 

Now what's interesting is one of their one of their projects has to do with getting kids pumped for high school graduation and access to university. And so one of their early wins - they call this a pudgy bunny. You guys call it that? No early wins - low lying fruit, pudgy bunny? Apparently you can catch a pudgy bunny. That’s an Adelaide thing.

 

It’s the Canada learning bond? Now do you have these learning bonds that where you get government top up to save for post secondary?

 

So it's a government programme that's out there, but the government can't exploit it. They're having trouble getting people to take up this learning bond and the learning bond is quite modest in size. But the research around savings programmes shows it's not just the money, it's getting you thinking about the future.

 

Regarding you with some level of intentionality, you think about post secondary education. So the early studies show that this looks pretty good that if you've got learning bonds, you're likely to increase the likelihood of people doing well at school and going to post secondary. The problem Is, uptake is low. Okay, so this was our organising effort. And they went through a five year process simply to get people to take up the learning bond, and I picked this example explicitly because it's not someone would say it's not rocket surgery.

 

Rocket Science brain surgery, get it. Okay. It’s pretty straightforward. So what does it take to even work on something this straightforward. So here is their strategy. Winnipeg Boldness is the convener, they have a dump load of partners, and I'm not going to tell you what they all did, but they're engaging a lot of people there. Their phrase was, anyone who's not moving, we're gonna get in their way. We want everyone to be moving on this. They have their strategies around enrollment identification, accessible banking, children and care and post enrollment, and they have systems change and population level results. Okay, that's their strategy.

 

Let me introduce a very cool methodology that helps track the unfolding of that strategy. And that's outcome harvesting. Anyone hear about outcome harvesting? Yeah, I love it. It's the most elegant thing you can do from 30,000 feet. And what it says is when you look at complex change initiatives, they really are all about an accumulation of behaviour changes by key system actors.

 

Right, outcomes. They contribute to an impact. In this case, there's going to be signup for the post secondary bond. But you need behaviour changes by system actors. And when we assess that we have to assess the significance of those changes, why they're important, those behaviour changes. And to what degree did we as the change makers contribute to their change?

 

If you're a backbone organisation, you're not running the show, you're behind the scenes, so we contribute to that change. Now, here's an example of Toronto. Toronto has something called the Toronto Renewal Housing Corporation renewal project, and they have 2000 towers in Toronto that were built between 1950 and 1970, that are terrible for energy GHG emissions, they're not multifunctional and they're disconnected from transit. The problem is you can't tear them down because they hold at least a million Torontonians, and they're the biggest absorption housing stock for new Canadians coming to Toronto and 55% of people who are in Toronto now are born outside the country.

 

Right. So, strategic stock. So the idea is how do you renew the stock in a way that deals with energy carbon emissions, reconnects them to transit and creates multifunctional nodes. Now, big project, lots of money involved in housing. This group has a whole bunch of things going on around public education, policy, nudges, demonstration projects, etc. To get the field aligned to dealing something like this. You can't - You can't manage that from the centre. There's too many moving pieces.

 

So here is an outcome - one outcome 2016 the Toronto City Housing Corporation, and the Toronto Atmospheric Fund had been launched a joint project to invest $4.2 million into seven retrofits. That's a behaviour change. Would you agree? There's some GHG stuff happening, but that's a behaviour change. That's important. Big housing owner and big investor are trying something. They're aligned with our mission. The project is learning rich, and we get some utility savings and 30% reduction in GHG emissions.

 

But here's why that's significant. This is an evaluation question. numbers by themselves don't mean anything, we have to ask why that outcome is significant. They have 2200 units.

 

So if they catch religion on this project, think of the scalability. This isn't just another housing provider. They're the largest owner of social housing in North America. So getting them in your engage is pretty significant. So we asked about the contribution and the contribution was, hey, we probably wouldn't have done this in the same way or even at all, had there not been the Toronto region.

 

Had there not been this partnership - you nudge just into that direction, and you shaped our delivery processes, so they give us a high contribution. This is a little vignette of a long-term journey towards systems change. Now, I'm going to give you an example or two that shows you what this looks like when you zoom out. Here's a project in Burundi just developing capacity, For public sector reform.

 

This is taken from a World Bank site, World Bank report. Look at all those little bubbles. You don't even have to know what's in them. All of them are little behaviour changes. This happened. And each of them could be a story like this, folks, one bubble could be that you want to zoom in, story upon story, this led to that.

 

And they've combined them by leadership and priority setting policy changes, public sector delivery reforms. It's the accumulation of things, everyone that leads to big moments of change. And when we're in the field of social innovation, we're always looking for the big knockout punch. We did it boom. But we're not, we don't have the patience to handle all the tactical things required to get there. Change accumulates over time. And it's the accumulation of little things that leads to big moments.

So here's another example. Solid Waste Management in Reform in Bosnia - bable bable bable - this leads to that that leads to this, this is terrific because you don't even need a theory of change to track this. You're tracking something as it emerges and unfolds doesn't even have to fit nicely into a strategy.

 

This is how change happens, folks. There are steps here that I'm not really going to touch on too much, which is reviewed documentation. Better pretend you were going to be tracking somebody for policy change, what you and your team would do was you would start down and saying, here's what we think all of our little outcome statements are. You would produce them and you might put them in little bubbles. Then you go to your stakeholders, and with hunched shoulders, you go, what do you think?

 

And you you, you're evoking response, because guess what, they're going to think something - they're going to go what the hell? Who were you, you didn't get that right. Well, what about this, you forgot about this - you're, you're challenging them to fix up the outcome story. And they're going to say you forgot about this and you forgot about that. So you're evoking response. Hopefully the stories you get back is twice or twice as good.

 

Because you've got 360 degree intelligence on this, then if you said we have some high stake claims here in some of these stories, and we might be drinking our own Kool Aid on this - let's actually get verification and substantiation from third parties who know about that story. And they can tell us whether it really unfolded like that – is this making sense?

 

This is interesting because rather than have to be perfect about this, what we're trying to get is roughly right, reasonable approximation of what happened, when and who is involved and how it’s significant. So we're not trying to prove a model to get scaled. We're trying to get multiple stakeholders to track and make sense of an unfolding change process. Now, the reason I want to show you that is this is where this group in Winnipeg is, here's their outcome harvest by their five, five are different streams of strategy, different times different bubbles, big things happened.

 

Now, when we zoom out and saying let's look a little bit more deeply into all these little accumulative changes they are claiming. We have some interesting things - operational changes.

 

The development and delivery of workshops to train local organisations and how to build awareness and increase uptake in the CLB. In the North End, I forgot what the numbers were, but it was up like 25. So they put these workshops out. They organised them, they're taking a high contribution. And why was that significant?

 

It expands a number of variety people in Winnipeg North End who are on on top of this topic, and in fact, they have stats here about how many people came to their workshops. So boom, that's an outcome. Postnatal engagement pilot, two pilots developed a pilot – ah, two hospitals developed a pilot to address errors in birth registration, because this has been identified as a bit of a barrier to painting, primary identification, boom, they're trying out.

 

We don't have to love these folks. I don't even really understand this issue. But what they're doing is trying to say here are little things that happened on our way to the big things. Provincial government ID pilot, they influenced the provincial government. It even goes into policy and investment changes. Look at this with number five - direct billing practices extended across the province, that's pretty big - STC investments agree to invest $12.5 million - that's reserved flows.

 

Right? Are these by themselves? Huge? I don't know, probably not. But again, it's the accumulation of those little things. So here's what happens when you look at all those systems changes, all of which we could have talked about here and see what the outcomes were so - outcomes for individuals or programmatic interventions, 630 persons directly signed up to the Canada learning bond through those delivery agents. This is a neighbourhood of about 10,000 people.

 

Targeted anecdotal evidence of increased uptake in First Nations communities. We don't have evidence, we're scared, now we want evidence - what the heck, how did we not capture this? So we got to go look for targeted evidence. Here's population level evidence. The signup rate for CLB increased in Point Douglas at 65% a greater rate than larger Winnipeg rating, and two times the Canadian average. Everyone's hustling on the CLB but Point Douglas is making more progress faster than anyone else.

 

So we know we have a population level impact just on the uptake. We still, as you know, to have questions about downstream education, outcome harvesting is a 30,000 foot executive summary of the unfolding of a systems change over time. It allows for the messy accumulation of that data and to help social innovators and their supporters get a roughly right sense of are they making progress on their their big ambitious goals?

 

I'm going to finish with the third one and then let you chat. So the plotline again. Are we moving drivers seeing behaviour changes? Is that translating into people outcomes of some kind of order and scale?

 

The third one is strategic learning. What are we learning when we engage in a system and get strategic intelligence about the nature of the problem, the conditions that keep it in place and our own ability to make change?

 

So, racism in Edmonton, we have racism. There's, and this is considered a fundamental cause of a lot of the inequities in Northern Alberta. So, one group took on a racism lab - how do you use human centred design to uncover what generates racialized behaviour and create little prototypes to help people who might be experiencing racism, particularly persons of colour, getting housing in Edmonton.

 

So they have this thing called the shift lab and many of you have your own human centred design process. This is just a variation on the theme, ethnographic research, brainstorming, rapid prototypes, field prototypes, and then if you find something that works, you go look for people to adopt and scale. We're going to talk more about this tomorrow.

 

Feeling it? So they pull a group together, and they get some pretty interesting prototypes. They go through around it, they have 25 people that meet for four months, and they come up with three prototypes, all of which have been adopted since, right. They have been sustained - one is mobile Legal Aid, if you're experiencing challenge with a landlord who you feel is breaking the law, you make this call and people Zoom Zoom out to you in a van with lawyers etc. And they go here your options we're going to, we're going to go toe to toe with this tenant with this landlord.

 

Pretty aggressive, someone funded that, believe it or not pretty cool. Number two, diversity certification programme for housing suppliers. This is to say you want to be a housing supplier of choice. Come take our diversity training. And we'll tell you what it's like for different cultures to be involved in housing, including simple things like if you're in Northern Alberta, and you have indigenous people staying in your place, they often do smudging - and I just learned smudging is not big here. Smudging is where you take sweet grass and you smoke it and it's like a prayer and if you had a meeting if we had a meeting, we would start with this smudge and he would cleanse your head, your mind, your ears, your eyes, your heart and your whole body and you will proceed.

 

It's like a meditation. It's not racially motivated. But it has a racialized outcome that most tenants saying you can't smoke smudge in your apartment. Fire code. It's just European neurosis about that stuff. So they're saying, here's what it's like. And this is a tradition. And it's actually here's how it doesn't actually affect fire code. So imagine someone goes through that, you get the stamp of approval. And when the boom is on in Alberta, and it goes up and down all the time, or it's off and you want to attract people to your apartment. So this programme has taken off.

 

The final one is, you have NIMBY here? So they said, yes, let's get yes in my backyard. And they have this whole thing about engaging neighbourhoods with data to show that housing prices don't go down and safety isn't an issue, etc, because they're bringing evidence to bear to say most of these things are myths.

 

So, folks, the reason I like these is they're all terrific, but the conclusion was none of these is going to have much in the way of programmatic impact for a while. And their dosage is not big enough to have targeted or population level impact. Why the hell not? So they went through the strategic learning thing, using a method called Triple Loop Learning. Many of you have used this with reflective practice. And they said, here's our result - un satisfactory, is it because of what we're doing, what we're thinking or how we're being.

 

Now, here's what they found out. They have 11 key insights about the strengths and limitations of their own workshop design. There was 11 tactical things they could have improved upon. We've all done workshops, we know what those are like, right? Not as long different facilitators, etc.

 

Thinking eight insights about the nature of racism, poverty in Edmonton, including racism can be internalised interpersonal and systemic, the different manifestations of racism. We learned a lot - the group. Including three insights about our lab methodology, which included the lab stuff is great, but it causes real friction with people who have a social justice stance, who think that’s incrementalism, managerial and it avoids the root cause. So we have a big blow-up in the lab about that. The final one was the most important - we struggle to have deeper meaningful conversations about race.

 

We always play Canadian nice. The moment someone feels a bit of tension, we kind of back off. I heard - I use this phrase the other day - why did the Canadian chicken cross the road? To get to the middle.

 

Okay, we aren't - we don't - we're not good at it. In fact, Canada's politeness is such that when some anthropologists did this, because we had such crazy diversity earlier on, it's like WD 40. It's meant to be the grease that allows for lots of interaction between different cultures but it's got a dark side it means it's hard to go deep on stuff. So here's what we did. We use something called theory you to interpret that and some people have used their you how many people have used it. Otto Sharma, etc.

 

What it says that if you want to actually get on the other side of something, to see transformational change, you have to actually go on a journey that goes downward, where you uncouple your thoughts and your sense of what you think the problem is you get tested, you even get battered a little bit to the point where you hit the bottom of the U, where you're kind of open, right? You've deconstructed yourself a little bit.

 

And then you're open to new ways of seeing things. Now, this is actually a constant process. But they said, when you look at the theory U, when we try and understand what we did, it offers a lot of insight from a reflective practice point of view, including four types of conversations. Level one - talking nice. Level two - talking tough debating. Level three - reflects the dialogue; and level four - generative dialogue. And here's what we found. We were always talking nice.

 

Someone would say something about racism, someone else would get a little bit perturbed and there'd be a little bit of tension in the room and we'd all - we'd back off and we never entertained any of it. People in the post-review set I brought 5% of myself to this I was so scared of saying anything for fear of getting contested or unleashing something we couldn't stop, that we didn't talk. We never even got to talking tough.

 

And one fellow actually stood up and says, I don't have a position, I have some perspectives that I'm willing to get on the other side of, but I can't do that with dialogue. Like how am I supposed to get on the other side of it? I can't. You've asked me to stay inside. I don't know if I believe this. I just it's my opening move. So then reflexive dialogue, Matthew, let's say you and I were arguing about something. The reflexive dialogue is, what is Matthew trying to say?

 

I've given up trying to communicate my own stuff, I've given up I've suspended that and saying, how does he see this issue and I need insight and empathy into his experience and his worldview? Because if we can do that, we can get to generative dialogue.

 

Folks, this sounds soft, but it's hard. This was the hardest lab I've ever evaluated - the most exciting, because they were the ones that took this on seriously. And so here's what they said, because we did this, we mostly stayed here. We did not go deep enough in our interventions. And therefore we had a very shallow U.

 

Our prototypes are interesting, but they're incremental. They're not big enough or profound enough to have any kind of impact. So if we want impact, we have to go to Shift Lab 2.0. And Shift Lab 2.0 was to support the evolution of the original three and a strong emphasis on developing capacity for incredibly tough conversations in Edmonton. So here are the three speakers we brought simply to shake up the Edmonton environment.

 

Shelley Tochluk, from Los Angeles - witnessing whiteness, ‘they need to talk about race and how to do it’. She said - she calls everyone whitey, because if you're whitey, here's your problem. And she really delved into it, it was really quite anxiety inducing, actually. But it was really good. So she gave us all these tools and tips just cracked, disrupted, normalised, socialised something.

 

Then we brought Darrell Davis, who's a blues musician. Has anyone heard of Dale? Dale converts Klu Klux Klan members. He goes and says, I'd rather talk with you than fight with you. And he spends time with him. He's on the news, check them on YouTube. He's a real personality. And he's collected 18 robes of people who've thrown in the robes after spending time with them. And he talks about what it's like to be fierce in these conversations and normalise that for us.

 

So many people should know Trevor Phillips, anyone know Trevor – No? UK diversity tzar under Tony Blair. They ran a campaign that dealt with racism in England, but they focused on diversity. And he talked about all the things not to do - to saying you think you're doing the right thing, but you're going to get counterproductive results. Including having signs saying here's the brain from Africa. Here's a brain from Asia - and they're all the same size from Europe. And he goes and here's the racist brain - and it's about this big, and we caught such hell for that. Because what we did is we diminished people, we threatened them, and then they got repressed. And that repression showed up in very odd ways, in inappropriate ways all over the place. We would never do that again.

 

So what I love about this, he's not giving solutions, but they're now creating a container for us to talk about this stuff, where Canadians have a real hard time talking about it. I won't tell you much about Ship Lab 2.0, because the evaluations being done right now, it's way better than 1.0. And it's partly because of this.

 

So folks, that's just a way of organising thinking about when we're tackling complex issues where systems change is the heart of it. We could easily default to traditional logic models, and quants and you know, comparison group designs on discrete interventions. That's not what systems changes. The unit of analysis is the system.

 

The intervention is emergent. There are multiple factors playing out, it unfolds in a messy way over time, hopefully clumsy off in the right direction. So how does one start asking questions that then leads us to the methods that we could use to begin to create a coherent picture of what the heck is going on.

 

So this is what we chatted about - why systems change? What does it look like, how do we  evaluate system change? And so what are the implications for you? These three examples, each reveals something different about this framework. But now what we're going to do with a little bit of time - perfect timing - is turn it over to you for just about five minutes…

 

Laura Barnes

Thank you so much.

Share or Print