How can program managers, policy makers and evaluators draw on the diverse range of community sector data to conceptualise and assess the social benefit that services provide?

Brentyn Parkin of My Community DIrectory talks about some of the ways his organisation has addressed this issue and puts forward suggestions for improving data collection and use in the sector.

The Australasian Evaluation Society Queensland and studioQ collaborate to bring you this lunchtime seminar on evaluation.


Today I wanted to talk really at a very much practitioner level about what we've been doing and some of the work we've done with evaluation frameworks.  And I guess for me, it's been very, very clear this idea about smarter not harder hasn't really sunk in across a lot of the community sector.  And it's not because we don't want to, it’s, I think, because we haven't really thought about how we can make it simpler.  And I know that as I look around the room and I see a number of evaluation professionals here and it's really good to be able to think about how this works for the community sector.  

Who are we? I'm going to start with that.  Community information support service is a not-for-profit that has been set up and established to help support information-sharing and help the community sector work together.  With that in mind, it's interesting that we've ended up working across the community sector, my background, and then technology platforms. Because it's often that link which is a really big problem.

The real question is why evaluation?  Why would I have a deep interest in evaluation when I'm working across information and IT platforms?  Why would evaluation be such a key tool?  One is cost.  I mean we've done three projects which we're going to talk about today and we've used an evaluation framework.  Those three projects we've been able to deliver within budget.  We've got several other projects that we've worked on without evaluation projects, without an evaluation framework and we have outgrown budgets beyond what we would expect.  And I truly believe that an evaluation framework, and we need to be thinking about how we sell evaluation.

Evaluation is around helping people understand, get on the same page, and obviously you guys as professionals, keep us on track.  So we've been doing that using that methodology.

We've got several different evaluation frameworks we've used. So this was one we worked with volunteer program logic, really quite simple, we were looking at the organisation policies, quality experience. If we have quality experience we're going to have better client outcomes.  Outcomes lead to community benefit.  Obviously it works the other way around.  It's that normal program logic.  

Here's another one that we worked on recently.  This was trying to help participants get the most appropriate services to meet their goals in the youth sector in Logan.  So putting the framework together gave us a really strong guiding way to move forward and keep people focused on what we'd set out to achieve so that we could truly evaluate.  

In the end we are now starting to evaluate "do we have a coordinated approach to quality employment services for marginalised young people?"  And we're able to now go back to where we started twelve months ago and now have a look at that, but again within the cost structure we've been able to maintain it because we don't keep running off in different directions.

So I think, often evaluation can be seen as expensive and the  community sector don't understand it's actually one of the good tools to keep you in budget and if we start selling and thinking about evaluation in that way, I think we can get different types of.

Being smarter is simply four things.  We found that using information that's already collected is the key.  And so often in an evaluation framework we're trying to asses and evaluate certain things but a lot of information has already been collected.  

When we've been able to use that information that they're collecting already in a format, we may change the mode.  We can make pretty good progress and it really contributes to the ease and being able to do evaluation in a smarter way.  

The other thing is map what's been done.  Talking to the organisations because we've been working across organisations, getting them to map those current relationships and what they're currently doing.  Taking that time has been part of the success.  

The other thing that I think is really important is linking with other information sources. I truly believe information is power but when you can overlay information with other sources; community information, census data, and you can start to overlay that information, it brings so much more richness to the community sector and their understanding and their buy in the evaluation process.  

And finally what we've done, and I, this is probably been a key, so each one of these is, we did it together and that's where the success came from by using evaluation to know that we've actually got a real life solution.  By sitting down with the community organisations and going through and saying, well this is really our goal, we've made sure that that evaluation does actually stack up.  

So we've got a solution that has is solving a real life problem.  I'm going to go through three examples today so we can just demonstrate how this works.  And the first one is service linker.  And service linker really was a response to quite a big problem.  That problem being clients were receiving duplicate services and not progressing towards employment.  

Now employment is expensive, trying to support someone to get a job. And we were finding with some of our organisations we worked with in Logan, people weren't progressing through those outcomes, or they were jumping around. And people in the employment sector couldn't get their head around it.  So what we did was we came up and we went through those four things.  We looked at what was already being collected, we mapped their work, we looked for other information sources and then we started with our evaluation framework.

And we're just going to go quickly to service linker, service linker was a really simple product and it was about getting organisations to use their current process, we have to get people to sign into our buildings anyway.  We're collecting their name, their number and a bit about them, who they need to see.  So we went through that same collection process and said, if we were to collect that information, and we knew why the person came to see us, would that help us understand how people are accessing services?  So we looked at that and then we mapped it across the services and we realized that information which was being collected and will still be collected, if we could change its mode, we could then understand how people access service delivery. 

We ended up with something quite simple.  Really important was keeping information safe.  So this is about client information.  So firstly organisations who use the system sign up, so we can check that they're part of the system.  This is the organisation Wesley Mission and part of the program, we're logged in.  And giving them control of what they're mapping.  Quite simple and then they can add users themselves.  Really simple system, but it’s what happens then, when they open their user attendant, this is what they leave sitting at their front desk.  And when people log in, they can simply say, "yep, I've signed in before," continue, what day were you born? Continue.  It's really simple, it takes about eight or nine seconds for people to log in.  When they log in, the information is put into the system.  It finds them if they've been to any of the participating services, so they're not duplicating entry of data.

It then just puts, it translates their information to the statistical linking key to protect privacy and then logs that entry into the database.  One of the challenges was quality service delivery.  When you're with the person, if they want to show where they've been, it can go back to the database and display all of the instances of service delivery, so that the person, the case manager working with them can talk to them about how they're progressing.  So, a really simple system using the data that was collected.  

The other thing that became really important was the services, the managers then wanted to go "well I want to know more about this thing."  So, we were able to then de-identify the data and then provide it back to the group of organisations. So now we can look at people that are attending multiple service deliveries, an important piece of work which was quite simple.  

Now with the evaluation data we're now collecting service usage, frequency of access, gender, age, location and the type of service that information is allowing services to really consider their effective behaviour.  What's working?  What isn't?  And challenge their practice.

It's also collecting information so we can see when people, if there are certain demographics attending multiple services.  So we use the information being collected from that front book, and we've really only added two other fields to make it unique so that we could sign in, in about eight or nine seconds.  If it was longer than that, it wasn't going to work.  

We mapped across the services and now we've got a map that is building over time and we can see all of these things which is mapping that service delivery across organisations.  Not been done before effectively in real time.  We linked with other information sources we can now actually overlay that with where youth are living and see how far they're travelling.  

And finally it's been great because the organisations involved have said that was a real problem and we've been able to solve that.  The final real solution to the problem when we rolled out that evaluation framework was the services also needed to keep that register at the front for list.  We've been able to do some modifications so every person including staff can log in and log out really quite simply.  So again that requirement meant that we went back and did a second round of changes.  

Another project I wanted to talk about where we used exactly the same methodology to get a real problem solved, was a product called volunteer alert.  Now volunteer alert, the problem that we were trying to solve was engaging community organisations and their volunteers into action at the right time in an emergency context.  When an emergency occurs, and we found this out while talking to organisations, if they don't engage their volunteers to help the people that they already look after, they end up with spontaneous volunteers.  They end up in a pool somewhere in a mud army.  Yet if we could utilise those volunteers to help Mrs. Jones, who we already have a relationship with, we can actually provide a different type of service for vulnerable people.  

Community sector often goes well, we couldn't really engage our volunteers so they all run off to the mud army then we can't support people we know best.  Volunteer alert was really quite a simple system again.  And our goal for volunteer alert was to try and look at ways to help managers communicate quickly, easily, simply.  

What we've done here using your my community directory login.  So we've already got a login and password to another system, you don't have to fill out all your details. We allowed organisations to login.  We created a really simple tool pulling together email and SMS.  For the reason being the volunteers needed to hear in their phone that they were needed, but often 160 characters just isn't enough.  

The community sector can now send an alert, purchase credits at just a commercial rate, but they can choose from their list of people that they are using, so they know all of these people.  You can actually just go alright, select all the people on my list.  Add them to the selected contacts list.  You could add everyone.  Every list together, but it will only send one SMS and one email because that's all we need to send.  You type your email, need to fill some shifts and up here you might say please see your email for details about those shifts.  Then you can just cut and paste your little table in there about what you're trying to fill.  

We worked with an organisation that helped us develop this and they found that they're saving over 50% of the time that they were using just contacting volunteers.  Again, using the evaluation framework, real life problem, mapping what they're doing.

So that was the first step, but with the evaluation process, we'd solved the problem to communicate but it's that next step that our evaluation process helped us identify and that was what else do we need to know if we were going to map this information.  Now using volunteer alert, we can actually look at the response rates for SMS's we can now look at comparisons by regions for responses and the types of messages.  

We can look at how organisations receive responses across a group of organisations, and look at response times.  The information the organisations were collecting, were the volunteers, the contact mobile, the phone number in a spreadsheet.  So we said well you're already collecting it, what if you could just upload that spreadsheet. 

So we used the information already collected from other systems.  We mapped again where those messages were going and the response times.  We were able to then link that within the emergencies that it's been used what was happening, so then you can overlay other data. And again it brought us back to where we started in looking at: did we solve a real problem.  

The final product I want to talk about which has been a much bigger project.  To be honest, we didn't start out with an evaluation framework for it.  In fact we've really been looking at some of those things more recently.  Building a way for more information to be shared.  

So the problem again was there was no common source of information for use by community organisations, public councils or government.  We know that there's places like white pages, yellow pages, Google.  So we're not saying as an individual you can't find information, because we know you can.  But what you can't get is a excel list of all of the disability service providers in your area, all of the sport and rec. groups.  Now that's what council need to use if they're going to fund and support communities.  That's what government need to know if they're going to understand service delivery. As a manager of a not for profit, I need to know the information to do my core business.  

So being able to give access to live data, whether you can manipulate, search and in an Excel spreadsheet which is how we do most of our business is the key to helping make that information useful. Again, we applied the idea that we needed to look at what that real problem was, however, as we've refined that and developed a little bit more of an evaluation framework and we're now developing it on the run. 

What we've found is that we've taken some different decisions.  We've looked at our product differently because we've asked our organisations what do you need to understand the sector.  Now I'm actually logging into a council level login.  When you log in, what was really important as we started to talk about how organisations can use this as part of their evaluation, understanding the sector and the work they do, this became really important.  In the background we've been collecting statistics for nearly a year, well we've collected them long before that but we changed our methodology almost a year ago and now we collect the actual pages, types of pages people click on.  We collect when searches show up in a result and all kinds of data to help us understand our database and how it was being used.  

However, it became really clear that information became important for the community organisations, so that they could understand the sector.  There are 997 organisations listed in Brisbane City Council area. 1400 outlets, physical locations, delivering over 2,300 different types of services.  Now we're not there yet, we've still got a long way to go.  But, we're working towards better data quality.

In August, we had 2,393 pages on our website pdf'd or printed.  So people are using it.  Very interesting when you start to look at what that means.  And so what we've just provided to them this month, after talking to them about what the need to understand the sector is some demand driven service delivery information.  So, in the Brisbane City Council pages, this month, so September, we've had 8422 health pages viewed. People have actually looked at pages containing health information. We've had 168 of those pages pdf'd or printed and 11 emails contacted through my community directory to the organisation.  

When you start looking at this information, we're now looking at what the public, the 150,000 page views a month, are looking for.  So it's really moving into demand driven service delivery, understanding what are looking for. Because when we spoke to community organisations they were saying, if we could overlay this type of information with the other information we're collecting, it becomes really powerful.  The services by type and different types of organisations.  So, that is now available to the community organisations, not just the council.  The other thing that community organisations really do every, every, you know, how many people are trying to service map.

What we've done is then just put all that information onto a map.  But the problem with service mapping is everyone has to do their own service map because I need an aged care and a youth service map and someone else needs a service map with different categories of information so what we've done for the sector, is allowed them to say actually I'm doing a project with oh disability services and indigenous services.

So now, we can overlay this service mapping information we need for our project. We can have a look at this and see who the manager is, what their contact details are, using a mapping based function because the sector needed to do their business.  And this has all come out about that idea of, what are we trying to achieve?  Not just us but the community sector.  What are those community goals at the top of our evaluation, you know our, our program logic?

And then we've worked back and said how could we provide that simply, cheaply and effectively.  And then you can look at who they are, what they're doing, and this is real data.  The really great thing is, as people update it and join, their data becomes part of the mix and they get better value.  What happened with the change of government that I'm pleased about is they start looking at open data.  

Now we can start overlaying public toilets. If you're working with someone who is incontinent you can then look at different ways of supporting them.  So, there's all this new data coming out enriches and overlays and allows us to go back to the work we're doing and see if we're going to reach those big goals. 

Members of 'My Community Directory' can actually download the data into an Excel spreadsheet.  So you can actually download organisations, outlets, contacts, just want a list of the contacts here because I'm looking for someone.  The organisation contacts that you have in your organisation and then the services.  So, that's about 2,000 lines of information.

Organisations are repeated because it's about what they do.  Again depending on what you're looking for, we're trying to serve up information that you need so that you can overlay that and understand the community sector.

When community organisations go to this communicate button, they can then use volunteer alert.  So they can communicate with their members.  But, councils now can log in, and partners, so if you're doing some evaluation work with council, make sure you check because they can now SMS and email. But Brisbane City Council can actually email a selection of not for profits across an area.

So when we start looking at evaluation and trying to engage and do consultation, we've got all this.  So they can actually go and send things out.  They can see who is accessing it and when that information's being used.  So with value added to Council so they come and use the information and help keep it up to date.  Evaluation data for My Community Directory we've done a lot of things that this product does, but the evaluation data that we've now been able to build in is demand by service type, comparison by location and we're now looking at availability and service gaps.  

That's My Community Directory and our projects and what we're doing.

Share or Print