We had a meeting recently where leadership was celebrating our incredibly high "successful referral" numbers.
I asked what exactly counts as a successful referral.
Apparently, it's when a caseworker gives someone a resource and checks a box in our system.
That's it.
Not when the person actually reaches the organization. Not when they're accepted into a program. Not when they receive any kind of service.
When we give them a phone number.
I've spent enough time doing direct service to know that half the places on our resource lists have six-month waitlists, disconnected numbers, or eligibility requirements that seem to change every week.
And yet our reports make it look like we're successfully connecting almost everyone to resources.
I want to be clear that I don't blame frontline workers for this. They're already drowning in documentation, and the last thing I want is another form for them to complete.
But our presentation literally said "families connected to services."
That's not what we measured. We measured families who were handed information about services.
I've suggested separating "referral provided" from "service accessed" and maybe following up on a small sample instead of making every caseworker track every referral.
The response was basically, "Well, the funder doesn't require that."
I understand that agencies have to report what funders ask for. I also understand that we can't magically create housing, therapists, or childcare slots that don't exist.
But shouldn't we at least be honest about what our numbers mean?
I'm curious how other people doing macro work, program evaluation, or administration handle this.
Have you successfully challenged an outcome measure that looked great on paper but told you almost nothing about whether people were actually getting help?