Lean Data: The “how” of measuring impact at lower cost for higher value

Acumen asks the impact enterprises it invests in, “What data can we help you collect?”

Operation Impact


Photo credit: SolarNow

Here’s an impact investing myth that needs busting: The proliferation of multiple frameworks for measuring social impact means that front-line employees, managers and investors are getting the kind of feedback and data they need to improve their operations — and deliver genuine impact.

With a tip of the hat to the Case Foundation’s Rehana Nathoo, who in her excellent opening to this Operation Impact series of essays called for continual myth busting, we’d like to bust “the metrics myth.” That’s what Jed Emerson, the originator of the term “blended value,” calls the misconception that the collection of proper impact metrics has become widespread. Sadly, it has not.

According to an annual survey of impact investors, unsophisticated measurement practice is one of the five biggest constraints to growing the impact investing market. With so many frameworks out there, why aren’t we making more progress?

We wrestled with this question as we began our own work creating the Lean Data approach to help build more impactful businesses by providing them with data on their social performance, customer feedback and behavior. Surely if we all agree it’s critical to measure impact, we’d be making greater progress? It’s not as if all that framework development is cost free. It consumes an awful lot of effort, for slim rewards.

The cart before the horse

A focus on frameworks, metric taxonomies and theories of change centers on “what” to measure. That’s all well and good, but in the absence of practical approaches to the tougher question of “how” to measure, these frameworks end up as little more than impact window dressing. They give the impression that impact is being measured, but boil down to either crude estimates using re-purposed operational measures (such as unit sales) or weak proxies from established research.

Focusing efforts in this way, puts the “what” cart in front of the “how” horse. Unpicking this conundrum rests on acknowledging that our impact measurement playbook has, to date, been poorly conceived.

For a start, impact measurement has a terrible name: monitoring and evaluation (M&E). Yeesh, who wants to be monitored? Secondly , measurement has by and large been excessively expensive. And lastly, and perhaps most importantly, the top-down framework fetish has meant that front-line organizations (those who should see greatest value from measurement) have little autonomy over, and hence gain little value from, impact measurement.

It’s pretty clear that an approach which is badly branded, costly, and not in demand by those who should be benefiting from the collection and use of data, is unlikely to scale.

“How” to turn measurement on its head

Based on the analysis above we’ve been working hard to develop an alternative approach that solves these challenges. Lean Data is rooted in two simple goals: drive down costs to measurement and significantly increase the value to enterprises of gathering data.

Photo credit: Acumen

To deliver the first, we’ve invested heavily in remote data collection. We’ve been leaders in experimenting with tools such as SMS, online surveys and simple phone calls for impact measurement. We have found they are highly effective at getting great data, especially in remote settings. And by focusing our data gathering and analysis on what investees want to learn, we’ve maximized the value generated. In practical terms this often means blending social performance measurement with other kinds of customer intelligence (e.g. Net Promoter Scorecustomer archetypescustomer care performance, and more).

This approach is not a free lunch. There’s the direct cost in terms of collecting data. But our bet is that once you start focusing efforts on data-collection rather than framework creation/metric-selection, you’ll find the value you create for each hour spent is dramatically improved. This bottom-up data approach also means that as funders or investors, we may have to accept that the metrics we want may not match those prioritized by front-line organizations.

Often, the fear is that this will prevent aggregation of impact data. Our experience is exactly the opposite. Companies in our portfolio have had surprisingly similar interests. Moreover, when they are made a data offer (“what data can we help you collect?”) as opposed to a data request (“you must collect this data for me”) our investees are much more excited about measurement.

Applying the principles of Lean Data, combined with a steely-eyed focus on getting end-consumer data at scale, we’ve transformed the measurement (and management!) of social impact at Acumen. In the past two-and-half years we’ve run more than one hundred Lean Data projects. In the process we’ve interviewed more than 35,000 investee-customers.

We’re still learning how to improve Lean Data in terms of both lowering costs and delivering ever greater value to investees. We’ve learnt one thing for certain: banishing the metrics myth is possible — if we re-orient our approach to measurement. It needn’t be so hard.

In Operation Impact, leading practitioners bust common impact investing myths with practical solutions. The series is brought to you by ImpactAlpha in partnership with the Case Foundation, Acumen, Bridges Fund Management, and ImpactSpace (ImpactAlpha’s open impact database). More from the series:

Source: Lean Data: The “how” of measuring impact at lower cost for higher value

Related posts: