Measuring Efficiency

What is the purpose of software for work? Whether for teachers, accountants, or doctors, the basic idea is to enable people to do their work faster or better. If software is slower or produces worse outcomes than the process it’s replacing, it has failed.

Measuring this isn’t always easy. The simple model is something like a manufacturing assembly line where each part of the process can be observed and measured. Automating the initial sorting of the parts either does or doesn’t reduce the average time. But the reality of most workflows is fragmentation across several layers of software and real world activity. 

Does gradebook software produce better education outcomes than paper gradebooks? Hard to say (probably not).

Let’s try narrowing: Does gradebook software reduce the amount of time teachers’ spend tabulating grades at the end of the semester? This can be easily measured, and is almost certainly a yes.

A foggy canyon lies between this measurable, microscopic element and the macro thing that really matters (revenue, time, customer satisfaction). 

The fastest way across is informed reasoning, but I often find my intuition is less informed than I thought! Some tactics that help:

Measure creatively and extrapolate. Find a way to measure somewhere in the middle. This clarifies the impact. If we can determine that teachers spend less total time on grades with the new system, we can be pretty confident the software works. 

Measuring the entire workflow before and after the change can work in some situations. If we’re building internal tools for say a customer service call center and can limit other changes to the workflow, we might be able to assess the effect. But this sort of data is usually too lagging for the product manager to wait on.

Ask the users. When selling software, this actually matters more than the measurement. Because if they don’t think it’s saving them time/money they won’t be renewing. But it’s just as valuable when building internal tools. Though of course people are fallible and sometimes don’t feel the change, even if the data shows otherwise, or in other cases they don’t like the change, even if it is better for the overall goal. 

Often faster is obviously better (less time adding up grades), but sometimes it’s worse (less time auditing expenses but worse accuracy rate). Maybe the most important thing to figure out, and the easiest to skip, is the quality measure that will tell you if things got faster but worse.