Healthy software metrics: How successful software teams measure their work

Author: Carol Lee, PhD, Morgan Ramsey, & Cat Hicks, PhD | Read time: ~ 17 mins 

In the first whitepaper in this series, we discussed the dilemma facing engineering leaders today: Increased scrutiny of engineering productivity, as well as increased pressure to retain the best talent by improving the developer experience. We proposed the Developer Thriving framework (in our first whitepaper) and the two kinds of visibility (in our second) as a way to improve both engineering performance and the developer experience.

But there’s something we haven’t addressed yet. Engineering leaders are facing pressure from board leadership to quantify engineering performance. This has led companies like Twitter and Salesforce to rely on severely flawed metrics such as lines of code written.

In part, because most engineering leaders haven’t yet had access to the data that allows them to own the story on their teams’ productivity, resulting in unhelpful measurements and poor developer experience.

Engineering leaders need to measure engineering performance in a way that’s accurate for complex technology work, and then communicate that information in a way that’s easily digestible for business leaders. But how can they do this without demotivating their engineers or making their developers feel micromanaged?

Our research showed that 87% of developers agreed that tracking parts of their coding process 

would help their teams make trade-off decisions, increase the visibility of technical work, and help them see their own productivity (Fig. 2).

Yet only 20 – 30% of devs reported teams using team-level metrics, and 26% of devs said they didn’t even know whether their managers and leaders used software metrics to understand their engineering work. 

There is a huge opportunity for current and aspiring engineering leaders to reimagine best practices for engineering measurement and evidence-based decision-making, therefore increasing clarity throughout their engineering organizations.

Our research found that:

• It is a misconception that developers don’t want to measure their work. They generally feel positively about using metrics, agreeing that they help with visibility, trade-off decisions, and understanding their own productivity.

• When used correctly, software metrics can address business needs (and questions from the board) and enhance developer experience by improving visibility in a scalable and less biased way.

• Racially minoritized developers were more likely to find it impactful to have parts of their coding work measured.

• There are patterns in the pitfalls that developers see with implementing software metrics on their teams. When developers whose teams use software metrics were asked to identify pitfalls in the ways their teams were using metrics, the most common pitfalls were not measuring enough and measuring without context.

In the rest of the paper, we cover:

• The current state of engineering metric usage

• The benefits of engineering measurement

• Common pitfalls when using engineering metrics

• Recommendations for better measurement