Product Metrics that Actually Matter

It’s alive! Your product is up and running and users are starting to que up for the Sign Up button.  Now, how do we know how to proceed? Clearly, our next step would be to optimize the leads funnel and performance. We need benchmarks and we need to decide what to measure. Product quality metrics should reflect our goals and achievements. We need to think like scientists.

Think Like a Scientist

“Believe nothing of what you hear and only half of what you see” – Edgar Allan Poe

Poe meant it in a spooky, ghost-infested way, but I find that it hits the nail on the head, with regard to selecting metrics and product progress. In other words – when making decisions that affect your product growth, trust only the numbers in your carefully selected product metrics.

When approaching the subject of software product metrics, we need to harness our ability to think like scientists and leverage critical thinking. Our thought process should be oriented to drawing conclusions, acting only on facts. We must avoid falling in love with a theory and forming a perception around it. For that, we need to select the right product performance metrics to act on.          

Mirror Mirror on the Wall, which is the most flattering product metric of them all?

Some product quality metrics are more tempting than others. They sound sexy, give off a super-relevant vibe and are more accessible than others. That doesn’t mean they are the ones to follow. Vanity metrics are metrics that make you feel really good but in fact are deceptive, in that they either don’t tell you anything substantial or else they divert your attention and energy to the wrong things. Some examples for vanity metrics:

• Number of hits
• Number of page views
• Number of downloads
• Number of visits
• Time spent on site
• Emails collected

If you’ll pay close attention, you’ll notice that none of these product performance metrics tell you how to proceed with agile development. They don’t show you where you fare better; they don’t tell you how to improve the traffic volume and they don’t indicate what you should improve. They are basically only good for patting yourself on the back and saying “good job!” to your reflection in the mirror. 

Good Metrics Change the Way You Act

So which are the right product development KPI metrics to follow? If I had to go all meta and name a metric for good product metrics, I would say the best way to test a metric’s effectiveness is by whether it alters your product development paradigm.

An actionable metric is Understandable, Comparative, a Ration/Rate and above all – Actionable.

Understandable, because it is too easy to go down analytics rabbit holes, drilling down to sub-categories of side-funnels or super-narrow specificities. The number of Android users that requested a demo on Wednesday before Thanksgiving – that number won’t unravel the secrets of site analytics or expose the greatest agile hack of them all. Find solid, understandable metrics you can return to and stick to them.

Comparative means context. Only if you have some reference or benchmark can you make sense of a metric. Same goes for ratability and actionability.

For example: the rate of conversions per website visits – is actionable. You can aspire to widen the funnel and expand the rate. The number of visitors to your website – is not an actionable figure, because without context or a reference – it is meaningless.

 

How to Decide Which Metrics Matter

There is a term coined by Donald Rumsfeld in the wake of 9/11, known as – “the known knowns”. This notion helps to map out the things we know for a fact we know, the things we know we don’t know, and the things we don’t know we don’t know. Originally borrowed from the field of psychology and self help, this exercise in inducing self-awareness helps in any strategic move.

The “known knowns” map is essential to the product development process – it can help you facts from intuition and focus you, the product manager, on the task at hand: do you need to obtain more facts? Are the facts you have sufficient, or do they require confirmation? Do you need to act on the definitive facts you have?

Another parameter that should factor into your product growth metrics is, of course, conversion. Here, too, we need to be careful with our perception of conversion. Conversions can be considered as one of two types:

• Actual conversions – or, as I like to call them, “mega-conversions”: sign-ups, demo requests, contact forms completed.
• Mini conversions – these are all the minor user achievements your funnel is built around: email submissions, social followings, video plays, free trials. Those also require monitoring and optimizing, but differently.

Analyze This

The final aspect of scientific approach to product growth is analysis. Selecting the right metrics to take in is one thing, but understanding what to do with the information is an entirely different feat, and just as important.

The scientist in you should aspire to find patterns and connections between things. Especially ones that evade the common eye. For instance: finding correlation between conversions and visits to a certain page. This is part of the product manager’s job – looking at the big picture and identifying the relationship between the different components.  

BONUS: example of product roadmap – download it for free at craft.io