We’ve transcribed some of our most popular Product to Product podcast episodes for you to peruse.
In this episode, we speak with Brian Balfour, founder and CEO of Reforge about all the product metrics that product and growth teams should use to measure long-term growth. (Reforge runs masterclasses around growth in product management by partnering with the leading VPs from top companies like Pinterest, Uber, HubSpot and more. Brian was also the VP of growth at HubSpot.)
If you’d like to listen to the entire episode, click below.
Highlights
(The highlights have been condensed and edited for clarity.)
In terms of product metrics, what is a North Star in the product industry?
Brian Balfour: The North Star Metric was originally defined as one identifiable thing that the team rallies around in order to know they’re making good progress. This was a solution to having an incredibly complicated set of metrics or goals that might be hard to communicate and that would send people in the wrong direction.
How do companies determine their North Star product metrics? Is there one method that you think is wrong?
Brian Balfour: There’s a number of wrong pathways to go down.
One would be copying the metrics of what a similar company uses. Investors may often be familiar with a metric because they have another company that uses it and is doing well. Logically, they think that all their other companies should be using it too. That's the first failure point. Don't do it that way!
The second method is to lean into some sort of revenue metric. Most companies exist to generate revenue and profitability, but I ultimately believe revenue is an output of usage. Usage is a better indicator that you’re delivering and solving a customer's problem.
Occasionally, there's complications within the organization where finance thinks it should be one thing, product thinks it should be another and marketing thinks it should be yet another. Every now and then, the metric gets decided upon depending on who has the most influence within the organization.
What are some ways the North Star product metric can lead a growth team or company astray?
Brian Balfour: At Reforge we always talk about outputs and inputs.
When you set a metric for an organization or team at a high level, you’re trying to set an indicator of “Are you making authentic progress on your goals?” What you're talking about is somewhat of an output metric. However, the challenge is that it should not be used to set goals for teams. The reason is that when you look at an output metric, it's too broad, even if you’re thinking of weekly active users or monthly active users. This leads you to say to your team, “Go improve weekly active users or monthly active users.” It’s difficult for the team to come up with a very focused set of ideas to move that number because there are so many levers that fit underneath that output metric. You commonly need to break it down into a slightly more granular set of input metrics or levers, so that if you improve those inputs, you will end up seeing the results in the output metric over time. When the team works toward those input metrics, they end up being much more focused. It gives them a clearer path to actually making improvements along the way.
Now the challenge comes into the second piece of it, which is that these output metrics are always lagging indicators. You might pull on the lever of an input metric and improve it, but you might not see that impact on the actual output metric for some time. This can be really hard, especially for executive teams in larger organizations who are most commonly looking at the output metrics.
You could see a team saying, "We improved this input metric X, Y, Z," in a celebratory mindset. But the executive might be looking at that and say, "Well, why haven't I seen it in revenue?" You have to understand that these output metrics are lagging indicators, which is why it's even more important for teams to focus on those input metrics instead.
Finally, we tend to find that teams almost oversimplify. That goes back to the first thing I said, which is taking this concept almost to the extreme.
There's never going to be a single metric that really captures all dimensions of your business. I like to think about it as a 3D object.
Typically, there are three different metrics that if you’re really trying to get a full representation of how the product is doing or how the business is doing, you need to look at it from different angles in order to understand all those pieces.
In most companies, those three dimensions tend to be some sort of breadth of retention metric, like monthly active users or weekly active users. Next is the depth of engagement metric, meaning out of those active users, how engaged they are and how their engagement is trending over time. The third metric is some sort of monetization metric. We can call this “a constellation of metrics.”
How do you distinguish the difference between an input metric and a vanity metric?
Brian Balfour: In the very early days when this concept emerged, I think it was a response to companies using things like cumulative metrics or number of signups, which I categorize in the vanity metric bucket.
That’s why I start with retention metrics––some form of DAU, MAU, weekly active user or depending on your product, an even longer cycle because if you're actually delivering value to a user, then they're going to come back and keep using your product.
Secondly, you need to take the business view of it.
The most basic example obviously would be if I started with a recurrent revenue number, like a monthly recurring revenue. I could break that down into monthly active customers multiplied by the average dollars per customer. I can also break down monthly active customers into: new customers, customers that have been retained and those that were previously dormant and are now active. From there, I can break it even further into individual pieces.
For example, retention, acquisition, resurrection all break down into their pieces.
So this means that product metrics can’t stand on their own and that they all impact each other.
Brian Balfour: There’s two ends of the extreme.
With metrics, there’s over simplification and over complication. You have to be in the middle.
It’s important you educate the team on the meaning of all of these metrics. This education goes to different levels of granularity depending on what those teams are working on.
On my team at HubSpot, educating new employees on key metrics was part of the onboarding experience. They would learn what the key output metrics are, what they actually mean, why we’ve set them and how it connects to our strategy. After that, they then understand how the things they’re working on connect to the metrics.
You talked about your output product metrics at HubSpot. How did you come up with those?
Brian Balfour: I always go through the same process to set it.
I start with the retention metric because it elicits a number of conversations.
The process that I go through is we first think about the natural frequency of this product. The best way to get to the natural frequency is to look at what problem you’re solving for the user. You ask, “How often does this problem occur in their lives?” Then you ask, “What are some indications that we’re solving the problem for them?”
After that, you get into what you’re measuring––is it a user, team or something else? However, there’s a branding component to it and it’s different for every company. For example, Pinterest uses weekly active re-pinners rather than weekly active users.
It’s important to remember that the metric should flow out of your strategy. At some point at HubSpot, we ended up evolving the strategy of the product that we were working on. As a result, we had to revisit that metric and redefine what we thought the natural frequency was and our definition of active.
That’s all to say, your metric can evolve over time and teams shouldn’t be afraid to do that. However, changing it on a monthly basis means you’re changing your strategy at the same speed, which isn’t really a strategy at all.
What are some of the most misleading North Star product metrics?
Brian Balfour: The most misleading North Star Metric I see often and that I don’t like is the revenue metric. This is specifically troublesome within the B2B world. Everyone rallies around MRR, ARR, but revenue is a massive lagging indicator for usage. What ends up happening in a lot of B2B products, is that they have these MRR or ARR numbers, but when you look underneath the surface at the actual usage metrics, you don't see a lot of usage. You see a lot of “zombie users.” This creates all sorts of misaligned incentives.
Many teams don’t want to make contact with these zombie users in fear that they’ll churn. But it’s more beneficial to know and fix the problem than to find out down the road.
Usually marketing teams focus on the total number of email subscribers, but I orient my team around the number of weekly active email subscribers. This changes the way we engage with our subscribers––being mindful not to blast them with irrelevant messages. However, I do see that most email tools just report a total list size because that’s how they charge customers and make their money.
It’s interesting how tools in the industry can point us in the wrong direction.
You recommend setting three product metrics: retention, engagement and monetization. Can you talk about them?
Brian Balfour: We think of retention metrics as a binary thing, whether a customer is retained or not retained.
To retain a user, they have to do an action that you’ve defined as a way to solve their problem. There are certainly levels and depths of engagement. So you want to figure out the spectrum because the deeper your engagement (meaning the more they do in your product within a certain time frame), the longer term retention usually happens.
For example, retention metrics at Pinterest are something like weekly active re-pinners. The depth of engagement of these re-pinners would be the average number of pins or re-pins per active re-pinner. A lot of teams then define different buckets within that. The most common would be to separate them into casual, core and power users based on the depth of activity.
The monetization metric depends on what your business model is. If it's advertising-driven, then you could do things like what a lot of the gaming or social sites use––some form of average revenue per user (ARPU). If it’s a subscription product, then you’re probably looking at things like MRR or ARR, depending on what that time period is.
So all three of those perspectives (retention, engagement and monetization), really give you the full perspective. Because I could be increasing routined users, but my average depth of engagement could be decreasing. This means that while on average I have a higher amount of retained users, they're not as engaged. Therefore, those users probably aren't going to retain for long. Once again, that’s going to eventually show up in the retention metric in the long run.
Or I could be increasing depth of engagement, but one of my other levers might not be moving along with that. And that can mean a totally different set of things! Just look at all three of these metrics and then specifically when you break those down into inputs and set goals. Then there's a whole other concept of how you’re thinking the decision through, which is called a tradeoff metric.
Do you have any advice for teams who might want to move away from this “one product metric to rule them all” idea?
Brian Balfour: I think companies shouldn't be afraid of moving towards higher data literacy. And clear communication is important in establishing the literacy within a company. If it’s too complicated, nobody understands it.
I go back to the three top level constellation metrics.
We look to track and monitor success over time. We determine how to set them and why they’re important to our strategy. Then you determine the major levers that you can pull on to improve these different metrics. I believe these things should be repeated at every company meeting, board meeting, team gathering so that everyone understands it.
The next thing you have to think about are the layers of details. If I'm going to break something down into another granular set of inputs, I need to make sure that the set of inputs is targeted for that team. Not only that, I want to make sure my team understands those elements. They should know how it ties up to the company’s lever metrics, why they’re important and how it’s representative of the strategy to improve the business.
The individual teams should know this, but to expect everyone in the company to hold the entire picture in their head is impossible. At the same time, you're going to need somebody in the organization that does understand the full picture. We call this the end-to-end quantitative growth model. We recommend that the product leader is the one who fully understands this concept.
Do you think North Star product metrics lead companies away from the idea that growth is spawned from the product itself?
Brian Balfour: I don't know if that's necessarily from the North Star metric.
I think in general, if we were to think about tech, we’ve been moving away from growth that’s just focused on sales and marketing. Instead, we’ve shifted to become a product-centered machine. You can see that in consumer social spaces like Facebook and Linkedin.
The majority of growth is being driven by product and technology. As a result, we need to evolve how we work as teams, how we think about metrics, how they all tie together and how we set goals. In addition, we need to evolve the way marketing and sales teams work cross functionally with product and engineering. These are all emerging things.
Is there anything else you’d like to add?
Brian Balfour: I think the other thing is all of this requires good instrumentation and mentality towards use of data within your company. Otherwise, this is something I call the data wheel of death.
Oftentimes, teams set these metrics, but then things change, break and the numbers look off. People lose trust in the numbers and then don’t use the data to help guide their decisions. This feeds itself into a loop.
At the foundation of all of this, a company has to say using data is not a binary project, but a core part of building their product as well as to be used in their marketing team. They need to know it’s iterative.
Want to how to set and achieve product goals using objectives and key results (OKRs)? Read about it here.