Experiment Your App Up To Growth
At Flurry from Yahoo, we interact with thousands of companies every day; from developers of small messaging apps to large game studios, everyone is looking for the right growth recipe. While there’s no magic ingredient, we’ll investigate the key parts of mobile growth. We’ll be talking to industry leaders in the next months in our new content series Keys to Mobile Growth (???), to discuss successful growth tactics and best practices.
To kick off ???, we had a chance to chat with Sean Ellis about how you can set your app up for “Growth” success. When growth comes to mind, not many names stand out as much as Sean’s does. As the first marketer of Dropbox, founder of GrowthHackers.com, to coining the actual term “growth hacking,” Sean is one of the leading industry experts. What better way to start this new series, than with him?
Anything in quotes below is attributed to Sean Ellis.
Building a growth mindset
First things first, being able to show long-term growth starts with a “growth mindset” across your team and ideally across your company. Sean actually sees a growth mindset as an experimentation mindset since optimization and consequently growth come through experimentation. For example, “Facebook’s whole culture is around experimentation. So somehow that’s not accidental. The Facebook app used to suck and now over time it really has become addictive. But how do you get people excited about experiments? Correct, experiments are addictive when they work.” So before you run experiments, at which metrics should you look at?
What are my KPIs?
It sounds like a basic question, but it’s an essential one on your path to growth. Sean puts it this way: “I like to start with the most important metric and then think about what are the proxy metrics here.” While this might be more difficult than just picking a few of the most important KPIs, it actually helps you a lot on measuring your progress. “What’s the one metric that we can all look at and then say we’re succeeding or we’re failing and everybody kind of agrees. If half the people look at one metric and the other half look at a different metric, then one half usually says you’re succeeding and the other one says you’re failing.” To avoid this, it’s important to find a main metric that reflects the value that your customers are getting from the product.
As an example, the main metric for Airbnb would be nights booked. “If I book a night, that’s valuable for me as a person that’s staying somewhere and also valuable for a host who now has someone staying at their place.” To build your sub-metrics, you can then take a look at the two levers that can increase the main metric: (1) The number of people that book at least one night and (2) the average number of nights booked per user.
Don’t be fooled…
While metrics such as onboarding or sign up completion can affect those levers, they’re not your main metric. “Back at LogmeIn, right after we’ve launched, we had over 1,000 people signing up, and there was this VC who got really excited and started congratulating me. Back in my head I knew that 90% or more of these users were actually never really trying the product. So those numbers did nothing to indicate that we were making progress on delivering value to people.”
OK, let’s talk retention
Retention has become an extremely strong buzzword among us mobile folks in the last couple months/years and at the same time it represents a challenge that pretty much every app developer comes across.
“Ultimately, retention is a proxy for value delivered. And that comes all the way back to product market fit. So if you’re not able to retain people, the truth is, you might just be tactically really good driving growth, but you haven’t created real value for the user yet.” While there’s no unique retention metric target for product market fit (Marc Andreesen popularized this term about 9 years ago), Alex Schultz, Facebook’s VP of Growth, recommends looking at comparable retention metrics in your vertical to define success in terms of product market fit.
Did you know? To make things easier, we recently published a new app loyalty matrix that includes average retention and frequency of use statistics across the different App Store and Google Play categories. So how do you get people to stick around more and lower high churn rates? You optimize your product for engagement, which brings us back to experimenting and testing.
Test your way up
The key for a successful testing environment is to put a framework in place that makes it easy for people to (1) contribute to upcoming experiments and (2) learn from results of previous tests. Sean came up with the idea scoring framework called ICE Score that helps prioritizing experiments and simplifies the process especially when a lot of people are involved. ICE stands for: impact, confidence, and ease. “Every idea that I have is scored on a potential impact on the business, how confident I am about it, and how easy it would be to implement.”
“Ultimately, on a mobile app, experimentation is harder than it is on the web.” Therefore, “on an ease level, you don’t want to look only at how much time it will take to implement a test, but also how much time does it actually take to get it released and approved and what the opportunity costs are.” Because of that level of inflexibility on mobile compared to web, you often can run fewer experiments than on web which makes the prioritization process even more important. “I want to make sure that everything I do on mobile experiments has a very high confidence score and probably impact score as well.”
In terms of monitoring, Sean emphasizes the importance of documenting your experiments. “The more consistent you are on how you define your experiments, the easier it is to organize the results information afterwards: of what worked and what didn’t work and to make all of that accessible to the team. I think a lot of that really starts with having this information systematically organized and accessible. And nail that before you start involving the rest of your organization or you’re just scaling a huge mess.”
What are your thoughts on Sean’s ideas and how do you plan to implement them into your growth initiatives? Who would you like to hear from next?