Learn how to raise your odds of success

 
 
 

How I am Measuring Product/Market Fit

Written by Ash Maurya

I can summarize my last post with a strong call to action:
Focus on Product/Market Fit First.

Here I’ll be laying the groundwork for how I plan to measure and optimize the process of achieving product/market fit. Metrics are, of course, the answer. But I have spent enough cycles getting lost in Google Analytics’ numbers, that it’s not just about numbers, but the right numbers. The right numbers vary over time and have to be actionable.

Dave McClure has some great models for startup metrics which I’ve decided to adopt. He talks about creating a 1-page business model, and identifying 3-5 actionable metrics that you live by (conversion dashboard).

There are 5 basic steps to any conversion process that Dave defines as :

  1. Acquisition: How do users find you?
  2. Activation: Do users have a great first experience with your product?
  3. Retention: Do users come back and use your product?
  4. Referral: Do users tell others about your product?
  5. Revenue: How do you make money?

While you should start laying the foundation for measuring all 5 steps from day one, you don’t have to start optimizing all of them. That said, you should have definitely thought through all the steps and be able to state how (you think) you’re going to address each one (Customer Development Hypotheses). It is also critical to have had some customers travel through the entire conversion funnel first (Customer Validation) before starting any optimization. Lastly, for any optimization to be statistically meaningful, you will need to establish a few channels that drive just enough new users entering the top of the funnel (Acquisition). These could be either through free channels (blogs, seo) and/or paid channels (adwords, banners). Just don’t break the bank (not yet anyway).

1-Page Business Model

For CloudFire, I am starting with 3 core metrics: Activation, Retention, and Revenue. Here’s my 1-page business model that summarizes my user types, conversion events, and priorities:

1-page-bm

Note: It’s important to frame the metrics as specific actions the user takes which makes them both measurable and actionable.

Activation is really critical to measure as CloudFire is a downloaded product which creates more friction to overcome. Retention is probably the most important metric for validating early traction and product/market fit. As I mentioned last time, Sean Ellis uses a 40% retention test to gauge product/market fit.

Some people advocate deferring Revenue to later but I believe getting people to keep paying for a product is the ultimate product/market validation. Since CloudFire is a premium service with no FREE plan, Retention and Revenue should follow each other quite closely. I believe that unless the end user isn’t the one ultimately paying, you have to charge from day one. You should already have asked the pricing question during Customer Discovery and should already have acquired a few paying customers from Customer Validation. Now it’s time to test if that pricing model will scale.

Referrals, while probably the most critical channel for CloudFire adoption, will take a back seat to Activation and Retention for now. People don’t refer products that don’t work or solve problems. You have to be remark-able first.

Parent Conversion Dashboard

Here’s my conversion dashboard for the Parent segment:

parents_dashboard

Note: Conversion dashboards let you visualize the entire conversion process but I am only going to *initially* focus on Activation, Retention, and Revenue.

The estimated values on the right are just that – estimated values that help quantify each event relative to the one real transactional revenue event. They will come in handy when I start optimizing a particular metric and want to measure the effectiveness of different approaches.

How I’m measuring Acquisition
This is the easiest metric to measure. What I’m looking for here are unique website visitors that didn’t bounce, spent enough time on the site, and made it to the Pricing page. Most analytics tools can provide this out-of-the-box. I was using Google Analytics but more recently have been using KISSmetrics.

While measuring overall Acquisition is easy, optimizing it is quite another story. Measuring the effectiveness across all campaigns in Google Analytics is a lot of work because the relevant data is spread out across multiple screens. I am not really at this stage yet, but I have come across 2 new services in private beta (KISSmetrics is one of them) that show a lot of promise here. KISSmetrics fundamentally uses people versus page views as the basic unit which makes funnel and conversions visualization a snap.

How I’m measuring Activation
I define Activation as the set of steps: Sign up -> Download -> Create First Gallery. Since the Activation steps start on the product website and end in the downloaded application, I had already starting building my own custom funnel analysis tool before I discovered Mixpanel.

Mixpanel is real-time analytics for your application. You can use it to track any application event using Javascript/PHP which can then be used for funnel analysis, retention, and macro-level A/B testing. So far I have found it to be very useful and was able to replicate my homegrown funnel report in a day. The presentation of the reports could be better though. I still use my homegrown report for other things but am increasingly depending on Mixpanel.

I have only started using KISSmetrics and I think that it too might be able to handle custom events. If it does, it would be interesting to compare the two. I definitely like the 1-page KISSmetrics reports and would prefer to use a single analytics tool, if possible.

How I’m measuring Retention
For now, I’m using a custom database report to find users that have shared at least one thing per month for the last 3 months.

Mixpanel seems to be designed for measuring user retention around a particular action. I simply send an event each time an album or movie is shared and Mixpanel tracks it and automatically generates cohort reports. The data is also available through an api which I’ll probably need for my specific metric.

Another approach to measuring Retention is to tie it to Revenue. Since CloudFire has a 14-day trial, it’s fairly easy to measure retention that way. Beyond the 14-days, CloudFire is priced annually and I am considering A/B testing a monthly pricing plan on Nivi’s suggestion. Another approach might be to offer a money back guarantee. The idea is to give users an easy way out. If they stay, you know it’s because of the product. There is no better measurement of Retention than recurring payment.

I haven’t had much experience running surveys, but it might also be interesting to conduct a survey based report using Sean Ellis’ 40% test.

How I’m measuring Referral
I am not currently measuring this metric but am using Mixpanel to track an event each time a user uses the “Tell a Friend” feature and let Mixpanel track it for me.

How I’m measuring Revenue
I am using a homegrown report to measure users that convert after the 14-day free trial.

Visitor Conversion Dashboard

Here’s my conversion dashboard for the Visitor segment:

visitors_dashboard

At this stage, I’ve defined but not implemented this dashboard. Since Parents come first in the sharing process, I’ve decided to implement that out first.

What’s next?

To reiterate, my sole focus is working towards Product/Market Fit. In Dave McClure’s model, that equates to optimizing for Activation and Retention. I’m going to measure but ignore the rest.

I am also a little weary of falling into the trap of just relying on numbers. Numbers tell you what is happening but not why. Yes, you can build yet more numbers to try to get to the bottom of an issue, but almost every time, it’s just easier to reach out and talk to a user.

Next time, I should have some actual numbers from my Parent Conversion dashboard to share with you and start diagnosing the health of Activation and Retention.


Update: If you liked this content, consider checking out my book: Running Lean which dedicates 50 pages alone on this topic.

You can learn more here: Get Running Lean.

  • http://www.firestarterlabs.com/ TJ Goan

    I really apprecitate you allowing us to follow your progress!

    Thanks for sharing and good luck to you,

    TJ

  • http://www.firestarterlabs.com TJ Goan

    I really apprecitate you allowing us to follow your progress!

    Thanks for sharing and good luck to you,

    TJ

  • Pingback: Twitted by ericries()

  • Pingback: uberVU - social comments()

  • http://www.arieldistefano.com/ Ariel Di Stefano

    Great Post, as usual.

    I really like your pragmatic approach using the best of each technique (LeanStartups, AARRR, etc).

  • http://www.arieldistefano.com Ariel Di Stefano

    Great Post, as usual.

    I really like your pragmatic approach using the best of each technique (LeanStartups, AARRR, etc).

  • Suhail

    Hi Ash,

    Thanks for using us and the explanation of how you’re using Mixpanel. Feel free to send me an email anytime regarding your feedback at suhail[at] mixpanel [dot] com.

    I agree, our funnel view could be better and I am interested in what you think would be better =)

    Suhail

  • Suhail

    Hi Ash,

    Thanks for using us and the explanation of how you’re using Mixpanel. Feel free to send me an email anytime regarding your feedback at suhail[at] mixpanel [dot] com.

    I agree, our funnel view could be better and I am interested in what you think would be better =)

    Suhail

  • http://easitour.com/ Dan Hodgins

    Will add your insights to my business system tonight.

    Of particular interest is the fact that you are measuring everything, but realizing that numbers must have context and meaning to be relevant and actionable.

    D.

  • http://easitour.com Dan Hodgins

    Will add your insights to my business system tonight.

    Of particular interest is the fact that you are measuring everything, but realizing that numbers must have context and meaning to be relevant and actionable.

    D.

  • http://padicode.com/blog/ Claudiu

    Hi Ash,

    Thanks a lot for your sharing. I love the one page dashboard and placing all your important KPIs in there. Is this something you’ve just started or has it been going for a while?

    I like it cause almost all of your KPIs talk about user engagement from your visitors. You focus a lot of attention on the users who don’t just browse but rather engage.
    However this assumes that people who just browse are not interested in your product that much… which is not always true.

    Well, just a thought. Anyway, I am going back to my drawing board and get my own cool dashboard as well. Thanks a lot.

  • http://padicode.com/blog/ Claudiu

    Hi Ash,

    Thanks a lot for your sharing. I love the one page dashboard and placing all your important KPIs in there. Is this something you’ve just started or has it been going for a while?

    I like it cause almost all of your KPIs talk about user engagement from your visitors. You focus a lot of attention on the users who don’t just browse but rather engage.
    However this assumes that people who just browse are not interested in your product that much… which is not always true.

    Well, just a thought. Anyway, I am going back to my drawing board and get my own cool dashboard as well. Thanks a lot.

  • http://www.ashmaurya.com/ Ash Maurya

    Thanks for the comment Claudiu –

    First, I’d like to point out that the only people ignored are those that bounce i.e. don’t even browse. All others get measured as engagement. Also, the analytics tool I’m using does track repeat visitors so even if they browse a few times before they signup, they are correctly tracked.

    The service is still fairly new and yes you do have to invest some time upfront to attract just enough level of engagement to move people further down the funnel to the product. A lot of people spend time optimizing the acquisition process but if the product isn’t right, that’s just wasted effort.

    Once the product fit is there, you will go back (with your new learning) and optimize the site for increased engagement.

    Cheers,

    Ash

  • http://www.ashmaurya.com Ash Maurya

    Thanks for the comment Claudiu –

    First, I’d like to point out that the only people ignored are those that bounce i.e. don’t even browse. All others get measured as engagement. Also, the analytics tool I’m using does track repeat visitors so even if they browse a few times before they signup, they are correctly tracked.

    The service is still fairly new and yes you do have to invest some time upfront to attract just enough level of engagement to move people further down the funnel to the product. A lot of people spend time optimizing the acquisition process but if the product isn’t right, that’s just wasted effort.

    Once the product fit is there, you will go back (with your new learning) and optimize the site for increased engagement.

    Cheers,

    Ash

  • http://openswipe.com/ Casey

    This stuff is definitely fascinating. Keep it up!

  • http://openswipe.com Casey

    This stuff is definitely fascinating. Keep it up!

  • http://www.personalmedicineinternational.com/ Natalie Hodge MD FAAP

    Thanks for sharing… just signed up for Kissmetrics beta, will be interesting to see if easier than google analytics, I’m hoping its free for duration of the beta… comparing to monthy analytics like hubspot… anyone out there using hubspot by the way??

    Natalie Hodge MD FAAP
    Medical Director
    Personal Medicine
    http://www.personalmedicineinternational.com
    ” Your Doctor Comes to You”

  • http://www.personalmedicineinternational.com Natalie Hodge MD FAAP

    Thanks for sharing… just signed up for Kissmetrics beta, will be interesting to see if easier than google analytics, I’m hoping its free for duration of the beta… comparing to monthy analytics like hubspot… anyone out there using hubspot by the way??

    Natalie Hodge MD FAAP
    Medical Director
    Personal Medicine
    http://www.personalmedicineinternational.com
    ” Your Doctor Comes to You”

  • http://hopshopper.com/ Marko

    Ash, this is great stuff. We are at more or less the same stage in the process and it’s of great help to see how you are defining your metrics. Keep it up!
    Marko
    http://hopshopper.com

  • http://hopshopper.com Marko

    Ash, this is great stuff. We are at more or less the same stage in the process and it’s of great help to see how you are defining your metrics. Keep it up!
    Marko
    http://hopshopper.com

  • http://www.twitter.com/MichaelZipursky Michael Zipursky

    Ash, amazing posts! Keep them coming when you can – great insights! Cheers.

  • http://www.twitter.com/MichaelZipursky Michael Zipursky

    Ash, amazing posts! Keep them coming when you can – great insights! Cheers.

  • Pingback: A First Look at Some Metrics Numbers()

  • http://crawdout.com/ Peter Crkon

    Only came across this yesterday, very interesting and hepful! Thanks and keep it up!

  • http://crawdout.com Peter Crkon

    Only came across this yesterday, very interesting and hepful! Thanks and keep it up!

  • Pingback: Achieving Flow in a Lean Startup()

  • http://www.twitter.com/MichaelZipursky Michael Zipursky

    The breakdown of how you’re measuring product/market fit is interesting.

    1. How/where did you learn to create dashboards?

    Reading Getting to Plan B (great book) – first real study of dashboards’ role in a startup.

    2. Wondering if you have any tips on creating/using them?

  • http://www.twitter.com/MichaelZipursky Michael Zipursky

    The breakdown of how you’re measuring product/market fit is interesting.

    1. How/where did you learn to create dashboards?

    Reading Getting to Plan B (great book) – first real study of dashboards’ role in a startup.

    2. Wondering if you have any tips on creating/using them?

  • Pingback: Achieving Flow in a Lean Startup « The Roach Post()

  • Anonymous

    Based on the same model AAARR! we build our funnel system.
    http://www.reedge.com/funnels-for-web-startup-pirates.html

    To figure out what are the key benefits of the product we are testing now the name and feature set. For example we are testing (A/B) if we should call our tool Conversion Rate Optimizer nd we change 50% of our homepage to focus on this vs the old focus. If we see more sign-ups we can assume that name works better. We will keep testing other names as well focusing each time on one feature and see what happens. We are not in the Sean Ellis stage where we have 40% begging us not to take the tool away… but we are working hard :-)

  • Pingback: Treasure BOOKMARKS for Entrepreneur « ZEST..!!()

  • Pingback: startupbug.com()

  • Pingback: » Lean Startup Tools | StartupNorth()

  • http://twitter.com/njvitto Nicola Junior Vitto

    Hi Ash,
    I’m looking for a tool to measure this kind of metrics but based on data that I have on my database.
    I’m already using Google Analytics and Mixpanel for some kind of metrics but…why I have to use “event-based” tools to do some cohort analysis and “startup metrics” analysis if I have all the data I need on my database?
    RJMetrics seems to be the best but it’s quite expensive…have you ever tried some products like this?

    Thx,
    Nicola.

  • http://www.ashmaurya.com/ Ash Maurya

    I’ve relied mostly on home-grown metrics. I outline how to do this in my book and am also in the process of productizing our own tool through USERcycle: http://www.usercycle.com.

  • Pingback: Startup Articles « Samuel R Solomon()

  • http://www.toolsjournal.com/agile Agile Tools Journal

    Ash genuinely impressed with not just this post couple of others i have gone thru. Needless to say subscribed to RSS. Cheers.

  • Pingback: Zapier Blog - 5 Entrepreneurs Who Inspire Strategy at Zapier()

  • Anonymous

    How I am Measuring Product/Market Fit
     Good points on AARRR
    Acquisition: How do users find you?Activation: Do users have a great first experience with your product?Retention: Do users come back and use your product?Referral: Do users tell others about your product?Revenue: How do you make money?

  • Pingback: 10 conceitos importantes para uma startup dar atenção | Blog da Concrete()

  • http://www.incion.com/ profesional web design

    These are very useful information. Great…

  • Pingback: Haven't read The Lean Startup? Then read this. Lean Startup, Management, Strategy - Informulate()

  • Mike

    Hi Ash, thanks for sharing with us. I went to see where to buy your book and found the link on the bottom of this page is broken