A brief information on constructing activation metrics for a product
In a earlier article, I talked concerning the Enter > Output > Final result framework, and the way “output” was the central piece, however not essentially simple to outline — simply since you need it to be moved by your inputs, however on the identical time, you’ll want to have a causal hyperlink together with your final result.
Consumer activation metrics fall below this class of metrics. “Activation” is the third stage of the Pirate metrics framework designed by Dave McClure (the well-known AAARRR framework — Consciousness, Acquisition, Activation, Retention, Referral, Income), and it’s often outlined as when your person handed the primary set of frictions, began utilizing your product, obtained some worth from it, and is now extra prone to be retained in the long term.
Some examples of product activation metric:Loom: Sharing a loom¹ Zappier: Setting a zap¹Zoom: Finishing a zoom assembly inside 7d of signup¹Slack: Sending 2,000+ staff messages within the first 30 days²Dropbox: Importing 1 file in 1 folder on 1 gadget inside 1 hour²HubSpot: Utilizing 5 options inside 60 days²
¹2022 product benchmark from Open View https://openviewpartners.com/2022-product-benchmarks/
²Stage 2 Capital: the science of scaling:https://www.stage2.capital/science-of-scaling
Measuring activation is vital as a result of it helps you perceive how properly your product is resonating with new customers and whether or not you’re successfully getting them to turn into “lively” customers. It’s the very first step towards person loyalty — that is the stage the place in case your customers are prone to stick round for the lengthy haul. If activation is low, it could possibly point out that there’s a drawback with the product or the onboarding course of, and it might be essential to make adjustments to enhance the person expertise and improve activation.
You need Activation to be a great predictor of Retention, however on the identical time, you need it to be easy sufficient as this needs to be a simple first step your customers are following.Principally, you’re searching for the smallest motion a person can take that may showcase the product’s worth for them, however you need this small motion to have a causal hyperlink with retention (nevertheless you outline it).As with all ‘main’ indicator, the causality piece (“doing motion Y results in long-term retention”) is tough. You often begin with observational information, and conventional information evaluation may not provide the full image, as it could possibly overlook confounding components that may impression activation/retention.
Utilizing a cohort evaluation, you can begin constructing some instinct round what person actions may good candidate in your activation metric.
The concept is to:
Group your customers primarily based on the place they signed-up for youu productSeparate them primarily based on in the event that they made it to the retain stage or notLook for the actions which are overwhelming executed by the customers you made it to the retain stage, however not a lot by the customers you didn’t.
Let’s say you run a health app. You begin creating month-to-month cohort, and also you discover that 70% of customers that add a minimum of one exercise inside the first week of signing up are nonetheless engaged with the app a 12 months later, vs 40% in the event that they don’t. This generally is a first thought for an activation metric.
A pre-requisite right here is so that you can get the thought of which motion to review. Within the instance above, you needed to have the thought to have a look at who tracked their exercises. That is the place quant meets qual, and when your ‘person acumen’/frequent sense comes into play. Or your networking expertise if you wish to ask the assistance of different subject material consultants.
Some recommendation:
You would possibly wish to provide you with only a few concepts of potential actions, not essentially look into too a lot of them, simply because because the adage goes: “in the event you torture the information lengthy sufficient, it should confess to something” (Ronald H. Coase). The extra actions you choose, the extra possible you will discover one thing, however you may be at excessive danger of it being a false constructive. So sticking to what is sensible and isn’t too far-fetched generally is a good rule of thumb.You would possibly wish to undertake a principled strategy to this, and solely search for issues that you just consider you’d be capable of transfer. When you provide you with one thing too sophisticated/area of interest, you may not be capable of transfer it, and so this can defeat the aim of the entire train.
With propensity rating matching, you may verify or infirm your earlier intuitions
When you’ve recognized your potential activation indicators, the following step is to verify they’re correct. That’s the place propensity rating matching can turn out to be useful — to grasp if the correlation you discovered beforehand may truly be causation. Though this isn’t the one resolution present, and it does require to have a bit of data round your customers (which could not all the time be the case) it may be comparatively simple to implement and may give you extra confidence in your outcome (till perhaps additional triangulation, with extra strong approaches akin to A/B testing).
The concept behind propensity rating matching is the next:
With the intention to discover the causal hyperlink between taking the motion and retainment, ideally you’d clone your customers that took the motion and have the clone not take the motion — to check the outcome.Since it isn’t doable (but?), the following neatest thing is to look inside your information, discover customers which are very comparable (virtually equivalent) to your customers that took the motion — however who didn’t take the motion.
Propensity rating matching is a strategy that means that you can discover these very comparable customers and pair them. Concretely talking, it’s about:
Coaching a mannequin to foretell the probability of your customers to take the motion you outlined (their propensity).Matching customers primarily based on the beforehand discovered probability (the matching half)
(Observe: you may have alternative ways to go about each steps, and a few nice tips can be found on-line concerning the way to choose a mannequin, the way to choose the fitting variable, what matching algorithm to pick out, and many others. — for extra data, see “Some Sensible Steering for the Implementation of Propensity Rating Matching”)
Taking our health app instance once more:
You’ve recognized that 70% of customers that add a minimum of one exercise inside the first week of signing up are nonetheless engaged with the app a 12 months later, vs 40% in the event that they don’t.You prepare a mannequin to foretell the probability of your person to add a exercise inside every week of signing up — and you discover out that the probability may be very excessive for customers which downloaded the app through a referral hyperlink from a big health websiteYou rank your customers primarily based on the probability, and begin doing a easy 1:1 matching (the first customers by way of probability that took the motion is matched with the first customers by way of probability that didn’t take the motion, and and many others.)Put up-matching, you see the distinction drop significantly, however nonetheless being vital so that you can take into account it as a possible candidate for an activation metric!
Cohort evaluation + Propensity rating matching may help you isolate the impression of a particular motion on person conduct, which is crucial for outlining correct activation metrics.
However this technique will not be a panacea —there are a bunch of speculation that comes with the methodology, and you have to to fine-tune it / have some validation to verify it really works in your use-case.
Particularly, the efficacy of PSM can be extremely depending on how properly you may predict the self choice. If you’re lacking key options, and the bias from unobserved traits is massive — then the estimates from PSM may be very biased and never be actually useful.
All this being mentioned — utilizing this technique, even in an imperfect method, may help having a extra data-driven strategy for metric choice, get you began on ‘what to give attention to’, till you get to the stage of working A/B testing and have a greater understanding of what drive long run success.
Hope you loved studying this piece! Do you may have any suggestions you’d wish to share? Let everybody know within the remark part!
And If you wish to learn extra of me, listed here are a number of different articles you would possibly like:
