NPS: The Volatile Indicator

We recently published an op-ed that highlights a range of issues related to Net Promoter Score (NPS). It does not measure word of mouth (since we do not measure the recommendation actually made), we have never demonstrated its impact on the real loyalty of customers (only on their "intention of loyalty" - declared), and the same NPS can correspond to very variable situations, etc... If the intention of recommendation is a useful indicator, it cannot be the ultimate indicator. Here we will focus on a particular point: the great volatility of the NPS.

Would you trust a doctor who told you he could diagnose your general health with a single indicator?

A quick look back at the NPS.
The NPS is calculated from a recommendation intention question, administered on an 11-position scale, from 0 to 10. Scores from 0 to 6 bring together so-called “detractors”, scores from 7 to 8 bring together so-called “passive” individuals, and scores from 9 to 10, “promoters”. The NPS is actually the balance of the percentages of promoters - detractors. The higher the NPS, the more favorable the company's posture, and vice versa.
Basically, we have nothing to object to the use of a recommendation question, it has always been used to offer a view (among others) on the customer's state of mind. We contest its predictive nature of the economic activity of the company and especially its capacity to become "The Ultimate Question" as its promoter Fred Reichheld asserts. See our previous remarks here.

In practice, a real problem of volatility
While the original question on which the NPS is calculated (i.e. recommendation intention rated out of 10) may be relatively stable from one customer segment to another or from one survey wave to another, the corresponding NPS can, on the other hand, display very high volatility.
On a purely mathematical level, this obviously poses a problem because if the only component of this NPS is stable (recommendation intention), it is not justifiable that the NPS itself could be unstable. It therefore tends to disrupt the diagnosis rather than clarify it.
On a managerial level, and this is the essential thing, it forces the researcher to put these variations into perspective, by returning to the original metric (the question of recommendation) to explain any surprising variations in the NPS. We can therefore basically wonder what the NPS is for.

Here is an illustration of this variation. We worked from a dataset of 4,600 real interviews relating to the recommendation intention of a major retail brand. The question is administered on an 11-point scale, the distribution of which is as follows, overall for the sample:

Here we have graphed the 23 averages of these subgroups and their 23 associated NPS.

AIn order to study the stability of the NPS, we carried out a random drawing without replacement allowing each interview to be allocated to 23 different groups each comprising 200 interviews (23 x 200 = 4600 interviews). This makes it possible to simulate the existence of 23 customer “segments” or 23 survey “waves”.

We can immediately observe the unstable nature of the NPS around the adjustment line. If for example we observe the two points marked in red with averages respectively (6.87 for the first and 6.90 for the second) we can calculate (standard deviations respectively 2.05 and 2.34) that their difference is not significant.
In other words, if these two averages corresponded to two successive waves of surveys, the recommendation intention should be considered stable. However, the NPS goes from -8% to -14%. How can we interpret this indicator, which is intended to be simple, educational and intuitive, given its instability and unpredictability? Not to mention the direction of variation which is visibly incorrect.
NPS is a metric that can show great volatility, without reflecting a significant change in recommendation intention. We think this is a serious point to consider before communicating about the NPS.

We renew our recommendations:

• Communicate on the average score or the original recommendation percentages,
• Collect the actual recommendation over the last x months, rather than the intention of recommendation,
• Build your customer relationship health diagnosis with several associated indicators, satisfaction, loyalty intention, customer seniority. Could we imagine a health diagnosis (on a medical level) with a single indicator? No. No more for a diagnosis of the state of the customer relationship...