Background

One of the objectives of a benchmark campaign is to be able to certify its results in terms of a statistically relevant statement on the ranking of the tested operators. In the by Omnitele applied “be-the-customer” benchmark approach these results indicate the Customer Experience levels rendered by the tested operators. Concerning the presentation of the Customer Experience, the following complications do arise:

• In mature markets the average results for Customer Experience of data- and OTT services are very close. Hence in order to achieve statistically significant differences between the performance level of the tested operators relatively large measurement sample size would be required
• There are no commonly agreed Mean Opinion Score methods for representing Customer Experience for OTT and data services, like the MOS score methods that are used for voice services.

Solution

With the purpose of overcoming the indicated obstacles Omnitele has introduced a Key Performance Indicator, called “% better than average Customer Experience”. For the calculation of this KPI, a Customer Experience threshold is defined as the average of the outcomes of all measurements for all tested operators for a particular data- or OTT service. Next the “% better than average Customer Experience” KPI is calculated for each individual operator by determining the number of measurement samples with outcomes that are better than the defined average Customer Experience threshold.



Customer benefits

The advantages of the “% better than average Customer Experience” KPI are as follows:

• The KPI defines a 2 level Customer Experience Mean Opinion Score scale for data- and OTT services, namely better and worse than the average Customer Experience level that is characteristic for the tested market
• The KPI gives a better representation of the service performance of an operator than only presenting the average of all measurement outcomes since it also takes bad performance areas into consideration which otherwise might have been averaged out
• The KPI will establish more pronounced differences in Customer Experience levels between operators, especially in mature markets where averages are close,

• Smaller measurement sample sizes will be required to obtain statistically significant differences between the outcomes of the tested operators and therefore more practical and thus more cost efficient benchmark measurement execution can take place
• The KPI gives a more profound indication of the Customer Experience level in the entire service area of an individual operator. An operator with a consistently high Customer Experience level throughout its entire coverage will score highly on the KPI results.

‍