Risk Management at Scale Part 2: collecting the data

Risk Management 2.JPG

{If you missed Risk Management at Scale Part 1, read it first and then come back here}.

“I wonder how many people live here.” - US founders, 1789 → US Census
“Do you know how fast you were going?” - Highway police, present day → Radar gun
“Are customers using the product the way we thought they would?” - You → ???

For better or worse, humans are obsessed with measurements and metrics. It helps us to compare our progress over a time range with our peers or our competitors.

To start, let’s define an important term that should help turn those above questions marks on product use into actions:

te·lem·e·try / tə-ˈle-mə-trē / noun: the science and technology of automatic measurement of data from remote or inaccessible points and transmitted to receiving equipment for monitoring.

Your team may already be accomplishing this telemetry: user analytics from Mixpanel to measure traffic, Zendesk reporting to keep track of tickets created per customer, and even Google news alerts for important customer events like acquisitions or board member changes. All of this is great information for your customer success team to collect and have top of mind for their next call. This is quantitative data.

But speaking of the call, how do you measure your relationship? This is qualitative. Below are examples of phrases to listen for that can help a CSM understand their customers temperature.

  • Are they talking about events far into the future (i.e. beyond their upcoming renewal date)?
  • Has your day-to-day mentioned that their manager is leaving and a new person is taking over the department?
  • Have they cancelled the last couple calls with no explanation?

While there is no specific KPI connected to these remarks, it tells you something about their specific experience interacting with the product and CS team. Typically, we recommend customers use either school (A - F) or traffic light (green - yellow - red) grading to evaluate the relationship. Updating the score, even if it’s refreshing the same grade because the relationship is still great, helps to ensure that everyone in the organization knows this is accurate.

Where you store the relationship score really depends on your suite of tools and budget. There are customer relationship management tools, like Salesforce and Gainsight, that have this functionality built-in. Changes in score can also trigger specific playbooks based on a positive or negative movement. In addition, we’ve seen companies simply use a shared spreadsheet with only four columns: customer, assigned CSM, score, and date updated. It’s worth noting that this seemingly simple spreadsheet, when organized correctly, can become the foundational documentation for when you upgrade to a CRM tool, as it provides a clear template for your implementation.

Working in tandem with CSM scoring, your product and engineering teams should have proprietary systems and metrics built into the software you sell, which helps understand customer behavior. When is the last time they logged in? Is the day-to-day only using one feature? Is the customer VP clicking on the ‘Review Plans’ page, comparing their basic package to the premium version?

Collecting isn’t the hard part (hopefully). Once you have all this data, ask yourself if it’s accurate. And, if it’s not accurate, why? It’s beneficial to absolutely no one if you find reasons to trim numbers: “Oh, ignore these customers because they are too big / small / odd.” While this practice may result in the pretty up-and-to-the-right chart to show your board, you didn’t learn anything and don’t know where to focus your efforts.

It’s also important to build a culture of transparency within your customer team.  If your team members feel overly incentivized to fib on their customer’s happiness (giving green when it should be yellow or red), you’re up a creek with no paddle.  It is much better to know there’s an issue and have a plan for action than to be falsely confident and blindsided by bad news. A customer should rarely, or never, go directly from green to churn.

Let the story unfold from the full data set, and accept it as the current picture. Ignorance is not bliss. It’s better to know that the numbers are not great, rather than have “good” heavily edited data… and then be surprised by a spike in churn.


So, you’re collecting the data. Next we’ll get our hands dirty figuring out what all this data tells us about today, and how that changes tomorrow (in Part 3). From here, we’ll learn how to leverage your data to prevent risk (Part 4) and do it consistently at scale (Part 5).  To get updates when we publish the additional parts of this series, be sure to follow Sandpoint Consulting on LinkedIn.

For more information about Risk Management, or to request a customized Risk Management Workshop for your team, send us a note at contact@sandpoint.io.