Helpful Banking Moments: Posh’s North Star Metric to Define AI Success

Moving forward with conversational AI, whether on digital or voice channels, is a major step to delivering a  better customer experience. But what’s the best way to measure the success of AI assistants? 

Introducing Helpful Banking Moments, a brand new metric we’ve developed to determine if our AI is doing right by the party that matters most: the end user.

Conversational AI: how do you measure success?

There are many metrics that can be used to measure conversational AI quality, but these are often lagging indicators of success. They don't address how the user is perceiving the experience. Here are some common benchmarks:

• Total conversations
• Containment rate (more on this below)
• Total time spent or total number of messages exchanged
• Accuracy of recognizing the user’s intent
• Customer satisfaction
• Wait times avoided
• Channels of customer engagement
• Abandonment (was the customer dissatisfied and abandoned the chat or call?) 

These metrics and KPIs (like containment rate) fall short of answering the question: “Did the conversational AI provide the best experience possible to the end user?”

In fact, focusing purely on containment rate may be concealing fatal shortcomings at the expense of your customers and members.

Why “Containment Rate” doesn’t tell the whole story

First, what is containment? Containment as a term simply means that a conversation was not forwarded from the AI bot to a live agent, or teed up in the call-back queue. So containment and escalation are opposites.

When an interaction is contained, is it safe to assume the user’s inquiry was answered? Does this mean that the bot was successful in solving the user’s problem? Well, maybe not.

Think about it this way: one easy way to guarantee 100% containment is simply to make it impossible for the AI to escalate to a human agent. So while the data may say the conversation was contained, whether a hang-up or chat closed, the truth is that the end user may have experienced high levels of frustration and just given up, when the optimal path would have been to get them to an agent.

Sometimes, the successful outcome is to intelligently route the customer. If the user is interested in more complex mortgage or loan assistance, then they will likely need to speak to a loan officer. In this case, the interaction isn't technically contained, but the bot appropriately assisted your customer to get to a lending expert.

Posh prides itself on working with clients to improve containment, which we know is important as call centers are dealing with record-high volumes and reduced staffing. But we ultimately want our partners to feel confident in how they assist their customers – delivering the optimal banking experience.

“We wanted to allow our agents to have a better, in-depth conversation with our members and really improve the overall member experience here at Associated [Credit Union of TX],” said Mike Procenko, Member Experience Center Manager (Contact Center). “[Ava] allows the agents to focus on the members with more in-depth questions. The response to the feedback survey we’ve added is overwhelmingly positive.”

It’s too limiting to rely on the idea of containment as the primary success metric. Containment is a metric, not the metric and making containment the primary measure of success (and potentially even pegging product pricing to that metric) does not align incentives towards delivering the best customer experience.

Introducing “Helpful Banking Moments”

We want our clients to be focused on the right measure of success relating to how the end users feel they were helped (or not). But all the common bot KPIs fell short in being this north star metric.

This problem is so important to us at Posh, that we sought to develop a brand new metric. A holistic one that can serve as a proxy for ROI and user satisfaction.

We define Helpful Banking Moments as conversations (or moments within a conversation) that solve the user’s problem. We want to leave the customer or member feeling like the conversational AI interaction added value, that it was helpful and positively impacted their experience.

What qualifies as a Helpful Banking Moment, you ask? Well if the user is simply asking about the routing number, then a helpful outcome is to simply respond back with that information. If the user is interested in investment opportunities, maybe the ideal path for them and the financial institution is to help schedule an appointment with an investment specialist. Or if they have a complex support issue that’s beyond the AI’s domain of knowledge, maybe the AI can 1. authenticate the customer, 2. recognize the higher level topic the issue falls under, and 3. intelligently route them to the proper rep while passing on contextual information like the customer’s identity and the inquiry.

Note that in these last two examples, neither interaction was technically contained. But both interactions led the customer down the optimal path to get the help they needed, and one of them was even flagged as a new revenue opportunity!

Financial Institutions deserve transparency in measuring improvements & ROI

You’ve probably heard the phrase: “You can't improve what you don't measure.”

Measuring success is critical to driving forward better ROI and banking experiences for your customers. And in order to measure success, transparency in data is crucial. At Posh, we provide our clients unfiltered access to every collected metric and transcription logs of every single interaction.

So, we aren’t here to tell you that metrics are not important. Quite the opposite. We just want to make sure that we (and our FI partners) are using the right lens to evaluate success from the standpoint of: “Did we deliver the most helpful banking experience possible to the customer?” 

“Our success really comes down to two things,” said Freedom First’s executive vice president and chief operating officer Sarah Andrews. “Our call center leadership and our partnership with Posh. That partnership is head and shoulders above our other vendors.”

Posh is here to be your AI partner. We believe financial institutions deserve transparency, baseline performance, and proof of improvement over time. We want to help our FI clients supercharge their customer experience. Come chat with us; we have lots of ideas about how to create a future full of Helpful Banking Moments. 

Request a demo.

‍