Title: System and method for annotating recorded information from contacts to contact center
Abstract: A system for forming a database indexed with content-based parameters, corresponding to recordings of contacts to a contact center, includes an analysis unit, a mining unit, and a correlation unit. The analysis unit performs automated-portion analysis techniques to ascertain predetermined events occurring in recordings of automated portions of the contacts to the contact center. The mining unit performs audio mining of recordings of agent portions of the contacts to the contact center. The correlation unit correlates an event occurring in an agent portion of the recordings with a corresponding one of a plurality of content-based parameters selected to be used as indices of the database, and makes an entry in the database based on the correlated event and the corresponding content-based parameter.
Patent Number: 6,898,277 Issued on 05/24/2005 to Meteer, et al.
| Inventors:
|
Meteer; Marie (Arlington, MA);
Peterson; Patrick (Cambridge, MA);
Suhm; Bernhard (Cambridge, MA)
|
| Assignee:
|
Verizon Corporate Services Group Inc. (New York, NY)
|
| Appl. No.:
|
090524 |
| Filed:
|
March 4, 2002 |
| Current U.S. Class: |
379/265.02; 379/88.18; 379/265.06; 379/265.07; 707/1; 707/100 |
| Intern'l Class: |
G06F 007/00; G06F017/40; H04M001/64; H04M003/52.3; H04M003/52.7 |
| Field of Search: |
379/8801,880.4,881.8,265.01,265.02,265.06,265.07,265.08,266.01
707/1,100
|
References Cited [Referenced By]
U.S. Patent Documents
Other References
D. Stallard, "Talk'N'Travel: A Conversational System for Air Travel Planning,"
Applied Natural Language Processing ANLP 2000, Seattle, WA.
E. Levin et al., "CHRONUS, The Next Generation," ARPA Workshop on Spoken Language
Technology, 1995, Austin, TX.
C.H. Lee et al., "On Natural Language Call Routing," Speech Communications, vol.
31, pp. 309-320, 2000.
H. Chang et al., "An Automated Performance Evaluation System For Speech Recognizers
Used In The Telephone Network," International Conference On World Prosperity Through
Communications, 1989.
D.S. Pallett et al., "1993 Benchmark Tests For The ARPA Spoken Language Program,"
ARPA Workshop on Spoken Language Technology, 1994, Princeton, NJ.
K. Edwards et al., "Evaluating Commercial Speech Recognition and DTMF Technology
for Automated Telephone Banking Services," IEEE Colloquium on Advances in Interactive
Voice Technologies for Telecommunication Services, 1997.
U.S. Appl. No. 10/090,205, filed Mar. 4, 2002, Peterson et al.
U.S. Appl. No. 10/090,579, filed Mar. 4, 2002, Peterson et al.
U.S. Appl. No. 10/090,546, filed Mar. 4, 2002, Godfrey et al.
U.S. Appl. No. 10/090,369, filed Mar. 4, 2002, Suhm et al.
U.S. Appl. No. 10/090,264, filed Mar. 4, 2002, Peterson et al.
U.S. Appl. No. 10/090,395, filed Mar. 4, 2002, Peterson et al.
U.S. Appl. No. 10/090,234, filed Mar. 4, 2002, McCarthy et al.
U.S. Appl. No. 10/090,522, filed Mar. 4, 2002, Peterson et al.
U.S. Appl. No. 10/090,210, filed Mar. 4, 2002, Peterson et al.
C.A. Kamm et al., Design and Evaluation of Spoken Dialog Systems, Proc. of IEEE
Workshop on Automatic Speech Recognition and Understanding, 1997 pp. 11-18.
Jeremy Peckham, A New Generation of Spoken Dialogue Systems: Results and Lessons
from the Sundial Project, Proc. Of the 3rd European Conference on Speech
Communication and Technology, Berlin, Sep. 1993, pp. 33-40.
S. Bennacef et al., Dialog in the RailTel Telephone-Based System, Intl. Conf.
On Speech and Language Processing, Philadelphia 1996, pp. 550-553.
M. Walker et al., PARADISE: A Framework for Evaluating Spoken Dialogue Agents,
Proc. Of the 35th Annual Meeting of the ACL and 8th European
Conf. Of the European Chapter of the ACL, Jul. 7-12, 1997, Madrid, pp. 271-280.
M. Cohen, Universal Commands for Telephony-Based Spoken Language Systems, Telephone
Speech Standards Committee, Common Dialog Tasks Subcommittee, SIGCHI Bulletin,
vol. 32, No. 2, Apr. 2000, pp. 25-29.
Bruce Balentine and David P. Morgan, How to Build a Speech Recognition Application,
Enterprise Integration Group, Inc., San Ramon, CA, 1999.
|
Primary Examiner: Hong; Harry S.
Attorney, Agent or Firm: Suchyta, Esq.; Leonard C., Wall, Esq.; Joel, Fitzpatrick Cella et al
Parent Case Text
CROSS REFERENCE TO RELATED APPLICATIONS
This application claims benefit of U.S. Provisional Appln. No. 60/273,710, filed
Mar. 5, 2001, and U.S. Provisional Appln. No. 60/276,266, filed Mar. 15, 2001,
which are each hereby incorporated herein by reference.
Claims
1. A method for forming a database indexed with content-based parameters corresponding
to recordings of contacts to a contact center, the contact center being operable
to present a contacting party with a contact that includes an automated portion
corresponding to interaction with an automated response interface and, at the contacting
party's option, an agent portion corresponding to interaction with an agent, events
occurring in the recordings of automated portions being ascertained by previously
executed automated-portion analysis techniques, said method comprising the steps of:
selecting content-based parameters to be used as indices in a database;
analyzing results of the automated-portion analysis techniques to detect occurrence
of events corresponding to the selected parameters;
analyzing one or more agent portions of recordings of contacts to ascertain events
occurring therein;
correlating events occurring in the one or more agent portions of the recordings
with corresponding ones of the selected parameters; and
making entries in the database based on the selected parameters and the correlated
events.
2. A method according to claim 1, wherein the contact center is a call processing
center, the contacts to the contact center are calls to the call processing center,
and the automated response interface comprises an interactive voice response (IVR) unit.
3. A method according to claim 2, wherein the content-based parameters include
parameters corresponding to semantic annotations, which include annotations corresponding
to characteristics of a call, a content of the call, and a quality of the call.
4. A method according to claim 1, wherein the content-based parameters include
at least one of a mention of an entity name, a completion of a specified transaction,
a request for a specified transaction, and a request for a specified service.
5. A system for forming a database indexed with content-based parameters corresponding
to recordings of contacts to a contact center, the contact center being operable
to present a contacting party with a contact that includes an automated portion
corresponding to interaction with an automated response interface and, at the contacting
party's option, an agent portion corresponding to interaction with an agent, said
system comprising:
analysis means operable to perform automated-portion analysis techniques to ascertain
predetermined events occurring in recordings of automated portions of contacts
to a contact center;
mining means operable to perform audio mining of recordings of agent portions
of the contacts to the contact center, said mining means including:
a speech/non-speech detector operable to identify speech and non-speech events
in an audio recording;
a speaker-change detector operable to identify speaker turns in an audio recording;
a speech recognizer operable to output a sequence of words for each speaker turn
identified by the speaker-change detector;
a topic detector operable to determine a topic of an audio recording; and
a named-entity detector operable to identify speech pertaining to an entity named
in an audio recording; and
correlation means operable to correlate an event occurring in an agent portion
of the recordings with a corresponding one of a plurality of content-based parameters
selected to be used as indices of a database, and to make an entry in the database
based on the correlated event and the corresponding content-based parameter.
6. A system according to claim 5, wherein the contact center is a call processing
center, the contacts are calls to the call processing center, and the automated
response interface comprises an interactive voice response (IVR) unit.
7. A system according to claim 6, wherein the plurality of content-based parameters
include parameters corresponding to semantic annotations, which include annotations
corresponding to characteristics of the call, a content of the call, and a quality
of the call.
8. A system according to claim 5, wherein the content-based parameters include
at least one of a mention of an entity name, a completion of a specified transaction,
a request for a specified transaction, and a request for a specified service.
9. A computer program product embodying a program for implementing a method for
forming a database indexed with content-based parameters corresponding to recordings
of contacts to a contact center, the contact center being operable to present a
contacting party with a contact that includes an automated portion corresponding
to interaction with an automated response interface and, at the contacting party's
option, an agent portion corresponding to interaction with an agent, events occurring
in the recordings of automated portions being ascertained by previously executed
automated-portion analysis techniques, said computer program product comprising
code for:
selecting content-based parameters to be used as indices in a database;
analyzing results of the automated-portion analysis techniques to detect occurrence
of events corresponding to the selected parameters;
analyzing one or more agent portions of recordings of contacts to ascertain events
occurring therein;
correlating events occurring in the one or more agent portions of the recordings
with corresponding ones of the selected parameters; and
making entries in the database based on the selected parameters and the correlated
events.
10. A computer program product according to claim 9, wherein the contact center
is a call processing center, a contact is a call to the call processing center,
and the automated response interface comprises an interactive voice response (IVR) unit.
11. A computer program product according to claim 10, wherein the content-based
parameters include parameters corresponding to semantic annotations, which include
annotations corresponding to characteristics of a call, a content of the call,
and a quality of the call.
12. A computer program product according to claim 9, wherein the content-based
parameters include at least one of a mention of an entity name, a completion of
a specified transaction, a request for a specified transaction, and a request for
a specified service.
Description
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates to systems and methods for assessing call center
performance and, in particular, to systems and methods that allow the performance
and current and potential levels of automation of a call center to be quantified.
2. Discussion of the Related Art
Telephone user interfaces are the most widespread class of human-computer
interfaces. Introduced more than a decade ago, touch-tone interactive voice response
(IVR) systems were adopted enthusiastically in many call-centers and promised to
provide customer service efficiently. However, calling customers (callers) have
exhibited antipathy towards touch-tone IVR systems, viewing them as difficult to
use. Further aggravating the callers is the fact that they often endure long waiting
times before they can speak to live agents. This dichotomy is not surprising considering
that most call-centers focus on minimizing operating costs and that usability and
its impact on call-center operations are poorly understood.
Since touch-tone IVR systems have been deployed for more than a decade, a significant
body of know-how on IVR systems has accumulated in the industry. Except for recent
attempts to define a style guide for (telephone) speech applications, as in Balentine,
B. and D. P. Morgan,
How to Build A Speech Recognition Application, 1999,
San Ramon, Calif., Enterprise Integration Group, and to introduce universal commands
in speech-enabled IVR systems, as in Cohen, M.,
Universal Command for Telephony—Based
Spoken Language Systems, SIG-CHI Bulletin, 2000, 32(2), pp. 25-30, this body
of knowledge is not well documented. The prevalence of usability problems in deployed
IVR systems suggests that designing good telephone interfaces is difficult and
usability engineering methods for telephone interfaces are not well developed.
Another area of related work is research on spoken it dialog systems, an
important application of speech recognition technology. Spoken dialog systems allow
the caller to communicate with a system in a spoken dialog, not necessarily over
the telephone. Many research articles on spoken dialog systems have been published,
e.g., Stallard, D.,
Talk'N'Travel: A Conversational System for Air Travel Planning,
in
Applied Natural Language Processing ANLP, 2000, Seattle, Wash.; Peckham,
J.,
A new generation of spoken language systems: recent results and lessons
from the SUNDIAL project, in European Conference on Speech Communication and
Technology EUROSPEECH, 1993, Berlin (Germany): European Speech Communication Association;
Levin, E. and R. Pieraccini,
CHRONUS: The Next Generation, in ARPA Workshop
on Spoken Language Technology, 1995, Austin (TX): Morgan Kaufmann Publishers, Inc.;
Bennacef, S., et al.,
Dialog in the RIALTEL telephone-
based system,
in
International Conference on Spoken Language Systems ICSLP, 1996, Philadelphia,
Pa.; and Lee, C. H., et al.,
On Natural Language Call Routing, in Speech Communications,
2000, 31, pp. 309-320.
Previous work on spoken-dialog system evaluation focused on quantifying
the performance of the underlying technologies, e.g., Chang, H., A. Smith, and
G. Vysotsky,
An automated performance evaluation system for speech recognizers
used in the telephone network, in
International Conference on World Prosperity
Through Communications, 1989; and Pallett, D. S., et al., 1993
Benchmark
Tests for the ARPA Spoken Language Program, in
ARPA Workshop on Spoken Language
Technology, 1994, Princeton (NJ): Morgan Kaufmann Publishers, Inc.
Some studies have evaluated the usability of telephone interfaces based on task
completion rates and post-experimental questionnaires, e.g., Edwards, K., et al.,
Evaluating Commercial Speech Recognition and DTMF Technology for Automated Telephone
Banking Services, in
IEEE Colloquium on Advances in Interactive Voice Technologies
for Telecommunication Services, 1997. More recently, PARADISE was introduced
as a "consistent integrative framework for evaluation" of spoken language systems,
as described in Walker, M. A., et al.,
PARADISE: A Framework for evaluating
spoken dialogue agents, in 35th
Annual Meeting of the Association of Computational
Linguistics, 1997, Madrid: Morgan Kaufmann Publishers, Inc. Basically, PARADISE
provides a method to identify measures that predict user satisfaction well, from
the large set of measures that have been used in the field. However, this work
does not address the cost for the call center, nor does it provide any guidance
for telephone interface redesign.
SUMMARY OF THE INVENTION
The present invention overcomes the deficiencies of earlier systems by presenting
a methodology for evaluating both usability and cost-effectiveness of telephone
user interfaces. The assessment methodology of the present invention, and the various
inventive techniques utilized within that methodology, provide usability practitioners
with tools to identify and quantify usability problems in telephone interfaces,
and provide call-center managers with the business justification for the cost of
usability-improvement engineering.
The present invention relates to a system for annotating recorded information
from calls to a call center, to enable assessment of the call center. Of course,
although the descriptions herein relate to call centers that process telephone
calls, other types of contact centers other than call centers are within the realm
of the present invention, including (but not limited to) Internet-based contact
centers where customers contact a company's contact center via the Internet using
known ways for transmitting messages and information via the Internet.
According to one aspect of the present invention, there is provided a method
for forming a database indexed with content-based parameters corresponding to recordings
of contacts to a contact center. The contact center is operable to present a contacting
party with a contact that includes an automated portion corresponding to interaction
with an automated response interface and, at the contacting party's option, an
agent portion corresponding to interaction with an agent. Events occurring in the
recordings of automated portions are ascertained by previously executed automated-portion
analysis techniques. The method comprises the steps of: selecting content-based
parameters to be used as indices in a database; analyzing results of the automated-portion
analysis techniques to detect occurrence of events corresponding to the selected
parameters; analyzing one or more agent portions of recordings of contacts to ascertain
events occurring therein; correlating events occurring in the one or more agent
portions of the recordings with corresponding ones of the selected parameters;
and making entries in the database based on the selected parameters and the correlated events.
According to another aspect of the present invention, there is provided
a system for forming a database indexed with content-based parameters corresponding
to recordings of contacts to a contact center. The contact center is operable to
present a contacting party with a contact that includes an automated portion corresponding
to interaction with an automated response interface and, at the contacting party's
option, an agent portion corresponding to interaction with an agent. The system
comprises: analysis means operable to perform automated-portion analysis techniques
to ascertain predetermined events occurring in recordings of automated portions
of contacts to a contact center; mining means operable to perform audio mining
of recordings of agent portions of the contacts to the contact center; and correlation
means. The mining means includes: a speech/non-speech detector operable to identify
speech and non-speech events in an audio recording; a speaker-change detector operable
to identify speaker turns in an audio recording; a speech recognizer operable to
output a sequence of words for each speaker turn identified by the speaker-change
detector; a topic detector operable to determine a topic of an audio recording;
and a named-entity detector operable to identify speech pertaining to an entity
named in an audio recording. The correlation means correlates an event occurring
in an agent portion of the recordings with a corresponding one of a plurality of
content-based parameters selected to be used as indices of a database, and makes
an entry in the database based on the correlated event and the corresponding content-based parameter.
According to yet another aspect of the present invention, there is provided
a computer program product embodying a program for implementing a method for forming
a database indexed with content-based parameters corresponding to recordings of
contacts to a contact center. The contact center is operable to present a contacting
party with a contact that includes an automated portion corresponding to interaction
with an automated response interface and, at the contacting party's option, an
agent portion corresponding to interaction with an agent. Events occurring in the
recordings of automated portions are ascertained by previously executed automated-portion
analysis techniques. The computer program product comprises code for: selecting
content-based parameters to be used as indices in a database; analyzing results
of the automated-portion analysis techniques to detect occurrence of events corresponding
to the selected parameters; analyzing one or more agent portions of recordings
of contacts to ascertain events occurring therein; correlating events occurring
in the one or more agent portions of the recordings with corresponding ones of
the selected parameters; and making entries in the database based on the selected
parameters and the correlated events.
These and other objects, features, and advantages of the present invention
will be apparent from the following description of the preferred embodiments considered
in conjunction with the corresponding drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a schematic diagram illustrating components of a complete system for
call center assessment, in accordance with preferred embodiments of the present invention;
FIG. 1A is a chart describing a workflow of an assessment in accordance with
the present invention;
FIGS. 2A-2C illustrate three options for recording incoming calls to a call center;
FIG. 3A is an exemplary IVR flow chart that maps out possible paths a caller
can take while interacting with an IVR system;
FIG. 3B, which is a composite of FIGS. 3C-1 and 3C-2, shows a
more detailed flow chart for an IVR system;
FIG. 3D is a schematic diagram of an IVR system;
FIG. 3E is a chart showing typical data collected for a call.
FIG. 3F shows an IVR log that includes an area for identifying the caller;
FIG. 4 is a diagram that illustrates a complete process of an IVR system analysis,
according to a preferred embodiment of the present invention;
FIG. 5 is a diagram illustrating an exemplary architecture for a computer-based
indexing system;
FIG. 6A is a flow chart illustrating a mini assessment process, according to
the present invention;
FIG. 6B, which is a composite of FIGS. 6B-1 through 6B-4, shows
an example of a coding sheet used in mini assessment;
FIG. 6C, which is a composite of FIGS. 6C-1 through 6C-4, shows
an example of tabulated results from an analysis of calls to a call center;
FIG. 6D is an example of an analysis report produced based on the tabulated
results illustrated in FIG. 6B;
FIG. 7 is a diagram illustrating a process for measuring existing automation levels;
FIG. 8 is an example of a spreadsheet generated to assist in IVR system automation analysis;
FIG. 9 is a chart showing an example of standard benefit assumptions;
FIG. 10 is a chart for calculating an automation benefit of a call center, in
accordance with an embodiment of the present invention;
FIG. 11 shows a chart useful for calculating an upper bound on call center automation;
FIG. 12 is a bar chart for comparing upper bounds of automation with existing
levels of automation in a call center;
FIG. 13 is a chart illustrating potential savings of agent time in a call center;
FIG. 14 is an exemplary user-path diagram, in accordance with a preferred embodiment
of the present invention;
FIG. 15 is a flow diagram illustrating a process for generating a user-path
diagram, in accordance with the present invention;
FIG. 15A is a diagram that illustrates how to read a user path diagram;
FIG. 16 shows an exemplary confusion matrix, developed in accordance with the
present invention;
FIG. 16A is a flow chart illustrating a method of manipulating call data files
so that they may be visualized in a confusion matrix;
FIG. 17 is an exemplary excerpt from a .sum file;
FIG. 18 is an exemplary life-of-call diagram;
FIG. 19 is a chart comparing cost savings in three different IVR systems;
FIG. 19A illustrates projected benefits for a touch-tone redesign versus a speech-enable
call center; and
FIG. 20 is a diagram illustrating a technique for monitoring IVR system performance.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The present invention provides techniques for assessing interactive voice response
(IVR) systems. The IVR system assessment techniques of the present invention provide
functionality for evaluating the performance of commercial call centers in detail
and can assist in providing concrete recommendations on how a call center IVR system
can be improved. As part of this process, the IVR system assessment as practiced
according to the present invention also quantifies the potential for improvement,
in particular, the maximum operational (cost) savings a re-engineering effort could
deliver. Advantageous results of an IVR system assessment include a detailed IVR
system usability analysis, a set of recommendations for redesigning the system,
and a business case for various redesign options.
I. Overview of Enumerated Techniques
Several new techniques are described below in the context of assessment of
call-center performance, in relation to call-center automation in general and to
IVR systems in particular. For ease of reference, these techniques will be identified
by reference numbers, as shown in FIG. 1, and may be referred to as such throughout
the specification. The techniques include: routing analysis
1, on-site and
off-site end-to-end call recording
2, IVR logging
3, IVR system performance
monitoring
4, mini assessment
5, IVR system analysis and prompt inference
6, call annotation and information retrieval
7, automation analysis
8, user-path diagram
9, and life-of-call diagram
10.
Initially, the above techniques will briefly be discussed in terms of
how they relate to the overall assessment method and to one another. According
to a preferred embodiment, the techniques enumerated above fit into the overall
assessment method of the present invention as follows.
The assessment technique proceeds in two major phases. In the first phase, data
from live calls is collected and processed into a format suitable for further analyses,
that is, a database of complete call event-sequences for every call. The event
sequence for every call should encompass both the IVR system-caller dialog and
agent-caller dialog. The latter is included if such a dialog occurred during the call.
FIG.
1
In the second phase, based on the call event-sequence database, caller behavior
is analyzed from different aspects and summary statistics are generated. FIG. 1
illustrates a complete process for an IVR system assessment and how the various
enumerated techniques fit into a preferred embodiment of the assessment method
and system of the present invention. Roughly speaking, the upper part of FIG. 1
illustrates various advantageous methods for obtaining a database of call event-sequences,
and the lower part shows various assessment analysis techniques in accordance with
the invention.
The capture of call event-sequences will be discussed next. The only complete
record of user and system behavior in telephone user interfaces are complete calls.
Therefore, any comprehensive assessment of telephone user interfaces must be based
on complete records of calls, which may be stored in the form of a database of
call event-sequences, shown as a central object in FIG. 1. A call typically begins
with a dialog with an automated (IVR) system, called the IVR system-caller dialog,
which may be followed by a dialog with a live agent, called the agent-caller dialog.
Three of the enumerated techniques are used to collect data from complete calls:
IVR logging 3, call recording 2, and (on-line) call monitoring 5.
First, in IVR logging 3, the IVR system itself provides a complete record
(log) of system prompts and user responses. To obtain an IVR log, the IVR program
is configured to write an event log at every significant state, as the call passes
through the call-flow logic of the program. All log entries for a specific call
constitute the complete event sequence for the IVR system-caller dialog of that
call, which sequence can be stored in the call event-sequence database. The IVR
logging technique is discussed in more detail below.
Second, end-to-end recordings of calls are obtained by passing incoming calls
through specialized recording hardware, which stores the audio signal for a complete
call into a file. Methods for obtaining end-to-end call recordings in accordance
with the present invention are described below in the detailed discussion of call
recording 2. Recordings of complete calls represent a large amount of data
that is difficult to analyze in its raw form. For example, one hour of recording
on one channel at 8 kHz with an 8-bit resolution corresponds roughly to 30 Mbytes
of audio data. To make the analysis of call data efficient, call recordings are
transformed into complete IVR event sequences using IVR system analysis and prompt
inference 6, as described below in the detailed description of that technique.
Finally, instead of recording, a call can also be monitored manually, resulting
in a rough, on-line annotation of the events of a complete call, including both
the IVR system-caller dialog and the caller-agent dialog. This method is used in
mini assessment 5, to be described in detail below. The main advantage of
a mini-assessment over the two data collection methods discussed above is its low
cost: instead of logging or recording and analyzing thousands of calls, only a
few hundred calls are monitored and annotated manually, thus trading cost for accuracy.
Call analyses, assessment diagnostics, and customer relationship management
(CRM) applications will be discussed next. Once event sequences have been obtained
for many calls, these call records can be used in three main ways: to (constantly)
monitor IVR system performance, to conduct a detailed IVR system assessment, and
to annotate and retrieve information, e.g., for customer relationship management
(CRM) purposes.
First, in IVR system performance monitoring 4, the performance of an
IVR system is monitored constantly using a few key performance statistics. These
performance statistics are obtained directly from IVR logs in a fully automated
fashion, allowing monitoring to be conducted on an on-going basis. Suitable performance
statistics include IVR system automation, detailed measures thereof, such as rate
of successful capture of caller ID or successful routing or delivery of information
to the caller, or total IVR system benefit. Techniques for gathering these statistics
are described in more detail below in relation to the technique for automation
analysis 8.
Second, IVR system assessment techniques are used to analyze the call records
in several ways to obtain detailed diagnostics for IVR system usability and efficiency.
In one such technique, a user-path diagram 9 is produced that represents
complete IVR event sequences visually in a state-transition diagram, which is annotated
with the IVR exit conditions and the levels of automation achieved for every call.
A user-path diagram essentially shows where callers went in the IVR system (what
choices they made), whether they received useful information in the IVR system,
where they abandoned the IVR system by hanging up, and if and where in the IVR
program they were transferred to an agent. An example of a user-path diagram technique
in accordance with a preferred embodiment will be described in detail below.
Routing analysis 1 is a technique that allows the performance of IVR
routing to be visualized by graphically comparing choices made in IVR to the true
reason for the call (or "call type"), as determined by annotations of agent-caller
interactions. A confusion matrix, to be discussed below, is an example of such
a graphical comparison.
Automation analysis 8 is a technique that quantifies the benefit
of the existing IVR system as well as the potential for increasing automation.
The benefit is calculated in terms of agent seconds (averaged across all calls),
i.e., agent time saved by completing transactions in the IVR system that otherwise
would have to be handled by an agent. The automation analysis 8 takes into
account the benefit for routing callers to the correct place in the IVR system,
which is measured in the routing analysis 1.
Finally, a life-of-call diagram 10 is a graphical technique that
allows vital timing information for broad sections of a call, such as IVR-caller
dialog, hold sections, and caller-agent dialogs, to be visualized. The timing of
these sections is categorized by different call types, such as calls abandoned
in the IVR system or while on hold for an agent, calls fully served in the IVR
system, and calls handled by various kinds of specialist agents.
A mini assessment 5 performs most of the analyses described above based
on manual annotations of a limited number of calls, which are monitored on-line.
Thus, a mini assessment 5 can be viewed as a version of an IVR system assessment
that delivers less accurate results but at a lower cost.
As shown by various arrows in FIG. 1, several of the IVR system assessment analysis
techniques shown in that figure, in particular techniques 1, 8, and
10, require specific annotations of caller-agent dialogs. Techniques for
gathering such annotations are described below as part of a more general method
of annotating calls.
Finally, calls can be annotated with semantic events, and a database of
call event-sequences enriched with semantic information can be accessed using information
retrieval techniques such as annotation 7. Calls can be annotated with semantic
information using at least three methods: by on-line monitoring (as in the mini-assessment),
by manually annotating recorded calls, or by automatically annotating recorded
calls. Semantic annotations can be limited to inputs required by an assessment,
such as the true reason for a call (call type) and transactions or exchanges of
information that are provided by the live agent but that could have been obtained
in an automated (IVR) system. Once such a semantic call database has been developed,
standard information retrieval techniques can be employed to access it. A detailed
discussion of the specific techniques of the overall assessment system will next
be described.
FIG.
1A
The following describes the workflow of an assessment according to the preferred
embodiment. The workflow is described with reference to FIG. 1A.
First, at step 1000 the recording method must be determined, i.e.,
how calls are being recorded end-to-end. The IVR platform employed in the call
center determines which recording method (as described above) is feasible. Preferred
are IVR platforms with a remote service-observation feature, because it makes end-to-end
recording possible with the least amount of configuration required.
At the same stage in the workflow as determining the recording method, call annotation
7 and prompt inference 6 are configured. To configure annotation
7, at step 1002 a coding sheet is developed that determines what
constitutes "significant" events in the agent-caller dialog for the call center
under investigation. As part of the configuration of prompt inference 6,
at step 1003 a non-deterministic finite-state automaton is defined that
models the call-flow (IVR) logic. In a preferred embodiment, the finite-state automaton
is defined using a flat text file called a "call flow specification" or ".cfs"
file. Once a .cfs file is established, a set of prompts that are to be detected
(rather than inferred) can be determined, as will be described in more detail infra.
At step 1004, end-to-end recordings of calls are made. Once end-to-end
recordings of calls are available, call annotation 7 begins, at step 1005,
and the various steps of prompt inference 6 can be configured. In a preferred
embodiment, annotation 7 is performed manually by human annotators. To complete
configuration of prompt inference 6, at step 1006 sample prompts
are obtained from sample recordings for each prompt that is to be detected. The
set of prompts that are to be detected have been determined earlier, based on the
call-flow specification or .cfs file. Also, once recordings are available, at step
1007 a DTMF (dual-tone multi-frequency) detector is run automatically to
obtain sequences of caller DTMF inputs for every recorded call. Furthermore, computer
programs that process thousands of recordings are configured to perform the steps
of IVR system analysis 6, i.e., to configure call splitting at step 1009,
to configure prompt detection at step 1008, and to configure prompt inference
at step 1011. This is accomplished by editing standard configuration files
to suit current needs.
With the completion of IVR system analysis 6 at steps 1007 and
1009-1013 and call annotation 7 at step 1005, a sequence
of significant events is available for every call, including both the IVR system-caller
and agent-caller dialogs, as applicable. Further processes to be discussed in detail
infra compile this event-sequence data to produce a user-path diagram 9,
a routing analysis 1, an automation analysis 8, and a life-of-call
diagram 10.
In a preferred embodiment, the user-path diagram 9 is generated using only
IVR event-sequence data. At step 1021 several computer scripts successively
compile the event-sequence data into a file that contains the IVR events for each
call in a single line, called a .sum file, fill in a two-way matrix that contains
IVR system state-transition information, called a .trans file, optionally collapse
the state-transition information by clustering states, and transform the state
transition information into a rough layout as a tree, similar to the examples of
user-path diagrams discussed infra. At step 1017 an analyst determines which
states of the call flow should be clustered and represented as a single state in
the user-path diagram, and a file must indicate the position and sizing of the
various states to be displayed in the user-path diagram. The rough layout of the
user-path diagram thus obtained is then loaded into a visualization tool (e.g.,
Microsoft™ Vision™) and at step 1025 the layout is cleaned-up manually.
At step 1019, routing analysis is performed using computer scripts that
extract, for every annotated call, the IVR exit point from the IVR event-sequence
and the call type from the agent-caller event sequence, and fill in a two-way routing
matrix with counts as described infra. To configure these scripts, at step 1015
a file is generated that defines the different IVR exit points and call types,
and how they appear in the IVR system-caller and agent-caller event sequences,
respectively. The configured scripts are cleaned up manually using graphing programs
available in standard spreadsheets (e.g., Microsoft™ Excel™).
Automation analysis 8 is performed based on the IVR event-sequence
data and results of the routing analysis 1. At step 1020 computer
scripts compile the IVR event-sequences into tables that collapse calls into various
automation profiles, and accumulate count statistics for each profile. These call-profile
count-statistics are read into a standard spreadsheet program. The spreadsheets
are configured at step 1016 to account for misrouting, as described infra.
Then, spreadsheet calculations produce a IVR system automation report and estimate
a total IVR system benefit.
The process for creating a life-of-call diagram 10 is similar. First,
a call-segment timing analysis is configured at step 1014 by indicating
the different call types that are to be distinguished and which call segments have
been annotated in calls. Next, at step 1018 computer scripts compile timing
information based on annotations of every call, and calculate average timings across
various call types. The output from the computer scripts, in table form, is read
into a standard spreadsheet program that is configured at step 1022 to produce
bar charts for the life-of-call diagram 10, as described infra.
Once all analyses have been completed, at step 1024 an analyst reviews
all the data to identify specific problems in the call flow and to infer recommendations
for call-flow redesign.
II. Detailed Description of Assessment Techniques
As outlined above, the present invention provides various techniques that form
part of an assessment methodology operable to evaluate both cost effectiveness
and usability of telephone interfaces based on a detailed analysis of end-to-end
recordings of thousands of calls. This assessment methodology is applicable to
both touch-tone IVR systems and speech-enabled IVR systems. Because agent time
is the major cost driver in call-center operations, the methodology advantageously
quantifies cost effectiveness in the form of a single number that measures how
much agent handling time a telephone user-interface (IVR system) saves.
To quantify usability, task completion is refined into a set of IVR system automation
rates, and the complete traffic through an IVR is represented as a tree, called
a user-path diagram. Beyond evaluation, the methodology of the present invention
has important implications for telephone user-interface design. Assessment results
provide concrete guidance on how to improve the interface. Furthermore, the benefit
of a redesigned interface can be estimated, thus providing the business justification
for telephone interface usability re-engineering.
As discussed above, in telephone user-interfaces, complete calls constitute the
only complete record of user and system behavior. Therefore, the methodology of
the present invention for performing a comprehensive usability assessment of telephone
interfaces is based on end-to-end recordings of calls. A call typically begins
with a dialog with an automated (IVR) system, which will be referred to as the
IVR section of the IVR system-caller dialog, which may be followed by a dialog
with a live agent, to be referred to as the agent-caller dialog.
The following sub-headings key into the various blocks shown in FIG. 1.
Collecting Data from Complete Calls (Recording) 2
Recordings of complete calls represent a large amount of data that is
difficult to analyze in its raw form. For example, as mentioned above, one hour
of recording on one channel at 8 kHz with an 8-bit resolution corresponds roughly
to 30 Mbytes of audio data. To make the analysis of call data efficient, various
techniques are used to transform the recordings into a complete trace of significant
events for each call.
Significant events of a call include, in the IVR section of a call, system
prompts and caller input (touch-tone or speech), and, in the agent-caller dialog,
exchange of various kinds of information, such as account numbers, dollar amounts,
etc., and description of the caller's problem, e.g., a question about a bill, an
inquiry into flight schedules. While most of the analyses are based on an event
sequence, the ability to switch between a call recording and its representation
as an event sequence is extremely advantageous throughout the analysis process.
The initial stages of the assessment method of the present invention involve
collecting data from complete calls and forming, from the collected call data,
a call event trace. The formed call event trace must encompass both the IVR section
and agent-caller dialog, if such a dialog occurs during a call. The reports generated
by conventional IVR platforms are inadequate and inaccurate, because they do not
track the event sequence for a call. Instead, they merely report "peg" counts,
which indicate how many times a prompt or menu was visited overall, without maintaining
references to specific calls. Conventional IVR reports are inaccurate because they
count any call that ended within the IVR section as "resolved", regardless of whether
the caller obtained any useful information.
The best method for capturing an IVR event sequence is an event log that is generated
by the IVR system itself, and is described in the disclosure corresponding to reference
numeral 3. However such IVR logging requires the IVR program code to be
modified to write to an event log at appropriate states. This process is error-prone
and intrusive to call-center operations. To avoid these disadvantages, the inventors
have developed a method that infers the complete IVR event sequence from a call
recording, and which can be performed after the fact. Since calls can be recorded
remotely, the event sequence can thus be captured in an unobtrusive fashion.
The following section describes techniques employed to collect data from complete
calls, in particular the end-to-end recording technique 2 and the IVR logging
technique 3.
FIGS.
2A-
2C
FIGS. 2A-2C illustrate three options for recording incoming calls to a call center.
In the first option, shown in FIG. 2A, recording is performed on-site at the
call
center by placing recording equipment at the call center site or by using a live
observer to listen to the calls and manually note the significant events. The recording
equipment or the live observer is connected directly to the call center's incoming
telephone lines and to a customer service representatives (CSR's) telephone handset.
Usually, the first option is used when a call center has a policy that prohibits
the off-site recording of calls or the recording of calls altogether. Therefore,
in order to be able to assess the performance of the call center, a live observer
is used to listen to (observe) calls to the call center and to manually make a
record of events that occur during the observed calls. During the IVR portion of
the calls, the observer uses a commercially available DTMF decoding device that
indicates which touch-tone buttons were pressed in response to various inquiries
made by the IVR system. If a call is transferred to a live agent, the observer
takes notes on (annotates) that portion of the call to manually create a record
of the questions, answers, and/or comments exchanged between the caller and the agent.
In practice, a recording operation according to the first option proceeds as
follows.
A caller 1101 places a call to a call center 1102 via a public switched
telephone network (PSTN) 1103, a trunking system 1104, and an automatic
call director (ACD) system 1107. The ACD system 1107 handles the
queuing and switching of calls to the call center 1102.
If the call is to be recorded or observed in real time, a voice response unit
(VRU) 1105 of the call center's IVR system 1106 notifies the caller
1101 that the call may be monitored for quality assurance or other purposes.
Then, an electronic recorder or a live observer 1108 makes a record of the
DTMF signals from the caller's touch-tone responses to inquiries or prompts from
the IVR system 1106. If the caller 1101 opts to speak with a live
agent or customer service representative (CSR) 1109, the recorder/live observer
1108 makes record of the caller-agent interactions.
In the first option, recording/observation takes place via lines 1110 between
the recorder/live observer 1108, the trunking system 1104, and the
CSR 1109, as shown in FIG. 2A.
In the second option, shown in FIG. 2B, recording is performed off-site at a
dedicated
recording facility or other suitable location. This option requires the call center's
ACD system 1107 to have one or more observation ports 1111 that enable
external dial-in observation. That is, the call center's ACD system 1107
must have a software "wire-tap" feature that allows calls to be listened to. Some
commercially available ACD systems, including ones made by Lucent™ and Aspect™,
have such a feature for monitoring call quality.
In practice, an off-site recording operation according to the second option proceeds
as follows. A VRU 1112 of a service observation system 1113 running
at an off-site recording facility 1114 places a call to the call center's
observation port 1111. The VRU 1112 provides the observation port
1111 with a DTMF security passcode, which effectively gives the VRU 1112
access to the call center's incoming calls. The VRU 1112 then specifies
an extension to be monitored and/or recorded. It should be understood that if the
ACD system 1107 has several observation ports then several extensions may
be monitored and/or recorded.
Similar to the first option, when a caller 1101 places a call to a
call center 1102, if the call is to be recorded, a VRU 1105 of the
call center's IVR system 1106 notifies the caller 1101 that the call
may be monitored for quality assurance or other purposes. Then, a recorder 1115
records the entire call, including the DTMF signals from the caller's touch-tone
responses to inquiries or prompts from the IVR system 1106, as well as the
caller-agent interactions, if any.
In the third option, shown in FIG. 2C, recording is performed off-site by using
a router 1116 to route incoming calls to a VRU 1112 of an off-site
service observation system 1113. The calls are then redirected or "tromboned"
back to the router 1116 by a trombone unit of the VRU 1112. Thus,
calls are diverted by the router 1116 to the VRU 1112 of the service
observation system 1113, where some or all of the calls may be recorded.
Optionally, the router 1116 may be programmed to decide which calls to divert
to the service observation system 1113.
In practice, a recording operation according to the third option proceeds as
follows.
A caller 1101 places a call to a call center 1102 via a PSTN 1103.
The call is then routed by a router 1116 to a service observation system
1113 of an off-site recording facility 1114 via a redirection line
11171. The VRU 1112 of the service observation system 1113
notifies the caller 1101 that the call may be monitored for quality assurance
or other purposes. The call is recorded by a recorder 1115 connected to
the service observation system 1113. A trombone unit 1118 of the
service observation system 1113 initiates a call back to the router 1116
via a redirection line 11172, and the router 1116 then routes the
call to the call center 1102 from which the call was diverted.
Then, similar to the first option, the call passes a trunking system 1104
to the ACD system 1107, where it is sent to the IVR system 1106.
The recorder 1115 records the entire call via the redirection lines 11171,
11172, including the DTMF signals from the caller's touch-tone responses
to inquiries or prompts from the IVR system 1106, as well as the caller-agent
interactions, if any.
FIGS. 2B and 2C show the recorder 1115 as a separate unit from the service
observation system 1113. It should be understood, however, that the recorder
1115 need not be a discrete unit but instead can be physically incorporated
into the service observation system 1113.
For the options described above, it should be understood that all of the incoming
calls need not be recorded and, instead, a selected percentage of all calls, a
selected number of calls, or only calls occurring during specified times of the
day may be recorded, for example.
Also, the options described above have various advantages and disadvantages,
and an option that may be ideal for one type of situation may be totally unsuitable
for another type of situation.
For example, the first option is the least flexible, because it requires the
installation and maintenance of recording equipment at the call center or the use
of a live observer set up at the call center to listen to calls. Also, recordings
must be transferred to another location for analysis, unless analysis equipment
(e.g., computers configured to run analysis programs) is installed at the call
center and trained personnel is available to run the programs at the call center.
Further, if a live observer is used, he or she must be trained as to the call flow
of the call center's IVR system. That is, the observer must know the options available
to the caller as the caller interacts with the IVR system. Also, the data collected
by the observer must be put into a form that can be analyzed by a computer.
Despite the disadvantages of the first option, if a call center does not
allow recording of calls or only allows recording to take place on its premises,
it may be the only way to obtain data necessary to make an assessment of the call
center's performance, using the assessment techniques described infra.
The second option is the most flexible, because it allows calls to be recorded
and analyzed at a facility dedicated to assessing call-center performance. It also
enables selective analysis of specific aspects of an IVR system. For example, a
call center's IVR system may initially request a caller to indicate whether the
call is related to a billing inquiry, an order inquiry, a product information inquiry,
or other inquiries. The second option enables the selective recording of, for example,
only those calls designated by the callers to be billing inquiries. This allows
the call center to target one aspect of its IVR system at a time for assessment,
by monitoring only particular extensions associated with that aspect, and prevents
the unnecessary recording of calls related to aspects not being assessed.
The second option may be implemented as an automated process by setting up the
service observation system to automatically place a call to the service observation
port of the ACD system at a designated time and provide the ACD system with the
appropriate access number. Recordings can then be made for calls to the call center
at selected hours of the day, at selected days of the week, for a selected number
of calls, for example. Because the recordings are made off site, that is, external
to the call center, there is minimal intrusion in the call center's operations.
Also, if the service observation system fails for whatever reason, this option
will not affect incoming calls to the call center.
Optionally, a call center that handles a large number of calls may use
multiple lines or a high bandwidth line, such as a T1 line, to handle multiple
calls simultaneously. Of course, the second option is not available if the call
center's ACD system does not accommodate external dial-in observation. In such
cases, the third option allows for calls to be recorded by redirecting (tromboning)
them through a service observation system. An advantage of tromboning is that it
allows for a random sampling of calls from geographically distributed call centers
to be recorded in a central location. Tromboning may also be used to enable live
observers to listen to calls at a central location, if recording of calls is not feasible.
A disadvantage of tromboning is it precludes the use of features such as caller-ID
to identify telephone numbers of callers, because the identified number will always
be the telephone number corresponding to the location of the service observation
system, i.e., where the calls are being redirected. Another disadvantage of tromboning
is that, should there be a break in any of the legs of a tromboned call, e.g.,
a break between the caller and the VRU of the service observation system or a break
between the VRU of the service observation system and the call centers ACD system,
the caller will be disconnected from the call center.
For the above options, calls can be recorded using, for example, a standard NT
workstation (not shown). According to a preferred embodiment, the calls are recorded
onto the workstation's hard disk, and the recorded data is later transferred to
a central server, which is connected to a network, so that the recorded data may
be accessed by various programs of the service observation system for analysis.
Transfer of the recorded data frees the hard disk to record additional calls.
