Choosing the Right Test Method

By Peter Leppik

Continued from page 2...

Pilot Testing
Pilot testing, or rolling out a new application to a limited number of live customer calls, can be a good way to understand application performance in the real world, but unfortunately, it comes too late in the project to make major design changes.

In addition, unless a concerted effort is made to follow-up with callers, there is no way to gather satisfaction data with this method. We recommend that pilot testing be viewed more as a roll-out strategy than a testing strategy, and that other methods be used earlier in the development process.

VocaLabs Survey
At VocaLabs, we have created a unique new method to evaluate customer service operations. A VocaLabs survey uses a very large number of individual participants-typically at least 500 people, and often 1,000 or more-in a process which is equally applicable to prototype applications as systems which are live in the field.

In a VocaLabs survey, each participant is given a specific task or tasks to try to accomplish through a call to a customer service operation. For example, some participants might be asked to check their account balance, while others are asked to activate a new service. After each call (and optionally before the call), each participant completes a questionnaire, which is matched to the individual's demographic data and a recording of the call.

A VocaLabs survey has a number of important advantages, including a large number of live callers, the ability to directly compare live and automated operations, the ability to test systems at any stage of development, the ability to gather both quantitative and qualitative data, and the ability to benchmark against industry norms.

Often, a VocaLabs survey is comparable in cost and turnaround time to other test methods, even ones which have serious flaws. While we admit to being biased, our survey method was designed from the ground up to address many of the limitations in other techniques, and we believe we have succeeded in developing a more cost-effective and meaningful way to evaluate customer service operations than any other method.

We have included a chart below that summarizes the strengths and weaknesses of each of these test methods. While no test can do everything, it is important to make sure the method you use is appropriate for what you are trying to measure.

indicates test method is suitable for the purpose indicates some suitability Empty box means no or very limited suitability		Test Method
		Contact Past Customers	Capacity Testing	Wizard Of Oz Testing	Technical Benchmarking	Focus Groups	Employee Test Calls	Pilot Testing	VocaLabs Survey
Desired Information	Technical Functionality What is the system capacity and performance?
	ROI Impact Will the system be cost effective?
	Meets Customer Needs Will the system help customers achieve their objectives?
	Customer Preferences Do customers like the system?
	Usability How easy is the system to use?
	Brand Loyalty Does the system increase customer's devotion?
	Customer Satisfaction Are customers satisfied?
	Contact Completion Do customers complete their objective in a single call?
	Consistent Experience How consistent is the experience for different customers?
	Demographic Breadth Do the results include a wide array of customer demographics?
	Statistically Accurate Are the results quantifiable and reproducible?
Target System	Pre-Launch Works on yet-to-be released new systems.
	Post-Launch Works on in-service systems.
	Automated System Measures automated system performance.

.

back to the top

 

Copyright © 2001-2003 VoiceXML Forum. All rights reserved.
The VoiceXML Forum is a program of the
IEEE Industry Standards and Technology Organization (IEEE-ISTO).