VoiceXML Conformance Testing

By Ken Rehor and Jonathan Engelsma

Continued from page 1...

Platform Testing Architecture

To run the test suite manually, a human tester calls into a VoiceXML platform (the System Under Test, or SUT) that is preconfigured to execute tests fetched from the test suite server as shown in Figure 1.


Figure 1. Manual VoiceXML testing architecture

Since there are over 700 tests, running all the tests could become time consuming. To automate the process, an Automated Test Driver System (ATDS, possibly a VoiceXML platform itself) replaces the human tester by placing calls to the SUT. The ATDS "talks" with the SUT via audio and DTMF prompts and commands. Results are automatically logged by the testing system.

The test driver employs a very simple architecture to minimize testing system complexity. By using simple predefined inputs and outputs, the ATDS can be developed with any common IVR testing systems or VoiceXML.

The Conformance Test Suite Development Process

The Conformance Committee is following essentially the same process used by the W3C Voice Browser Working group for developing the original implementation report test suite. Testable assertions are derived from the VoiceXML specification by methodically going through the text of the specification, section by section. These assertions are brief but reasonably precise English statements that describe a particular aspect of the language. For example, the following assertion (#362) was derived from section 3.1.6.1 in the VoiceXML 2.0 Candidate Recommendation:

"If a form-level grammar matches and returns a semantic interpretation for a field name or slot that is a non-scalar ECMAScript variable, the field will be filled with that non-scalar ECMAScript value. The selected property may be a compound object."

Next, the committee proceeds to review the assertion, ensuring that the assertion does in fact reflect the intent of the specification. Once the assertion is approved, a formal test script that exercises the behavior described by the assertion is authored. The test script implementations themselves are reviewed before they are added to the conformance test suite.

The committee's intent in the months ahead is to evolve and refine the set of assertions and associated test cases until it reaches a point where there is a reasonably high assurance of interoperability among the implementations that successfully pass the entire test suite. Periodic releases of our work in progress will be made available for download for VoiceXML Forum member companies in the members-only area of voicexml.org, and a public online version available to the general public. While we have not yet at the time of writing added any new extensions to the original W3C IR test suite, we have made a number of corrections to the test scripts that were reported in the W3C VBWG mailing list. These updates are reflected in the current "pre-release" download in the members-only area of voicexml.org.

The Conformance Test Script Language
The test script language (thanks to the Herculean efforts of Matt Oshry of Tellme, and other colleagues in the VBWG) is key in creating a highly flexible test suite. The language is essentially a superset of VoiceXML that introduces a dozen or so additional test-related elements via a separate XML namespace. Test scripts are transformed into pure VoiceXML by running them through an XSLT processor with the appropriate XSLT style sheet. This allows one to configure the test suite for a variety of different test configurations without having to rewrite the individual conformance tests themselves. There are a number of advantages to this. First, conformance testing in languages other than English can be accomplished by modifying the style sheet and regenerating the target VoiceXML scripts. Second, the test suite can be easily modified to account for differences in the test execution configuration such as an automated test environment vs. a manual test environment. Finally, while the VBWG is making progress in the semantic interpretation area, there are still a number of differences in commercial speech resources in terms of how semantic processing is specified. The differences in this area can be handled in style sheets rather than in each individual test. This problem will be discussed in more detail below.

An example test script for assertion #362 (cited above) and the resulting VoiceXML markup is shown below. The reader is referred to the W3C VoiceXML Implementation Report for more details on the test script language.

back to the top

 

Copyright © 2001-2003 VoiceXML Forum. All rights reserved.
The VoiceXML Forum is a program of the
IEEE Industry Standards and Technology Organization (IEEE-ISTO).