WebServices dot org

Freedom from Scripts: Web Services Needs a New Testing Landscape

Many organizations are still stuck in the mindset of the client/server era, when rules and configurations were far simpler. In fact, in many cases, even the most basic testing disciplines have not been extended to Web services. In a recent survey, we found that only a minority of developers, 36%, even use the most basic testing tools for their Web services installations. About use 31% use unit testing tools, and 21% use load testing and performance tools.

Narendra Patil, CEO, president, and co-founder of Optimyz , is determined to change this mindset, which he believes will undermine the performance of the next generation of Web services applications.

Quality has long been the Achilles Heel of the software world. Applications and software get rushed out the door so quickly that problems inevitably crop up. And, as we all have experienced, some vendors leave it up to end-users to unofficially do the final round of tests in version 1.0. Even when developers perform exhaustive performance and regression testing of their code, there are always unanticipated situations or combinations that will jam up applications.

There are a number of testing tools on the market, many of which are bundled into larger suites. Most prominent is Mercury Interactive , with its functional test tools, including WinRunner, QuickTest Professional, and LoadRunner. Mercury Interactive also recently announced a high-end tool, Business Process Testing, which is targeted at the business process level. IBM Rational Software and Compuware , and Segue Software also are leaders in the software testing market. At the low end of the market are a number of open-source testing tools, including Apache Jmaker, TestMaker, DieselTest and OpenSTA.

However, Web services poses a particularly vexing problem in the software quality equation. Web services are supposed to be designed to be built and deployed on the fly, all in Internet time. That means minutes, not months. And they are supposed to operate on any platform. We all are aware of the dismal state of many vendors’ software products that were supposedly tested for months or even years as beta code and then as version 1.0. And internal corporate developers are constantly being forced to second-guess where and how end-users will be using, abusing, and losing their applications. Applications built and deployed on Internet time are surely a recipe for disaster.

The task grows more complex as Web services infrastructures grow, Patil warns. “If you have a large number of Web services end points, which you are developing with a reasonable level of complexity, then testing becomes an extremely manual process,” he explains. “For example, if you have 10 WSDLs, and each has 10 Web services operations, then you're talking 100 combinations. And if each operation just has 10 test data points, then you're talking 1,000 combinations. With the IDE and open-source tools, sending 1,000 requests, and getting responses and manually verifying them is a one-by-one, very sequential process.” Such a capability needs to be automated, he says.

There’s more to testing than reviewing and debugging code and interfaces for particular applications. Web services are not designed as islands; they exist to interact and eventually form the fine-grained building blocks of larger service-oriented architectures. The interaction and process flow needs to be tested.

As of yet, this issue has yet to rear its head at most organizations, since few have huge Web services infrastructures. However, as more Web services begin to pick up the heavy lifting of mission-critical applications and interactions, the heat is on to test for conformance and orchestration. Developers need better ways to effectively test the way these Web services work together. A Web services-based application is a conglomeration of fine-grained components, and smooth functioning of such an application relies on dependencies and interactions between these components.

The need for standards conformance is acute. Web services is in a state of flux, with a new generation of specifications (the WS-* line) emerging from the standards bodies. “Web service is a standard-based technology paradigm and conformance is a very key aspect of testing the Web services,” he explains. “At least for a while, it's going to be very critical to be conformant with the standards and so we provide tools that will allow you to be conformant with WSI profiles. WSI profiles will specify rules which will be supported by different vendors to ensure proper interoperability on top of your SOAP or WSDL standards.” Interestingly enough, however, Patil believes this requirement may begin to fade over time, since in will become more incumbent on vendors to certify their platforms are in conformance with standards.

Orchestration is a capability that will increasingly be on the minds of developers, however. Orchestration enables fine-grained Web services to function as a larger system. An effective Web services testing tool needs to support Web services orchestration and workflow testing based on BPEL4WS, which allows companies to describe business processes that include multiple Web services and standardize message exchange internally and between partners.

Patil positions Optimyz’s flagship product, WebServiceTester , to directly address the intricacies presented by the Web services testing process, including standards conformance and orchestration.

For the most part, most testing tools are too one dimensional, tied to inflexible scripts, and built for client-to-server type applications, rather than the complexities of Web services frameworks, Patil explains. “There could be orchestration of Web services. There could be cascading of Web services. Or, one Web service may be calling another. There could be significant real-world business software dependencies between Web services, such as output of one Web service passed out into the other Web service, and so forth, without any GUI being involved.”

Many tools only look at the GUI layer. This is adequate some of the time, but needlessly complicates the developer’s job in a Web services environment, where links to back-end applications can change on the fly. “The fundamental problem is that the GUI always keeps changing,” Patil points out. “You find more bugs and problems up front if you have an API-level interface that checks the core functionality of the product.”

A good Web service-testing tool needs to be able to test SOAP messages, WSDL metadata, and interactions, Patil relates. His company, Optimyz, is attempting to provide a testing environment that covers all the bases involved in sophisticated Web services infrastructures.

Related articles:

WebServiceTester: Comprehensive Web Service Testing Solution