SAP Innovates Usability Testing

By Stephen Corbett, SAP AG – November 19, 2004

SAP is investing in new software and equipment to improve the way it evaluates usability. This article describes the current changes to usability testing at SAP.

Many people ask me: "How do you measure usability?" In fact there are many ways you can evaluate usability. This article concentrates on usability testing, one of the traditional methods used within SAP.

Imagine this: you walk into an unfamiliar office and there are three SAP people waiting for you. They introduce themselves and seem friendly. They explain that you will be evaluating software that they have built. They give you a list of jobs to complete and you sit down in front of a PC. The first thing you notice is that there is video camera pointing at you and the screen. You look behind you and you see that on one side of the room there is a wall of mirrors.

You start the software application and begin to work. As you work with the application, you have difficulties finishing a job. You notice everyone scribbling lots of notes, later on, you find a job particularly easy and the SAP people seem happy.

After you have finished your jobs, the development team members and usability expert sit down with you and review the positive and negative aspects of the software application. Then everyone says their goodbyes, shake hands and thank you for your participation. What you have just imagined is a usability test at SAP.

Since the formation of the first usability laboratory at SAP in Walldorf in 1993, usability testing has been one of the methods that SAP uses to evaluate the usability quality of its products. The idea behind usability testing is simple – observe people using software in a realistic setting, note any good things about the software and fix any problems that you see and do this before you get to market.

When I started at SAP, all usability testing was performed in the SAP usability laboratory. The usability lab consists of two rooms. A test room where the end users work with the software and an observation room where members of the development team observe people working. The two rooms are connected via a one way mirror, an intercom and by monitors that display the same screen that the evaluator sees. For some reason everyone who has visited a usability laboratory always remembers the wall length mirrors.

Watching people using their product can be a threatening experience for some development team members. For them the software application that they have created is equivalent to their baby. They are very protective of their baby and don't want to see people struggling using it.

However development teams generally like usability testing. They state that watching people use their application makes their work very real and energizes them to fix any problems that they see. It also takes the opinion out of the evaluation of their product. Everyone has opinions, but only when you see people using your product can you realistically evaluate the usability of your software application.

Over the years, usability testing at SAP has evolved and is no longer performed just in usability laboratories at SAP. Now SAP user interface specialists run usability tests at SAP User Group conferences or in 1 to 1 sessions at SAP customers' locations either in person, or remotely by using collaboration applications. The principle behind all these different approaches is always the same – observe people performing realistic tasks with SAP software and note what is good and bad about the software.

However while usability testing has developed, the infrastructure required to support it was getting out of date. To capture the users' interaction required a camera and other complex equipment to convert the computer screen to a video signal. This equipment was expensive and bulky and meant that testing outside of the lab was limited to using expensive portable equipment or just recording problems with pen and paper.

However this has changed with the advent of cheap memory, Web cams and powerful new software applications that digitally capture not only screen video but also keyboard and mouse usage. Due to these changes SAP decided to update its usability testing infrastructure.

Now instead of a professional video camera to record facial interaction there is a small Web cam perched on top of a monitor. The scan converter, expensive microphones, video mixer, video recorders and monitor have all been replaced by a single software application. This change will have many benefits for SAP in the future.

First, the savings have been tremendous. The usability test equipment for the first usability test laboratory cost around 30,000 Euro. Now the equivalent equipment with even more functionality is available at around 1000 Euros – a fraction of the cost. Such a low cost means more development teams will consider buying the testing equipment themselves which will stimulate greater usability testing usage.

Also the new equipment supports how usability testing solution is evolving at SAP. Now with the new set up, the minimum hardware required to have the same power as a fully fledge usability laboratory is a laptop with a Web cam. We expect to see more usability evaluations at SAP user conferences and in 1 to 1 sessions at SAP customers' locations over the coming months.

The new usability testing software records not only screen video but also keyboard and mouse interaction. It works by recording system events in real time and integrating them with captured screen and Web cam video, thus making it easier for the user interface specialist to measure end user interaction and detect more usability issues.

Being able to create digital movies will allow better sharing of the usability testing results with all members of the development team, especially those that could not attend the usability evaluation, so that everyone can participate in the evaluation process.

Finally, the biggest benefit of the changes will be noticed by our customers. By focusing on improving usability, SAP not only enhances the quality of its products but also delivers other key benefits for SAP customers, including reducing total cost of ownership (TCO). TCO drivers such as technical support costs, administration costs, training costs and end-user operations/downtime are all reduced with improved usability. Such reductions in TCO are a lasting advantage for companies and in these days of razor-sharp competition such distinctions can make all the difference between making a profit or loss.


top top