During the STAR WEST '99 conference many of the speakers who discussed test automation indicated that in their experience "Capture/Playback simply doesn't work". In most cases this comment seemed to be made in the context of GUI testing (although this was usually not actually stated), and our own experience certainly backs this up. Tests recorded (Captured) using tools designed for GUI testing almost always need some form of editing before they can be used to "Playback" the test reliably. The editing required depends on the tool used and the type of test, but in every case the tester must understand the "scripting" language used and be familiar with a large range of additional functions that can be added to the test "script" to ensure that the test will replay correctly each time it is used. Testing using this type of tool can only be done successfully by people who are trained programmers, and creation of the tests can be a difficult, error prone and time consuming task because of the programming involved.
However, if implemented in the right way, Capture/Playback tools which are designed specifically for testing host based applications do NOT suffer from this type of problem. "Captured" tests WILL play back reliably every time without the need for ANY editing.
If the application you are testing changes (e.g. new fields are added to a screen) then you may need to edit your tests, depending on whether the changes affect the input data steam, the output data, or both. However, if the tool is designed to easily support application changes, then editing need be no more complex than editing a typed document, because there is no special scripting language used, and no need for programming knowledge or skills. TALC2000 supports this kind of editing for all components of a test case, and in the Windows NT version almost any edit function can be performed with the mouse alone. If application changes affect application output only, then test cases can be easily updated automatically, without perfoming ANY editing.
How do you know if an automated test tool will allow simple and reliable "Capture/Playback" operation ? Obviously the best way to find out is to try it. However, you probably will not want to try every test tool available in the market, and will need some way of narrowing the range of possible options. Start by eliminating every tool which can also be used for GUI testing. There is currently no such thing as an easily used, general purpose, "one size fits all" test tool, and there probably never will be. If a tool is designed to support GUI testing, then by default it will use a full "scripting" language of some kind, and will need skilled programmers to use it effectively.
To allow easy and reliable "Capture/Playback" operation for host based applications, an automated test tool must provide:
1. Fully automatic adjustment to variations in host response times. Simply playing back the test at the same speed as it was recorded will not work.
2. "Capture" of the underlying data displayed by your host application. This data needs to include the row and column location and attribute of every field and character on the screen, but be completely independent of the emulator you are using, the emulator font, the window size and location and anything else that does not directly relate to what your host application is doing. If you need to rely on a bitmap image for any form of host output comparison or checking then changes in any of these items will invalidate your test.
If the tool you are considering does not provide these features, then it will not support reliable "Capture/Playback" operation.
TALC2000 fully supports these and many other functions which are critical to providing an easily used and reliable testing environment.
Even with the best tools, there are some functions that "Capture/Playback" cannot provide. For example, suppose your test application provides a unique computer generated number during execution of a test case, and that you then need to use that number in a later part of the test or in some different test. This type of problem can only be solved by some kind of "programming". TALC2000 solves it by providing a function that allows special needs to be accessed from a list box at any time during test "Capture". Someone with programming skills can set up the procedure to do this (it requires one function call) and catalog it so that it can be re-used as and when required by whoever is creating the test. The test operator simply requests a special function by a click on a button, then selects the function required. This is done as the test is being recorded, and has no effect on the test application.
TALC2000 provides many other unique ways to support totally reliable and extremely simple test "Capture", without in any way limiting the testing tasks that can be undertaken. If in doubt, please ask us for the names of some existing users who can verify this, or arrange for a free limited time evaluation copy to trial in your own environment.
TALC2000 Showcase List