This started as a comment on Dana Epp's blog, and I have decided to reproduce and enhance it here.
We have been using QuickTest as our primary testing tool for over 6 months at my company.
Pros
- QuickTest's IDE is fine.
- Automated runs are possible using the Windows Script Host. One of the first things you will want to do is dump the crappy batch job runner that comes with it. I have 150 lines of JScript that are WAY more reliable.
- Documentation is OK.
- Very versatile - QuickTest can interact with pretty much anything that ever appears on your screen.
Cons
- Pricing. Our initial quote on a single-seat license was CDN $6,000.
- OS compatibility is sticky. Mercury uses an Activation-based approach for their software, so you'll have to activate each install. To get it to run on a Windows Server-family system, you'll need to purchase the concurrent licensing scheme (we were quoted $10,000 CDN per user for that). We tried to use our seat license over remote desktop (only one user) but the Mercury licensing prevented this entirely (and we had to call in for a reactivation).
- QuickTest tries to be all things to all people. It's definitely not "lightweight".
- Custom actions are in VBScript. This sounds ok and is easy to start non-programmers on, but becomes really, really annoying once the users learn how to code.
- The Mercury forums and telephone support are really hit-and-miss.
- No built in automated notifications. They want you to buy Test Director (~$10,000) to do this. We're writing a Web Service based custom reporting system that's substantially cheaper (and also not dependent on Mercury software).
- Working in Subversion is difficult. QuickTest scripts are actually entire directories full of files. Some are text (like the VBScript commands), but most are binary. The text is useless without the corresponding binaries.
- QuickTest puts a mapping layer between your app and your test scripts. This is convenient when the name of a control changes... you simply update the mapping and all is well. Unfortunately, the mapping is stored in a proprietary binary format. Furthermore, the tool for managing it has no way to add controls after the first recording session. This leads to the entire mapping being dumped and re-recorded frequently, which breaks things regularly. We dumped the mapping component entirely and wrote code generators to look at HTML and generate interface classes (that talk to QuickTest). I have seen other shops write a custom mapping layer, or simply do without and hard-code control names and properties.
- Recorded QuickTest scripts are really brittle. Small changes to the application frequently breaks them in ways that require re-recording.
- Recording is a terrible way to QA, in my opinion. The app must be in a known-good state before it can be recorded. This prevents writing the test scripts before the app is ready. Worse, bugs that go unnoticed during recording become part of the script and get "passed" by the automated tests in future test cycles.
- QuickTest stores each "action" as a separate set of files in its own directory, and has reimplemented parameter passing between them. This is a nightmare to debug.
Overall, I'm not enormously impressed by QuickTest. Its core feature, interacting with the Application Under Test, works very well, but it's a "high maintenance" solution. In the long run, we will probably be moving to a script-first (ie: non-recorded) open source tool like WATIR or Selenium as these mature. Already WATIR outperforms QuickTest in many areas.
Your mileage, of course, may vary. I would highly recommend downloading the demo version. Instead of being blowing away by how cool everything looks, and how easily you can accomplish a trivial task, take your hardest test and try to automate it. That will give you a much better feel for the tool.
Comments