freealf.com

Remember your checklists!

A passage in a book I was reading today reminded me of the importance of simple checklists in testing. 

"NASA had learned how to put a man on the Moon: by using checklists."
(Kirshner, R. P. 2002. The Extravagant Universe. 1st ed. Princeton: Princeton University Press. p246.)

Gene Kranz, former flight director for the Apollo 11 lunar landing, talking about checklists:

"I always had a checklist on my console. When I was under stress, I would always go through the checklist. Get the data, call the backups in, start running d-logs (data logs), etc. So that carries you through the first moments of this crisis. It also carries the team through these first moments of the crisis. So you get through."
(Asaravala, Amit. NASA Veteran Weighs In on Launch.  Wired News, Jul 25, 2005.  p2.)

As a tester, you may be the closest thing to an expert on how an application should behave.  Remember to write it down!  If I had a nickel for every time some subtlety was missed due to my sick days, vacation days, blonde days or simple forgetfulness, I would be a much wealthier man!

Load Testing with Random Walks, Part Two

Read "Load Testing with Random Walks, Part One" here.

This article is a little late coming.  That's OK.  We're here to examine how to interface with a webpage from a simple script.  For this discussion, I'm going to be using Microsoft JScript and the XMLHTTP object provided by Microsoft (this is the same magical object that makes Google Maps work).  A quality reference that I will be using is "Dynamic HTML and XML: The XMLHttpRequest Object" published by Apple.  This article covers the use of XMLHTTP on the client-side for webpages.  We'll be using it in a stand-alone command line application.  Also see the official MSDN reference.

The goal here is to wrap some sort of component that does HTTP requests.  The actual component that is used is unimportant.  I chose XMLHTTP for this because it's a different object than the one I use with my employer.  I'd hate to get into an intellectual property spat.  The nice thing about the wrapper is that you can use it and then chuck it away when you're done.

First off, we need to define a data structure for a page and a few constants:

// "Constants" var GET  = "GET"; var POST = "POST"; function Page() { this.url = ""; this.requestType = GET; this.postData = ""; this.responseCode = 0; this.pageSource = ""; this.timeToLoad; }

Testing a Stapler

I rediscovered one of my favorite articles on software testing today at work whilst cleaning out my bookmarks.  How many test cases can you write for a stapler?  Destined to be a future interview question.

Security Concern When Implementing Unicode

The Unicode Consortium has a draft paper up about security considersations when implementing Unicode.  Good reading for testers and developers alike.

There are many potential problems, the most prominent being International Domain Name (IDN) spoofing.  In this attack, a malicious user sets up a site at a domain name using Unicode characters that look nearly (or completely - this could depend on the font your browser uses) to an existing site.  They then go phishing.  The average users is completely lost as their address bar is displaying an apparently perfect URL.

Once again, thanks to Bruce Schneier for the heads up.

Using Test Patterns in Software Testing

I've recently had success with using "test patterns" in software testing.  A test pattern is probably familiar to you from other manufacturing processes.  Your local TV station broadcasts one in the middle of the night.  The box of cereal you eat every morning has a few coloured squares printed next to each other to check the alignment of the various colours used in printing.

Test patterns in software work using the same principle - it is easier for a human being to remember and recognise a pattern than a set of details.  As an example, when asked to enter a budget amount for each month, I entered the numbers 1 through 12 sequentially.  Wherever else this data was accessed, I could immediately check the pattern, instead of the individual values, to ensure that everything was working correctly.

I find this sort of testing to be more effective than the usual approach of hammering in "random" values on the keyboard and then saying "this looks right" or of entering "realistic" values which may not have a unique entry for each field.  A human tester can easily recognise off-by-one errors and the like by noticing that a pattern is incorrect.

Other examples include picking dates that increment by one year, using paragraphs of text beginning with sequential letters or creating graphical patterns using multiple radio buttons or checkboxes.

Update:  A few other test patterns I find useful:

French Characters

ÀÁÄÈÉÊËÎÏÔŒÙÛÜŸàâäèéêëîïôœùûüÿÇç«»

HTML / SQL Server / JScript

<!--<"'\"\'""''&''""\'\"'">-->

What's Wrong with This Dialog?

Now what could possibly be wrong with this picture?

Bug: the application gives no feedback to the user indicating that it is calculating install steps.  The "Next" button magically appears as soon as this is done.

Kudos to Simon for pointing this one out.

Load Testing with Random Walks, Part One

A recent project of mine has been testing with Random Walks.  This was inspired by the random models of gases in statistical mechanics that models each particle of a gas as a simple entity obeying simple rules.  My intention is to create a similar model for web applications - each user of the application is a simple entity that follows simple rules.

I work at a web application shop, so this model is designed primarily for webpages.  It works quite well.  The steps are:

1. User GETs a page.
2. User examines page and determines the next action.
3. User either POSTs or GETs the next page.
4. Follow redirects as necessary.

We now replace the user with a "Random Walker."  The Walker talks to a "Browser" object that handles all of the communication with the server.  We'll leave the details of the Browser to a future post (and believe me, the devil is in those details).

The Random Walker, at its simplest, scans through the HTML it gets and counts the number of links it could follow, plus the number of forms it could fill out.  It then picks one action randomly.  Links are the easiest - it requests a GET from the Browser for the new URL.  In the case of forms, it analyzes the input fields (INPUTs, SELECTs, TEXTAREAs, etc) and computes a random form data string.  It then requests a POST from the browser, with that form data.

Now we add some useful features.  The Walker excludes actions that will take it off-site.  If it finds 0 possible actions, it backs up to a point where it could have made a different choice.

We can run multiple walkers simultaneously to simulate real, random, user interaction.  If you feel some actions are more likely than others, you can certainly add a weighting to the averages.  Another good enhancement is data typing for form fields (although submitting invalid data is certainly realistic from actual users).

What you'll get out of this is an enormous log file of page hits and response times.  Running some statistics and analysis can reveal some quality load data.

Next part: Getting the Browser object to work.

Mercury QuickTest Reviewed

This started as a comment on Dana Epp's blog, and I have decided to reproduce and enhance it here.

We have been using QuickTest as our primary testing tool for over 6 months at my company.

Pros

• QuickTest's IDE is fine.
• Automated runs are possible using the Windows Script Host.  One of the first things you will want to do is dump the crappy batch job runner that comes with it. I have 150 lines of JScript that are WAY more reliable.
• Documentation is OK. 
• Very versatile - QuickTest can interact with pretty much anything that ever appears on your screen.

Cons

• Pricing.  Our initial quote on a single-seat license was CDN $6,000.
• OS compatibility is sticky. Mercury uses an Activation-based approach for their software, so you'll have to activate each install. To get it to run on a Windows Server-family system, you'll need to purchase the concurrent licensing scheme (we were quoted $10,000 CDN per user for that). We tried to use our seat license over remote desktop (only one user) but the Mercury licensing prevented this entirely (and we had to call in for a reactivation).
• QuickTest tries to be all things to all people.  It's definitely not "lightweight".
• Custom actions are in VBScript. This sounds ok and is easy to start non-programmers on, but becomes really, really annoying once the users learn how to code.
• The Mercury forums and telephone support are really hit-and-miss.
• No built in automated notifications. They want you to buy Test Director (~$10,000) to do this. We're writing a Web Service based custom reporting system that's substantially cheaper (and also not dependent on Mercury software).
• Working in Subversion is difficult. QuickTest scripts are actually entire directories full of files. Some are text (like the VBScript commands), but most are binary. The text is useless without the corresponding binaries.
• QuickTest puts a mapping layer between your app and your test scripts. This is convenient when the name of a control changes... you simply update the mapping and all is well. Unfortunately, the mapping is stored in a proprietary binary format. Furthermore, the tool for managing it has no way to add controls after the first recording session. This leads to the entire mapping being dumped and re-recorded frequently, which breaks things regularly. We dumped the mapping component entirely and wrote code generators to look at HTML and generate interface classes (that talk to QuickTest). I have seen other shops write a custom mapping layer, or simply do without and hard-code control names and properties.
• Recorded QuickTest scripts are really brittle. Small changes to the application frequently breaks them in ways that require re-recording.
• Recording is a terrible way to QA, in my opinion. The app must be in a known-good state before it can be recorded. This prevents writing the test scripts before the app is ready. Worse, bugs that go unnoticed during recording become part of the script and get "passed" by the automated tests in future test cycles.
• QuickTest stores each "action" as a separate set of files in its own directory, and has reimplemented parameter passing between them. This is a nightmare to debug.

Overall, I'm not enormously impressed by QuickTest. Its core feature, interacting with the Application Under Test, works very well, but it's a "high maintenance" solution. In the long run, we will probably be moving to a script-first (ie: non-recorded) open source tool like WATIR or Selenium as these mature. Already WATIR outperforms QuickTest in many areas.

Your mileage, of course, may vary.  I would highly recommend downloading the demo version.  Instead of being blowing away by how cool everything looks, and how easily you can accomplish a trivial task, take your hardest test and try to automate it.  That will give you a much better feel for the tool.

We're Hiring

My company is hiring a part-time student to help out in the QA department.  Lots of great, hands-on experience and decent pay.  You'll definitely not be put to work doing the dregs...

I like to maintain a bit of independence between my blog and my company, so please contact me at alf@freealf.com if you're interested.  I'll point you towards the application.

Update Mar 13, 2005: The position has been filled.  Thanks for your interest.

An Excellent Article on Unit Testing

Alberto Savoia has written an excellent article on the benefits of unit testing.  The article is entitled  The Developer Testing Paradox.  In it he delves into the interesting idea that there is a threshold at which a program undergoes a "bugterial infection" in which bugs begin to multiply and compound on each other.  I would very much like to look into any non-linear trends in bug rates.  This will require consistent data from a few projects that undergo regular testing and bug reporting.

He also postulates seven laws of unit testing:

1. The Law of Management Commitment
2. The Law of Team Buy-In
3. The Law Of Metrics
4. The Law Of Targets
5. The Law Of Training And Coaching
6. The Law of Automation
7. The Law Of Failure

The Law of Failure will be near and dear to any tester's heart: Good Tests Fail.