freealf.com

Load Testing with Random Walks, Part One

A recent project of mine has been testing with Random Walks.  This was inspired by the random models of gases in statistical mechanics that models each particle of a gas as a simple entity obeying simple rules.  My intention is to create a similar model for web applications - each user of the application is a simple entity that follows simple rules.

I work at a web application shop, so this model is designed primarily for webpages.  It works quite well.  The steps are:

1. User GETs a page.
2. User examines page and determines the next action.
3. User either POSTs or GETs the next page.
4. Follow redirects as necessary.

We now replace the user with a "Random Walker."  The Walker talks to a "Browser" object that handles all of the communication with the server.  We'll leave the details of the Browser to a future post (and believe me, the devil is in those details).

The Random Walker, at its simplest, scans through the HTML it gets and counts the number of links it could follow, plus the number of forms it could fill out.  It then picks one action randomly.  Links are the easiest - it requests a GET from the Browser for the new URL.  In the case of forms, it analyzes the input fields (INPUTs, SELECTs, TEXTAREAs, etc) and computes a random form data string.  It then requests a POST from the browser, with that form data.

Now we add some useful features.  The Walker excludes actions that will take it off-site.  If it finds 0 possible actions, it backs up to a point where it could have made a different choice.

We can run multiple walkers simultaneously to simulate real, random, user interaction.  If you feel some actions are more likely than others, you can certainly add a weighting to the averages.  Another good enhancement is data typing for form fields (although submitting invalid data is certainly realistic from actual users).

What you'll get out of this is an enormous log file of page hits and response times.  Running some statistics and analysis can reveal some quality load data.

Next part: Getting the Browser object to work.

Mercury QuickTest Reviewed

This started as a comment on Dana Epp's blog, and I have decided to reproduce and enhance it here.

We have been using QuickTest as our primary testing tool for over 6 months at my company.

Pros

• QuickTest's IDE is fine.
• Automated runs are possible using the Windows Script Host.  One of the first things you will want to do is dump the crappy batch job runner that comes with it. I have 150 lines of JScript that are WAY more reliable.
• Documentation is OK. 
• Very versatile - QuickTest can interact with pretty much anything that ever appears on your screen.

Cons

• Pricing.  Our initial quote on a single-seat license was CDN $6,000.
• OS compatibility is sticky. Mercury uses an Activation-based approach for their software, so you'll have to activate each install. To get it to run on a Windows Server-family system, you'll need to purchase the concurrent licensing scheme (we were quoted $10,000 CDN per user for that). We tried to use our seat license over remote desktop (only one user) but the Mercury licensing prevented this entirely (and we had to call in for a reactivation).
• QuickTest tries to be all things to all people.  It's definitely not "lightweight".
• Custom actions are in VBScript. This sounds ok and is easy to start non-programmers on, but becomes really, really annoying once the users learn how to code.
• The Mercury forums and telephone support are really hit-and-miss.
• No built in automated notifications. They want you to buy Test Director (~$10,000) to do this. We're writing a Web Service based custom reporting system that's substantially cheaper (and also not dependent on Mercury software).
• Working in Subversion is difficult. QuickTest scripts are actually entire directories full of files. Some are text (like the VBScript commands), but most are binary. The text is useless without the corresponding binaries.
• QuickTest puts a mapping layer between your app and your test scripts. This is convenient when the name of a control changes... you simply update the mapping and all is well. Unfortunately, the mapping is stored in a proprietary binary format. Furthermore, the tool for managing it has no way to add controls after the first recording session. This leads to the entire mapping being dumped and re-recorded frequently, which breaks things regularly. We dumped the mapping component entirely and wrote code generators to look at HTML and generate interface classes (that talk to QuickTest). I have seen other shops write a custom mapping layer, or simply do without and hard-code control names and properties.
• Recorded QuickTest scripts are really brittle. Small changes to the application frequently breaks them in ways that require re-recording.
• Recording is a terrible way to QA, in my opinion. The app must be in a known-good state before it can be recorded. This prevents writing the test scripts before the app is ready. Worse, bugs that go unnoticed during recording become part of the script and get "passed" by the automated tests in future test cycles.
• QuickTest stores each "action" as a separate set of files in its own directory, and has reimplemented parameter passing between them. This is a nightmare to debug.

Overall, I'm not enormously impressed by QuickTest. Its core feature, interacting with the Application Under Test, works very well, but it's a "high maintenance" solution. In the long run, we will probably be moving to a script-first (ie: non-recorded) open source tool like WATIR or Selenium as these mature. Already WATIR outperforms QuickTest in many areas.

Your mileage, of course, may vary.  I would highly recommend downloading the demo version.  Instead of being blowing away by how cool everything looks, and how easily you can accomplish a trivial task, take your hardest test and try to automate it.  That will give you a much better feel for the tool.

We're Hiring

My company is hiring a part-time student to help out in the QA department.  Lots of great, hands-on experience and decent pay.  You'll definitely not be put to work doing the dregs...

I like to maintain a bit of independence between my blog and my company, so please contact me at alf@freealf.com if you're interested.  I'll point you towards the application.

Update Mar 13, 2005: The position has been filled.  Thanks for your interest.

An Excellent Article on Unit Testing

Alberto Savoia has written an excellent article on the benefits of unit testing.  The article is entitled  The Developer Testing Paradox.  In it he delves into the interesting idea that there is a threshold at which a program undergoes a "bugterial infection" in which bugs begin to multiply and compound on each other.  I would very much like to look into any non-linear trends in bug rates.  This will require consistent data from a few projects that undergo regular testing and bug reporting.

He also postulates seven laws of unit testing:

1. The Law of Management Commitment
2. The Law of Team Buy-In
3. The Law Of Metrics
4. The Law Of Targets
5. The Law Of Training And Coaching
6. The Law of Automation
7. The Law Of Failure

The Law of Failure will be near and dear to any tester's heart: Good Tests Fail.

SHA-1 is Broken

The SHA-1 Hashing algorithm has been broken.  Go change your password hashing scheme if you use it!

Looking at Performance Test Results

I've been heavily involved in performance testing for the past few days.  We have a home-grown solution that works reasonably well (and happens to have been my pet project).  Given the size of our organization, we were more concerned with getting a good "feel" for the results for a reasonable cost, as opposed to the nitty-gritty fine details a product like LoadRunner would have provided (at it's enormous fee).

One area I've been looking at is how to visualize test results.  We receive as input a set of page load times for a webpage, and are asked "is it fast?"  Calculating averages and plotting histograms were the first two things we tried.  More recently, I've decided I like this approach (click on the thumbnail for a larger view):

This graph shows the percentage of requests completed versus the time it took.  I like it because we can quickly determine management-friendly statements like "90% of request are fulfilled within X milliseconds".

Incidentally, the data shown was from a technology evaluation.  We were trying to determine which solution worked best under load.  The dashed red line was pure ASP.NET, the blue was pure ASP, red was a custom COM component called by ASP and the other two were in-house variations on this theme.  I was really impressed by how well ASP.NET held up under load.

Does anyone else have any thoughts or tips for visualizing data like this in a productive manner?

Now THAT's a regular expression!

The Daily WTF, in today's post, showed a potentially difficult regular expression.  I'm a reg ex geek, so I really liked this one, which was pointed out by an anonymous poster:

(?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t]
)+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:
\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(
?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ 
\t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\0
31]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\
](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+
(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:
(?:\r\n)?[ \t])*))*|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z
|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)
?[ \t])*)*\<(?:(?:\r\n)?[ \t])*(?:@(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\
r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[
 \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)
?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t]
)*))*(?:,@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[
 \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*
)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t]
)+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*)
*:(?:(?:\r\n)?[ \t])*)?(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+
|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r
\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:
\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t
]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031
]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](
?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?
:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?
:\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*)|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?
:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?
[ \t]))*"(?:(?:\r\n)?[ \t])*)*:(?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\] 
\000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|
\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>
@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"
(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t]
)*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\
".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?
:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[
\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*|(?:[^()<>@,;:\\".\[\] \000-
\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(
?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n)?[ \t])*(?:@(?:[^()<>@,;
:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([
^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\"
.\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\
]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*(?:,@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\
[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\
r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] 
\000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]
|\\.)*\](?:(?:\r\n)?[ \t])*))*)*:(?:(?:\r\n)?[ \t])*)?(?:[^()<>@,;:\\".\[\] \0
00-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\
.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,
;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?
:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*
(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".
\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[
^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]
]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*)(?:,\s*(
?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\
".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(
?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[
\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t
])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t
])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?
:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|
\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*|(?:
[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\
]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n)
?[ \t])*(?:@(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["
()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)
?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>
@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*(?:,@(?:(?:\r\n)?[
 \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,
;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t]
)*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\
".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*)*:(?:(?:\r\n)?[ \t])*)?
(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".
\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:
\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\[
"()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])
*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])
+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\
.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z
|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*\>(?:(
?:\r\n)?[ \t])*))*)?;\s*)

For those of you who don't find that inherently obviously, it's used by a Perl module to validate strings against RFC-822 (the standard defining the format of email messages and email addresses on the Internet).  Anything that matches the above reg ex is a valid email address.

The author of the module, not without a sense of humour, comments that this reg ex "somewhat pushes the limits of what it is sensible to do with regular expressions."

Christian Principles and the founding of the United States

CBS' The Nation has an excellent article examining the relationship between the early United States (before the Civil War) and the "Christian Principles" which President Bush is sure it was founded on.  Their conclusion is that the founding fathers, and the constitution, were founded on the principles of the Enlightenment - small "L" liberalism and big "R" Reason.

A wonderful little quote from Benjamin Franklin:

"A man compounded of law and gospel is able to cheat a whole country with his religion and then destroy them under colour of law"

Navigate the Streets Race

This is just too cool.  I read an article on Spacing magazine's website about the Navigate the Streets race here in Toronto (and also Ottawa, Montréal, and Winnipeg by the look of it).  Essentially it's a public scavenger hunt.  Teams of two, on foot or by transit only, compete to decipher the list of clues and find the destinations mentioned in the list.

An excerpt from the article:

Our course brings us face to face with cute chicken graffiti on the side of a poultry shop in Kensington Market, the statue of a bear outside the Olga Korper gallery near Roncesvalles, and The Communist’s Daughter, a discreet little watering hole for hipsters at Dundas and Ossington.

Strategy is also part of the game. We duck through parking garages and alleyways to escape the competition, and contemplate whether it would be faster to use the wildcard clue, which would take us to the giant dominoes at Bloor and Spadina, or clue # 9, which would lead us toward the wind turbine by the CNE.

This is a great public space hack and I want in.  As an avid automobile non-owner, I think I can take the locals when it comes to finding every nook and cranny downtown.  Anyone want to be my partner?

Don't Repair the Hubble - Replace It!

I was thrilled today to see a proposal for the Hubble Origins Probe, which would be a replacement for the Hubble Space Telescope.  Due to the absurd cost of a Space Shuttle launch (Robert Zubrin estimates, in Entering Space, that one costs approximately $600M USD) this would actually be cheaper than repairing the existing telescope!

The design of the Hubble would be re-used (which itself is just a re-hashed KH-11) and the new intruments that have already been constructed to go up with the next maintenance launch would be included in this new probe.  A new ultra-wide field camera would be added, which would improve the overall science mission at less cost.

Kudos to the HOP team for figuring out this little economic coup.  Now here's hoping that the politicians will listen!