Tonight I finished reading "How to Break Software" by James A. Whittaker, 203 (C), ISBN 0-201-79619-8. I think the book is/was used as a text book on software testing at Florida Institute of Technology.
There were a couple of things that stuck in my mind from reading the book.
- Ultimately, and conceptually most accurately, all 'application software' interacts only with the Operating System. It is one job of the Operating System (OS) to provide access to various forms of I/O and other resources to the application software.
- When this book was written, various special testing applications were used as wedges between the software under test and the Operating System to supply bogus I/O and envirounments to the tested software as part of the test program.
The time period since this book was written has seen the rise of OS virtualization in importance. Perhaps this has already been worked out, but it seems the ability to dynamicly inject I/O reactions and make arbitrary changes on demand to the the percieved environment of the application/OS system seems to be a logical developement to OS virtualization. By this I mean to literally dial in/make changes on the fly. This would effectively replace these 'wedge programs' with virtualized OS's, and would also extend the testing methodologies Whittaker documents to prepackaged app/virtual OS bundles.
Also I can't recall much in the way of specific web application testing information in the book. Sun proclaimed "The network is the computer", and the client is as much a part of that computer as the server and any intermediate routers are. One continually hears about the difficulty of testing web applications against all possible browsers the people out in user land may be using. An ideal client would of course adhere to all web standards strickly, but today no one is using such an abstraction in the real world. An ideal *test* client would be able to dial in the actual behaviour of all real world browsers as well as the ideal, perfect standard complient browser. Such a test tool would not only be a breakthrough in web testing, but the ideal chamleonic browser for the user, to deal with all the weird, poorly tested 'web apps' on the Internet.
Perhaps someone out there knows more about these subjects and can expand on these thoughts.
4 comments:
Dallas,
You raise some excellent points in your post.
As you know I recently presented on web application testing tools to the UUASC chapter, and also blogged on the subject at:
http://charlesnw.blogspot.com/2008/03/non-website-testing-tools.html
You bring up an interesting point with:
An ideal *test* client would be able to dial in the actual behaviour of all real world browsers as well as the ideal, perfect standard complient browser.
In terms of the actual behavior, tools such as Watir:
http://wtr.rubyforge.org/
drive the browser (IE on Windows. Firefox on Windows/Mac/Linux). Which bottom line are the most popular os/browser combinations in the wild today.
However it would be very cool to have a tool which shows deviations from the w3c specs that all browsers have.
To my knowledge no way exists to test via a 100% to spec client, because no one has built said client.
I believe we need to move to the IDL model of development used by companies such as Microsoft for their entire server suite. See http://en.wikipedia.org/wiki/Interface_description_language for more details on that.
Perhaps use something such as http://rspec.info/ to build software. Should developers spend massive amounts of time writing programming code? Or should the open source process be put into a spec generation system that takes in specs and spits out perfect code?
Centralize the effort into one project, one validation suite etc.
Things to think about.
I need to ponder the virtualization testing idea more... you have touched on something that I need to devote serious brain cycles to later today. It may in fact be the quality assurance revolution that is sorely needed in the world of software.
Dallas,
Looks like he wrote another book specifically on web software. See
http://www.amazon.com/How-Break-Web-Software-Applications/dp/0321369440
for more info.
That may explain why he didn't cover it specifically in the book you mention. The amazon link for that is:
http://www.amazon.com/How-Break-Software-Practical-Testing/dp/0201796198/ref=sr_11_1?ie=UTF8&qid=1205943018&sr=11-1
It's interesting how he had a co author for the Web book. Does this indicate its a bigger subject/problem space?
Thanks for the thoughtful, extensive and excellant comments, Charles. Especially for relating this to your presentation, and pointing out Whittaker's book on Web App testing. I was vaguely aware of another book he wrote, 'How to break software security : effective techniques for security testing' ISBN-13: 978-0321194336. (That might take off in another direction of speculations.) If you haven't already done so, J.A.W. probably deserves an Amazon search. :-)
One other point in relation to Whittaker's book "How to Break Software Security..." - even in the plain ol' "How to Break Software: ...", he refers to the verious things to check for not as tests, but "attacks". :-)
Post a Comment