COMP 325 Lecture 5: Testing Overview
major resources: Object-Oriented and Classical Software Engineering 6ed, Schach 2005, Object-Oriented Software Engineering, Schach 2008.

[ previous | schedule | next ]

Introduction

• testing historically defined as process for revealing faults
• Testing can only prove the presence of bugs, not their absence (Edsgar Dijkstra)
• some have expanded meaning to include techniques for preventing faults
• testing should be done throughout the lifecycle
• Unified Process includes Test Workflow
• testing is integral to verification and validation
• testing is integral to software quality assurance (SQA)
• you are most familiar with execution-based testing
• non-execution-based testing is also critical

Software Quality and Testing

• quality defined as how well product meets its specification
• example software quality attributes: reliability, security, robustness, performance, understandability, adaptability, usability, reusability, learnability, portability
• developer's quality culture also includes process quality (CMM, ISO 9000)
• ideal project includes quality plan, which outlines quality attributes, quality assessment process, organizational standards
• independence is a key testing issue
• project development and SQA should be under different management. Why?
• non-execution-based testing should be done by someone other than the person(s) who produced the document. Why?
• execution-based testing should be done by someone other than the programmer(s) who produced the code. Why?

Non-execution-based testing

• also known as static testing
• two major practices are reviews and correctness proving
• reviews can be performed in any workflow
• goal is to detect faults as early as possible. Why?
• major risk is temptation to use information from reviews for personnel performance evaluation
• two major review practices are: walkthroughs and inspections

Static Testing: Overview of walkthroughs

• participants: chaired by rep from SQA, rep(s) from document producing team, manager for that workflow, rep from team for next workflow
• for specification and analysis document walkthroughs, a client rep should also participate
• process:
• documents distributed in advance for participants to analyze individually
• chair will manage and facilitate the walkthrough. Meeting 2 hrs max.
• walkthrough itself usually document-driven
• document producer presents document orally, walking through each section at a time
• participants interrupt with comments (prepared in advance or spontaneous)
• presence of faults are determined and recorded
• notably, no attempt is made to correct faults. Why?

Static Testing: Overview of inspections

• more formal that walkthroughs
• participants: moderator, rep from document producing team, rep from team for next workflow, tester
• 5 step process
• (pre-inspection) document producer gives overview then distributes document
• (pre-inspection) participants prepare by studying document in detail, sometimes supplemented with checklists of things to look for and lists of faults uncovered in previous inspections
• the inspection is held with document walkthrough followed by fault finding. Meeting 2 hrs max. Moderator produces written inspection report by the next day.
• (post-inspection) document producer reworks the document to resolve faults listed in the written inspection report
• (post-inspection) moderator will follow up by check the revised document against the written inspection report to assure all issues are resolved.
• perform statistical analysis of recorded faults (number, type, severity) for process measurement
• can compare fault patterns on this project versus previous
• can detect fault patterns in specific workflows
• can detect fault patterns in specific units (e.g. classes)

Static Testing: Overview of correctness proving

• mathematical technique for proving unit (usually code or design) is correct
• two practices are assertions and loop invariants
  • an assertion is a condition specified to be true at the point it is inserted (e.g. between program statements)
  • a loop invariant is a constraint that should be satisfied by every iteration of a loop (Webster online dictionary).
  • in other words, a condition specified to be true before entering the loop, at the end of each loop iteration, and upon loop exit
  • assertions are also used to specify pre- and post-conditions (input and output specifications)
  • assertions and loop invariants are specifications, our goal is showing the program meets its specifications
• proof is only as good as its assertions and loop invariants!
• assertions and loop invariants are only as good as the specification upon which they are based!
• correctness proving is in general difficult, time-consuming if done manually
• there is correctness proving software to automate this task (can it prove itself to be correct?)
• consider costs and benefits, and risks (are human lives at stake?)
• one early published example of provably-correct program (25 lines of code) was found upon execution-based testing to have 7 bugs!
• correctness proving as a software engineering practice continues to be controversial.
• Assertions and invariants are also relevant to Programming by Contract
  • program module (e.g. method) has contractual relationship with its clients (those who invoke it)
  • Precondition - condition the caller promises will be true at the time the call is made
  • Postcondition - condition the module promises will be true at the end of the call
  • Invaniant - condition that should always be true
• Java has assertions (since 1.4) See, for example, http://java.sun.com/developer/technicalArticles/JavaLP/assertions/
  • assert <boolean expression> ;
  • assert <boolean expression> : "message" ;
  • Assertions disabled by default; to enable, use -ea switch (or enable-assert option of an IDE) when running program
  • Statement can be placed anywhere; if statement is executed and expression evaluates false, execution will halt with message

Execution-based testing

• also known as dynamic testing
• applies to code artifacts
• should be performed by someone other than programmer, such as SQA group (Why?)
• test must be designed to make the code fail!
• there are a number of practices
• unit testing
• integration testing
• acceptance testing (who does this?)
• regression testing (maintenance)
• unit testing techniques include
• black box testing (aka functional testing) based on pre- and post-condition specifications
• glass box testing (aka structural testing) based on code structures, exercise all statements
• basic testing process
• design test cases
• prepare test data
• run program with test data
• compare results with test cases
• dynamic testing should be automated as much as possible to speed the process and facilitate regression testing
• integration testing concerns both interface testing and unforeseen affects due to coupling
• exhaustive testing is not in general feasible or possible
• real-time systems are more difficult to test due to non deterministic nature of events