Unit Testing Database Applications

Claus A. Christensen Steen Gundersborg Kristian de Linde Kristian Torp

{cac, earser, kdl, torp}@[Link]

Department of Computer Science Aalborg University & Logimatic Software A/S

Outline - Overall
Testing in general Unit testing
The xUnit family of unit test frameworks

Database applications
A table wrapper example

Unit testing database applications

What is wrong with the current unit test frameworks The DBMSUnit test framework

Experiences using DBMSUnit

Based on experiences from Logimatic Software A/S

Finding more information on software testing Conclusions

Future work
Livslang Lring 20. august 2004 2

About Myself
Education
Master in computer science (database systems) 1994 Ph.D. in computer science (database systems) 1998

Work four years in software industry

Three years at Logimatic Software A/S

Currently associated professor, Database and

Programming Systems Unit, Aalborg University Research interests
Programmatic access to databases XML and databases Moving objects and databases
Livslang Lring 20. august 2004 3

My Road to Working with Software Test

By introducting big enough errors in released software Trying to increase quality of work Software developed may contain severe errors
Missing sleep and bad nerves Unsatified customers

Unable to understand existing code

Written by morons

Questions from management

Does it run? Is it error-free?

Afraid to make changes to existing code Nice emails from the open source community
Livslang Lring 20. august 2004 4

Disclaimer

Livslang Lring 20. august 2004

[Courtesy: Tom Kyte]

Outline - Testing in General

Motivation Purpose of test Types of tests

Livslang Lring 20. august 2004

Why Focus on Test?

The importance of software testing and its
implications with respect to software quality cannot be overemphasized M. Deutch

Error removal is the most time-consuming phase of a

[software] life cycle R. Glass.
20% analysis, 20% design, 20% implementation, 40% test

Current available positions at Microsoft (2004-08-19)

Software development Software testing
Livslang Lring 20. august 2004

556 304
7

Test Purpose
Test of performance
TPC*C, TPC*H, TPC*R, and TPC*W ([Link]) Home-grown benchmarks
N

90% all user requests finished within a minute

Test of design or idea

Prototyping

Test of correctness
Formal verification Code reviews and walk-throughs Compile-and-run Stocastic testing (n-solutions) Unit testing
Livslang Lring 20. august 2004 8

Unit, Integration, and System Tests

A Need Finished System Success Requirements Failures System Test Success High-Level Design Failures Integration Test Success Low-level Design Failures Unit Test

Code
Livslang Lring 20. august 2004

[source [Link]

Unit, Integration, and System Tests, cont.

System Test
Testing overall requirements to the system Does the system works as a whole?

Integration Test
Testing of interfaces between subsystems How well do the subsystems fit together?

Unit test
Testing a single unit (or module) of code Uses stubs or drivers for interfaces to other units Is the foundation working?
Livslang Lring 20. august 2004 10

Outline - Unit Testing

Motivation for
unit testing JUnit

The xUnit family The JUnit test framework

A framework for Java

Extensions to JUnit

Livslang Lring 20. august 2004

11

Motivation for Unit Testing

The single most effective technique to better code! Very simple [and cheap] to get started
Return On Investment very high

Test code very informative to read

Internal and informal documentation End-users typically use code snips in documentation as an outset for development!

Supports incremental development

System can be fully tested frequently, e.g., each night

Better motivated for writing test cases if these are

intensively used.
Livslang Lring 20. august 2004 12

Motivation JUnit
It is nearly impossible to do a good job at software
error removal without tool support Used as model for many other unit test frameworks Instant binary feedback to developers Easy to use Widely used (even students use it, under pressure!) Open source (and free)
There are many extremely expensive commercial products related to software testing

Itegrated into many IDEs

Eclipse, Idea, JBuilder, JDeveleoper, jEdit, etc.

Written by very compentent people

Kent Beck and Eric Gamma
Livslang Lring 20. august 2004 13

Unit Test Characteristics

Supports Code a little, test a little development
analysis
start project

design

imp.

test
stop project

start project

stop project

Done by programmers, for programmers! Not a formal validation and verification of a program!

Livslang Lring 20. august 2004

14

The xUnit Family

JUnit (Java, master copy) **
[Link]

nUnit (all .Net languages)

[Link]

PerlUnit (Perl)
[Link]

PyUnit (Python)
[Link]

RubyUnit (Ruby)
[Link]

utPLSQL (Oracles PL/SQL) **

[Link]

Unit++ (C++)
[Link]

VBUnit (Visual Basic, commercial product)

[Link]
Livslang Lring 20. august 2004 15

The JUnit Test Framework

Software framework
Inversion of control (fill in the blanks programming) Do not call us, we will call you!

The basic test class is TestCase

TestCase run(TestResult) runTest() setUp() tearDown()

Supplied by JUnit

MyTestCase setUp() tearDown() testM1() testM2()

Supplied by programmer
16

Livslang Lring 20. august 2004

The JUnit Test Framework, cont

Three import concepts in JUnit
Test suite Test case Test method

Test Suite

Test Case

Test Method

Livslang Lring 20. august 2004

17

The setUp and tearDown Methods

public class TestHardDrive extends TestCase{ private HardDrive seagate1; private HardDrive seagate2; /** Sets up the test fixture. */ protected void setUp(){ seagate1 = new HardDrive("Seagate", "Deskstar", 1000.00, 40); seagate2 = new HardDrive("Seagate", "Deskstar", 1000.00, 40, [Link]); }

/** Tear down the test fixture */ protected void tearDown(){ // empty } }
Livslang Lring 20. august 2004 18

The Test Methods

public class TestHardDrive extends TestCase{ <snip> /** First test method */ public void testIntel1(){ assertEquals("Make not equal", [Link](), "Seagate"); assertTrue("Price not equal", [Link]() == 1000.00); }

/** Second test method */ public void testEquals(){ assertTrue("HardDrives are not equal", [Link](seagate2)); <snip> } }
Livslang Lring 20. august 2004 19

Overview JUnit Test Framework

Livslang Lring 20. august 2004

[Source: [Link]]

20

Consequences of using Unit Tests

Refactoring code becomes much easier
cost cost

time

time

Without unit tests

Using unit tests

Have more KLOC test code than real code Makes your low-level code much more reliable!
Solid foundation for integration and system tests

Decrease the use of a debugger

A debugger is a review tool not a test tool!
Livslang Lring 20. august 2004 21

JUnit Extension
Test coverage
Clover (commercial) [Link] NoUnit (open source) [Link]

MockObjects
MockMaker [Link] EasyMock [Link]

Performance testing
JUnitPerf [Link] Daedalos [Link]

Database testing
DBUnit [Link]
N

Import/export oriented. Truncates tables!

Most extensions use the Decorator Design Pattern

Livslang Lring 20. august 2004 22

Outline - Database Application

An example
table wrapper API

Overview of interfaces provided by table wrapper API The wrapper in an application stack

Livslang Lring 20. august 2004

23

A Table Wrapper API

Features illustrated using Java
getAttribute(<id>) return String

Livslang Lring 20. august 2004

Table 1 Table 2

setAttribute(<id>, String) count() return int exist(<id>) return boolean

equal(<id> 1, <id> 2) return boolean update(<id> , values) return boolean delete(<id>) return boolean id exist(<id>) return boolean id

24

Examples of Wrapper Interfaces

Table
list(<predicate>) getName(<id>) setName(<id>)

update( list <id>, list values)

get(<id>) update(<id>, values)

Livslang Lring 20. august 2004

25

Table Wrapper in Application Stack

Web-services XML API

Views

Tables

Table wrapper API more advanced than CRUD

For example Hibernate ([Link] and Cayenne ([Link] Customized for each single table
Livslang Lring 20. august 2004 26

Outline - DBMSUnit
Motivation
Problems existing frameworks

Test framework design and implementation

Static structure Test framework design Dynamic structure via sequence diagrams

Extensions Integration with utPLSQL unit test framework Oracle specifics

Livslang Lring 20. august 2004

27

Testing seen from 30.000 Feet

input

Environment

System Under Test

output

Testing functions (input/output) Testing procedures (side effects)

Livslang Lring 20. august 2004 28

Problems Existing Unit Test Framworks

Coupling between test methods
Test methods are independent must always build test fixture Foreign keys in a database context

Persistency
Focus on main memory data structures Side effects (procedures not functions) Manual clean up between executing test cases necessary

Two-valued logic
In a database context three-valued (true, false, and unknown)
Livslang Lring 20. august 2004 29

A Sample University Schema

Name StudentID Name Date CourseID SemID

SemID

Student k

Participant

Course h

CPR 1 Semester 1

SemID

StartDate

EndDate

Livslang Lring 20. august 2004

30

Database Application Test Framework

Coupling between
Test methods Test cases

Dependent of the state of the database Persistency

Must test for side effects Must clean up after use

Livslang Lring 20. august 2004

31

Coupling Between Test Methods

Setup only executed once for each test case
Also done by some JUnit extensions Reason: speed it takes milliseconds to a create database connection JUnit DBMSUnit Framework
setup() test_1() teardown() setup() test_n() teardown()

Test Case

Framework
setup() test_1() test_n() teardown()

Test Case

Livslang Lring 20. august 2004

32

Coupling Between Test Methods, cont.

Test methods are no longer independt
Must be executed in a specific order!
public class MyTestCase extends TestCase{ // insert rows into underlying table(s) public void testInsert1(){ } public void testInsert2(){ } // query and update rows inserted public void testUpdate1(){ } public void testQuery1() { } public void testQuery2() { } public void testQuery3() { } // delete rows from underlying table(s) public void testDelete2(){ } public void testDelete1(){ } }
Livslang Lring 20. august 2004 33

default order of execution

Coupling Between Test Cases

To test student a valid semesterID must exist To test course a valid semesterID must exist To test participant a valid courseID and studentID must
exist Reasons
Controllability Simpler to build text fixture Participant Student Course

Semester
Livslang Lring 20. august 2004 34

Building Test Fixtures in JUnit

Single table
public void setup(){ // semester table insert(row1) insert(row2) }

Two tables
public void setup(){ // semester table insert(row1) insert(row2) // course table insert(row3) insert(row4) }

We want to avoid the repetition of code in the setup

methods
Livslang Lring 20. august 2004 35

Coupling Between Test Cases, cont.

Test suite JUnit Semester Student Course Participant

Participant Student Test suit DBMSUnit Course Semester Time

Livslang Lring 20. august 2004 36

Static Structure of Test Case

JUnit
TestCase setup() teardown()

DBMSUnit
TestCase setup() teardown() use() disuse() TestSemester semesterId = sm47 NotSemesterId = sm99 prerequests = null test1() test2() test3()

TestSemester test1() test2() test3()

Livslang Lring 20. august 2004

37

Static Structure of Test Case, cont

TestSemester semesterId = sm47 notSemesterId = sm99 prerequests = null test1() TestStudent semesterId = [Link] studentId = St43 prerequests = TestSemester test1()

TestCourse semesterId = [Link] courseId = Co343 prerequests = TestSemester test1()

Livslang Lring 20. august 2004

TestParticipant semesterId = [Link] studentID = [Link] courseID = [Link] prerequests = TestCourse, TestStudent test1()
38

Dynamic Structure of Test Case

Setup method
Responsible for setting up the text fixture

Teardown method
Responsible for tearing down the text fixture

Use method
Responsible for make the public variables exist in the test database.

Disuse method
Responsible for make the public variables non-existing in the test database.

Run method (called by developer)

Calls all test methods in specified order
Livslang Lring 20. august 2004 39

Avoiding Repeated Setups

If we test participant the setup method of semester is
called twice
If both calls inserts data then it leads to integrity constraints violations Participant Student Course

Semester

Livslang Lring 20. august 2004

40

The Test Stack

Controls that setup method is only executed once for
each test case in a test session Controls the clean-up of the database Singleton design pattern
only one Stack object (connected to database table)
Test Case Public constants run() setup() teardown() use() disuse() test1()
Livslang Lring 20. august 2004

Stack push(TestCase) TestCase pop() TestCase top() isActive(TestCase) cleanup(TestCase)

41

Setup and Teardown Methods

void setup() // setup what this test case depends on for unit_test in prerequests unit_test.use() // tell that test case is in use [Link](self) // setup not part of any test method <snip>

void teardown() // own teardown not part of any test method <snip> // tell that test case is longer in use [Link]() // teardown what this test case depends on for unit_test in reverse(prerequests) unit_test.disuse()
Livslang Lring 20. august 2004 42

Use and Disuse Methods

void use() // setup myself [Link]() // make public variables legal testInsert1() // reuses insert test methods testInsertN()

void disuse() // make public variables illegal testDeleteN() testDelete1() // teardown myself [Link]()
Livslang Lring 20. august 2004 43

The Goal of the Test Framework

Easier to build test fixtures
Extensive reuse between test cases

Minimal number of restrictions on application

developer in test methods
None
N

Insert, update primary key Select, update other than primary key (on specific data) Insert specific primary keys, update (on specific primary keys)

Slight modification
N

Trade-off
N

We must be able to clean up

Truncating tables is not an option!
Livslang Lring 20. august 2004 44

Setup Sequence Diagram

P:run()
setup()

P St C Se Stack

Participant
use()

Course
use() use()

Semester

Student

use()

test1()

Livslang Lring 20. august 2004

45

Teardown Sequence Diagram

P:run() Participant Course Semester

P St C Se Stack

Student

teardown() disuse() disuse() disuse() disuse()

Livslang Lring 20. august 2004

46

Requirements to Users
Must publish keys
Inserted in underlying table(s) Updated in underlying table(s) Optional keys guaranteed not to be in underlying table(s)

Must comply with rules for modification Specific order in which test methods are executed
Defaults to the textual order of the file

Must specify test cases that are prerequested

Livslang Lring 20. august 2004

47

DBMSUnit Extension
Support multiple users running tests concurrently
Goal: Speed up execution of large test suites

Avoiding repeated setups

Goal: Speed up execution of large test suites

Livslang Lring 20. august 2004

48

Supporting Multiple Users

Testing large applications Testing non-connected concepts
Student Course

Extensions

Semester

Stack user, time to live, being tested, being setup Locking

Livslang Lring 20. august 2004

49

Avoiding Repeated Setups

Minimize testing time Ordering of unit tests
Directed acyclic graph Topological sort

Teardowns Large schemas

Student

Course

Semester

Livslang Lring 20. august 2004

50

Integrating with utPLSQL

Benefits
Assert library Large user community

Extensions
Setup and teardown methods must be redefined Stack must be added Requirements must be followed
N

Cannot be checked by compiler!

Livslang Lring 20. august 2004

51

Oracle Specifics
Flashback queries
Multi-transaction rollback Ensures that flash back is to a consistent database state Removes the need for the teardown methods
N

Database garbage collection

Workspace management
CVS-like multiple versions of rows Saves changed rows to private copy of the database
N

Private to the user that changed the rows Public to all users of the database

Checks-in rows to shared database

N

Allows multiple users to run concurrently

Livslang Lring 20. august 2004 52

The Framework Offers

A systematic way of testing a database applications Automatic cleanup after testing
System failure Error in unit test

Minimal effort to build test fixture

Data only inserted once!

Simple to use
Looks-and-feel of existing xUnit frameworks

Livslang Lring 20. august 2004

53

Outline - Experiences
Overall experiences
The technical side The human side

Test patterns
Ideas for how to unit test database applications

Livslang Lring 20. august 2004

54

Experiences
It is non-trivial to get the test framework to work! The first unit test takes a long time to get the up and running due
to dependencies Building unit tests takes quite a long time and is repetitious in nature (reads boring)
Auto-generation can make it more interesting

When unit test first build, repeated execution is very simple! Good error messages are essential for quickly finding the test
methods (and test case) that failed. Ripple effects could not be avoided due to coupling between test methods and test cases!
However, not that hard to find the source of the error

Brute-force method for cleaning-up the database is nice to have

can be difficult to build, due to integrity constraints
Livslang Lring 20. august 2004 55

Experiences, cont.
It is important that the test cases can use the data that is
delivered with the database application.
Can be modelled with public variables (and no insert methods)

Invariants are good for detecting that an error occurred but not
which error
As an example, a simple count on the number of rows in a table

Encapsulate all calls to the system clock such that the time can
be frozen and the test always be done at a specific time.
Should also be applied to random value function Mock objects can be used for this

A simple coverage analysis is better than none!

All method testes at least once? However, remember to test for behavior not method

Livslang Lring 20. august 2004

56

Experiences, cont.
It is important to separate test code from the real
code. Symmetric methods are a thrill to test (high testability)
e.g., object2String, string2Object

Stocastic testing using trace files are not worth the

effort. We made changes more aggressively and later in the project than usual. Do not return values from test methods
There is nothing to use it for

Use dynamic SQL where possible!

May lead to hard-to-find errors in the code
Livslang Lring 20. august 2004 57

Experiences - The Human Side

The technique is fairly easy to learn
If knowledge of, e.g., JUnit before hand then even simpler!

Test cases should not be used for building permanent

test databases Manager : Does the work work? Developer: Yes!

Livslang Lring 20. august 2004

58

Test Patterns
Minimal/maximal pattern
Minimal case only the required attributes specified Maximum case all the attributes specified

NULL pattern (the ugly face of three-valued logic)

Replace all the attribute in minimum case with NULL values one by one

Repeat pattern
Run the insert test methods twice, should raise exception Run the copy test methods twice, should raise exception

Commit pattern
Execute all test method in a single transaction Execute each test method in a separate transactions
Livslang Lring 20. august 2004 59

Future Work

Stepping through the execution of each test method Fully integration with the utPLSQL framework Making DBMSUnit a JUnit extension Performance measurements on proposed performance improvements

How to use test methods for program documentation

Livslang Lring 20. august 2004

60

Finding More Information

[Link] FAQ, good place to start
[Link] Danish site dedicated to software test [Link] Better Software magazine [Link] asp Softwaretest - kom godt i gang med testen, booklet, Poul Staal Vinje og Klaus Olsen, Dansk IT, 2004.

Livslang Lring 20. august 2004

61

Finding More Information, cont.

Robert Glass Facts and Fallacies of Software
Engineering, Addison-Wesley, ISBN 0-321-11742-5
Good introduction and arguments. No low-level details.

David Astels, Test-Driven Development A Pratical

Guide, Prentice-Hall, ISBN 0-13-101649
Overview of various unit test framework and very large example. Covers unit test of GUIs.

Glenford Myers, The Art of Software Testing, John

Wiley & Sons, 1979, ISBN 0-471-04328-1 (2nd Edition ISBN: 0-471-46912-2)
Easy to read and quite pragmatic.

Boris Beizers, Software Testing Techniques Van

Nostrand Reinhold, 1990 ISBN: 0-442-24592-0
Livslang Lring 20. august 2004 62

Reklameblok

Modelbaseret test
Vejen til mere effektiv test
Fr. Bajers Vej 7E Rum B2-109 Tid: Onsdag 25. august 13.00 16.00 [Link]

Livslang Lring 20. august 2004

63

Conclusion
Unit testing
is an inexpensive test approach is very helpful in software maintainance phase. can be the difference between success and fiasko

There is no single best approach to software error

removal R .Glass With respect to testing tools one-size-fits all does not apply! DBMSUnit is better for testing database than JUnit!
Can you live with dependencies between test cases and test methods?
Livslang Lring 20. august 2004 64

Stable requirements Accurate estimates

Livslang Lring 20. august 2004 65

JUnit Screenshot

Livslang Lring 20. august 2004

[Source: [Link]]

66

Two Concrete Test Examples

University example schema Semester test example
Single table

Student test example

Dependent tables

Livslang Lring 20. august 2004

67

Semester Test Example

Semester
SemID

1 2

Se Stack

Semester procedure run begin setup(); test_insert(); test_update(); test_exist(); teardown(); end;
Livslang Lring 20. august 2004

procedure test_insert setup test_update teardown test_exist begin update(SemID, SemID_update); if (exist(not_SemID)) then [Link](); delete(SemID_update); insert(SemID); raise not_not_found_exception; public if delete(SemID); (not constant SemID := 1; then end if; exist(SemID_update) public commit; raise constant SemID_update:= 2; if (not update_not_found_exception; exist(SemID)) then end; end if; not_found_exception; public [Link](); constant notSemID := 3; raise end; end if; end;
68

Student Test Example

Student
SID SemID

Semester
SemID

St Se Stack

Student
procedure run begin setup(true); test_insert(); teardown(); end;

procedure teardown setup procedure procedure test_insert begin begin begin [Link](false); delete(SID_update); insert(SID, [Link]); [Link](); delete(SID); public constant SemID := 1; commit; if (not exist(SID, [Link])) then public constant SemID_update:= 2; [Link](); raise not_found_exception; public constant notSemID := 3; [Link](false); end if; end; end; end;
69

Livslang Lring 20. august 2004