System tests
From Informatics
PLEASE READ THROUGH THE ENTIRE DOCUMENT BEFORE PERFORMING TESTS
Contents |
Best practices for System tests
What are system tests
System tests (manual tests, GUI tests) are test procedures for manually testing software components. In the case of geWorkbench, there is (should be) a system test for each modular component that is accessible through the GUI. Other tests, such as Unit tests, are performed automatically during development and before releases.
List of System Tests
There is an Excel work sheet under sharepoint/Testing called "geworkbenchv1.1 components.xls" link that based on Ken's works lists all components and the status of the system tests.
This list shows all tests under development and is not the list to work from when performing system tests
Execution of system tests
System tests are located on SharePoint (https://sharepoint.cu-genome.org/c2b2/Testing/) and are MS Word documents. These documents and the associated data (to be found in the same directory hierarchy) should be downloaded to the local machine for execution. The word documents are "MS-Word forms" with instructions on what to do and drop down selections at critical points to indicate whether a step passed or failed. At the end of the document there is space for comments. A word macro (Visual Basic) transfers the data from the word document and the word document itself to a database. The DB is called "system_tests" and the table is "log" and located on AFDEV. The project name, username, date/time, status of worst result from a system test, and module tested (filename) are stored.
Installation requirements
In order for the system tests to work correctly, some features have to be installed/configured. The system has been tested in the following enviornments:
- Windows XP - Word 2003
- Windows XP - Word 2007 (several security issues need to be resolved, and the system test has to be unprotected before executing the macro)
- Windows Vista - Word XP SP3 (several security issues have to be resolved).
If the macro was not signed (no attached certificate of trust) and you have verified the source of the unsigned macro and believe you can trust it, you can enable the macro with the following procedure:
- Select the Tools menu option and then select Macro and Security. In the resulting Security dialog, set the security level to Medium by clicking the Medium radio button.
- Close the file and any other instances of the application currently running on the computer.
- Open the file again and click the Enable button when prompted to allow for the unsigned macro to run.
MySQL ODBC
A MySQL ODBC is needed for the connection to the centralized database (located on AFDEV). The drivers can be downloaded from mysql.com (http://dev.mysql.com/downloads/connector/odbc/3.51.html). We have succesfully used Version 3.51.12-win32 locally. The driver, once installed, should not require frequent updating.
MS Template
a template file (systemTests.dot available here: https://sharepoint.cu-genome.org/c2b2/Testing/) needs to be copied to c:\workbench on the local computer. This documents holds the macro and UIs needed to perform the system tests. It would be possible to move these scripts to the individual word documents but then each time the script changes, all scripts have to be changed. That is impractical and more error prone.
When downloading the file from SharePoint, please make sure that the files ends in ".dot" and NOT ".doc". If downloaded as ".doc" just change the extension manually.
VB macros
WARNING: it is NOT necessary anymore to have the systemTest.dot template file open when executing the macros. (5-16-08)
Short description on how the macro works
This macro / VB Script will
- ask for the Operating System used for testing (cancel option)
- go through the current document (ActiveDocument),
- search for all drop-down fields,
- collect the "worst" result,
- collect the comments at the end of the document,
- confirm the worst result (cancel option)
- store the information in the MySQL database,
- save the document with the username and date added to original name next to the original,
- open the original file
- upload the saved file to the database
- print a confirmation message
(Closing the file is necessary for uploading the document into the database.) The VB Script is now stored in a template that has to be loaded (see instructions below) The system test documents contain a field-code bookmark with the tag "PROJECT" having a value of the project name "geWorkbench". This way the execution of the macro in unintended documents can be circumvented.
Location of test scripts
The scripts are located on SharePoint (https://sharepoint.cu-genome.org/c2b2/Testing/ VersionX), where X will indicate the number of the release currently being tested. Test scrips have to be downloaded to the local computer. Each executed test script will be saved by the macro on the local machine with the user name and date attached to the original file name in the directory where the downloaded version was opened. The file will be also transfered to the database. <P> All data files needed for a given test are located in sub-directories beneath the respective script (so it is advisable to download the entire directory structure for a test from Sharepoint to avoid unexpected errors). It is easier to copy entire directories from SharePoint when using the Explorer view. To change to that view, click on "Explorer View" on the left of the screen (this applies to Windows XP, Vista has some problems here):
Working with system test documents
In order to be able to use the macro for all system tests the following procedure is suggested: download the MS Word template "systemTests.dot" from https://sharepoint.cu-genome.org/c2b2/Testing/ into "c:\workbench"
(at some point in the near future we could write a small script to automatically update that file..)
When opening a system test you should see the following tool bar entry:
If you don't see that, you have to activate the tool-bar to show the button that executes the script: within the tool-bar section of a System test MS Word document right click and select "System Tests" (last entry in list).
Executing test scripts
The MS Word documents have detailed instructions on what to test and what to compare/check. Each check point is associated with a drop-down box with four values: pass, inconclusive, fail, error.
- Pass - when everything works as expected.
- Inconclusive - you suspect correct behavior of the program but find an error in the system test script.
- Fail - if the test failed
- Error - if the program crashes or you see something really bad happening otherwise.
The order defines the severity. The most severe result will be stored in the database, with "Error" being the most severe
The task of the tester is to select the appropriate values during the test. A comment field at the end of each test script gives room for any comments. Please use this field extensively. It is not always the case that a system test will be executed until the very end. Sometimes further execution will be hindered by a defect. In such a case the defect should be reported (see below) and made note of in the comment section.
After a test script has been completely executed, the macro has to be executed to pass the results to the database. By pressing the "WB" button (see above) this macro will be executed.
If a test script fails
In case of failure do the following:
- Use the Comments section of the test script to document exactly what went wrong. The text that you enter there will be (conveniently) displayed in the test script status page (http://afdev/systemtest/BrowseLogs.php) under the column titled "Comments".
- If the reason of the failure is a reproducible defect (as it should be in the majority of cases) then report the defect in Mantis (unless one is already there - CHECK FIRST). See the following wiki section for a description of the defect reporting process: Bug reporting. NOTE: remember to correctly set the values of the fields "Product version" and "Scheduled for release" to indicate the CVS branch the testing is performed on and the target release number.
- After entering a new defect report in Mantis (or discovering that one already exists) go to the script status page here, find the row corresponding to the test script at hand, click the "Edit" link in the "Edit Bugs" column and enter the defect number (not the defect URL, just the number; multiple numbers can be entered using commas to separate them). This allows linking a failed script with its corresponding Mantis entries: this way, one can decide easily if it's time to attempt and re-run the script (when all linked Mantis defects have been resolved).
Location for test results
Michael Honig created a web page for looking at the current results: http://afdev/systemtest/BrowseLogs.php
