Method and system for object level software testing Number:7,334,219 from the United States Patent and Trademark Office (PTO) owispatent A system, method, and computer program product for software testing, including analyzing the source code under test to generate a scan file, including a map of the source code under test and a tree structure for the source code under test; generating a stub file for the source code under test based on the scan file; generating a driver test script file based on the scan file and the stub file; generating a test driver based on the driver test script file; and running the test driver on the source code under test to generate a results file for summarizing the results of the software testing.<H software, testing, Method, and, syst, 7334219.html, Patent, pto, 7334219

Title: Method and system for object level software testing

Abstract: A system, method, and computer program product for software testing, including analyzing the source code under test to generate a scan file, including a map of the source code under test and a tree structure for the source code under test; generating a stub file for the source code under test based on the scan file; generating a driver test script file based on the scan file and the stub file; generating a test driver based on the driver test script file; and running the test driver on the source code under test to generate a results file for summarizing the results of the software testing.

Patent Number: 7,334,219 Issued on 02/19/2008 to Cebula, et al.

Inventors: Cebula; James David (Conklin, NY), Gee; Raymond A. (Windsor, NY)
Assignee: ENSCO, Inc. (Falls Church, VA)

Appl. No.: 10/674,064

Filed: September 30, 2003

Related U.S. Patent Documents


Application Number	Filing Date	Patent Number	Issue Date
60414325	Sep., 2002

Current U.S. Class: 717/124 ; 714/38

Current International Class: G06F 9/45 (20060101)

Field of Search: 717/124 714/38

References Cited [Referenced By]

U.S. Patent Documents


5359546	October 1994	Hayes et al.
5541863	July 1996	Magor et al.
5615333	March 1997	Juettner et al.
5651111	July 1997	McKeeman et al.
5754860	May 1998	McKeeman et al.
5781720	July 1998	Parker et al.
5790117	August 1998	Halviatti et al.
5799266	August 1998	Hayes
6067639	May 2000	Rodrigues et al.
6071316	June 2000	Grossen et al.
6202199	March 2001	Wygodny et al.
6301701	October 2001	Walker et al.
6311327	October 2001	O'Brien et al.
6332211	December 2001	Pavela
6353896	March 2002	Holzmann et al.
6370681	April 2002	Dellarocas et al.
6662312	December 2003	Keller et al.
6769114	July 2004	Leung
7062753	June 2006	Ward et al.
2002/0038401	March 2002	Zaldi et al.
2003/0023856	January 2003	Horne et al.
2004/0010735	January 2004	Paternostro et al.
2004/0031015	February 2004	Ben-Romdhane et al.
2004/0060044	March 2004	Das et al.

Primary Examiner: Chavis; John
Attorney, Agent or Firm: Nixon Peabody LLP Costellia; Jeffrey L.

Parent Case Text

CROSS REFERENCE TO RELATED DOCUMENTS

The present invention claims benefit of priority to commonly assigned, now abandoned, U.S. Provisional Patent Application Ser. No. 60/414,325 of Gee et al., entitled "OBJECT LEVEL SOFTWARE TEST SUITE," filed Sep. 30, 2002, the entire disclosure of which is hereby incorporated by reference herein.

Claims

What is claimed is:

1. A computer-implemented method for software testing, comprising: analyzing the source code under test to generate a map of the source code under test or a tree structure for the source code under test; and to generate a scan file of all or a portion of the map or tree structure of the source code under test; generating a stub file for all or a portion of the map or tree structure based on the map or tree structure of the source code under test; generating a driver test script file for all or a portion of the map or tree structure of the source code under test based on the scan file and the stub file; generating a test driver for all or a portion of the map or tree structure of the source code under test based on the driver test script file; and running the test driver on all or a portion of the map or tree structure of the source code under test to generate a results file for summarizing the results of the software testing.

2. The method of claim 1, wherein said analyzing step is automatically performed by a code scanner configurable by a graphical user interface.

3. The method of claim 1, wherein the step of generating the stub file is automatically performed by a stub generator configurable by a graphical user interface.

4. The method of claim 1, wherein the step of generating the driver test script file is automatically performed by a driver script generator configurable by a graphical user interface.

5. The method of claim 1, wherein the step of generating the test driver is automatically performed in a driver build directory configurable by a graphical user interface.

6. The method of claim 1, further comprising displaying the results file with a graphical user interface.

7. The method of claim 1, wherein the map includes a list of executable source code names, package names, and procedure or function names for the package names.

8. The method of claim 7, further comprising generating the list of executable source code names for the stub file and the source code under test.

9. The method of claim 7, further comprising generating the tree structure by performing calls until the procedures or function for each of the packages are determined.

10. The method of claim 1, further comprising providing a graphical user interface for performing the analyzing step, the step of generating the stub file, the step of generating the driver test script file, the step of generating the test driver, and the running step.

11. The method of claim 10, further comprising employing the graphical user interface for specifying variable and variable type definitions.

12. The method of claim 10, further comprising employing the graphical user interface for documenting the software testing.

13. The method of claim 10, further comprising providing a FTP capability via the graphical user interface.

14. The method of claim 10, further comprising providing multi-language support for the source code under test via the graphical user interface.

15. The method of claim 10, further comprising providing mixed-language software testing via the graphical user interface.

16. The method of claim 10, further comprising displaying the scan file, the source code under test, the driver test script file, the test driver, and the results file, via the graphical user interface.

17. The method of claim 1, further comprising implementing the method via object-oriented programming.

18. The system of claim 1, wherein said analyzing means is automatically activated by a graphical user interface.

19. A system comprising a processor for software testing, comprising: means for analyzing the source code under test to generate a map of the source code under test or a tree structure for the source code under test; and to generate a scan file of all or a portion of the map or tree structure of the source code under test; means for generating a stub file for all or a portion of the map or tree structure based on the map or tree structure of the source code under test; means for generating a driver test script file for all or a portion of the map or tree structure of the source code under test based on the scan file and the stub file; means for generating a test driver for all or a portion of the map or tree structure of the source code under test based on the driver test script file; and means for running the test driver on all or a portion of the map or tree structure the source code under test to generate a results file for summarizing the results of the software testing.

20. The system of claim 19, wherein the means for generating the stub file is automatically activated by a graphical user interface.

21. The system of claim 19, wherein the means for generating the driver test script file is automatically activated by a graphical user interface.

22. The system of claim 19, wherein the means for generating the test driver is automatically activated by a graphical user interface.

23. The system of claim 19, further comprising means for displaying the results file.

24. The system of claim 19, wherein the map includes a list of executable source code names, package names, and procedure or function names for the package names.

25. The system of claim 24, further comprising means for generating the list of executable source code names for the stub file and the source code under test.

26. The system of claim 24, further comprising means for generating the tree structure by performing calls until the procedures or function for each of the packages are determined.

27. The system of claim 19, further comprising means providing a user interface for accessing the analyzing means, the means for generating the stub file, the means for generating the driver test script file, and the means for generating the test driver, and the running means.

28. The system of claim 27, further comprising means for employing the user interface for specifying variable and variable type definitions.

29. The system of claim 27, further comprising means for employing the user interface for documenting the software testing.

30. The system of claim 27, further comprising means for providing a FTP capability via the user interface.

31. The system of claim 27, further comprising means for providing multi-language support for the source code under test via the user interface.

32. The system of claim 27, further comprising means for providing mixed-language software testing via the user interface.

33. The system of claim 27, further comprising means for displaying the scan file, the source code under test, the driver test script file, the test driver, and the results file, via the user interface.

34. The system of claim 27, wherein the user interface includes a graphical user interface.

35. The system of claim 19, further comprising means for implementing the system via object-oriented programming.

36. The system of claim 19, wherein the means for analyzing, the means for generating the stub file, the means for generating the driver test script file, the means for generating the test driver, and the means for running the test driver comprise devices of a computer system.

37. The system of claim 19, wherein the means for analyzing, the means for generating the stub file, the means for generating the driver test script file, the means for generating the test driver, and the means for running the test driver comprise computer-readable instructions stored on a tangible computer storage medium.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to methods and systems for testing software, and more particularly to a method and system for testing software at the object level.

2. Discussion of the Background

Customers that develop and maintain software prime source code (e.g., avionics source code, etc.) are in need of object level and unit testing for newly developed code. Such object level and unit testing typically is performed by a multitude of methodologies that vary from project to project. In addition, very few standards and guidelines are shared among software test engineers. Typically, each computer component grouping is tested vastly different from the next, even within a common project. Furthermore, such testing typically is based upon the particular preferences of the individual software testing engineers, resulting in various problems, and inefficiencies.

SUMMARY OF THE INVENTION

Therefore, there is a need for a method and system for software testing that addresses the above and problems with software testing methodologies. The above and other needs are addressed by the exemplary embodiments of the present invention, which provide an exemplary system and method for software testing. The exemplary embodiments include novel automated processes in the form of a software testing tool suite for performing code analysis, for providing variable and variable type definition, for providing driver creation, for providing stub generation, and the like. Advantageously, the exemplary system and method for software testing can provide a novel platform for standardized test procedure development, for example, including repeatable test cases to prove software verification, a consistent and standard look and feel for test procedures, results file, and test case documentation, and a re-usable test procedure for subsequent regression efforts.

Accordingly, in aspects of the present invention, a system, method, and computer program product for software testing are provided. The system, method, and computer program product include analyzing the source code under test to generate a scan file, including a map of the source code under test and a tree structure for the source code under test; generating a stubbed file based on external calls of procedures/functions within the source code under test; generating a driver test script file based on the scan file and the stub file; generating a test driver based on the driver test script file; and running the test driver on the source code under test to generate a results file for summarizing the results of the software testing.

Still other aspects, features, and advantages of the present invention are readily apparent from the following detailed description, simply by illustrating a number of exemplary embodiments and implementations, including the best mode contemplated for carrying out the present invention. The present invention also is capable of other and different embodiments, and its several details can be modified in various respects, all without departing from the spirit and scope of the present invention. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 illustrates test suite directories of a software testing tool suite, according to an exemplary embodiment;

FIGS. 2-5 illustrate systems for testing object level software, according to various exemplary embodiments;

FIGS. 6-11 illustrate processes for software testing that can be performed by the exemplary embodiments of FIGS. 1-5;

FIGS. 12-14 illustrate a test setup file, a driver script file, and a results file, according to exemplary embodiments;

FIGS. 15-37 illustrate screen shots of the software testing tool suite of FIGS. 1-5, according to various exemplary embodiments;

FIG. 38 illustrates a software calling hierarchy, according to an exemplary embodiment; and

FIG. 39 illustrates an exemplary computer system, which can be programmed to perform one or more of the processes of the exemplary embodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention includes recognition that typical software testing requires a large amount of redundant effort for performing various related tasks, such as code analysis, test driver creation, variable and variable type definition, stub generation, regression test availability, and the like. In addition, the present invention includes recognition that test development can be dependent on the style and interpretation of an individual designer or tester and that the content of resulting test procedures and test documentation can vary greatly from desired standards.

The test suite of the exemplary embodiments addresses the above and other problems by providing novel automated processes, in the form of an exemplary software testing tool suite, for performing code analysis, for example, via an exemplary code scanner, variable and variable type definition, driver creation, for example, via an exemplary driver script generator, stub generation, for example, via an exemplary stub generator, and the like. The exemplary test suite further addresses the above and other problems by providing a novel platform for standardized test procedure development, for example, including repeatable test cases to prove software verification, consistent and standard look and feel for test procedures, results file, and test case documentation, re-usable test procedure for subsequent regression efforts, and the like.

The exemplary test suite can be configured to be platform independent, for example, via an internal File Transfer Protocol (FTP) application, can provide a multi-language test capability, for example, for C, Ada, FORTRAN, and the like, can provide a mixed-language test capability, can provide an easy to use graphical user interface (GUI), can provide automation of test set-up and support activities, reducing the time of non-test related efforts, can be configured as an object oriented design, allowing for flexibility in functionality and features, can provide a user easy access to source code and data files (e.g., by double clicking on file to display contents thereof), and the like.

Advantageously, the exemplary test suite can provide improved productivity, for example, when creating new tests, performing regression testing, and the like. The exemplary test suite can be configured to provide ease of software certification (e.g., DO187 certification for avionics software), can provide testing repeatability, including consistent documentation, full featured result files, and the like. The exemplary test suite can be configured to provide stub files for an entire build being generated automatically, can automatically perform analysis of code to identify testable gates by searching for keywords, reducing time to determine coverage, and the like. The exemplary test suite can be configured to provide metrics, for example, on source lines of code (SLOC), number of page description language (PDL) lines, procedures, and functions, and the like.

The exemplary test suite can be configured to provide creation of a driver that is consistent in format, can be used to set and read signals from or to software under test, can provide input/output variables and types extracted from easily accessible source code to reduce error, save time, and the like. The exemplary test suite can be configured to provide expected and actual test results, tolerances, pass/fail status, in a consistent and repeatable format. The exemplary test suite further can facilitate entry of information for requirement traceability, software version, test descriptions, software design, and the like. The exemplary test suite can be configured to provide selection of multiple code modules to perform integration level testing, and the like.

The exemplary test suite can be employed to assist a user in, for example, complete testing of software development, software code, and the like. The exemplary test suite can be implemented, for example, as a PC-based, Windows driven application that, for example, can be configured to allow the user to select functions that can test prime software at the procedure or function level, the object or string level, the systems integration level, and the like. The user can easily select the exemplary test suite to be configured for testing source code written in, for example, Ada, C, FORTRAN, and the like. However, the exemplary test suite can be used with other operating systems and can be used to test other types of source code, as will be appreciated by those skilled in the relevant art(s).

In an exemplary embodiment, exemplary internal tool functions can be configured to handle error conditions, and the like, that may occur. For example, an integer error value can be returned to a main function, wherein a returned value of zero can be used to indicate no error, a non-zero value can be used to indicate an error, and the like. The exemplary error handler, as further described, can provide a listing for specific errors and associated error numbers. In an exemplary embodiment, a storage media, such as a CD ROM, and the like, can be created and configured to include software, documentation, install files, and the like, needed to run the exemplary object level software test suite, for example, on a Windows 95 or newer PC, and the like.

Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, and more particularly to FIG. 1 thereof, there are illustrated test suite (TS) directories, according to an exemplary embodiment. In FIG. 1, the exemplary test suite directories can include a test suite directory 102 that can be created by running the install executable, a build directory 104, including files employed for compiling and linking, including symbolic linking, to build an executable file used for testing, a configuration management (CM) source directory 106, including configured files, which also can include symbolic linking, a TS output (ts_output) directory 108, including files generated by the exemplary tool suite, and the like.

In an exemplary embodiment, an installation executable (e.g., called install.exe) can be created and when invoked can install files and executables employed for running the exemplary object level software test suite package. In an exemplary embodiment, during installation, a note can be displayed indicating to the user to modify the setup file for particulars of the user. A file (e.g., called readme.txt) also can be installed. In an exemplary embodiment, the default installation directory can be set to C:\test_suite\, and the like, but otherwise the default directory can be user specified.

In an exemplary embodiment, questions that can be displayed to the user, prompted to the user, and the like, when the installation executable is invoked, for example, can include: 1. Enter installation directory: (Default directory C:\test_suite\). 2. Display README.txt File (Y/N): 3. Run test suite now (Y/N):

In an exemplary embodiment, an uninstall executable (e.g., called unistall.exe) can be created, such that when invoked can remove files and executables related to the exemplary test suite from the default directory, C:\test_suite\. In an exemplary embodiment, the default directory for uninstalling also can be user specified.

FIGS. 2-5 illustrate systems for testing object level software, according to various exemplary embodiments. In FIG. 2, graphical user interfaces (GUI) 204 can be used to bind a test suite 202 together and allow the user to easily perform the various tasks involved, for example, by use of pages, windows, and the like, of the GUI 204. In an exemplary embodiment, one or more GUIs 204 (e.g., implemented using Visual C++, Visual Basic, and the like) can be employed and can include stub generator functions 206, code scanner functions 208, driver script generator functions 210, helper functions 212, and the like. In further exemplary embodiments, in the exemplary systems 300 and 400, as shown on FIGS. 3 and 4, build directory functions 314, tool utility functions 316, such as an FTP Utility 418, and the like, also can be provided.

The exemplary system 200 can include the CM source directory 106, including source code 222, and user inputs 224, the TS output directory 108, including stub files 228, scanned files 230, and driver scripts 232, the software (SW) test build directory 104, including interface software 236, an execute source under test component 238, test results report 240, and the like. In further exemplary embodiments, in the exemplary systems 300 and 400, as shown on FIGS. 3 and 4, an execute test driver component 342, a setup and run test environment component 344, and the like, can be provided. In FIGS. 2-4, components shown with a dashed line can be content management controlled items.

In an exemplary embodiment, the exemplary test suite can be run as a Windows application, providing the user with a simplified approach to software testing via the exemplary GUI 204. The functional software of the exemplary test suite, for example, can be executed from the command line, for example, by executing the helper functions 212, which can be written in C, C++, and the like. Advantageously, this feature can allow the exemplary test suite to be run on non-Windows platforms, such as UNIX platforms, DOS platforms, and the like.

In an exemplary embodiment, as previously described, the directory structure can be created, for example, including the CM source directory 106, the TS output directory 108, the build directory 104, and the like, as shown in FIG. 1. The CM source directory 106 can include source code to be tested, as well as the interfacing software 236. The TS output directory 108 can include files created by the exemplary test suite, such as stubbed files, scanned files, test script files, test driver files, and the like.

In FIG. 5, a further exemplary object level software testing system 500 can include a user interface 504, for example, implemented as a Java script, and the like, for providing the noted user interface functions, an executable component 502, including a translator component 506, a driver component 508, a source component 510, a shared memory component 512, and the like. The shared memory component 512 can further include a controller component 518, an input parameters component 514, an output parameters component 516, which can be picked, chosen, and the like, by a user, via the corresponding components 520 and 522 (e.g., implemented using eXtensible Markup Language (XML), etc.), and the like. In an exemplary embodiment, the source component 510 can include the object under test, specs, children, stubs, and the like, that are employed for complete compiling and linking.

In an exemplary embodiment, the translator 506 can be configured to receive data, via the user interface 504, and convert the received data into another format, for example, such as an XML format, and the like. The exemplary translator 506 can further be configured to take in the XML formatted data and convert such data into a language specific format for the layout of the shared memory 512 partition. The exemplary translator 506 can be configured to handle various data formats, such as integer byte, integer half word, integer word, float, character data, 1553 bit pattern, 422 bit patterns, and the like.

In an exemplary embodiment, a scheduler sub-component (not shown) can be provided and can be configured for calling the driver 508 at a rate defined by the user from the user interface component 504. The execution rate of the driver 508 can be configured for various rates of executions, such one shot execution, 50 Hz execution, and the like. The exemplary scheduler also can be configured for ramping of specified values of the input parameters 514 and/or the output parameters 516. The exemplary scheduler also can be configured to set data based on test case number, iteration count, and the like (e.g., set weight_on_wheels flag to true for iterations 1-6, set to false for iteration count 6 through to the end, etc.). The exemplary scheduler further can be configured to set data based on test case number (e.g., set weight_on_wheels flag true for test cases 1-20, set false otherwise, etc.). The exemplary scheduler can be configured to include a preset fixed number of test cases (e.g., FFF or 4095, etc).

In an exemplary embodiment, the driver 508, and the like, can be configured to call the actual source module under test. The exemplary driver 508 can be language specific, for example, based on the module under test. The exemplary driver 508 can be configured as a high level module overseeing parent and children under test. The exemplary driver 508 can be configured to be non-generic and can be based on the object under test. The exemplary driver 508 can be separated by test case numbers, which correspond to actual calls to procedure or function calls to the object under test. Each test case can include corresponding input parameters and output parameters. The input parameters 514 can be derived from the user interface 204 or 504 and the desired outputs can be output to the results file (e.g., called ".results"). The exemplary driver 508 also can be configured to correlate the test case numbers from the scheduler to actual calls to procedure or function calls to the object under test.

In an exemplary embodiment, the test case numbers can be generated, for example, via a code scanner, such as the exemplary code scanner 208, and the like. Advantageously, the exemplary scheduler can be configured to be generic and need not be based on the module under test. The exemplary scheduler can be configured to include one or more interfaces (e.g., Pragma, etc.), for example, including an interface configured for input data parameters, and an interface configured for output data parameters. In an exemplary embodiment, such interfaces can be configured, for example, as C language data modules configured to map the data parameters to the source 510 under test, to the user interface 504, and the like.

In an exemplary embodiment, the shared memory 512 sub-component, for example, can be configured as C based routine configured to map the user interface 504 data parameters to the data parameters that the source 510 under test can utilize. The exemplary routine can be called, for example, via one or more of the above-noted interfaces (e.g., Pragma, etc.) for the case of an Ada source 510 module. The size of the shared memory 512 can be configured to be a fixed, relatively large, size. In an exemplary embodiment, information within the XML data can be configured for determining how much of the shared memory 512 can be utilized. In an exemplary embodiment, the types of data that can be included in the shared memory 512 space, for example, can include scheduler control parameters 518, input parameters 514, output parameters 516, and the like.

FIG. 6 illustrates an exemplary process that can be performed by the executable 502 of the exemplary system 500 of FIG. 5. In FIG. 6, at step 602, user commands are read and utilized by the executable 502 form the shared memory 512. At step, 604, ramped input data can be set. At step 606, the driver 508 can be called. At step 608, output data can be read and further processing of user commands can be performed by returning control to step 602.

FIG. 7 illustrates an exemplary process that can be performed based on the build directory 314 of the exemplary systems 300 and 400 of FIGS. 3 and 4. In FIG. 7, at step 702, user selected stub files can be copied. At step 704, user selected driver files can be copied. At step 706, user selected source under test files can be copied.

FIG. 8 illustrates exemplary user interface processes that can be performed based on the exemplary systems, 200-500 of FIGS. 2-5. In FIG. 8, at step 802, user interface menu functions can be performed. At step 804, language specific functions can be performed, including, at step 806, running of the stub generator 206, at step 808, running of the code scanner 208, at step 810, running of the driver script generator 210 to set up the driver, and at step 812, running of the translator 506 to perform source code translation. At step 814, the control driver 508 can be generated and at step 816, the input parameters 516 can be set and ramped.

FIG. 9 illustrates further exemplary user interface processes that can be performed based on the exemplary systems 200-500 of FIGS. 2-5. In FIG. 9, at step 902, user interface menu functions can be performed. At step 904, language specific functions can be performed, including, at step 906, running of the stub generator 206, at step 908, running of the code scanner 208, and at step 910, running of the driver script generator 210 to set up the driver 508, and the like. At step 916, the input parameters 514 can be set and ramped, a test case can be set up, the code under test can be executed, and the output parameters 516 can be displayed.

FIG. 10 illustrates exemplary code scanner processes that can be performed based on the exemplary systems, 200-500 of FIGS. 2-5. In FIG. 10, at step 1002, mapping of source name, package and procedure/functions can be performed, including, at step 1004, creating a list of executable source names located in stub and source under test directories. At step 1006, for each of the source names, package names can be added. At step 1008, for each of the package names, procedure or function names can be added, completing the process 1002 of mapping of source name, package and procedure/functions.

At step 1010, a tree structure of source name, package and procedure/functions can be generated, including, at step 1012, for each procedure/function of a major package, sub module calls can be made. At step 1014, for each procedure/function of a minor package, sub module calls can be made. At step 1016, processing is repeated until the last procedure/function can be found, completing the process 1010 of generating a tree structure of source name, package and procedure/functions. At step 1018, a resultant file, including a map/tree structure of a complete object under test can be generated. In an exemplary embodiment, the resultant file can be configured, for example, as an ASCII text file, and the like.

FIG. 11 illustrates an exemplary process for performing software testing in the exemplary embodiments described with respect to FIGS. 1-10. In FIG. 11, step 1102 can be used to identify the software to be tested. Step 1104 can be used to create the scanned file of the software to be tested, for example, as an aid in identifying the various input and output variables involved. Step 1106 can be used to determine the different test cases and how the inputs can be manipulated in order to test suitable conditions. Step 1108 can be used to create the employed stubbed files, for example, to eliminate interference from the interfacing software. Using the exemplary test suite, step 1110 can be used create the test script file which can be used to describe the test.

Step 1112 can be used to generate a test driver from the test script file, using the exemplary test suite. Step 1114 can be used to place copies of the test driver, stubbed files, source under test, interfacing software, and the like, in the build directory and to compile and build the test driver executable. Step 1116 can be used to run the test driver executable file to perform the test and create the results file. Step 1118 can be used to analyze the results and determine if problems exist in the source code under test or in the test driver, as can be determined by steps 1120 and 1122. Step 1124 can be used to make changes, as necessary, to the source code or the test script. If the test script is changed, as determined at step 1126, control can transfer back to step 1112 to generate a new test driver and then recompile and build the new test driver. Steps 1116 and 1118 can then be employed to rerun the test driver executable file and once again analyze the results.

The exemplary test suite can include an initialization component, a user interface component, an executable component, a stub generator component, a code scanner component, and the like. In an exemplary embodiment, during initialization, the user interface can be invoked and call a read_setup_file function that can read in a setup text file (e.g., called setup.txt). The setup file can be parsed and global variables included within the main component can be populated. The read_setup_file function can be invoked, via the main component, for example, during the initialization phase of the exemplary test suite. Global variables included within the main component then can be populated.

Once the set up is completed, the main loop can be invoked and can display a Main Menu page of the user interface. In an exemplary embodiment, the user interface 204 can be configured as the main executive that invokes the other various functions and subroutines. For example, the user interface 204 can be configured to run in a loop until the user decides to exit the exemplary test suite, for example, via a Quit button, and the like.

FIG. 12 illustrates an exemplary test setup file 1200. In FIG. 12, the exemplary text file 1200, setup.txt, can be generated automatically or manually. The file can be configured, for example, as an ASCII text file that can include initialization parameters that the exemplary object level software test suite can utilize. In an exemplary embodiment, comment lines can be prefaced with two dashes, e.g., "--". FIGS. 13 and 14 illustrate an exemplary driver script file 1300 and a driver results file 1400. In an exemplary embodiment, the driver results file 1400 can be configured to list expected results, actual results, any errors associated with differences therebetween, and the like.

FIGS. 15-17 illustrate a SW test suite Main Menu page or window 1500, according to various exemplary embodiments. In an exemplary embodiment, the test suite Main Menu 1500 can be configured to allow a user to select a language of unit under test (UUT), FTP files across different platforms, create stubs, create listing of gates, setup test cases and drivers, compile, link and execute drivers, monitor system messages that occur during execution, and the like.

In FIGS. 15-17, the exemplary SW Test Main Menu page 1500 can include, in a SW Test Menu portion 1504, a Select Language button 1506, and an FTP button 1622, as shown in FIG. 16. The FTP button 1622 can be configured to be used to generate an FTP page or window 1800, as shown in FIG. 18.

In FIG. 18, the FTP page 1800 of the FTP utility 418, allows the user to port source software from one platform to another. For example, this feature allows software residing on a UNIX based platform to be ported, copied, and the like, on to the computer executing the software test tool.

In an exemplary embodiment, once a driver is created, the driver can be run on the computer or ported back, via the built in FTP utility 418, to the UNIX platform and run there. In an exemplary embodiment, the FTP utility 418 window 1800 includes an Address drop down menu 1808 and Connect button 1810 of a Connection Information portion 1802 that allows a user to enter or recall the address of a remote machine, platform, and the like.

In an exemplary embodiment, the window 1800 can include a Remote portion 1804 that can be used to list the files located on the remote machine, platform, and the like, in the directory specified in the field 1812 in a window section 1814. The Remote portion 1804 further can include Delete and Make directory (MkDir) buttons 1816 and 1820 for performing the indicated functions.

In an exemplary embodiment, the window 1800 can include a Local portion 1806 that can be used to lists the files in a directory located on the local PC as specified in a directory address field 1822 and as displayed in a window portion 1824. The Local portion 1806 further can include Delete and Make directory (MkDir) buttons 1826 and 1828 for performing the indicated functions. The Connect button 1810 can allow a user to connect to the remote address specified in the address field 1808, a button 1830 can allow the user to copy the highlighted file from the remote machine to the local PC, and a button 1832 allows the user to copy the highlighted file from the local PC to the remote machine, platform, and the like. The FTP window 1800 can further include a Done button 1834 and a Help button 1836 for performing the indicated functions.

In the window 1500, the user can click on the Select Language button 1506, which can then pop up the window 1900, including software language (e.g., Ada, C, FORTRAN, etc.) selection buttons 1904-1908 of the window portion 1902, as shown in FIG. 19. In an exemplary embodiment, once the language is selected, the remaining buttons 1508-1516 and 1622 can be made selectable. As shown in FIGS. 15-17, the window 1500 also can include a System Messages portion 1502, including a Clear Messages button 1520, a Cancel button 1518, a Help button 1626, a logo portion 1624, and a language configuration identification portion 1728. On the main window 1500, the Select Language button 1506 can be made selectable. When the user first selects the button 1506, the Select Language page 1900 can appear.

FIG. 19 illustrates a Select Language page 1900, according to an exemplary embodiment. In FIG. 19, the exemplary window 1990 can include language buttons, for example, such an Ada button 1904, a C button 1906, a FORTRAN, button 1908, and the like. When the user selects one of the buttons 1904-1908, the Main page 1500 can be active again and the exemplary test suite can be configured for testing in the selected language. The window 1900 can further include a Cancel button 1910 for performing the indicated function. Once the source code language can be determined, the Stub Generator button 1508 on the main menu 1500 can be selected. In an exemplary embodiment, when the user selects the button 1508, a stub generator page can be displayed.

FIGS. 20-22 illustrate a Stub Generator page 2000 (e.g., when set to Ada), according to various exemplary embodiments. In FIGS. 20-22, the exemplary Stub Generator page 2000 can include a listing of directories 2004 and 2006 separated by a Single File button 2008, an All files button 1010 of a Create stub portion 2002, a Copy stub & Rename button 2116, a Finished button 2012, a Body Only button 2220 and a Spec & Body Pair button 2222 of a window portion 2218, and the like. The exemplary window 2000 can further include a Cancel button 2014 and a Help button 2224 for performing the indicated functions.

In an exemplary embodiment, the CM directory portion 2004 can be configured for listing and displaying the files in the cm_source directory 106 and which can be selected, for example, via highlighting, and the like. The stub directory portion 2006 can be configured for listing and displaying the stubbed files, for example, as they are generated. After the user selects a source file to be stubbed, the user can then select the Single File button 2008 of the Create Stub window portion 2002. The appropriate stubbed files can then be generated. The user can also generate stub files for all the source files by selecting the All Files button 2010. The stub generator page can be configured to invoke functions, for example, including a list_dir function, and a stub_gen function.

In an exemplary embodiment, selecting the Spec & Body Pair button 2222 creates a spec and a compatible body that contains variables that the user can set in the driver. The values of such variables can be returned to a calling routine at the appropriate time. Advantageously, this allows the stub to return particular values based on a test case number.

In an exemplary embodiment, the stub generator window 2000 can be configured to include the CM Directory window portion 2004 that lists the files under configuration management, the Stub Directory window portion 2006 that lists the stubbed files created by the test tool in the ts_output directory 108, the Body Only button 2220 that allows a user to stub a package body only, the Spec & Body Pair button 2222 that allows the user to stub both spec and body of a package, the Single File button 2008 that allows the user to select one file from CM directory 2004 and which can be stubbed and stored in the ts_output directory 108 and displayed in the Stub Directory window portion 2006, and an All files button 2010 that allows the user to select all files in the CM Directory window portion 2004 for stubbing and storage in the ts_output directory 108 and displaying in the Stub Directory window portion 2006.

In an exemplary embodiment, the list directory functionality, via the list_dir function, can be invoked to generate a file named cm_list.txt or stub_list.txt based on input parameters. The text file can include a directory list of the source files located in the passed in path. The corresponding inputs and outputs can be given, as follows:

TABLE-US-00001 inputs: Path - cm or stub Mode - 1=cm, 2=stub Spec_suffix - .ads Body_suffix - .adb outputs: C:\test_suite\cm_list.txt C:\test_suite\stub_list.txt

In an exemplary embodiment, the stub generator functionality, via the stub generation function (e.g., called stub_gen) can be provided. The stub generator can receive a cm source module as an input and output a stubbed source module. In an exemplary embodiment, calculations and logic included within functions and procedures can be removed. The created stubbed file can be configured to compile error free. The corresponding inputs and outputs can be given, as follows:

TABLE-US-00002 inputs: source_path - source path directory source_name - the name of the source to be stubbed dest_path - destination path directory outputs: source_name_stub.adb

In an exemplary embodiment, when the user selects the code scanner button 1510 on the Main Menu page 1500, a code scanner window can be displayed. FIG. 23 illustrates a Code Scanner Window 2300, according to an exemplary embodiment. In FIG. 23, the window 2300 can include a listing of directories 2302 and 2304 separated by a Create Scan button 2306, a Finished button 2012, and the like. The exemplary window 2300 can further include a Cancel button 2310 and a Help button (not shown) for performing the indicated functions.

In an exemplary embodiment, the CM directory listing window portion 2302 can be configured to display the files of the cm_source directory 106 and which can be selected, for example, via highlighting, and the like. The Scan Directory window portion 2304 can be configured for display the files of the ts_output directory 108 and which can be selected, for example, via highlighting, and the like. The user can selects a source file to scan from the CM directory listing window portion 2302, and select the Create Scan button 2306, which can generate the corresponding scanned file (e.g., text format, and the like), and which can be displayed in the Scan Directory window portion 2304.

In an exemplary embodiment, the Code Scanner window 2300 can be configured to include the CM Directory window portion 2303 that can list files under configuration management, Scan Directory window portion 2304 that can list files created by the exemplary tool suite, the Create Scan button 2306 that can allow the user to select a file from the CM Directory window portion 2303 for scanning. In an exemplary embodiment, a text file including test gates, PDL, source metrics, and the like, can be generated and stored in the ts_output directory 108, which can be displayed in the window portion 2304.

FIG. 24 illustrates an exemplary scan file 2400 that can be displayed in a word-processing application, such as Windows Notepad, and the like, by selecting, for example, by double clicking, and the like, a newly created scan file in the window portion 2304. In an exemplary embodiment, the generated scan file 2400, for example, can include a synopsis of gates included in the scanned source file, corresponding metrics, and the like.

In an exemplary embodiment, when the user selects the Driver Script button 1512 on the Main Menu page 1500, a driver script generator window 2550, for example, as shown in FIGS. 25-32, can be displayed and defaulted for allowing input of parameters, for example, after the user is prompted information via the windows or pages 3300-3500 of FIGS. 33-35. In FIG. 33, the Source Code Version page 3300 allows for entry a version of source code under test, in the field 3304, when the page 3300 is automatically displayed. The exemplary window 3300 can be configured to allow the entered data to be inserted into a test script, via the Insert Comments button 3302, and can include a Cancel button 3306 for performing the indicated function.

In FIG. 34, the System Requirements page 3400 can be configured to allow entry of system requirements, and the like, that code under test is to be tested under in a window portion 3404. The exemplary window 3400 can be configured to be displayed automatically for each different test case. The exemplary window 3400 can be configured to allow the entered data to be inserted into a test script, via the Insert Comments button 3402, and can include a Cancel button 3406 for performing the indicated function.

In FIG. 35, the Test Case Comment page 3500 can be configured to allow entry of comments, for example, for describing the test being performed in the corresponding section of the test script, and the like, in a window portion 3504. The exemplary window 3500 can be configured to be displayed automatically for each different test case. The exemplary window 3500 can be configured to allow the entered data to be inserted into a test script, via the Insert Comments button 3502, and can include a Cancel button 3506 for performing the indicated function.

FIGS. 25-32 illustrate a Driver Script Generator window 2500, according to various exemplary embodiments. In an exemplary embodiment, the window 2500, when initially displayed, can be defaulted with an input button 2552 selected for allowing entry of the input parameters. Then, by selecting an output button 2554, entry of the output parameters can be performed.

In an exemplary embodiment, the window 2500 can include a menu portion 2682 or 2882, for example, including an Options button 2686 or pull-down menu 2886, and the like, for allowing selection of various options for the exemplary tool. An auto sequence option can be provided via the Options button 2686 or menu 2886 to allow sequencing of the various menus and buttons as the data is entered, advantageously, allowing the data to be entered in a correct or appropriate order. The exemplary window 2500 can include a Driver Script Editor screen portion 2510 configured to display script commands, and the like, that are automatically inserted, as the various buttons, menus, fields, and the like, of the exemplary window 2500 are selected.

In an exemplary embodiment, when updating a previously created script, the auto sequencing option can be unselected, and a cursor can be placed in an area where the script is to be edited and the appropriate changes, such as additions, deletions, and the like, can be made to the script. The exemplary window 2500 can include a Reference Packages section 2504 configured to allow selection of a directory for viewing. The user can select, for example, by double clicking, and the like, on a source module shown in a window portion 2528 and which can be displayed in a notepad window. Such a feature, advantageously, allows the user to view, copy, paste, and the like, variables, data, parameters, and the like, into a script page, driver page, and the like, in a seamless manner.

The Driver Script Generator window 2500 can include file selection menus, buttons, fields, and the like, for example, including the menus sections 2682 and 2882, including a File button 2684 or menu 2884 configured to activate pull-down menu 2788, the Options button 2686 or menu 2886 configured to activate pull-down menu 2790, and the like. The exemplary window 2500 further can include a Help menu 2894 or button (not shown), a Cancel button 2580, and the like, configured to perform the indicated function.

In an exemplary embodiment, the Reference Packages section 2504 of the window 2500 can include a CM button 2522 configured to list the files in the CM source directory 106, a Stub button 2524 configured to list the stub files in the TS output directory 108, a Scan button 2526 configured to list the scan files in the TS output directory 108, and the like, wherein selecting of a file in the window portion 2528, for example, by double clicking, and the like, can allow the corresponding file to be opened in a specified editor, for example, a word processing application, such as Windows Notepad, and the like.

In an exemplary embodiment, a Source Selection section 2502 of the exemplary window 2500, when configured for Ada, can include a Spec files pull-down menu 2516 configured to allow, for example, selection of an Ada Spec file of the UUT, and the like, a Packages pull-down menu 2518 configured to allow, for example, selection of an Ada Package name of the UUT, a Procedures/Functions pull-down menu 2520 configured to allow, for example, selection of an Ada procedure/function of the UUT. In an exemplary embodiment, if a corresponding script of the selected procedure or function exists, the script can be loaded into the Driver Script Editor window 2510.

In an exemplary embodiment, the Menu selection portions of 2682 and 2882 of the exemplary window 2500 can include the File button 2684 or menu 2884 configured to activate pull-down selections menu 2788, for example, that allows saving of the current script file, creation of a driver from the generated script file, and the like. The Options button 2686 or menu 2886 can be configured to activate pull-down selections menu 2790, for example, that can include selections for turning on and off the Auto Sequence mode that can be used to activate appropriate menus, buttons, and the like, at a correct time or in correct order, selections for specifying when to be prompted to enter the system requirements text, the test comments text, and the like.

In an exemplary embodiment, the exemplary window 2500 can include a Test Case Setup portion 2506, including a Test Case portion 2530 having a field configured for entry of a desired test case number in a suitable range (e.g., 1 to 10, 100, 1000, etc.), a Renumber button 2532 configured to allowing automatic renumbering of the test cases, and the like. The exemplary Test Case Setup portion 2506 can further include a Start Test Block button 2563 configured to add a delimiter to the Driver Script Editor window 2510, for example, stating the start of a test case, an End Test Block button 2538 configured to add a delimiter to the Driver Script Editor window 2510, for example, stating the end of the test case, an End Test File button 2550 configured to add a delimiter to the Driver Script Editor window 2510, for example, stating the end of the test file, a Total Iterations field 2534 configured to allow specifying of a number of loops in the test case in a suitable range (e.g., 1 to 10, 100, 1000, etc.), a Current Iteration field 2542 configured for displaying the current iteration of the test case being created, and the like.

In an exemplary embodiment, the exemplary Test Case Setup portion 2506 can further include a Start Iteration (Iter) Block button 2540 configured to add a delimiter to the Driver Script Editor window 2510, for example, stating the start of current iteration, an End Iter Block button 2544 configured to add a delimiter to the Driver Script Editor window 2510, for example, stating the end of current iteration, an input Code Block button 2546 configured to add a delimiter to the Driver Script Editor window 2510 to declare inputs and which displays a Variable Selection portion 2508 configured for input mode, an output Code Block button 2548 configured to add a delimiter to the Driver Script Editor window 2510 to declare outputs and which displays the Variable Selection portion 2508 configured for output mode, and the like.

The exemplary window 2500, in input mode, for example, with the input button 2552 selected, allows for entry of information, for example, including the source to be tested, the number of cycles the selected tests are to be performed (e.g., as an integer that can be passed to the control driver) and which parameters can be used by a scheduler, and the like. The exemplary window 2500 further allows the user to select fixed input, for example, via highlighting, selecting, and the like, and an Insert Fixed Value button 2560, and to set a corresponding value in a Value field 2558. The user can also be allowed to select ramp input, for example, via highlight and select, and an Insert Ramp Values button 2564.

In an exemplary embodiment, when the Insert Ramp Values button 2564 is selected, the user can be allowed to input a starting value and an ending value for that particular input in corresponding fields 2566 and 2568. Such values can be passed to the control driver and the scheduler can utilize the corresponding parameters. The user can select, for example, via highlighting, and the like, the test numbers that are to be run, and which can be inputted to the control driver and the corresponding parameters can be utilized by the scheduler.

In an exemplary embodiment, the Variable Selection portion 2508 of the exemplary window 2500 can include the input button 2552, which can be selected, for examp