TPTP provides several automatable services which can be invoked from outside the context of the eclipse workbench. The services include test execution, test result interrogation and test result publication.
This help topic explains how to use the built-in TPTP automatable services to do the following:
The test execution service is provided for the programmatic launch, execution and generation of test results. The test execution service wraps existing test execution harness functionality and obsoletes the current standalone test execution harness methods. Using the test execution service, tests can be launched using a headless Eclipse from ant scripts, shell scripts and arbitrary programs. Multiple tests can be run at the same time given enough memory on the executing machine (each service execution currently requires a separate Eclipse host instance). Even if tests are running on remote machines using TPTP remote test execution features and deployment, the service consumer's machine is the machine that runs the Eclipse host instances.
Refer to the reference material on the supported test execution service properties -- the same properties are valid across client automation adapters. For ant scripts, a set of simple ant tasks are provided to set service properties and execute the service. For shell scripts, the properties are specified in the expected command line paradigm (using dashes for properties followed by space and then a value for instance). For Java programs, the automation client jars will be referenced and used for setting properties and controlling the service execution. Some client adapters might have additional options that are required for that particular client environment (or offered as extra features from that adapter).
The TPTP automatable service components are packaged within the org.eclipse.hyades.execution plug-in represented with three jar files, some XML in the plugin.xml and some entries in the MANIFEST.MF file.
tptp-automation-client.jar (the
lightweight bit of code needed to establish the client side of the service
bus, launch Eclipse if needed and execute services headlessly, this jar will
be referenced when running tests from the command line and shell scripts as
well as when running tests from arbitrary Java code).tptp-automation-server.jar (the TPTP
automation server, loaded on-demand as services are interrogated and executed
using the framework, implements the core Eclipse applications extension point,
used as the broker and service locator)ant-tptp.jar (this jar has the same
contents as the tptp-automation-client.jar but is renamed to be
consistent with other ant task libraries)Ensure ant is properly installed if tests are to be run outside of a running Eclipse instance -- refer to the ant documentation to set this up properly. In addition to the standard ant installation, also copy the ant-tptp.jar into the ant lib directory alongside the other tasks. If running inside of Eclipse, ant is already installed although be sure to include the ant-tptp.jar in the additional tasks tab of the ant launch configuration, or in the Global Entries section of the Ant Classpath preference page. The embedded sample ant script below is an example of an ant script named "test.xml" that executes a test named testA within the unit-tests folder of the test project. The test project is located in the specified workspace of "d:\development\workspace" and the Eclipse host instance is specified to be "d:\development\eclipse". Refer to the reference list of supported properties for other valid combinations of properties.
The example below uses the TPTP test task although it is also possible to execute a test using the TPTP automation task. The more general automation task is an ant task for executing arbitrary TPTP automatable services which contrasts the more specific TPTP test task. The test task is a type-safe task specifically created for launching the test execution service. More ant script examples are included in the org.eclipse.hyades.execution plug-in's src-automation-client source folder's org.eclipse.hyades.automation.test package.
<project name="TPTP Ant Tasks Test Script" default="test" xmlns:tptp="antlib:org.eclipse.hyades.automation.client.adapters.ant"> <property name="eclipse.home" value="D:\development\eclipse"/> <description> This ant script tests the TPTP automation ant task client adapters and also can be used to test the underlying execution of TPTP automatable services via the TPTP automation server </description> <!-- Execute TPTP test suites --> <target name="test" description="Executes TPTP Test Suites using default TPTP Ant Tasks"> <!-- Execute a TPTP test suite using the test task and providing the workspace --> <tptp:test workspace="D:\development\workspace" project="tests" suite="unit-tests/testA.testsuite" synchronicity="synchronous"/> </target> </project>
Ensure that you have properly configured ant to run TPTP automatable services
(refer to the above section on running tests from ant.) The example below shows how
to the use the tptp:execution task to execute multiple tests, and how to use the
tptp:interrogation task to interrogate the results of those test executions.
Both of these ant tasks accept multiple ant filesets and/or
filelists, allowing you to use wildcards or to directly
specify multiple files as inputs to the tasks. In this example, we execute
all of the test suites in the TPTPJUnitTests project, and the set of
all resulting .execution files is returned to us as a filelist
in the ant variable that we supply (in this case "tptp.test.results".) We then
pass that filelist as input to the tptp:interrogation service,
which will interrogate each of those execution files to determine an overall
verdict and will assign it to the ant variable that we supply (in this case
"tptp.test.verdict".)
<project name="TPTP Automatable Services Tutorial"
default="main" xmlns:tptp="antlib:org.eclipse.hyades.automation.client.adapters.ant"> <!-- Define common properties for all automatable services --> <property name="tptp.test.workspace" value="C:\eclipsecon\eclipse\workspace"/> <property name="tptp.test.project" value="TPTPJUnitTests" /> <property name="tptp.test.connection" value="tptp:rac://localhost:10002/default"/> <property name="tptp.automation.eclipse" value="C:\eclipseconwb_birt\eclipse" />
<!-- Define script-local properties, that vary by installation --> <property name="project.dir" value="${tptp.test.workspace}\${tptp.test.project}"/>
<target name="main" depends="test-execution, test-results-interrogation"/>
<!-- Execute test suites using default results name --> <target name="test-execution"> <echo message="Executing test suite..." /> <tptp:execution resultsrefid="tptp.test.results"> <fileset dir="${project.dir}"> <include name="**/*.testsuite"/> </fileset> </tptp:execution> </target>
<!-- Interrogate test suite results for verdict --> <target name="test-results-interrogation"> <echo message="Interrogating test suite results..." /> <condition property="tptp.test.success"> <tptp:interrogation verdictproperty="tptp.test.verdict"> <filelist refid="tptp.test.results"/> </tptp:interrogation> </condition> <echo message="The overall test result verdict is: '${tptp.test.verdict}'" /> </target>
</project>
Ensure that you have properly configured ant to run TPTP automatable services
(refer to the above section on running tests from ant.) The example below shows how
to the use the tptp:execution task to execute multiple tests, and how to use the
tptp:publication task to generate BIRT based reports from the results of those
test executions. The publication task also accepts multiple ant
filesets and/or filelists, allowing you to use
wildcards or to directly specify multiple files as inputs. In this example,
we execute all of the test suites in the TPTPJUnitTests project, and the set of
all resulting .execution files is returned to us as a filelist
in the ant variable that we supply (in this case "tptp.test.results".) We then
pass that filelist as input to the tptp:publication service,
which will generate a TPTP execution report using the default report template
(TestExecution.rptdesign, found in the org.eclipse.tptp.test.report.birt plugin),
placing it in the specified location (in this case c:\temp\report.html.)
<project name="TPTP Automatable Services Tutorial"
default="main" xmlns:tptp="antlib:org.eclipse.hyades.automation.client.adapters.ant"> <!-- Define common properties for all automatable services --> <property name="tptp.test.workspace" value="C:\eclipsecon\eclipse\workspace"/> <property name="tptp.test.project" value="TPTPJUnitTests" /> <property name="tptp.test.connection" value="tptp:rac://localhost:10002/default"/> <property name="tptp.automation.eclipse" value="C:\eclipseconwb_birt\eclipse" />
<!-- Define script-local properties, that vary by installation --> <property name="project.dir" value="${tptp.test.workspace}\${tptp.test.project}"/> <property name="report.publication.location" location="c:/temp" />
<target name="main" depends="test-execution, test-results-publication"/>
<!-- Execute test suites using default results name --> <target name="test-execution"> <echo message="Executing test suite..." /> <tptp:execution resultsrefid="tptp.test.results"> <fileset dir="${project.dir}"> <include name="**/*.testsuite"/> </fileset> </tptp:execution> </target>
<!-- Publish test suite results report -->
<target name="test-results-publication"> <echo message="Publishing test suite results report..." /> <tptp:publication report="${report.publication.location}/report.html"> <filelist refid="tptp.test.results"/> </tptp:publication> </target>
</project>
You may customize the behavior of the tptp:publication task by specifying a different BIRT report template, for instance TabularReport.rptdesign found in the templates directory of plugin org.eclipse.tptp.test.report.birt. You may also customize the behavior by using the publish-testsuites publication service and providing test suites as the input to the publication service instead of test execution results files. If you use the publish-testsuites publication service and pass test suite files as input to the publication service, you must also specify a start and end date for the report window (which will be used to query the workspace to find all execution results within that time window.) The example below shows how to run a report on all of the test suites in the Test1 project, selecting every execution result from the past 60 days, and generating the report using the Tabular report design.
<project name="TPTP Automatable Services Tutorial"
default="main" xmlns:tptp="antlib:org.eclipse.hyades.automation.client.adapters.ant"> <!-- Define common properties for all automatable services --> <property name="tptp.test.workspace" value="C:\eclipsecon\eclipse\workspace"/> <property name="tptp.test.project" value="TPTPJUnitTests" /> <property name="tptp.test.connection" value="tptp:rac://localhost:10002/default"/> <property name="tptp.automation.eclipse" value="C:\eclipseconwb_birt\eclipse" />
<!-- Define script-local properties, that vary by installation --> <property name="project.dir" value="${tptp.test.workspace}\${tptp.test.project}"/> <property name="report.publication.location" location="c:/temp" /> <property name="tptp.publication.service" value="org.eclipse.tptp.test.report.birt.publish-testsuites"/>
<!-- Run the report --> <target name="main" depends="test-results-publication"/>
<target name="set-date-range"> <!-- Use standard ant tstamp mechanisms to specify the desired date range --> <!-- This example uses a range between today and 60 days days prior --> <tstamp> <format property="startDateTime" pattern="MM/dd/yyyy hh:mm aa" offset="-60" unit="day"/> </tstamp> <tstamp> <format property="endDateTime" pattern="MM/dd/yyyy hh:mm aa"/> </tstamp> </target>
<!-- Publish test suite results report --> <target name="test-results-publication" depends="set-date-range"> <echo message="Publishing test suite results report..." /> <tptp:publication report="${report.dir}/report.html" reportTemplateLocation="C:\TPTP_TestPass\ReportBuild\eclipse\plugins\org.eclipse.tptp.test.report.birt_4.2.0.v200605180959\templates\TabularReport.rptdesign" startDateTime="${startDateTime}" endDateTime="${endDateTime}" > <fileset dir="${project.dir}"> <include name="**/*.testsuite" /> </fileset> </tptp:publication> </target>
</project>
Tests can be executed directly from the command line and shell scripts with the test properties taking the form of command line arguments. The TPTP automation client adapters provide a .bat and .cmd file for use within Windows and a .sh file provided for use within the Linux operating system. It is possible to specify long sets of arguments and/or frequently repeated arguments in a config file to minimize the amount of typing necessary to run the test execution service from the command line. The shell automation client adapter manifests as a text file (for the batch file or script) and a complementary Java component -- both must exist to execute tests from the command line. All the TPTP provided automation client adapters can be found in the tptp-automation-client.jar although the ant-tptp.jar replicates some of the adapter code in the form of an ant jar (this was done in order to keep the TPTP task jar named consistently with the other available ant tasks).
The test execution service's properties are set by prefixing the property names with a hyphen making sure to also specify the Eclipse home or root directory.
Tests can be launched from any Java application and in fact, the other client automation adapters such as the ant and shell adapters, use the Java adapter internally to hook into the service bus. The only requirement for executing tests from some Java code is to make sure the tptp-automation-client.jar is properly referenced (in Eclipse this translates to having the library in the Java project's build path) and of course all the typical TPTP requirements apply as well, such as the agent controller must be running.
The Java automation client adapter provides a very simple API for executing tests with all specifiable properties being contained in a standard Java properties object. An example follows for executing the same test as introduced in the previous ant script example. Note that the service identifier passed in to the execute method is the service identifier for the TPTP test execution service (this service supports all the base TPTP test types) -- it is also possible to execute other automatable services, if available, but varying the identifier and properties configured in the code below.
// Create the Java adapter associated with the specified Eclipse home
AutomationClientAdapter automation = new AutomationClientAdapter
("d:\\development\\eclipse");
// Create and configure a properties object
Properties properties = new Properties();
properties.setProperty("workspace", "D:\\development\\workspace");
properties.setProperty("project", "tests");
properties.setProperty("suite", "unit-tests/testA.testsuite");
// Execute the service named below using the configured properties
automation.execute("org.eclipse.hyades.test.tools.core.execute", properties);
Related concepts
Overview of the automatable services framework
Related tasks
Launching tests from scripts and applications
Related references
Supported test execution service properties
(C) Copyright IBM Corporation 2005,2006. All Rights Reserved