VxATP - Vortex Automated Test Platform
A module containing tools to write tests in Python with Vortex is made available in the binary directory.
This module is called vxatp3.
Structure of vxatp3 folder
vxatp3 has the following directory structure:
vortex \bin\vxatp3 : the Python package
vortex \bin\vxatp3_*.bat : batch helpers to run tests from the command line
vortex \resources\vxatp\ : resources used by vxatp3, e.g. setup documents
Setting the environment to run vxatp3
The batch command vxatp3_set_env.bat details how to properly set the environment when starting from the binary installation directory.
vxatp_set_env
set VXTK=%~dp0..\
set PATH=%VXTK%\bin;%VXTK%\plugins;%PATH%
set PYTHONPATH=%VXTK%\bin;%PYTHONPATH%
Any IDE or Python interpreter would have to set up or inherit the same environment.
How to script a test
The vxatp3 test scripts are similar to standard scripts using the python package unittest: unittest — Unit testing framework — Python 3.8.20 documentation
From the configuration given, a VxApplication
can be created to perform operations on assets.
A helper is provided in VxATPConfig
that return a setup application ready to be tested : VxATPConfig.createApplication(self, prefix_name, config_name).
In addition, VxATPConfig.createApplication
sets to the test case a configuration member that contains useful information in the context of vxatp3.
The configuration can be accessed with self._config
.
The configuration contains the following:
self._config.app_config_file
: the setup file. It can be overridden by the test itself to add or remove specific vortex modules.self._config.app_log_prefix
: the prefix for the log file. It can be overridden by the test to specify the name of the test case in it.self._config.output_directory
: the output directory desired by the test suite. It should be used by the test to output data in a safe place (SVN and other versioned locations must be avoided).
Vortex Python module must also be imported in order to use the Vortex Python binding of the SDK.
Example from unittest documentation adapted to vxatp3
import os
import unittest
from Vortex import *
from vxatp3 import *
class TestMethods(unittest.TestCase):
directory = os.path.dirname(os.path.realpath(__file__))
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
def test_isupper(self):
self.assertTrue('FOO'.isupper())
self.assertFalse('Foo'.isupper())
def test_loading(self):
# create a VxApplication while applying a setup file contained in the test directory
application = VxATPConfig.createApplication(self, setup='%s/config.vxc' % self.directory)
# load content and verify it has been correctly loaded
file = '%s/MyAsset.vxscene' % self.directory
object = application.getSimulationFileManager().loadObject(file)
self.assertIsNotNone(object)
if __name__ == '__main__':
vxatp_run_thisfile('my_vxatp_output_directory')
Helpers for default setups
VxATPConfig
provides optional helpers to get default predefined setup of the application :
VxATPConfig.getAppConfigWithoutGraphics()
return absolute path to default config not containing graphics moduleVxATPConfig.getAppConfigWithGraphics()
return absolute path to default config containing graphics module
Helper to change application mode
VxATPUtils.requestApplicationModeChangeAndWait(self.application, Vortex.kModeEditing)
Sets the new mode and wait for the change to be effective. Use to switch between kModeEditing and kModeSimulating.
Helper for tests parametrization
vxatp3 provides a decorator that allow tests to be parametrized.
The following gives an example of its usage.
Example using VxATPUtils.parametrize
import os
import unittest
from Vortex import *
from vxatp3 import *
class TestMutipleDiff(unittest.TestCase):
directory = os.path.dirname(os.path.realpath(__file__))
@VxATPUtils.parametrized([['diff_one', 3, 5, -2],
['diff_two', 13, 8, 6],
['diff_three', 13, 8, 5]])
def test_diff(self, test_name, a, b, expected):
self.assertTrue(a-b == expected, '%d-%d does not match expected value %d.' % (a,b,expected))
if __name__ == '__main__':
vxatp_run_thisfile('.\\results\\')
The following is the output generated by the test.
example of parametrized test output
-------------------------------------------------------------------------
Running Test : test_difference
-------------------------------------------------------------------------
======================================================================
FAIL [0.000s]: test_diff_diff_two (test_difference.TestMutipleDiff)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\CM Labs\Vortex Studio 2020b\bin\vxatp3\vxatp_utils.py", line 108, in parametrized_func_wrapper
return fn(*a, **kwargs)
File "S:\scripts\test_difference.py", line 13, in test_diff
self.assertTrue(a-b == expected, '%d-%d does not match expected value %d.' % (a,b,expected))
AssertionError: 13-8 does not match expected value 6.
test_difference.TestMutipleDiff - test_diff_diff_one: Pass
test_difference.TestMutipleDiff - test_diff_diff_three: Pass
test_difference.TestMutipleDiff - test_diff_diff_two: Fail
Ran 3 test cases in 0.000s
FAILED (failures=1, errors=0)
Helper for tests tagging
vxatp3 provides a decorator that allow tests to be tagged, you can then select to run only the tagged tests.
The following gives an example of its usage.
tags are case insensitive
multiple tags can be put on a test
vxatp_run_thisfile('.\\results\\, tag='one') will run only the tests with the designated tag in this file
vxatp_run_thisdirectory('.\\results\\, tag='one') will run only the tests with the designated tag in this directory
Example using VxATPUtils.tag
import os
import unittest
from Vortex import *
from vxatp3 import *
class TestTagging(unittest.TestCase):
directory = os.path.dirname(os.path.realpath(__file__))
@VxATPUtils.tag("one")
def test_one(self):
self.assertTrue(True)
@VxATPUtils.tag("ONE")
@VxATPUtils.tag("two")
def test_one_two(self):
self.assertTrue(True)
@VxATPUtils.tag("three")
def test_three(self):
self.assertTrue(True)
def test_none(self):
self.assertTrue(False)
if __name__ == '__main__':
vxatp_run_thisfile('.\\results\\', tag='one')
The following is the output generated by the test. In this example, only the tests tagged “one” will be run.
example of parametrized test output
Found 2 test(s) with tag "one"
-------------------------------------------------------------------------
Running Test : test_vxatp_tags
-------------------------------------------------------------------------
test_vxatp_tags.TestTagging - test_one: Pass
test_vxatp_tags.TestTagging - test_one_two: Pass
Ran 2 test cases in 0.000s
SUCCESS
How to call a vxatp3 test from the command line
There are two ways a vxatp3 test can be called from the command line :
Calling a vxatp3 test like any unittest.TestCase
See unittest#command-line-interface (e.g. python -m unittest test_my_test.py
).
A convenience batch file is also provided to set up all Vortex required environment variables.
running one vxatp3 test script
vxatp3_run_onetest.bat my_test.py
Calling vxatp3 test using vxatp_launcher
vxatp_launcher can be used to run all tests in a given directory or recursively or interactively
running all vxatp3 test scripts in a given directory
vxatp_launcher.py -p my_test_directory -o my_output_directory_for_results_and_logs
running all vxatp3 test scripts recursively
vxatp_launcher.py -p my_test_directory -o my_output_directory_for_results_and_logs -r
running one vxatp3 test script interactively
vxatp_launcher.py -p my_test_directory -o my_output_directory_for_results_and_logs -i -r
running all vxatp3 test scripts recursively with forcing graphics
vxatp_launcher.py -p my_test_directory -o my_output_directory_for_results_and_logs -g -r
Naming Convention
To be found by discovery, test scripts must follow the default pattern of the unittest package : "test*.py".
"test_my_test_name.py" is the recommended syntax.
In addition to running all test cases, the runner outputs xml formatted logs that can be used by any JUnit compatible parser.
example of console output
-------------------------------------------------------------------------
Running Test : content
-------------------------------------------------------------------------
test_content_rover.TestContentRover - test_validate_content_rover_assembly: Pass
test_content_rover.TestContentRover - test_validate_content_rover_mechanism: Pass
test_content_rover.TestContentRover - test_validate_content_rover_scene: Pass
Ran 3 test cases in 1.285s
SUCCESS
an example of xml output
<testsuite duration="1.285" errors="0" failures="0" name="content" tests="3">
<testcase classname="test_content_rover.TestContentRover" duration="0.400" name="test_validate_content_rover_assembly" start_time="2016-01-07 14:30:25"/>
<testcase classname="test_content_rover.TestContentRover" duration="0.376" name="test_validate_content_rover_mechanism" start_time="2016-01-07 14:30:26"/>
<testcase classname="test_content_rover.TestContentRover" duration="0.509" name="test_validate_content_rover_scene" start_time="2016-01-07 14:30:26"/>
</testsuite>
More batch helpers
As a convenience, the following batch files are provided in the bin directory
set environment variables to use vxatp_launcher.py
vxatp3_set_env.bat
running one vxatp3 test script
vxatp3_run_onetest.bat my_test.py
running all vxatp3 test scripts recursively from the test_directory
vxatp3_run_tests.bat test_directory output_directory
running one vxatp3 test script interactively discovered recursively from the test_directory
vxatp3_run_tests_interactive.bat test_directory output_directory
running one vxatp3 test script with graphics discovered recursively from the test_directory
vxatp3_run_tests_interactive.bat test_directory output_directory
How to use vxatp3 in Visual Studio 2015
Visual Studio has Python tools available as an add-on.
It includes among others functionalities an editor and a debugger.
To use it with vxatp3, you need first to configure your Python environment to be compatible with Vortex.
Next, in your Python project properties the 'Working Directory' must be the binary directory of your Vortex installation.
To be able to run and debug, the 'Search Paths' of the 'Debug' property tab must contain the binary directory of your Vortex installation, the vxatp3 path of your binary installation, and optionally the path containing your test scripts.
The settings should be enough to run and debug your vxatp3 based verification scripts in Visual Studio.