VxATP - Vortex Automated Test Platform

A module containing tools to write tests in Python with Vortex is made available in the binary directory.

This module is called vxatp3.

Structure of vxatp3 folder

vxatp3 has the following directory structure:

vortex \bin\vxatp3 : the Python package
vortex \bin\vxatp3_*.bat : batch helpers to run tests from the command line
vortex \resources\vxatp\ : resources used by vxatp3, e.g. setup documents

Setting the environment to run vxatp3

The batch command vxatp3_set_env.bat details how to properly set the environment when starting from the binary installation directory.

vxatp_set_env

set VXTK=%~dp0..\
set PATH=%VXTK%\bin;%VXTK%\plugins;%PATH%
set PYTHONPATH=%VXTK%\bin;%PYTHONPATH%

Any IDE or Python interpreter would have to set up or inherit the same environment.

How to script a test

The vxatp3 test scripts are similar to standard scripts using the python package unittest: https://docs.python.org/3.8/library/unittest.html?highlight=unittest#basic-example

From the configuration given, a VxApplication can be created to perform operations on assets.

A helper is provided in VxATPConfig that return a setup application ready to be tested : VxATPConfig.createApplication(self, prefix_name, config_name).

In addition, VxATPConfig.createApplication sets to the test case a configuration member that contains useful information in the context of vxatp3.

The configuration can be accessed with self._config .

The configuration contains the following:

self._config.app_config_file : the setup file. It can be overridden by the test itself to add or remove specific vortex modules.
self._config.app_log_prefix : the prefix for the log file. It can be overridden by the test to specify the name of the test case in it.
self._config.output_directory : the output directory desired by the test suite. It should be used by the test to output data in a safe place (SVN and other versioned locations must be avoided).

Vortex Python module must also be imported in order to use the Vortex Python binding of the SDK.

Example from unittest documentation adapted to vxatp3

import os
import unittest
from Vortex import *
from vxatp3 import * 

class TestMethods(unittest.TestCase):
  directory = os.path.dirname(os.path.realpath(__file__))

  def test_upper(self):
     self.assertEqual('foo'.upper(), 'FOO')

  def test_isupper(self):
     self.assertTrue('FOO'.isupper())
     self.assertFalse('Foo'.isupper())

  def test_loading(self):
     # create a VxApplication while applying a setup file contained in the test directory 
     application = VxATPConfig.createApplication(self, setup='%s/config.vxc' % self.directory)
     # load content and verify it has been correctly loaded
     file = '%s/MyAsset.vxscene' % self.directory
     object = application.getSimulationFileManager().loadObject(file)
     self.assertIsNotNone(object)

if __name__ == '__main__':
    vxatp_run_thisfile('my_vxatp_output_directory')

Helpers for default setups

VxATPConfig provides optional helpers to get default predefined setup of the application :

VxATPConfig.getAppConfigWithoutGraphics() return absolute path to default config not containing graphics module
VxATPConfig.getAppConfigWithGraphics() return absolute path to default config containing graphics module

Helper to change application mode

VxATPUtils.requestApplicationModeChangeAndWait(self.application, Vortex.kModeEditing) Sets the new mode and wait for the change to be effective. Use to switch between kModeEditing and kModeSimulating.

Helper for tests parametrization

vxatp3 provides a decorator that allow tests to be parametrized.

The following gives an example of its usage.

Example using VxATPUtils.parametrize

import os
import unittest
from Vortex import *
from vxatp3 import * 

class TestMutipleDiff(unittest.TestCase):
  directory = os.path.dirname(os.path.realpath(__file__))
  
  @VxATPUtils.parametrized([['diff_one', 3, 5, -2], 
                            ['diff_two', 13, 8, 6], 
                            ['diff_three', 13, 8, 5]])
  def test_diff(self, test_name, a, b, expected):
     self.assertTrue(a-b == expected, '%d-%d does not match expected value %d.' % (a,b,expected))

if __name__ == '__main__':
    vxatp_run_thisfile('.\\results\\')

The following is the output generated by the test.

example of parametrized test output

-------------------------------------------------------------------------
Running Test : test_difference
-------------------------------------------------------------------------
======================================================================
FAIL [0.000s]: test_diff_diff_two (test_difference.TestMutipleDiff)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\CM Labs\Vortex Studio 2020b\bin\vxatp3\vxatp_utils.py", line 108, in parametrized_func_wrapper
    return fn(*a, **kwargs)
  File "S:\scripts\test_difference.py", line 13, in test_diff
    self.assertTrue(a-b == expected, '%d-%d does not match expected value %d.' % (a,b,expected))
AssertionError: 13-8 does not match expected value 6.
test_difference.TestMutipleDiff - test_diff_diff_one:           Pass
test_difference.TestMutipleDiff - test_diff_diff_three:         Pass
test_difference.TestMutipleDiff - test_diff_diff_two:           Fail
Ran 3 test cases in 0.000s
FAILED (failures=1, errors=0)

Helper for tests tagging

vxatp3 provides a decorator that allow tests to be tagged, you can then select to run only the tagged tests.

The following gives an example of its usage.

tags are case insensitive
multiple tags can be put on a test
vxatp_run_thisfile('.\\results\\, tag='one') will run only the tests with the designated tag in this file
vxatp_run_thisdirectory('.\\results\\, tag='one') will run only the tests with the designated tag in this directory

Example using VxATPUtils.tag

import os
import unittest
from Vortex import *
from vxatp3 import * 

class TestTagging(unittest.TestCase):
    directory = os.path.dirname(os.path.realpath(__file__))

    @VxATPUtils.tag("one")
    def test_one(self):
        self.assertTrue(True)

    @VxATPUtils.tag("ONE")
    @VxATPUtils.tag("two")
    def test_one_two(self):
        self.assertTrue(True)
            
    @VxATPUtils.tag("three")
    def test_three(self):
        self.assertTrue(True)
 
    def test_none(self):
        self.assertTrue(False)

if __name__ == '__main__':
    vxatp_run_thisfile('.\\results\\', tag='one')

The following is the output generated by the test. In this example, only the tests tagged “one” will be run.

example of parametrized test output

Found 2 test(s) with tag "one"

-------------------------------------------------------------------------
Running Test : test_vxatp_tags
-------------------------------------------------------------------------

test_vxatp_tags.TestTagging - test_one:         Pass
test_vxatp_tags.TestTagging - test_one_two:             Pass

Ran 2 test cases in 0.000s

SUCCESS

How to call a vxatp3 test from the command line

There are two ways a vxatp3 test can be called from the command line :

Calling a vxatp3 test like any unittest.TestCase

See unittest#command-line-interface (e.g. python -m unittest test_my_test.py).

A convenience batch file is also provided to set up all Vortex required environment variables.

running one vxatp3 test script

vxatp3_run_onetest.bat my_test.py

Calling vxatp3 test using vxatp_launcher

vxatp_launcher can be used to run all tests in a given directory or recursively or interactively

running all vxatp3 test scripts in a given directory

vxatp_launcher.py -p my_test_directory -o my_output_directory_for_results_and_logs

running all vxatp3 test scripts recursively

vxatp_launcher.py -p my_test_directory -o my_output_directory_for_results_and_logs -r

running one vxatp3 test script interactively

vxatp_launcher.py -p my_test_directory -o my_output_directory_for_results_and_logs -i -r

running all vxatp3 test scripts recursively with forcing graphics

vxatp_launcher.py -p my_test_directory -o my_output_directory_for_results_and_logs -g -r

Naming Convention

To be found by discovery, test scripts must follow the default pattern of the unittest package : "test*.py".

"test_my_test_name.py" is the recommended syntax.

In addition to running all test cases, the runner outputs xml formatted logs that can be used by any JUnit compatible parser.

example of console output

-------------------------------------------------------------------------
Running Test : content
-------------------------------------------------------------------------
test_content_rover.TestContentRover - test_validate_content_rover_assembly:             Pass
test_content_rover.TestContentRover - test_validate_content_rover_mechanism:            Pass
test_content_rover.TestContentRover - test_validate_content_rover_scene:                Pass
Ran 3 test cases in 1.285s
SUCCESS

an example of xml output

<testsuite duration="1.285" errors="0" failures="0" name="content" tests="3">
  <testcase classname="test_content_rover.TestContentRover" duration="0.400" name="test_validate_content_rover_assembly" start_time="2016-01-07 14:30:25"/>
  <testcase classname="test_content_rover.TestContentRover" duration="0.376" name="test_validate_content_rover_mechanism" start_time="2016-01-07 14:30:26"/>
  <testcase classname="test_content_rover.TestContentRover" duration="0.509" name="test_validate_content_rover_scene" start_time="2016-01-07 14:30:26"/>
</testsuite>

More batch helpers

As a convenience, the following batch files are provided in the bin directory

set environment variables to use vxatp_launcher.py

vxatp3_set_env.bat

running one vxatp3 test script

vxatp3_run_onetest.bat my_test.py

running all vxatp3 test scripts recursively from the test_directory

vxatp3_run_tests.bat test_directory output_directory

running one vxatp3 test script interactively discovered recursively from the test_directory

vxatp3_run_tests_interactive.bat test_directory output_directory

running one vxatp3 test script with graphics discovered recursively from the test_directory

vxatp3_run_tests_interactive.bat test_directory output_directory

How to use vxatp3 in Visual Studio 2015

Visual Studio has Python tools available as an add-on.

It includes among others functionalities an editor and a debugger.

To use it with vxatp3, you need first to configure your Python environment to be compatible with Vortex.

Next, in your Python project properties the 'Working Directory' must be the binary directory of your Vortex installation.

To be able to run and debug, the 'Search Paths' of the 'Debug' property tab must contain the binary directory of your Vortex installation, the vxatp3 path of your binary installation, and optionally the path containing your test scripts.

The settings should be enough to run and debug your vxatp3 based verification scripts in Visual Studio.