Performance Analysis

Owned by Marc Lievin
Mar 14, 2024
6 min read
The Performance Analysis guide provides step-by-step procedures for finding and resolving performance bottlenecks.

Evaluating Performance in Simulation

In the best case scenario, a virtual training simulator should produce a high level of psychological fidelity. That is, it should reproduce real world situations as realistically as possible to allow students to successfully transfer their training from the simulator to the real world. Good simulation performance is a key requirement to achieving this. A well performing simulator should thus have the following characteristics:

  1. Fluidity measures the level at which objects appear to be moving smoothly and continuously rather than jumping from one position to another in distinct, noticeable increments. The human eye will notice a low level of fluidity or no fluidity at all when the frame rate (rate at which the images on the screen are updated) falls below a certain threshold. This threshold (at which the appearance of fluidity in the simulation is lost), is directly related to the simulation content, that is, the speed at which cameras and objects move, and the process time in each simulation. Slow moving simulations can get away with lower framerates though the target is usually 60 Hz.
  2. Monitor synchronization concerns having all the monitors rendering the same frame in sync with each other. If there are performance issues, observable desynchronization will mostly happen when the camera/viewpoint rotates since this is what triggers the biggest visual difference compared to the previous frame. The observer will perceive this desynchronization as stuttering. In a well performing simulator, all display monitors are synchronized and the user perceives no stuttering (noticeable delays between frames).
  3. Realistic physics at expected speed. In simulations with low framerates, real-time physics simulation can often not be achieved, making objects move around slower than expected, thus yielding unrealistic behavior. Usually, a physics simulation in Vortex Studio is run at a simulation frequency of 60 Hz, that is, the simulated time increases by 1/60 seconds per frame. For real-time simulation, the processing time for a single simulation frame may thus not exceed this time duration.

There are three considerations to keep in mind:

  1. Target frame processing time < 1/60 seconds (for a simulation running at 60 Hz simulation frequency): The frame processing time is the amount of time it takes for the whole simulation to generate and use a frame. The frame generation includes things like processing dynamics and visualization data (i.e., positioning triangles on the screen based on the dynamics data). The usage of the frame includes how the data is used and/or sent to the user for interaction. For instance, visualization is displayed on multiple monitors, sound is played through speakers, joysticks are moved, etc. There are quite a lot of dependencies between the simulation modules, so the generation of data depends on many factors. Consequently, users should always aim for frame processing times below 1/60 seconds to account for some additional overhead in processing (e.g., OS and graphics drivers need some time in order to send the data in time for the vertical sync).
  2. No spikes in frame processing time: Even if the frame processing time is generally under 1/60 seconds, spikes (processing times above the target) can happen. This results in delays in the frame visualization. This, in turn, might then trigger unexpected behavior like the graphics drivers forcing the simulation to lower the synchronization, rather than 60 Hz when vertical sync is enabled. This situation is perceived as stutter.
  3. Stable frame processing times: Even if most of the frames are generated in times below the 1/60th of a second, frame processing times can still be quite unstable. Most frames will take a different amount of processing time since inputs and the simulation's environment will change, but great care should be taken to ensure similar processing times from one frame to the other. The more unstable the processing times in a simulation are, the higher the chances a frame duration will spike over the target. For instance, consider a scene made of several machines (e.g., cranes or excavators) which each take 10 ms to simulate each frame. If in the simulation only one machine is used at a time, the frame processing time should stay below the target of 1/60 second. But if, at some point in time, more than one machine is used for an unexpected reason, the processing time would then go well above 1/60 second and create stuttering or simulation slow downs.

Identifying the Bottleneck

Performance can be affected by many factors, but, as stated above, a user will mostly recognize simulation performance issues through problems in the fluidity of its visualization. The simulation is performed through a series of steps in a pipeline, in which each stage depends on its predecessor. For example, in order to render an object on the screen, the graphics module needs to know where this object is currently located, information which is provided by the outputs of the dynamics module. Many modules are involved in the simulation pipeline including graphics, dynamics and networking.

Simulator Setup

The first thing to do to identify bottlenecks is to ensure the setup of the simulator topology is done properly. A proper synchronization of all nodes in this topology which are involved in the simulation (e.g., a dynamics master node, and several image generator slave nodes) depends on having a clean and optimal setup.

After this is confirmed and the simulation still shows performance issues, you can start using Vortex Studio tools outlined below to identify the problem.

Network Processes

The first tool to use is The Network Processes tab of the Vortex Studio Player.

In this tab, the user can see a list of the nodes with their corresponding frame processing times (Cycle Time). Those time values give the larger picture of the simulation's performance and are often the biggest hints needed to continue the investigation. There may be multiple issues causing poor performance but a proper investigation should start with the most problematic item, i.e., the slowest node, which is the one with the largest Cycle Time value. After identifying a faulty node (e.g., in case of 60 Hz as the targeted simulation frequency, this would be a node with a Cycle Time larger than 1/60 second), the Profiler, running on this particular node, is needed in order to gain a better understanding of the problem.

Profiler

Note that the Vortex Studio Editor includes some tools for diagnosing performance issues, but Vortex Studio Player application has features designed specifically for investigating performance problems.

The Profiler displays the time each module takes, the two major modules being dynamics and graphics. Remember that each node has a predefined responsibility. The dynamics usually runs on one specific node and its results are communicated to the other nodes every frame. Multiple graphics modules are used to generate the data for a specific monitor, meaning the graphics may be running on multiple nodes (commonly referred to as Image Generators in this case) for different camera angles.

Node running the dynamics

Node Balancing

If one node is running more than just dynamics or graphics, great care should be taken regarding node balancing. The different nodes obviously all have limited resources (e.g., CPU or GPU time) but what is not as obvious is how to assign the nodes' responsibilities. Those resources have to be given based on the different needs and balanced properly in order to have as much content at as low frame processing times as possible.

  • The engine should always be running on a separate node if possible. If not, share the node's resources with something that does not hinder it more than needed.
  • The main monitor (the one which displays the most graphical content) will require a lot of resources so this needs much consideration. Try not to use the same node for both the engine and the main monitor.

Graphics

In Vortex Studio, the graphics module (usually called GraphicsModule in your application setup; see the Vortex Player (Slave) image above) performs all calculations related to graphics processing and rendering. If the Profiler points out a significant time consumption in the graphics module, please refer to Graphics Performance Analysis for steps to take in order to analyze and tune graphics performance.

Dynamics

In Vortex Studio, the dynamics module (usually called DynamicsEngine in your application setup; see screenshot above) performs all calculations related to the simulation of dynamics. If the profiler points out a significant time consumption in the dynamics module, please refer to Dynamics Performance Analysis for steps to take in order to analyze and tune dynamics performance.

Sub-Topics

The following topics are covered in this section: