If VR bugs can make you (cyber)sick: use testing as a diagnostic tool

Defects in Virtual Reality (VR) experiences whether through poor design or performance can trigger nausea, disorientation and other physical symptoms. Although the physical effects are rarely long term, they can last beyond the end of the experience and may do enough psychological harm to discourage the user from wanting to try other VR experiences.

I covered much of the background in my last post and don’t want to repeat it here. In this post I suggest some ways to test VR applications for cybersickness in the hope of discovering bugs or recognising potential issues before they have a chance to “infect” anyone.

Caveats

A lot of what follows is not as simple as pass or fail. There are currently no internationally adopted standards for VR, the research is not fully comprehensive and the technology is still developing. It is also necessary to consider the context. The following is a heuristic checklist to help get you started.

Unanticipated movement (or lack of it)

One of the primarily causes of cybersickness is when there’s a disconnect between perceived movement in the virtual environment and the actual (physical) or expected (virtual) movement. This disconnect can be as a result of poor performance, but design also plays a part.

Performance

It is essential to track head movement at all times in VR, even delays of microseconds can cause the user to feel nauseous or become disorientated. To mitigate for issues, high-end VR hardware manufacturers like Vive and Oculus recommend maintaining a high frame rate of at least 90 frames per second. For mobile devices, most high end phones (at the start of 2018) only have a refresh rate of 60Hz so that should be the target there.

Steam provides tools to measure fps for PC based hardware. On mobile platforms, the application GameBench features a useful interface with associated screenshots that show you what was happening at intervals throughout the experience. There are also tools to monitor the CPU/GPU usage and temperature - these can help identify parts of your experience for extra attention.

On finding an area of the application where the CPU and/or GPU are being taxed, perform activities that are likely to cause additional load:
  • Perform rapid headset movement and changes in direction (being careful to protect your neck - the headset is an additional off-centre load) 
  • Trigger multiple interactions simultaneously or in quick succession
Frame from GameBench showing VR performance
Screenshot captured from GameBench shows how the application has a consistently high refresh rate of around 60fps which drops significantly at one point.

TestExpected ResultNotes
- Start performance tracking software
- Navigate throughout entire experience
- In parts of the experience where the CPU or GPU are being taxed heavily try the adding extra load
High refresh rate maintained throughout experience90 fps [for pc based experiences]
60 fps [for mobile experiences]
Head tracking constantHead motion is tracked at all times, no observable lag, stuttering or screen tearing*.
Exception 1If refresh rate has to drop below the expected rate for some reason - the display is faded out so no motion or lack of motion can be perceived.
Exception 2Some locomotion options may cause virtual movement in the headset, where this is the case, the movement must be controlled / triggered by the user or at least anticipated.

Design

Performance isn’t the only cause of unanticipated movement, it could also be part of the design.
Whilst navigating through the VR experience watch out for
  • Places where the user is moved outside of their direct control, this can be uncomfortable unless anticipated 
  • Places where there is a lot of movement, particularly if it fills a large part of the visual field 
Test
Expected Result
Notes
Look for any place that the user is being moved outside of their control.The movement is anticipated and within context of the experience.

The user maintains full control of the “camera” at all times - the head tracking is constant.

Observe any part of the experience where a large part of the visual field moves (e.g. a river) and get as close a as possible to that feature.If the moving area covers a large part of the visual field there needs to be some static reference points to help orientate the user (e.g. rocks in the river, or riverbank).These reference points could be artificial or could be options that are on by default.
Move around the environment or observe when movement is triggered.Motion should be constant, without acceleration or elliptical motion.If acceleration or elliptical motion is desirable, then have it as an optional setting.

Screenshot from End Space VR showing complex scene
If your application has lots going on like this shot from End Space VR it might be as well to limit access to only high performing devices (in this case iOS 8.0 or later), or consider optimising for slower devices.

Locomotion

TestExpected ResultNotes
Try moving in different directions.Unnatural movement such as walking backwards and sideways (strafing) are either not possible, or not required and do not deliver any advantage in a multi-player or competitive setting.
Check the menu, settings or options where available.There is at least one option of a different style of locomotion from the default.Users may opt for different locomotion styles for a number of reasons:
- Accessibility
- Personal preference
- Realism / immersion
- If the default causes them to suffer from cybersickness
Watch for the use of visual effects throughout the experience, particularly when moving.Effects such as “Head bob" or "weapon sway" are off by default.

There is no distortion or blurring, especially around the periphery.
These effects can be optional, but should not be the default.
If the user has an in-experience avatar - check how the avatar movement corresponds to the real physical movement with particular attention to any untracked parts of the body.All avatar movement corresponds to real physical movement. It is not possible to position the body in a way that is not matched by the avatar.It is common for only the head and hands to be tracked, so attention should focus on arms and legs.

Screenshot from Job Simulator showing disembodied hands
A lot of applications use disembodied hands as getting the untracked arms to synchronise with the user’s real arms is incredibly difficult and has consequences if not done accurately. (Source: Job Simulator)

Visual and environmental effects

TestExpected ResultNotes
Observe the visual display for a static frame of reference.Examples of a static frame of reference are: - Distant horizon - Window frame - Cockpit or canopy - Helmet edges - Reticle** - Virtual nose Where a suitable “in-context” reference is not possible, an artificial one could be provided as an optional setting.This is less important for seated or static experiences.
Look out for any flashing lights or flickering images. These could be intentional (e.g. a warning light) or an unintentional attribute (e.g. aliasing or overdrawing).No flickering or flashing occurs.Intentional effects could be optional, in which case a warning should be displayed.The W3C suggest three flashes or less but this is for websites, VR will occupy a much larger part of the user’s visual field - the safest thing would be to avoid all together.

Screenshot showing display reticle
Where the horizon is not present or obscured, having a reticle (in this case a white circle) can help to stabilise the user. (Source: Dave's Code Blog)

Control

Giving the user some control over how the application is setup can give them a better experience and likely to reduce some of the symptoms or delay the occurrence of cybersickness.

TestExpected resultNotes
- Adjust the base brightness of the device on which the application is running
- Start application
- Adjust the base brightness level
The application maintains the base brightness level. Any adjustments to the base brightness level are mirrored by the application.
- Adjust the base volume level of the device on which the application is running
- Start application
- Adjust the base volume level
Application adopts the base volume level and reacts to any changes.Alternatively, the application could give the user the ability to adjust the volume within the experience.
Whilst in the experience, close then restart the application.The application provides the ability to do at least one of the following:
- Gives the user the option to save/reload their progress
- Gives the user the option to save/reload their progress
- Automatically saves and loads progress on restarting the application
- Automatically saves and loads progress on restarting the application
- Notifies the user when they have reached a checkpoint. On restarting, the experience starts from the last checkpoint by default
- Notifies the user when they have reached a checkpoint. On restarting, the experience starts from the last checkpoint by default
- Provides the option to pause/resume progress
- Provides the option to pause/resume progress

Comfort

VR when is done well it has the ability to trick the brain into giving the user the feeling of being present in the artificial environment. Be considerate of this and treat the user kindly.
TestExpected ResultNotes
Consider the position of objects, effects, UIs and the environment. Unless trying to make the user feel uncomfortable (e.g.in a horror experience), give them personal space: - Objects (and UI) are presented at a reasonable distance from the user - Confined areas that could trigger feelings of claustrophobia are avoided
Consider how the user has to interact with the environment.- The user does not have to repeatedly refocus on different objects - There is no requirement to stand awkward positions for long periods (e.g. looking over shoulder) - Repetitive motions that would not be done in normal life are not required frequently (e.g. flicking the head from side to side) Where possible give the user options of how to interact.For example, provide options to choose between using a controller or gaze control.
This checklist is still in development, please let me know if you have any edits or suggestions for other tests.

-

*See notes from last post
**This is a centre spot, often presented like a weapon site or focus ring

Comments