Why tracking matters
For a virtual scene to feel believable, the headset needs to know where your head is, which way it is pointing, and where your hands are. When you lean forward, the world should get closer. When you turn, the view should turn with you, immediately and smoothly. If tracking is slow or inaccurate, the illusion breaks, and some people feel uncomfortable or queasy. Headset tracking is the set of techniques that handles this.
Six degrees of freedom
Your head can move in two basic ways: it can rotate and it can translate (shift position). Rotation has three parts: turning left or right, nodding up or down, and tilting side to side. Translation has three parts: moving forward or back, left or right, and up or down. Three plus three gives six, which is why high-end tracking is described as having six degrees of freedom, often shortened to 6DoF.
Simple viewers track only rotation, so the scene turns when you turn but does not respond if you lean or walk. That is called 3DoF. Full 6DoF tracking is what lets you crouch to look under a virtual table.
Motion sensors: fast but drifting
Inside nearly every headset is an inertial measurement unit, or IMU. It typically combines an accelerometer, which senses acceleration, and a gyroscope, which senses rotation speed. Many also include a magnetometer, which senses magnetic fields in a way similar to a compass. These are the same kinds of tiny sensors found in phones and other wearable devices.
IMUs have a great advantage: they report extremely quickly, many times per second, so they respond with very little delay. Their weakness is drift. Because position is worked out by adding up many small measurements, tiny errors accumulate, and the estimated position slowly wanders away from the truth. An IMU alone is therefore good at telling you how you just moved, but poor at telling you where you are over time.
Cameras: slower but anchored
To correct drift, headsets use cameras. There are two broad designs.
Inside-out tracking. Cameras mounted on the headset look at the room. Software picks out distinctive features such as edges, corners and textures, and watches how they shift between frames. From that, it calculates how the headset itself must have moved. Combining camera-based position with the IMU’s quick motion data is often called sensor fusion. Related techniques are used in robot sensing, where machines also need to work out where they are from camera and motion data.
Outside-in tracking. External sensors or base stations placed around the room watch the headset and controllers, which may carry markers or light sensors. This can be very precise, but it requires setting up equipment and a fixed play area.
Most current consumer headsets use the inside-out approach because it needs less setup. A consequence is that it depends on what the cameras can see. A dim room, a plain blank wall, very shiny surfaces or strong sunlight can give the software too little to work with, and tracking can wobble or lose its place.
Tracking your hands
Hands can be tracked in two main ways.
- Controllers. Handheld devices have their own motion sensors, plus patterns of lights or markers the headset cameras can see. Buttons, triggers and thumbsticks supply input that cameras cannot, and some controllers also sense finger touch or grip.
- Camera-based hand tracking. The headset cameras look at your bare hands, and software, usually machine learning models trained on many example images, estimates the position of each joint and finger. It requires no extra hardware, but can struggle if hands are hidden, held behind the body, or moving very fast.
Because hand tracking relies on what the cameras can see, your hands must be in view of the headset. Controllers with their own sensors can keep working a little longer when briefly out of sight.
Eyes and the body
Some headsets also track where your eyes are looking, using small infrared cameras inside the device. This can help with sharpening the center of the image or with interaction, but gaze is personal data. It raises questions about data privacy because where someone looks can reveal attention and interest. Full-body tracking, using extra trackers or camera-based estimation, exists but is less common.
Keeping it fast
Tracking must be quick, but it also must be predictive. Between the moment a sensor reading is taken and the moment the image reaches your eyes, a few milliseconds pass. Systems therefore often predict where your head will be a moment later and render for that predicted pose. Total delay from movement to display is sometimes called motion-to-photon latency, and engineers work to keep it low.
A lot of the processing happens on the device itself, or close to it, in the spirit of edge computing, because sending data far away and back would add delay.
Where it still falls short
Tracking is good but imperfect. Drift, lost tracking in poor lighting, hand occlusion and jitter all happen. Safety boundaries, which warn you before you reach the edge of a cleared space, depend on the tracking being right. Researchers continue to look for ways to make it more robust, but exactly how quickly it will improve is hard to predict.