A major technical challenge for VR is to make head tracking as good as possible.
The metric that matters is called motion-to-photon latency. For mobile VR
purposes, this is the time that it takes for a user’s head rotation to be fully
reflected in the rendered content.
The simplest way to get up-and-running with head tracking on the web today is
to use the deviceorientation events, which are generally well supported across
most browsers. However, this approach suffers from several drawbacks which can
be remedied by implementing our own sensor fusion. We can do even better by
predicting head orientation from the gyroscope.
I’ll dig into these techniques and their open web implementations. Everything
discussed in this post is implemented and available open source as part of the
WebVR Polyfill project. If you want to skip ahead, check out
the latest head tracker in action, and play around with this motion
sensor visualizer.
The trouble with device orientation
The web provides an easy solution for head tracking through the
deviceorientation event, which gives Euler angles corresponding to your
phone’s 3-DOF orientation in space. This orientation is calculated through an
undisclosed algorithm. Until very recently, the spec didn’t
even specify whether or not these events should give your phone’s orientation in
relation to north or not. However, recently accepted spec
changes make this behavior more standard across
browsers.
In Android, the JavaScript deviceorientation event was implemented using
Sensor.TYPE_ORIENTATION in Android, which fuses accelerometer, gyroscope and
magnetometer sensors together to give a North-aligned orientation. The trouble
is that the magnetometer’s estimate of magnetic North is easily affected by
external metallic objects. On many devices, the North estimate continually
changes, even when you are not looking around. This breaks the correspondence
between motion and display, a recipe for disaster.
Another issue in some implementations is that the deviceorientation sensor
ramps up and down in firing rate depending on the speed of the phone’s rotation.
Try opening up this diagnostic page on Android. This variation in
sensor update rate is not good for maintaining a reliable head track.
To top it off, a recent regression in Android M broke
deviceorientation for Nexus 5s. Why do bad bugs happen to good people?
What is to be done?
We implement our own sensor fusion with devicemotion, which provides lower
level accelerometer & gyroscope events. These fire at a regular rate. When you
search for “sensor fusion”, jumping into the rabbit hole will quickly take you
into the realm of Kalman Filters. This is a bit more firepower than we will need
for the moment, although I did finally get a better sense of the concept with
the help of a boring but understandable explanation.
Luckily, there are simpler alternatives such as the Complementary Filter, which
is what we’ll talk about next.
Your sensing smartphone
Let us start with the basics: sensors. There are three fundamental motion
tracking sensors in your smartphone.
Accelerometers measure any acceleration, returning a vector in the phone’s
reference frame. Usually this vector points down, towards the center of the
earth, but other accelerations (eg. linear ones as you move your phone) are also
captured. The output from an accelerometer is quite noisy by virtue of how the
sensor works. Here’s a plot of the rotation around the X-axis according to an
accelerometer:
Gyroscopes measure rotations, returning an angular rotation vector also in the
phone’s reference frame. Output from the gyro is quite smooth, and very
responsive to small rotations. The gyro can be used to estimate pose by keeping
track of the current pose and adjusting it every timestep, with every new gyro
reading. This integration works well, but suffers from drift. If you were to
place your phone flat and capture it’s gyro-based position, then pick it up,
rotate it a bunch, and place it flat again, its integrated gyro position might
be quite different from what it was before due to the accumulation of errors
from the sensor. Rotation around the X-axis according to a gyroscope:
Magnetometers measure magnetic fields, returning a vector corresponding to the
cumulative magnetic field due to any nearby magnets (including the Earth). This
sensor acts like a compass, giving an orientation estimate of the phone. This is
incredibly useful combined with the accelerometer, which provides no information
about the phone’s yaw. Magnetometers are affected not by the Earth, but by
anything with a magnetic field, including strategically placed permanent
magnets and also ferromagnetic metals which are often found in substantial
quantities in certain environments.
Intuition: why do we need sensor fusion?
Each sensor has its own strengths and weaknesses. Gyroscopes have no idea where
they are in relation to the world, while accelerometers are very noisy and can
never provide a yaw estimate. The idea of sensor fusion is to take readings from
each sensor and provide a more useful result which combines the strengths of
each. The resulting fused stream is greater than the sum of its parts.
There are many ways of fusing sensors into one stream. Which sensors you fuse,
and which algorithmic approach you choose should depend on the usecase.
The accelerometer-gyroscope-magnetometer sensor fusion provided by the
system tries really hard to generate something useful. But as it turns out, it
is not great for VR head tracking. The selected sensors are the wrong ones, and
the output is not sensitive enough to small head movements.
In VR, drifting away from true north is often fine since you aren’t looking at
the real world anyway. So there’s no need to fuse with magnetometer. Reducing
absolute drift is, of course, still desirable in some cases. If you are sitting
in an armchair, maintaining alignment with the front of your chair is critical,
otherwise you will find yourself having to crank your neck too much just to
continue looking forward in the virtual world. For the time being, we ignore
this problem.
Building a complementary filter
The complementary filter takes advantage of the long term accuracy of the
accelerometer, while mitigating the noise in the sensor by relying on the
gyroscope in the short term. The filter is called complementary because
mathematically, it can be expressed as a weighted sum of the two sensor streams:
This approach relies on the gyroscope for angular updates to head orientation,
but corrects for gyro drift by taking into account where measured gravity is
according to the accelerometer.
Initially inspired by Pieter’s explanation, I built this filter by
calculating roll and pitch from the accelerometer and gyroscope, but quickly ran
into issues with gimbal lock. A better approach is to use quaternions
to represent orientation, which do not suffer from this problem, and are ideal
for thinking about rotations in 3D. Quaternions are complex (ha!) so I won’t go
into much detail here beyond linking to a decent primer on the
topic. Happily, quaternions are a useful tool even without fully understanding
the theory, and many implementations exist. For this filter, I used the
one found in THREE.js.
The first task is to express the accelerometer vector as a quaternion rotation,
which we use to initialize the orientation estimate (see
ComplementaryFilter.accelToQuaternion_).
quat.setFromUnitVectors(new THREE.Vector3(0, 0, -1), normAccel);
Every time we get new sensor data, calculate the instantaneous change in
orientation from the gyroscope. Again, we convert to a quaternion, as follows
(see: ComplementaryFilter.gyroToQuaternionDelta_):
quat.setFromAxisAngle(gyroNorm, gyro.length() * dt);
Now we update the orientation estimate with the quaternion delta. This is a
quaternion multiplication:
this.filterQ.copy(this.previousFilterQ);
this.filterQ.multiply(gyroDeltaQ);
Next, calculate the estimated gravity from the current orientation and compare
it to the gravity from the accelerometer, getting the quaternion delta.
deltaQ.setFromUnitVectors(this.estimatedGravity, this.measuredGravity);
Now we can calculate the target orientation based on the measured gravity, and
then perform a spherical linear interpolation (SLERP). How much to
slerp depends on that constant I mentioned before. If we don’t slerp at all, we
will end up only using the gyroscope. If we slerp all the way to the target, we
will end up ignoring the gyroscope completely and only using the accelerometer.
In THREE parlance:
this.filterQ.slerp(targetQ, 1 - this.kFilter);
Sanity checking the result, we expect the filter output to be roughly parallel
to the gyroscope readings, but to align with the accelerometer reading over the
long term. Below, you can see the accelerometer and gyroscope (green and blue)
and compare them to the fused output (orange):
Predicting the future
As your program draws each frame of rendered content, there is delay between
the time you move your head and the time the content actually appears on the
screen. It takes time for the sensors to fire, for firmware and software to
process sensor data, and for a scene to be generated based on that sensor data.
In Android, this latency is often on the order of 50-100 ms with sensors firing
on all cylinders (the technical term for 200 Hz) and some nice graphics
optimizations. The web suffers a strictly worse fate since sensors often fire
slower (60 Hz in Safari and Firefox), and there are more hoops of abstraction to
jump through. Reducing motion-to-photon latency can be done by actually reducing
each step in the process, with faster sensor processing, graphics optimizations,
and better algorithms. It can also be reduced by cheating!
We can rely on a dead reckoning inspired approach, but rather
than predicting position based on velocity, we predict in the angular domain.
Once we predict the orientation of the head in the (near) future, use that
orientation to render the scene. We predict based on angular velocity, assuming
that your head will keep rotating at the same rate. More complex schemes are
possible to imagine too, using acceleration (2nd order) or Nth order prediction,
but these are more complex, and so more expensive to calculate, and don’t
necessarily yield better results.
var deltaT = timestampS - this.previousTimestampS;
var predictAngle = angularSpeed * this.predictionTimeS;
The way this works is pretty straight forward, using angular speed from the
gyroscope, we can predict a little bit into the future to yield results like
this:
Notice that the predicted signal (in red) is somewhat ahead of the fused one (in
orange). This is what we’d expect based on the motion prediction approach taken.
The downside of this is that there is noticeable noise, since sometimes we
over-predict, and are forced to return back to the original heading.
Plotting graphs
Although still in very active development, Mathbox2 is already a
formidable visualization toolkit. It is especially well suited to output in 3D,
which I used actively to debug and visualize the filter.
I also used Mathbox2 to generate plots featured earlier in this blog post. I
wrote a live-plotting tool that can compare gyroscope, accelerometer, fused and
predicted streams on each axis, and also let you tweak the filter coefficient
and how far into the future to predict.
You too can try the plots live on your phone. After all, it’s just a
mobile webpage! Many thanks to Pierre
Fite-Georgel and Julius
Kammerl for lending their incredible
filter-building skills to this project.
Powered by WPeMatico
