Magnetic Input for Mobile VR

It’s easy to do, just follow these steps:

1. Cut two holes in a box
2. Put your phone in that box
3. Look inside the box

And that’s the way you do it.

Your smartphone is now in a box, so how do you do input? Now that we have a
paper accepted to ISWC 2015, I can tell you!

It’s not easy being in a box

Let me remind you: your smartphone is still in a box. This means that your
fingers can’t reach the touch screen or the volume buttons. Let’s consider a
couple of input alternatives:

• Cameras and microphones require extra app permissions, are inefficient to keep
  always on, and may face many false positives.
• External electronic devices cost money. Plugging them in and out is a
  usability nightmare.

Ok, let’s nix those. How about permanent magnets? They are inexpensive, robust,
require no power to operate, and do not degrade over time. The vast majority of
smartphones have a magnetometer, which is used for the compass. Intriguing…

Fun with permanent magnets

In 2009, Chris Harrison and Scott Hudson published Abracadabra, a
magnetic ring form factor for finger interactions with small devices:

In 2010, Hamed Ketabdar and others published Magitact, instead using
a magnetic rod for more varied interactions near smartphones:

In 2011, Daniel Ashbrook and others published Nenya, which is similar
to Abracadabra, but focused more on the eyes-free input aspects:

In 2013, Sungjae Hwang and others published Maggetz, which used
passive magnets to build all sorts of widgets around the device:

Fucking magnets: how do they work?

Magnets affect the magnetometer in an inverse-cubic relationship, so
distance between magnet and magnetometer really makes a dramatic difference in
signal strength. We empirically determined that in most phones, the sensor is
placed at the top of the device, near the earpiece:

Smartphone Model	Sensor Location
Moto X	Top
Nexus 4	Top
Nexus 5	Top
Samsung S4	Top
Galaxy Nexus	Top
Samsung S3	Bottom
Moto G	Bottom

Some magnetometers are really screwy, like the one found in the first revision
of the HTC M7, or broken, like in some models of the Galaxy Nexus we tested
with. There’s little that we can do in these cases, but luckily they are quite
rare.

The way you access the magnetometer on Android is via the sensor stack,
requesting the TYPE_MAGNETIC_FIELD sensor. This is a calibrated sensor, since
it’s primarily used to determine the direction of magnetic north for the
compass. Calibration means that somewhere deep inside Android, software and
hardware periodically calibrates the output of the sensor. When calibration
occurs, magnetometer readings effectively reset to some new coordinate system.

Calibration can happen at any point, and the calibration pattern can look quite
different depending on the device. In some cases, it’s a gradual calibration,
not a sudden spike as above. This limitation restricts what we can reliably
detect, which is why we chose a pull-and-release interaction. Android already
has provisions for an uncalibrated magnetometer via
TYPE_MAGNETIC_FIELD_UNCALIBRATED, but this sensor is not nearly as ubiquitous
as its calibrated cousin. Even so, we should be robust to phone insertions and
removals from Cardboard, which can also look like calibration events.

Magnetic input for VR

As you can see, the interaction involves pulling the magnet downward, and
releasing it. The magnetic ring automatically returns to its rest position
because of the force exerted on it by an internal magnet. The external magnet is
also held in-place by the same force, and while it’s possible to pull the magnet
off the cardboard side, it takes concerted effort to do so. The motion of the
magnet is constrained by a cardboard indentation, so it can only move downward.
The thing I find most elegant about this design is that both the digital signal
to the smartphone and the physical mechanism itself relies on the same
principle: magnetism.

We collected a bunch of data for this pull-and-release interaction from many
devices. We found that most devices behave predictably well. Here’s a combined
plot of normalized, superimposed positives and negatives from all phones which
we collected data from, with each dimension of the magnetometer vector plotted
separately.

The detector we built was not based on a template learned from all of the data
above, but a simpler state machine based on thresholding. The thresholds
themselves were learned empirically. Here’s the simple state machine:

The basic idea is that we take a sliding window approach, normalizing all of the
data relative to the last value in the window. For each window, we calculate min_1,
which is the smallest value of the first half of the window, and max_2, the
largest value in the second half. Next, we compare to empirically determined
thresholds perform the appropriate transition in the state machine. I won’t bore
you with details of normalizing the data, etc but you can find all of the
details in the paper. Oh, and all of the code is also available on
github.

What’s next?

A lot more can be done using passive magnetic input. With uncalibrated
magnetometers, there is no fear of calibration events, so we could implement a
faster detector based on just the down motion of the magnet. We could reliably
detect long presses and double clicks. Alternatively, extensions to the existing
input can be implemented by simply changing the geometry of the physical
constraints, such as a joystick form factor.

I’m incredibly happy that Cardboard has been doing so well. Thanks to the great
team working so hard on it, there are now over 1 million units shipped.
The press has been happy with it too, with kind reviews from many tech
publications.

Techcrunch said:

This funny little cardboard faux-Rift has something even the original Rift
itself does not: a built-in button. Your phone is able to sense the magnet’s
movement, allowing it to act as a ridiculously clever little button. Yeesh.

Techradar said:

What’s also somewhat amazing is the magnet on the side. […] The little magnet
on the side is actually a quite ingenious design aspect of Google Cardboard.
It’s a button!

Engadget said:

One of the things I liked most was a switch located on the left temple, which
consists of just a couple of magnets and a metal ring.

Future versions of Cardboard are switching to a different input method
using a conductive button which brings your body’s capacitance to the screen,
similar to how a touch stylus works. It’s cheaper without them, and the new
input works well, but I’ll definitely miss the magnets!