One of the first problems presented to students of deep learning is to classify
handwritten digits in the MNIST dataset. This was recently ported to
the web thanks to deeplearn.js. The web version has
distinct educational advantages over the relatively dry TensorFlow tutorial.
You can immediately get a feeling for the model, and start building intuition
for what works and what doesn’t. Let’s preserve this interactivity, but change
domains to audio. This post sets the scene for the auditory equivalent of MNIST.
Rather than recognize handwritten digits, we will focus on recognizing spoken
commands. We’ll do this by converting sounds like this:

Into images like this, called log-mel spectrograms, and in the next post,
feed these images into the same types of models that do handwriting recognition
so well:

The audio feature extraction technique I discuss here is generic enough to work
for all sorts of audio, not just human speech. The rest of the post explains
how. If you don’t care and just want to see the code, or play with some
live demos, be my guest!

Why?

Neural networks are having quite a resurgence, and for good reason. Computers
are beating humans at many challenging tasks, from identifying faces and images,
to playing Go. The basic principles of neural nets is relatively simple, but the
details can get quite complex. Luckily non-AI experts can get a feeling for what
can be done because a lot of output is quite engaging.
Unfortunately these demos are mostly visual in nature, either examples of
computer vision, or generate images or video as their main output. And
few of these examples are interactive.

Pre-processing audio sounds hard, do we have to?

Raw audio is a pressure wave sampled at tens of thousands times per second and
stored as an array of numbers. It’s quite a bit of data, but there are neural
networks that can ingest it directly. Wavenet does speech to
text and text to speech using raw audio sequences,
without any explicit feature extraction. Unfortunately it’s slow: running speech
recognition on a 2s example took 30s on my laptop. Doing this in real-time, in
a web browser isn’t quite ready yet.

Convolutional Neural Networks (CNNs) are a big reason why there has been so much
interesting work done in computer vision recently. These networks are designed
to work on matrices representing 2D images, so a natural idea is to take our raw
audio and generate an image from it. Generating these images from audio is
sometimes called a frontend in speech recognition papers. Just to
hammer the point home, here’s a diagram explaining why we need to do this step:

The standard way of generating images from audio is by looking at the audio
chunk-by-chunk, and analyzing it in the frequency domain, and then applying
various techniques to massage that data into a form that is well suited to
machine learning. This is a common technique in sound and speech processing, and
there are great implementations in Python. TensorFlow even has a
custom op for extracting spectrograms from audio.

On the web, these tools are lacking. The Web Audio API can almost do
this, using the AnalyserNode, as I’ve shown in the past, but
there is an important limitation in the context of data processing:
AnalyserNode (nee RealtimeAnalyser) is only for real-time analysis.
You can setup an OfflineAudioContext and run your audio through the analyser,
but you will get unreliable results.

The alternative is to do this without the Web Audio API, and there are
many signal processing JavaScript libraries that might
help. None of them are quite adequate, for reasons of incompleteness or
abandonment. But here’s an illustrated take on extracting Mel features from raw
audio.

Audio feature extraction

I found an audio feature extraction tutorial, which I followed closely
when implementing this feature extractor in TypeScript. What follows can be a
useful companion to that tutorial.

Let’s begin with an audio example (a man saying the word “left”):

Here’s that raw waveform plotted as pressure as a function of time:

We could take the FFT over the whole signal, but it changes a lot over time.
In our example above, the “left” utterance only takes about 200 ms, and most of
the signal is silence. Instead, we break up the raw audio signal into
overlapping buffers, spaced a hop length apart. Having our buffers overlap
ensures that we don’t miss out on any interesting details happening at the
buffer boundaries. There is an art to picking the right
buffer and hop lengths:

• Pick too small a buffer, and you end up with an overly detailed image, and
  risk your neural net training on some irrelevant minutia, missing the forest
  for the trees.
• Pick too large a buffer, and you end up with an image too coarse to be useful.

In the illustration below, you can see five full buffers that overlap one
another by 50%. For illustration purposes only, the buffer and hop durations are
large (400 ms and 200ms respectively). In practice, we tend to use much shorter
buffers (eg. 20-40 ms), and often even shorter hop lengths to capture minute
changes in audio signal.

Then, we consider each buffer in the frequency domain. We can do this using an
Fast Fourier Transform (FFT) algorithm. This algorithm gives us complex values
from which we can extract magnitudes or energies. For example, here are the FFT
energies of one of the buffers, approximately the second one in the above image,
where the speaker begins saying the “le” syllable of “left”:

Now imagine we do this for every buffer we generated in the previous step, take
each FFT arrays and instead of showing energy as a function of frequency, stack
the array vertically so that y-axis represents frequency and color represents
energy. We end up with a spectrogram:

We could feed this image into our neural network, but you’ll agree that it looks
pretty sparse. We have wasted so much space, and there’s not much signal there
for a neural network to train on.

Let’s jump back to the FFT plot to zoom our image into our area of interest. The
frequencies in this plot are bunched up below 5 KHz since the speaker isn’t
producing particularily high frequency sound. Human audition tends to be
logarithmic, so we can view the same range on a log-plot:

Let’s generate new spectrograms as we did in an earlier step, but rather than
using a linear plot of energies, use can a log-plot of FFT energies:

Looks a bit better, but there is room for improvement. Humans are much better at
discerning small changes in pitch at low frequencies than at high frequencies.
The Mel scale relates pitch of a pure tone to its actual measured frequency. To
go from frequencies to Mels, we create a triangular filter bank:

Each colorful triangle above is a window that we can apply to the frequency
representation of the sound. Applying each window to the FFT energies we
generated earlier will give us the Mel spectrum, in this case an array of 20
values:

Plotting this as a spectrogram, we get our feature, the log-mel spectrogram:

The 1s images above are generated using audio feature extraction software
written in TypeScript, which I’ve released publicly. Here’s a demo that
lets you run the feature extractor on your own audio, and the code on
github.

Handling real-time audio input

By default the feature extractor frontend takes a fixed buffer of audio as
input. But to make an interactive audio demo, we need to process a continuous
stream of audio data. So we will need to generate new images as new audio comes
in. Luckily we don’t need to recompute the whole log-mel spectrogram every time,
just the new parts of the image. We can then add the new parts of spectrogram on
the right, and remove the old parts, resulting in a movie that feeds from the
right to the left. The StreamingFeatureExtractor class implements this
important optimization.

But there is one caveat: it currently relies on ScriptProcessorNode, which is
notorious for dropping samples. I’ve tried to mitigate this as much as possible
by using a large input buffer size, but the real solution will be to use
AudioWorklets when they are available.

Wrapping up

An implementation note: here is a comparison of JS FFT libraries which
suggests the Emscripten-compiled KissFFT is the fastest (but still 2-5x slower
than native), and the one I used.

Here is a sanity check comparing the output of my web-based feature extractor to
that of other libraries, most notably librosa and from AudioSet:

The images resulting from the three implementations are similar, which is a good
sanity check, but they are not identical. I haven’t found the time yet, but it
would be very worthwhile to have a consistent cross platform audio feature
extractor, so that models trained in Python/C++ could run directly on the web,
and vice versa.

I should also mention that although log-mel features are commonly used by
serious audio researchers, this is an active area of research. Another audio
feature extraction technique called Per-Channel Energy Normalization
(PCEN) appears to perform better at least in some cases, like processing
far field audio. I haven’t had time to delve into the details yet, but
understanding it and porting it to the web also seems like a worthy task.

Major thanks to Dick Lyon for pointing out a few bugs in my feature
extraction code. Pick up his “Human and Machine Hearing” if you’re ready
to delve deeper into sound understanding.

Ok, so to recap, we’ve generated log-mel spectrogram images from streaming audio
that are ready to feed into a neural network. Oh yeah, the actual machine
learning part? That’s the next post.