Example 04 · CNN

Wake Word Detection

An always-on listener that wakes a device when it hears its name, without streaming the microphone anywhere. Our 14,818-parameter network listens for one word in every second of audio. We trained and tested it on real recordings from Google’s Speech Commands dataset, and tested it only on voices it never heard in training.

Read the write-up on GitHub

A smart speaker shown cut away: a MEMS microphone at 16 kHz feeds an always-on microcontroller running log-mel features and a CNN, which wakes a sleeping main processor only for the wake word. Beside it, what the device hears: 'Marvin', other words such as 'house', 'go' and 'stop', and background noise like a running tap. — The wake-word detector sits between the microphone and everything that costs power, and only wakes the main processor when it hears its word.

of wake words caught from voices it never heard, at under 1% false triggers: 95%
parameters in the whole network: 14,818
of flash, as 8-bit integers: 21 KB
of wake words caught on hours of continuous speech, at 6–8 false wake-ups an hour: ~90%

Where it's used

Listening all the time, on almost no power

A doorbell, a smart speaker or a hearing aid that responds to its name needs something listening all the time. Streaming the microphone to a server costs bandwidth, power and privacy, and keeping a full speech recognizer awake drains the battery.

A wake-word detector gets around both. It’s a small model on a low-power processor that checks the last second of audio for one word, and wakes the rest of the system only when it hears it.

A doorbell that answers to its name
A smart speaker that wakes only when it’s called
A hearing aid that responds when it hears its name
No audio has to leave the device to listen for the word

The data

Real voices, real recordings

We used Google’s Speech Commands dataset: one-second clips of single spoken words from thousands of people, recorded on their own devices. The wake word is “marvin”. The network learns from 1,710 clips of it, set against more than 9,000 clips of other words and background noise, so it learns to stay quiet through both.

We split the data by speaker, using the dataset’s own lists, so every voice in the test set is one the network has never heard. That is what a product shipped to new users faces.

The wake word is just a setting. Training for a different word is a rerun, not a rewrite.

Training · 38,344 clips: what the network learns from: whole clips, each “marvin” also shifted a little in time, and one-second windows cut from runs of words, the way a listener hears them
Validation · 10,020 clips: checked after each round of training, and used to set the decision point
Test · 11,048 clips: 195 “marvin” clips and 10,853 others, kept aside until the very end. The scores below come from these.

Waveforms and log-mel feature images for three one-second clips: the word 'marvin' with two voiced syllables, the word 'house' from the same speaker with one burst and a hissing ending, and background noise from a running tap that fills all bands evenly. — Three clips as the network hears them: the waveform on top and the sound features below. “marvin” and “house” are the same speaker.

How we built it

Sound in, one decision out

Two pieces work together. The first turns each second of audio into 49 short frames of 20 sound-energy bands, the same first step full speech recognizers use, and it runs identically on the device against the live microphone. The second is a small convolutional network that reads all 49 frames at once, as a picture of the second, and decides whether the clip was the wake word.

01
Frame
32 ms windows, every 20 ms: 49 per second
02
Transform
a 512-point FFT of each window
03
Bands
20 mel bands from 0 to 8 kHz, then the log of each
04
Normalize
every clip scaled the same way, whatever its volume

Pipeline: microphone at 16 kHz, frames of 512 samples every 320, Hann window, FFT and 20 log-mel bands, per-clip normalization; then the whole clip as a 49 by 20 image into three stages of 3 by 3 convolution with GELU and 2 by 2 max pooling, with 16, 32 and 32 filters, down to 6 by 2 by 32; a two-way softmax output; and a decision: wake word if its probability is at least the threshold. — The whole pipeline. The network sees the full second at once, time down and sound bands across, and makes its call in one step.

Loud or quiet, it hears the same thing. Real recordings vary enormously in volume and microphone quality. Each clip is normalized before the network sees it, which takes those differences out of the picture.

Results

95% of wake words, from voices it had never heard

We trained the network five times and tested every run on the same 11,048 clips from speakers it never heard in training: 195 of them “marvin”, and 10,853 other words and background noise. With the decision point set for one false trigger in a hundred clips, every run caught between 93.8% and 95.9% of the wake words.

Tuned to your product

Every listener balances two kinds of mistake: missing the wake word, and waking up when it shouldn’t. Raising the decision point trades one for the other, and each product wants that balance somewhere different. A doorbell can afford the odd false wake-up, and a device that starts recording when it wakes can’t.

We set the decision point on a separate set of 10,000 clips, so it lands where you need it and stays there on new voices. A recurrent network a third its size, listening frame by frame, comes close: at the same setting it misses about 9% of wake words, against 5%.

False-reject rate against false-accept rate, on a log scale, as the threshold moves from 0.05 to 0.95, for one CNN run and one GRU run. The CNN curve runs from 0.5 percent missed at 5.8 percent false accepts to 8.7 percent missed at 0.15 percent; the GRU curve lies just above it. Five black diamonds mark the five CNN runs at their validation-chosen thresholds, at 4 to 6 percent missed and 0.6 to 1.0 percent false accepts. — Missed wake words against false triggers, as the decision point moves. Lower and further left is better. The diamonds are our five runs at the point we chose for them.

Tested the way it listens

A device never hears neat one-second clips. So we also laid every test clip end to end into a 5.7-hour stream over background noise and slid the detector along it ten times a second, the way it runs in a product. It caught about 90% of the wake words, waking by mistake 6 to 8 times an hour.

Getting there took training on windows cut from runs of words, not just whole clips. The same network trained on clips alone scored almost as well per clip, and on the stream it woke by mistake dozens to hundreds of times an hour. A test that only uses clips would never have shown it.

A second check cuts the false wake-ups further: the device wakes only when a small recurrent network agrees, three windows in a row. That brought every run under one false wake-up an hour on the stream, while still catching about four wake words in five.

Missed wake words against false wake-ups per hour, on a log scale, on a 5.7-hour test stream at 20 dB signal-to-noise ratio. The CNN trained on whole clips only never gets below 25 false wake-ups per hour, where it misses 16 percent. The CNN trained with streaming windows reaches 3.7 per hour at 11 percent missed and 6.4 per hour at 9 percent; the GRU with streaming windows reaches lower rates only by missing far more. Four black diamonds mark four CNN runs at thresholds chosen on a validation stream, at 6 to 8 false wake-ups per hour and 9 to 14 percent missed. A fourth curve, the CNN confirmed by a GRU over three windows in a row, continues below 1 false wake-up per hour; five black squares mark its runs at 0.2 to 0.5 per hour and 17 to 28 percent missed. — On a continuous stream: missed wake words against false wake-ups per hour. Lower and further left is better. The dashed line is the same network trained on whole clips alone; the green line adds the second check.

On the device

An always-on listener on a microcontroller

On the device, the same feature code runs against the microphone buffer and our library runs the network. Each second of audio takes 49 small FFTs and about 1.8 million multiply-adds, and a live listener slides that one-second window along the stream.

Our tools convert a trained model to 8-bit integers, with no measurable loss of accuracy, and to a format that runs straight out of flash: 21 KB in all. Where RAM is tighter, a smaller layout fits in 21 KiB, at the cost of missing more wake words.

microphone audio, read a second at a time: 16 kHz
multiply-adds for each second of audio the network checks: ~1.8M
of flash as 8-bit integers, with no loss of accuracy: 21 KB
of RAM, which a 128 KiB microcontroller fits: 78 KiB

Get the code and the full write-up on GitHub

Want this in your product?

We build embedded AI like this into medical and industrial devices, from the sensors and firmware to the network running on the chip. Tell us what yours needs to recognize.

Let's talk about embedded AI See all eight examples

Founded in 2008 · Hudson, Ohio · info@coledd.com