Wake Up Browser · engineering notes

Browsing by voice in a room where people talk, without a push-to-talk key or the cloud

Saying back, next page or a link's number to a browser sounds like a weekend project. It is not, because a microphone that is always listening in a room with people and a TV in it will press keys nobody asked for. I built a small Windows tray app that opens the microphone only when it can help, recognises a fixed set of phrases offline, and acts only when an utterance is exactly one command. These notes are about how it decides, and what it got wrong first.

The short version
35commits
11working days, September to November 2026
339unit tests, three running synthesised speech through the real recogniser
29 of 29steps passed in the last live run against a real Chrome window
69commands in the default set
15,909lines of Python across 65 modules

01Why I built it

Browsing is a steady stream of small actions: go back, open a tab, return to the tab you were just on, scroll, pick a link, turn to the next page of results. I wanted to do those without reaching for the keyboard, while still being able to talk to someone in the same room.

The usual tools take one of two routes. A push-to-talk key defeats the point of being hands-free. A cloud recogniser that tries to understand everything is slower, sends audio off the machine, and is generous about what it thinks it heard. I went the other way: a small, fixed vocabulary recognised offline, and a microphone that is open only when it could be useful.

02Architecture at a glance

It is a Python tray app with a Qt window. The speech side, the browser side and the interface run on separate threads, and the interface hears about everything through one channel of events, so a slow page never freezes the window.

Listening gateDecides, twenty times a second, whether the microphone should be open at all, and stands aside while another dictation app is in use.
Recogniser and endpointerVosk, an offline recogniser, fed by a loudness-based endpointer that decides when an utterance has ended.
Live grammarAn editable TOML file of commands, narrowed at each moment to the phrases that make sense on the current page.
Conversation guard and journalDecides whether a heard command was meant, and records every decision with the features it used.
Browser driverKeyboard shortcuts through Windows SendInput, and tabs, links and scrolling through Windows UI Automation.
Saved pagesRemember this stores a page's address, title and text in a local SQLite archive with full-text search.
Installer and doctorFinds or installs a suitable Python, pins every package, downloads the model, and repairs only what is missing on a second run.

Two rules hold the shape together. Everything that talks to Windows UI Automation happens on one worker thread, because those objects released on the wrong thread crash the app without a word. And the Qt interface is touched only from its own thread.

03The gate decides the microphone

Most voice tools listen all the time and decide afterwards whether to act. I wanted the decision earlier. By default, the microphone stream opens only while a browser is the active window, the mouse or keyboard was used in the last thirty seconds, and the pointer is over the browser. Otherwise the stream is closed, not just ignored, and the Windows microphone indicator goes away.

The gate also stands aside while another dictation app's push-to-talk key is held, or while any other program is recording. Windows keeps a record of which programs are using the microphone for its privacy indicator, and the gate reads it.

For rooms with a TV there is an optional wake. With it on, commands work only for a short window after a spoken wake phrase, and each command carried out restarts the window. While the wake is on, the microphone also listens away from the browser, but only for the wake.

04One utterance, one phrase

The central rule is that an utterance fires only if every word is in the grammar and the words exactly equal one phrase. Go back to the shop contains go back, and does nothing. The recogniser's partial guesses are never acted on.

The grammar is live. It holds the commands that make sense right now, plus 104 common catch words and a token for anything else. YouTube commands exist only on YouTube, the names of open tabs become sayable after go to, and words like faster and up join only while the page is scrolling. The grammar is rebuilt in about two milliseconds and swapped in between utterances, never during one.

The catch words matter more than they look. Without them, a short word in ordinary speech gets forced onto the nearest command. With them, it has somewhere else to go.

Speed came from the endpointer. It ends an utterance after a quarter second of quiet. When the words already form a phrase that nothing longer continues, it waits only 120 ms. In the project's own recorded test that halved the typical response, to about 110 ms.

05Telling a command from conversation

Whole-utterance matching stops most accidents. But in talk, a word like stop or back sometimes lands alone between two pauses, and the recogniser hears it perfectly. The question is no longer what was said, but whether it was meant.

My own log of real use answered that. Real commands followed long quiet, and only a small share had other speech in the seconds before them. The accidental fires came out of runs of talk with almost no quiet on either side. So during a conversation, a command must stand alone, be about as loud as my usual commands, and not be followed by more speech.

In the project's own replay of 355 logged commands, 346 still fired at once and three of four accidental fires were stopped. An undo command is the safety net for the rest.

Every decision is journalled with the features it used: the quiet before and after, the loudness against my usual level, recent mouse or keyboard use. Review tools replay the journal under nudged settings and report which change would fix the marked mistakes without causing new ones. A learned model trained on the same journal runs in the shadow and decides nothing until it beats the hand-written rules on enough marked decisions.

06Noise, and the TV across the room

I built a simulator that mixes my recorded commands with hiss, a fan, a TV and a party at chosen levels, and ran every idea through it before shipping it.

A noise suppressor helped with hiss and fans and hurt with a TV or other voices, because it keeps all voices and blurs mine. Always on, it also cost a few of my real commands in a quiet room. So it switches itself on only for steady noise, judged by how much the room's level rises and falls between words, and only between utterances.

Other voices needed a different answer. My microphone is close and the TV is across the room, so while other voices are detected, quieter speech is ignored. In the simulator, with a TV at a tenth of my level, commands heard alone went from 4 of 30 to 16 of 30.

07Driving the browser without an extension

Nothing is installed in the browser. Most commands press the browser's own shortcuts through Windows SendInput, exactly as a keyboard would. Anything that needs to know what is on screen uses Windows UI Automation, the interface screen readers use.

Through it the app reads the tab strip. Last tab returns to the previous tab even in another window, and go to followed by a word finds a tab by its title, its site or a nickname. Ordinary browsing never reads a page's content; only the commands that need it do, such as show links or remember this.

Scrolling taught me that the wheel cannot glide. In Chrome a notch moves 100 pixels and a smaller step moves nothing, so reading at a steady pace had to set the page's scroll position directly instead.

Show links asks the page once for everything clickable and puts a numbered badge beside each, up to 199. Saying a number or a word from the link clicks it. While the numbers are up, they own the grammar: only numbers, the offered words and a few controls are sayable. In the project's recorded test, numbered links were picked correctly 99.5% of the time across five voices.

Next link walks a list of results: open the first item, read it, say next link, and the app opens the second. That needs to know which page was the list. A page map remembers, for each tab, the pages it showed and the items on each list page, so the app can step back to the list and open the item after the one you last read.

09How it is tested

There are 339 unit tests, mostly pure logic. Three synthesise speech with the Windows text-to-speech voices and run it through the real recogniser. Tests never press real keys and never touch the real saved-pages archive.

A live end-to-end check serves five test pages from a local server, opens them in a new Chrome window, and feeds synthesised speech into the real engine with the microphone replaced. It covers tabs, muting, saved pages, scrolling, show links, next link and ignoring ordinary sentences. It last passed 29 of 29.

The recognition figures come from a recorded test set of 1,520 clips: every command phrase in five Windows voices at three speeds, plus 920 things that must not fire. In that test, 99.8% of commands were recognised and about 1% of natural sentences fired one. Chrome is verified live; Edge shares Chrome's tab strip but has not been run live, and the Firefox path is untested.

10Where it stands

It is at version 1.0.0 and in regular use on Windows with Chrome. It uses well under one percent of a CPU core when not listening and about two percent while listening.

Two optional parts sit around the core. The wake suits rooms with a TV. An assistant can be called up by voice to answer a spoken question from a local model or a hosted one, in a searchable chat window. Neither changes the core rule that a browser command fires only when the whole utterance is that command.

11Ten lessons

Almost every one of these was found in a log of real use, not in a test. The fix was usually small; the finding was the work.

01A steady noise held the endpointer open

The endpointer decides an utterance has ended when the sound drops back toward the room's noise floor. With a steady noise like a fan, the floor estimate drifted up with it, and the endpointer could stay convinced someone was still speaking.

Now: The noise floor is the quietest block of the last three seconds, so a steady noise cannot latch it, and a unit test pins it.

a baseline that follows the signal cannot tell you when the signal stopped.

02The speech level, not the wording

Commands in my own voice were missed far more often than synthesised ones, and I assumed the phrases were badly chosen. A tuning session showed otherwise. A third of my takes never crossed the loudness threshold, so they were only ended by the recogniser's own slow endpoint.

Now: The threshold was lowered and the pause retuned. In that session, commands heard exactly went from 68% to 96%. Wording mattered once: mute tab was heard as new tab, so mute this tab comes first.

measure the input before tuning the vocabulary.

03Scroll up failed eleven times, and it was not the wording

A log showed scroll up misheard eleven times in a row as other words. The phrase was fine. A simplified mode was on, and scroll up was not part of it, so it was not in the grammar at all and the recogniser had to pick the nearest thing that was.

Now: The simplified mode includes scrolling and page movement.

when a recogniser keeps mishearing one phrase, first check that the phrase is in what it is allowed to hear.

04Forty finished early, and forty two clicked forty

Early finishing ends an utterance quickly when the words already form a complete phrase. Among the link numbers, forty is a complete phrase, so forty two clicked link forty. Picking among links 100 to 199 was right only 95% of the time.

Now: Only phrases that nothing longer continues may finish early. In the recorded test, links 100 to 199 went to 99.5%.

a shortcut that is right for the words alone can be wrong for what follows them.

05A common word in a command removed it from the catch list

Catch words give ordinary speech somewhere to go. A word cannot be both a catch word and part of a command, so a new phrase containing the quietly removed it from the catch list, and false fires rose.

Now: A test pins the catch list, and the phrase that caused it was dropped.

adding to a vocabulary can take something away from it.

06The first conversation guard took the TV for a conversation

The first guard worked in a quiet room. In the noise simulator, with a TV playing, it treated the TV as ongoing talk and refused real commands, doing worse than no guard at all.

Now: The second version weighs loudness against my usual level and my own recent mouse or keyboard use. In the simulator it held its quiet-room result and recovered most of what the first guard lost to the TV.

test a guard in the conditions it exists for, not only the ones it was written in.

07Chrome hid the page it had just opened

Next page worked once and then every page command failed. A page opened from a link the app had clicked got a new render window that Chrome left out of its accessibility tree entirely, for as long as you waited. Focus changes did not wake it.

Now: When no document is found, the app asks what is at a point on the page, which makes Chrome build the tree. It worked five turns in a row.

an accessibility tree is built on demand, so sometimes you have to ask before there is anything to read.

08Released on the wrong thread, the app died silently

UI Automation objects created on the worker thread and garbage-collected on another released their Windows COM handles in the wrong place. The result was an access violation with no message and no log.

Now: Those objects are handed back and released on the thread that made them, and a test pins the shutdown path.

an object with thread affinity has to be destroyed where it was born, not wherever the garbage collector happens to be.

09A settings change that looked like a freeze

Changing a setting sometimes froze the window for nearly four seconds. The window was waiting on the engine's lock while the engine stopped and restarted things under it.

Now: Nothing slow happens under the lock, and the interface never waits on it. A settings change takes milliseconds.

a lock held by slow work turns every waiter into a hang.

10A wake sound that woke for the wrong reasons

The first wake was two tongue clicks, trained on my own recordings. Two ordinary taps a fraction of a second apart were taken for clicks while another program was in front, and a go back fired twelve seconds later with nobody noticing. The laptop's own microphone filter also stripped the clicks out.

Now: A badge shows whenever the wake window is open, and a spoken wake phrase became the main option. In its recorded test it was heard 10 of 10 times, with no false wake in 45 minutes of recorded talk or in 200 sound-alike sentences.

a trigger that can fire unseen needs to be visible when it does.

12What I would do differently

The app works and I use it. If I started again, I would change the order I did things in more than the design.