Earshot: a walking guide that lives in your pocket (Gemma on-device, no signal needed)

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Every "explore your city" app I've used has the same problem: it wants you to look at it. You end up walking through a beautiful old town staring at a blue dot.

Earshot flips that around. It's a walking guide where the phone goes in your pocket and stays there. Your guide is Wren, a friendly voice in your earbuds. Under the hood, Wren is Google's open-weight Gemma model running on your phone.

1. Plan (one minute, while you still have Wi-Fi). Pick "around me" or type a place. Earshot pulls the interesting corners within a short walk from OpenStreetMap and Wikipedia: forts, old baths, cathedrals, a famous tree, a memorial. Then Wren (Gemma, running on your own device) writes a short spoken piece about each one.
2. Walk (no screen, no signal needed). Put your earbuds in and your phone away. When GPS says you've reached a place, Earshot tells you which side it's on and talks about it for about 20 seconds. Every narration ends with a small physical invitation: look up at the carved door, put your hand on the coral stone, listen for the call to prayer.
3. In between, when it's been quiet for a few minutes, it nudges you to notice something: three shades of green, the smell of the sea, the sound under your feet.

The screen is the shortest part of the walk. The walking screen is deliberately all black, with one pulsing dot.

Who it's for: anyone who wants to see a neighbourhood rather than scroll it. That includes visitors in a new town, locals who've walked past the same building for years without knowing its story, and people walking somewhere with bad or expensive mobile data.

Demo

🔗 Try it: https://alphaolomi.github.io/earshot/ (Chrome or Edge on desktop, or Chrome on Android 12+)

No time to go outside? Plan a walk anywhere (try "Stone Town, Zanzibar"), then tap Preview it from here. It fast-forwards a stroll past every place so you can hear the whole tour at your desk.

Code

flowchart LR
  subgraph PLAN["PLAN · online, about a minute"]
    direction TB
    OSM["OpenStreetMap<br/>(Overpass mirrors)"] --> P["Places + facts"]
    WIKI["Wikipedia<br/>geosearch"] --> P
    P --> G["<b>Wren</b> = Gemma, on the phone<br/>via WebLLM + WebGPU<br/>rewrites the facts

Plain JavaScript and Vite, about 800 lines, MIT licensed. No backend, no API keys, no accounts.

How I Built It

Diagram made with Mermaid.

The open pieces doing the work:

• Gemma 2 2B Instruct (open weights, 4-bit quantized) as the default narrator, with Gemma 3 1B as a "lite" option for older phones. Both run through WebLLM on WebGPU, inside a Web Worker so the UI stays smooth. The weights download once (1.5 GB or 540 MB), then load from the browser cache.
• OpenStreetMap through the Overpass API for places and their tags (historic, heritage, start_date, inscription, tree species…).
• Wikipedia's geosearch API for nearby articles and their intro paragraphs.
• The browser's built-in Web Speech API for the voice, and Geolocation + Screen Wake Lock for the walk.

Grounding a small model. Gemma is never asked to know anything about a place. It's asked to rewrite the facts I hand it (OSM tags plus the Wikipedia intro) as two spoken sentences, then add one sentence inviting you to look at, touch or listen to something. Then a check: every number in the narration (years, heights) must appear in the source text. If Gemma says "built in 1878" and Wikipedia says 1888, it retries at low temperature, then falls back to reading the facts plainly. Places with no facts get a different prompt: "state no facts, just tell them what to look for."

Things that bit me (and might save you time):

• Gemma 3 in WebLLM refused to load: Only one of context_window_size and sliding_window_size can be positive. WebLLM's prebuilt record sets a 4096 context, while the weights declare a 512-token sliding window. My first fix, turning the sliding window off, loaded fine but produced looping nonsense ("Okay, here's what I'll say. Okay, here's what I'll say…"). The right fix was the opposite: { context_window_size: -1, attention_sink_size: 0 }.
• …which means short prompts matter a lot. With a 512-token window, my first prompt (rules + a worked example + a Wikipedia paragraph) pushed the instructions out of view. Gemma 3 1B confidently described a 19th-century Persian bath house as "the museum of the Great Pyramid." Cutting the prompt to one line, "Rewrite these facts as 2 short spoken sentences…", made the same model accurate.
• A 295 MB weight shard. Gemma 2 2B's vocabulary embedding is one big file, and some browsers refuse to put it in Cache Storage. Earshot retries, then switches WebLLM's cacheBackend to IndexedDB. (Watch out: the option was renamed from useIndexedDBCache in recent versions, and the old name is silently ignored.)
• Public Overpass servers are flaky. On my build day, the main one was timing out and the mirrors were returning 500s. Earshot now queries several mirrors in parallel and Wikipedia's geosearch, merges the results, and dedupes places mapped twice ("Freddy Mercury House" vs "Freddie Mercury's House"). One OSM node for Zanzibar's Old Slave Market even linked to the Wikipedia article for a slave market in South Carolina. Earshot now ignores Wikipedia links whose coordinates are more than 2 km from the place.
• People don't want to meet a model ID. Early versions said things like gemma-2-2b-it-q4f16_1-MLC running on your GPU. Now the app has one persona, Wren, who introduces itself and talks you through every step ("First time we've met! I'm settling into your phone…"). The settings say "Full" or "Light" storytelling instead of model names.
• Mobile browsers pause GPS when the screen sleeps. So Earshot holds a Screen Wake Lock and shows a pure-black screen: cheap on OLED, invisible in a pocket. It isn't elegant, but it works without a native app.

Why Does Open Innovation Matter?

For this project it isn't a nice-to-have. Without open weights, the idea doesn't work.

• It works with no signal. The narration is generated before you leave and saved on the device. During the walk there's no network call at all, so it works in the old town's narrow alleys, on a ferry, or on a SIM with no data left. With a closed API, every stop would need a round-trip to someone's server.
• Your location stays with you. A walking app knows where you are every few seconds for an hour. With Earshot, that trace never leaves the phone. While planning, only a coarse (~100 m) point goes to OpenStreetMap and Wikipedia, both open, non-profit projects. There's no account and no server of mine in the loop.
• It costs nothing to run. No API key, no per-token bill, no backend. That's why I can publish it as a static site and not worry if a thousand people try it. In places where mobile data is expensive, "download the model once on Wi-Fi" matters a lot.
• I could swap the brain, and see inside it. Earshot ships two Gemmas: 2B for quality, 1B for older phones, chosen from a dropdown. Swapping is one string. When the 1B model misbehaved, I could open mlc-chat-config.json, find the 512-token sliding window, and design the prompts around it. A closed API would have given me a black box and a bill.
• The data is open too. Every place Earshot knows about was mapped or written by a volunteer. If a building in your neighbourhood is missing, you can add it to OpenStreetMap, and Earshot will talk about it on the next walk.

Prize Categories

• Best Use of Gemma: Gemma 2 2B and Gemma 3 1B (open weights) run entirely on-device through WebLLM/WebGPU and write every narration. No cloud model is involved at any point.