Wiring Android's ML Kit Barcode Scanner to a On-Device LLM for Real-Time Product Intelligence

---
title: "Android Barcode Scanner + On-Device LLM: The Latency Budget That Ships"
published: true
description: "Wire ML Kit barcode detection into MediaPipe LLM Inference with frame throttling and structured JSON output. P95 latency targets for Pixel 9."
tags: [kotlin, android, mobile, architecture]
canonical_url: https://blog.mvpfactory.co/android-barcode-scanner-on-device-llm-latency-budget
---

## What We Are Building

By the end of this walkthrough, you will have a working Android pipeline that chains ML Kit's barcode scanner directly into MediaPipe LLM Inference — no round-trips through a remote API, no main-thread bottlenecks, and a P95 end-to-end latency under 500ms on a Pixel 9. You will also understand *why* naive implementations blow past 700ms and exactly which three decisions pull you back under budget.

**Prerequisites:** Android Studio Hedgehog or later, a physical device (emulator GPU delegation is unreliable for profiling), familiarity with CameraX and Kotlin coroutines.

---

## The Architecture Problem Nobody Talks About

Let me show you a pattern I use in every project that involves on-device vision-to-language pipelines: treat the vision stage and the language stage as a *single pipeline on a single background dispatcher*, not as two modules talking through the main thread.

Most teams get this wrong. They shuttle barcode results back to the main thread before dispatching to the LLM. Here is what that costs you on a Pixel 9 with Tensor G4:

| Stage | Naive (main-thread hops) | Optimized (background dispatch) |
|---|---|---|
| Camera frame capture | ~16ms | ~16ms |
| ML Kit barcode decode | ~35ms | ~35ms |
| Main-thread context switches | ~80ms | ~0ms |
| LLM prompt construction | ~8ms | ~5ms |
| Time to first token (TTFT) | ~520ms | ~310ms |
| **P95 end-to-end** | **~720ms** | **~410ms** |

That 300ms gap is the difference between a product that feels instant and one that feels broken.

---

## Step 1: Throttle Your Frame Analysis

CameraX's `ImageAnalysis` will happily deliver frames at 30–60fps. Your barcode scanner does not need them. Here is the minimal setup to get this working with an 8fps ceiling:

`STRATEGY_KEEP_ONLY_LATEST` drops frames the analyzer cannot keep up with instead of queuing them. The 120ms gate enforces your fps ceiling explicitly. This cuts redundant ML Kit invocations under typical handheld scanning conditions and — critically — preserves the thermal headroom your LLM inference stage depends on.

---

## Step 2: Eliminate the Main-Thread Hop

Here is the gotcha that will save you hours: the silent latency killer is not raw compute, it is thread choreography. Barcode output from ML Kit is a plain string — there is no GPU tensor to share — but naive implementations post results back to the main thread before dispatching to the LLM, accumulating 60–80ms of context-switch overhead that compounds at the P95 tail.

Keep your result handling on `Dispatchers.Default`, build your structured prompt immediately in that context, and dispatch directly to LLM inference:

The docs do not mention this, but GPU delegation at LLM initialization is the other half of this stage — without it, inference falls back to CPU and the latency budget collapses entirely:

---

## Step 3: Enforce Structured Output From Day One

This is the architecture decision most teams defer too long. Unconstrained LLM output forces a parse-and-retry loop that blows your latency budget on the 10–15% of responses that deviate from schema.

MediaPipe LLM Inference API does not yet expose first-class grammar constraints the way llama.cpp does — so prompt engineering carries the full load. The schema injection pattern above (schema in prompt body + `JSON:` suffix) achieves consistent schema-valid output on Gemma 2 2B with temperature ≤ 0.2.

Here is how the models stack up on Pixel 9:

| Model | Avg TTFT | Schema validity | Recommended temp |
|---|---|---|---|
| Gemma 2 2B (INT4) | ~280ms | ~94% | 0.1–0.2 |
| Gemma 2 2B (FP16) | ~480ms | ~96% | 0.1–0.2 |
| Gemma 3 1B (INT4) | ~190ms | ~89% | 0.1 only* |

\* Gemma 3 1B exhibits a higher hallucination rate at this parameter scale; temperatures above 0.1 measurably increase schema deviation in structured output tasks.

*Measured on Pixel 9 (Tensor G4), MediaPipe 0.10.x, cold-start excluded, 50-token output cap, n=200 prompts per model.*

For P95 targets, INT4 quantized Gemma 2 2B is the practical sweet spot on Pixel 9.

---

## Gotchas

**Thermal throttling invalidates your benchmarks.** Run sustained load tests, not cold-start microbenchmarks. A 410ms P95 measured over 10 prompts will look nothing like production after 2 minutes of continuous scanning.

**`withContext(Main)` anywhere between barcode decode and LLM dispatch destroys your P95.** It is invisible in unit tests and brutal in production traces. Audit your coroutine chain before you ship.

**`setMaxTokens(50)` is not optional.** MediaPipe LLM Inference will generate until EOS if uncapped. For a three-field JSON schema, 50 tokens is sufficient and prevents runaway generation from blowing your latency ceiling.

**Do not profile on an emulator.** GPU delegation falls back silently on many emulator configurations. Every number in this post was measured on physical hardware.

---

## The P95 Ceiling That Matters

A user pointing a phone at a product barcode has approximately 600–800ms of patience before the UX registers as slow. The optimized pipeline — 8fps frame throttling, no unnecessary thread hops, INT4 Gemma 2 2B with GPU delegation — lands at roughly **410ms P95** on Pixel 9. That is your shipping target. Budget the remaining 290ms for UI rendering, network enrichment calls, or caching misses.

Three decisions get you there: throttle frames aggressively, never post barcode results to the main thread before LLM dispatch, and enforce structured JSON output via prompt schema injection from the first sprint. Skip any one of them and you ship a frustrating UX instead of a snappy one.

**Relevant docs:** [ML Kit Barcode Scanning](https://developers.google.com/ml-kit/vision/barcode-scanning/android) · [MediaPipe LLM Inference API](https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android) · [CameraX ImageAnalysis](https://developer.android.com/training/camerax/analyze)