LLM Food Recognition in Production: What Shipping soba Taught Me

TL;DR

I spent the last few months building soba, an iOS app that photographs a meal and returns carbs, glycemic index, and portion weights for people who count carbohydrates. The recognition backend went through one model migration, one full prompt rewrite, and a stack of validation code. Three things carried almost all of the improvement: picking the model with a benchmark instead of vibes, forcing strict JSON Schema through OpenRouter, and rewriting the prompt around scene scale rather than food identity. Model choice turned out to be the smallest lever of the three. Every model I tested misjudges portion weight by 25–35%, so the UX for correcting weight beats another round of model shopping.

What soba does

The app is aimed at people with type 1 diabetes: carb counting several times a day, every day. You point the camera at a plate, and a few seconds later you get a list of items with weights, net carbs per 100 g, and a glycemic index estimate. There’s a barcode path for packaged food too. Under the hood it’s a Go server (internal/recognize/) that sends the photo to a multimodal LLM and turns the answer into something a diabetic can act on.

That framing shaped every technical decision. A person dosing insulin needs grams of digestible carbohydrate, an honest error bar, and no surprises when the model has a bad day.

Picking the model: benchmark first, then a live A/B

I picked the starting model off a benchmark instead of marketing pages. DiningBench evaluates 29 models on 3,021 dishes for exactly this task. Gemini-3-Pro led both fine-grained dish classification and nutrition estimation, with a 24.45% nutrition MAPE against 32.17% for GPT-5. That settled the starting lineup: soba launched on gemini-3.1-pro-preview.

Then I A/B-tested gemini-3.6-flash against the pro model on the live recognition path. Same prompt, same photos. Quality differences never rose above the noise floor of gram estimates. What did change:

A scan that costs 58% less and returns in as little as a third of the time, at the same practical accuracy, was an easy call. The lifecycle column mattered more than it looks, too: preview models get deprecated whenever the vendor decides, and a health-adjacent app can’t afford to wake up to a dead endpoint.

All of these models (pro, flash, GPT) miss portion weight by 25–35%. That error dwarfs any gap between frontier models on food identity. The accuracy budget went into scale anchoring in the prompt and a weight-editing UX in the app, and I stopped following model releases with any anxiety — the opposite of what I found comparing frontier models for coding, where the gaps are real. A multi-dataset study in Nutrients reports the same pattern, with prompt design moving nutrition accuracy as much as a model swap, and portion size has been the weak link in photo-based estimation since long before LLMs.

One more model decision that saves real money: not every subtask deserves the full recognition call. Estimating the GI of a barcode-scanned product is a lookup against well-known food knowledge, so it runs on the fast flash model at temperature 0 and costs a rounding error compared to a photo scan.

The API call, hardened

Everything goes through OpenRouter, and the request body carries most of the production hardening. Here’s the shape of it, abbreviated:

{
  "models": ["google/gemini-3.6-flash", "google/gemini-3.5-flash", "openai/gpt-5.1"],
  "messages": [
    { "role": "system", "content": "<recognition prompt, below>" },
    { "role": "user", "content": [
      { "type": "text", "text": "Item names and notes language: German.\nThe photo is approximately 31 cm wide at the distance of the food.\nAnalyze this photo." },
      { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,..." } }
    ]}
  ],
  "temperature": 0.2,
  "reasoning": { "effort": "low" },
  "response_format": {
    "type": "json_schema",
    "json_schema": { "name": "meal", "strict": true, "schema": { "...": "..." } }
  },
  "provider": { "require_parameters": true, "data_collection": "deny" },
  "usage": { "include": true },
  "user": "u_9f3ab2c1"
}

Line by line, the parts that earn their place:

The models array is the whole failover story. Primary is gemini-3.6-flash; if Google has an incident, OpenRouter walks down to gemini-3.5-flash and then across vendors to openai/gpt-5.1. One line of config replaces a routing layer I never had to write, and a provider outage doesn’t reach the user.

Structured outputs with strict: true mean the response is guaranteed to match my JSON Schema, so parsing is literally json.Unmarshal and nothing else. Strict mode has sharp edges: every field must be listed in required, and every object needs additionalProperties: false. Miss either and the request is rejected before any tokens flow. The abbreviated schema for a meal item looks like this:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["name", "name_en", "grams", "grams_min", "grams_max",
               "weight_basis", "carbs", "fiber", "protein", "fat",
               "kcal", "gi", "gi_confidence"],
  "properties": {
    "grams":      { "type": "number" },
    "grams_min":  { "type": "number" },
    "grams_max":  { "type": "number" },
    "weight_basis": { "type": "string" },
    "gi":         { "type": ["number", "null"] },
    "gi_confidence": { "type": "string", "enum": ["high", "medium", "low"] }
  }
}

provider.require_parameters: true closes the gap the schema leaves open: it makes OpenRouter route only to endpoints that actually support json_schema. Without it, your request can land on a provider that silently ignores the schema and returns prose, and you’ll waste an evening blaming the model for a routing problem.

reasoning.effort: "low" is the setting I’d check first on any cost surprise. Gemini 3.x and GPT-5.x burn hidden reasoning tokens that bill as output, and identifying rice doesn’t need chain-of-thought (my post on efficient LLM reasoning covers why reasoning-token budgets got out of hand in the first place). Left at default, this setting alone can double the scan price.

Food photos are health data, so provider.data_collection: "deny" opts them out of provider-side training.

usage.include: true asks OpenRouter to attach the request’s exact cost to the response (newer API revisions include the usage block by default). Every scan’s cost is written to the database, and the server has a daily budget kill switch sitting on top of those numbers. When something goes wrong (a retry loop, a pricing change upstream) the stop-loss trips before the invoice does.

And user carries a stable pseudonymous ID so OpenRouter’s abuse detection bans an abusive account rather than my whole API key.

On the client side, photos get downsampled to 1024 px via ImageIO before upload. They land at 100–250 KB of JPEG. I tested bigger inputs and they bought nothing: gram accuracy is limited by scene scale ambiguity, and more pixels don’t resolve that.

The prompt: scale first, honesty second

The first version of my prompt read like everyone’s first food prompt (“identify the foods in this photo and estimate nutrition”) and produced confident nonsense. The rewrite starts from the observation the A/B forced on me. The dominant error is portion weight. Weight is unguessable without scene scale. So the system prompt now opens by assigning the model a job and a method, in that order. Verbatim:

You are a clinical nutrition analyst helping a person with type 1 diabetes count carbohydrates. Analyze the food photo and identify every distinct edible item (dishes, single products, packaged foods).

Then the scale hierarchy, the part that did the most work:

Work scale-first:

1. Anchor the physical scale. If the user message includes a depth-camera width measurement, it is the primary reference. Otherwise infer scale from objects of known size: dinner plate 24-26 cm, salad/dessert plate 20 cm, bowl 15-18 cm, fork/knife 19-21 cm, tablespoon 20 cm, glass 7-8 cm wide, credit card 8.6 cm, adult palm ~9 cm wide.
2. From that anchor estimate each item’s covered area and layer height, then weight = volume x typical density of the cooked food.
3. For countable pieces (eggs, bread slices, syrniki, cookies, dumplings, sushi) prefer count x typical unit weight (egg ~55 g, bread slice 30-40 g, pelmeni ~12 g, syrnik ~60 g) over volume reasoning.

On devices with LiDAR or stereo depth, the app measures the physical width of the frame at the food’s distance and passes it in. Everything else in the hierarchy is a fallback for when hardware can’t help. And rule 3 exists because counting beats volume math. “2 eggs” lands within a few grams; “a pile of scrambled egg” doesn’t.

Two fields in the schema exist purely to keep the model honest. weight_basis forces it to name the anchor it actually used: “plate 26 cm, rice layer 2 cm”. That disciplines the estimate and makes failures debuggable, because you can see exactly where the reasoning slipped. And instead of one confident number, the model returns a grams_min/grams_max bracket with instructions to earn its width:

Keep it tight where the weight is nearly certain (label or package weight, countable pieces: 2 eggs, 3 slices) and widen it where it is not (a mound of rice seen from above, an opaque bowl, a sauce, a partly hidden portion, no scale anchor in frame). … Do not default to a fixed percentage and do not collapse the bracket to hide uncertainty — an interval that turns out to contain the true weight is worth more to this user than a confident single number.

The rest of the prompt is domain policy, compressed. Effort goes where errors hurt: weight accuracy is prioritized for carb-dense items (bread, rice, pasta, sweets), while butter and greens get rough estimates by design. Nutrition comes back per 100 g in the as-served state (cooked pasta, not dry), with carbs meaning net digestible carbs and fiber in its own field. GI is nullable with an explicit ban on inventing numbers, plus a gi_confidence grade. If a nutrition label is visible in the frame, transcription beats vision, with fiber subtracted from carbs for US-style labels. And the model never gives insulin dosing advice; that line is not negotiable in a diabetes app.

The system prompt sets the method; a dynamically assembled user message carries each scan’s variables. Four moving parts, composed per request:

Item names and notes language: German.
The photo is approximately 31 cm wide at the distance of the food.
User hint: this is buckwheat, not rice.
Analyze this photo.

Language first (without it, a German grocery shelf comes back with English item names), then the depth measurement when hardware provides one, then a free-text hint that the prompt instructs the model to trust over its own eyes. When there’s no photo at all, the same pipeline accepts a text description of the meal (“two syrniki and a bowl of soup”) and estimates typical portions from words alone.

Validation catches what the schema can’t

JSON Schema only guarantees the shape of the response; a perfectly well-formed answer can still claim 150 g of carbs per 100 g of product. The Go server runs its own validator after unmarshal: carbs ≤ 100 g/100 g, macro sum ≤ 105 g/100 g, GI within [0, 110], and calories cross-checked against the Atwater formula:

atwater := 4*it.Carbs + 4*it.Protein + 9*it.Fat + 2*it.Fiber
if it.Kcal > 0 && math.Abs(it.Kcal-atwater)/atwater > 0.25 {
    it.Kcal = math.Round(atwater) // silently recompute — don't fail the scan
}
if it.GramsMin > it.GramsMax {
    it.GramsMin, it.GramsMax = it.GramsMax, it.GramsMin
}

The validator repairs instead of rejecting wherever it can. An inverted weight bracket gets swapped. A grams value outside its own bracket gets clamped. A meaningless bracket narrower than a gram gets zeroed out. The user paid for this scan with a real LLM call, so it never fails over a decorative field.

Violations are tiered by severity. Hard nonsense like carbs at 150 g/100 g triggers one retry with feedback: the model receives “Your previous answer was invalid: . Return corrected JSON” and a second chance. In practice one feedback round is enough: a paid scan almost never dies on a bad first answer. Cosmetic issues, like a word where the emoji field should be, just get cleaned up server-side without spending tokens on them.

There’s plumbing around the edges that earns its keep, too. Client retries are idempotent via an X-Scan-ID UUID that lives for the scan’s whole lifecycle, so a network error after the paid LLM call can’t bill twice. Local development runs on MOCK_LLM=1, a deterministic mock that mirrors the real model’s heuristics, so tests and UI iterations burn zero tokens. This kind of contract-first setup will be familiar if you’ve done structured document extraction; the failure modes rhyme even when the domain doesn’t. And the broader discipline of packing exactly the right variables into each request — nothing more — is the same instinct I described in context engineering.

FAQ

Which LLM is best for food recognition?

On DiningBench (3,021 dishes, 29 models), Gemini-3-Pro leads nutrition estimation with 24.45% MAPE versus 32.17% for GPT-5. In my production A/B, though, gemini-3.6-flash matched the pro model within the noise of gram estimates at 58% lower cost — for this task, the flash tier is the sweet spot.

How accurate are LLMs at estimating food weight from a photo?

Portion weight is where the pipeline is weakest: every model I tested lands 25–35% off. A depth-camera scale reference, known-size objects in frame, and counting pieces instead of estimating volume all narrow the error, but an editable weight in the UI is still the most reliable fix.

How do you get strict JSON from Gemini through OpenRouter?

Use response_format with json_schema and strict: true, put every field in required, set additionalProperties: false on each object, and add provider.require_parameters: true so the request only routes to endpoints that honor schemas. The response then parses with a plain json.Unmarshal.

Can an LLM read nutrition labels from a photo?

Yes, and it should — my prompt makes a visible label override visual guessing entirely, the same way transcription beats inference in document OCR pipelines. The one trap is US-style labels counting total carbs: the prompt has to spell out subtracting fiber to get net carbs, or packaged foods come back inflated.

Sources

• DiningBench — arXiv:2604.10425 — the 29-model food recognition benchmark that picked my starting model
• soba on the App Store — the app all of this ships in
• OpenRouter structured outputs docs — json_schema, strict mode, and require_parameters
• Prompt engineering and model selection for LLM-based nutritional estimation — Nutrients — reports prompt design moving accuracy on par with model choice
• Calorie estimation from photos, crowdsourcing study — JMIR — portion-size error predates LLMs by a decade

Bottom line

The recognition stack in soba got better three times: when a benchmark replaced my gut feeling, when strict schemas replaced “please return JSON”, and when the prompt started arguing about centimeters instead of cuisine. It never got noticeably better from a model upgrade alone. If you’re building anything that turns photos into numbers a person will act on, spend your week on scale anchors, honest intervals, and repair-first validation. The model is the least interesting part of the pipeline, which is exactly how you want it.