Multilingual Production LLM Apps in October 2026: Pin the Response Language, Retrieve Across Languages, and Evaluate Every Locale You Ship

Most LLM apps are built and tested in English, then quietly shipped to users who write in Spanish, Japanese, Hindi, Portuguese, or a mix of two languages in the same sentence. The model usually copes, which is exactly why the problems go unnoticed: answers drift back into English halfway through a conversation, retrieval misses the one document that would have answered the question because it was written in another language, and nobody has a test that would catch either failure. This guide covers the practical pieces of running a multilingual LLM app in production: deciding which language to answer in, keeping that decision stable, retrieving across languages, handling the cost differences, and evaluating each locale you actually serve.

Image: APK via Wikimedia Commons (CC BY 4.0)

Image: Albinfo via Wikimedia Commons (CC BY-SA 3.0)

Why multilingual failures are easy to miss

A multilingual bug rarely throws an error. The response is fluent, it just has the wrong property. Common symptoms look like this:

• Language drift. The user writes in German, the first answer is in German, and after a long tool result in English the model switches to English for the rest of the session.
• System-prompt bleed. Your system prompt, few-shot examples, and tool descriptions are all in English, so short or ambiguous user messages get English replies.
• Retrieval blind spots. The knowledge base has a French policy document and a user asks in Italian. A monolingual embedding model puts the two far apart, so the right chunk never reaches the prompt.
• Formatting mistakes. Dates, decimal separators, currencies, and units come out in US format for a user in Brazil or Germany.
• Quality gaps you never measured. Your eval set is English only, so a prompt change that hurts Korean answers ships with a green dashboard.

The fix is to treat language as explicit request state, not something you hope the model infers correctly every turn.

Step 1: Decide the response language explicitly

Pick a single source of truth for the response language on every request. A reasonable priority order is:

1. An explicit user setting (account locale or a language picker in the UI).
2. A language the user asked for in the conversation ("please answer in English").
3. The detected language of the current user message, if detection is confident.
4. The language already pinned for this conversation.
5. The app default.

Store the result as a BCP 47 tag such as de, pt-BR, or zh-Hant on the conversation record. Region and script matter: Brazilian and European Portuguese differ in vocabulary, and Traditional and Simplified Chinese are not interchangeable for readers.

Language detection that is good enough

You do not need the LLM to detect language. Small open-source detectors are fast and cheap: fastText's published language-identification model, Google's CLD3, and the lingua library are common choices. A few practical rules:

• Do not trust detection on very short text. "ok", "gracias", or a product name is not enough signal. Below a minimum length, keep the pinned language.
• Strip code, URLs, and stack traces before detecting. A Spanish question with a pasted Python error will otherwise look English.
• Use a confidence threshold. Only change the pinned language when confidence is high and the new language persists, for example across two consecutive messages, unless the user explicitly asks to switch.
• Expect code-switching. Users mixing Hindi and English, or Spanish and English, are normal. Pick the dominant language for the reply and do not flip-flop on every message.

Step 2: Pin the language in the prompt, close to the end

Once you know the target language, tell the model directly, and put the instruction where it carries weight. Long contexts full of English tool output are exactly where drift happens, so a single line at the top of a long system prompt is not enough. A practical pattern is a short instruction in the system prompt plus a reminder appended right before generation:

System: ...You are a support assistant for Acme...
Response language: pt-BR. Always reply in Brazilian Portuguese,
even when documents or tool results are in another language.
Keep product names, code, and error messages in their original form.

[conversation, retrieved documents, tool results]

Reminder (inserted just before the model turn):
Reply in pt-BR.

Two details prevent most complaints. First, tell the model what not to translate: code, identifiers, commands, quoted error strings, brand names, and legal names. Translated variable names break copy-paste. Second, if your app produces structured output, keep JSON keys and enum values in English and only localize the human-readable string fields. Your parsers stay stable and the user still sees their language.

Verify the output language

Run the same language detector on the final response, at least on the user-visible text fields. If the detected language does not match the pinned one with good confidence, you have options in order of cost: log it as a drift event, regenerate once with a stronger reminder, or pass the answer through a translation step. Logging alone is valuable, because drift rates by locale tell you where your prompt is weak.

Step 3: Retrieve across languages

If your knowledge base contains documents in several languages, or only in English while users ask in many, retrieval needs attention. There are three main approaches.

Multilingual embeddings

Models trained to place translations near each other in vector space let a query in one language match a document in another. Open options such as BAAI/bge-m3 and the intfloat/multilingual-e5 family are widely used, and several hosted embedding APIs also advertise multilingual support. Check the model card for the languages it was trained on rather than assuming coverage, and benchmark recall on your own query and document pairs, because cross-lingual matching is usually weaker than same-language matching.

Query translation

Translate the user query into the dominant document language (often English) and search with both the original and the translated query, then merge the results. This works with any embedding model and is easy to add, at the cost of one extra fast model call per request. Keep the original query too, because translation can lose domain terms.

Document-side translation

For a small, high-value corpus such as policies or help-center articles, you can translate documents at index time and store each translation as its own chunk, linked to the source document ID. Retrieval then runs mostly same-language. The trade-off is index size and keeping translations current when the source changes, so tie re-translation to your normal document update pipeline.

Whichever you choose, add the document's language as metadata on every chunk. That lets you filter (for example, prefer the user's language when both versions exist), show the right citation link, and debug why a chunk was or was not retrieved. Hybrid keyword search also needs care: BM25-style search depends on tokenization, and languages without spaces between words, such as Japanese, Chinese, and Thai, need a proper word segmenter or n-gram tokenizer in your search engine.

Step 4: Budget for token and latency differences

Tokenizers are trained mostly on English-heavy data, so the same content often takes more tokens in other languages, and the gap tends to be larger for non-Latin scripts. More tokens means higher cost, slower time to first token on long prompts, and less room in the context window. Do not guess the multiplier. Measure it:

1. Take a sample of real or representative messages per locale, ideally parallel translations of the same content.
2. Count tokens with the tokenizer of the model you actually call.
3. Record the ratio against English per locale and use it in context budgets, truncation limits, and cost forecasts.

If you set a fixed character limit for user input or retrieved context, convert it to a token budget instead. A character limit that is comfortable in English can overflow the context in another language, or be far too strict in a language with dense characters.

Step 5: Normalize text and format locale-sensitive values

Unicode has several ways to encode the same visible text. Normalize user input and documents to NFC before hashing, caching, deduplicating, or exact-match lookups, otherwise identical-looking strings miss each other. Be careful with NFKC: it folds compatibility characters, which helps search but can change meaning in some contexts, so apply it to search keys rather than stored content.

For dates, numbers, and currency, do not ask the model to format them. Have tools return raw values (ISO 8601 dates, plain numbers with a currency code) and format them in your application with a locale-aware library such as the ICU-based Intl APIs in JavaScript or Babel in Python. The model can still write the surrounding sentence, but the number itself comes out correct and consistent.

Step 6: Evaluate every locale you ship

A multilingual app needs a multilingual eval set. Start small and grow it from real traffic:

• Per-locale golden cases. For each supported language, collect at least a few dozen representative questions with reference answers or grading criteria. Use native speakers to write or review them; machine-translated test sets miss the phrasing real users write.
• Language-correctness check. An automatic check that the response is in the pinned language. This is cheap and catches drift regressions immediately.
• Cross-lingual retrieval check. Pairs where the question is in one language and the correct source document is in another. Track recall at k for these separately from same-language pairs.
• Formatting check. Cases with dates, prices, and units for each locale.
• Per-locale reporting. Report scores broken down by language. An average across languages hides a regression in the smallest locale.

If you use an LLM judge, give it the target language in the rubric and spot-check its grades with native speakers, since judge reliability can vary by language too.

A minimal request flow

Putting the pieces together, a request in a multilingual app looks like this:

1. Load the conversation's pinned language and the user's locale setting.
2. Clean the new message (remove code and URLs), detect its language, and update the pin only if the rules above say so.
3. Retrieve with a multilingual embedding model or with the original plus translated query, and keep chunk language metadata.
4. Build the prompt with the language instruction, a do-not-translate list, and a closing reminder.
5. Generate, then verify the response language and log the result with the locale.
6. Format dates, numbers, and money in the app using the locale.

Common mistakes to avoid

• Inferring the language from the browser or IP address and overriding what the user actually typed.
• Translating code, commands, or error messages inside answers.
• Using an English-only embedding model for a corpus with mixed languages.
• Applying the same character limits to every language.
• Shipping a prompt change after testing only in English.

Takeaway

Multilingual support is not a model feature you switch on, it is a set of small engineering decisions: store the response language as state, instruct the model clearly and near the end of the prompt, verify the output, retrieve across languages on purpose, budget tokens per locale, and keep per-language evals. Each step is cheap on its own, and together they turn "it mostly answers in the right language" into behavior you can measure and trust.