Mechanistic Understanding and Mitigation of Language Confusion

Mechanistic Understanding and Mitigation of Language Confusion
in English-Centric Large Language Models
Ercong Nie1,2 Helmut Schmid1 Hinrich Schütze1,2
1Center for Information and Language Processing (CIS), LMU Munich, Germany
2 Munich Center for Machine Learning (MCML), Germany
nie@cis.lmu.de
Abstract
Language confusion—where large language
models (LLMs) generate unintended languages
against the user’s need—remains a critical chal-
lenge, especially for English-centric models.
We present the first mechanistic interpretabil-
ity (MI) study of language confusion, com-
bining behavioral benchmarking with neuron-
level analysis. Using the Language Confu-
sion Benchmark (LCB), we show that confu-
sion points (CPs)—specific positions where lan-
guage switches occur—are central to this phe-
nomenon. Through layer-wise analysis with
TunedLens and targeted neuron attribution, we
reveal that transition failures in the final layers
drive confusion. We further demonstrate that
editing a small set of critical neurons, identified
via comparative analysis with a multilingual-
tuned counterpart, substantially mitigates con-
fusion while largely preserving general com-
petence and fluency. Our approach matches
multilingual alignment in confusion reduction
for many languages and yields cleaner, higher-
quality outputs. These findings provide new
insights into the internal dynamics of LLMs
and highlight neuron-level interventions as a
promising direction for robust, interpretable
multilingual language modeling. Code and
data are available at: https://github.com/
ercong21/lang_confusion.
1 Introduction
Current Large Language Models (LLMs), such
as GPT-4 (Achiam et al., 2023), PaLM 2 (Anil
et al., 2023), and Llama 3 (Grattafiori et al., 2024),
have demonstrated exceptional linguistic compe-
tence across a wide range of complex tasks that
require abstract knowledge and reasoning (Dong
et al., 2024; Wei et al., 2022; Nie et al., 2025; Wang
et al., 2025b). Early LLMs were

Achiam et al., 2023), PaLM 2 (Anil
et al., 2023), and Llama 3 (Grattafiori et al., 2024),
have demonstrated exceptional linguistic compe-
tence across a wide range of complex tasks that
require abstract knowledge and reasoning (Dong
et al., 2024; Wei et al., 2022; Nie et al., 2025; Wang
et al., 2025b). Early LLMs were predominantly
trained on massive amounts of English text data,
with some limited exposure to other languages, re-
sulting in initially constrained multilingual capa-
bilities (Touvron et al., 2023). Recent advances,
what 	is 	network security
Layer
Input
Illustration of TunedLens 	Generation of	
Confusion Point (CP)
User Prompt
LLM Response
الكريم؟ القرآن في وا0دينة ا0كية السور ب6 الفرق ما
English translation: What is the difference between the
Meccan and Medinan surahs in the Holy Quran?
The difference between Meccan and Medinan surahs in the
Quran is a significant aspect of Islamic studies. Here's a
brief overview:
**Meccan Surahs (مكية سور):**
1. **Time period:** The Meccan surahs were revealed during
the early years of Prophet Muhammad's prophethood, …
(a)
(b)
Example of Language Confusion
0
100
ar 	pt 	tr 	zh 	id
Llama3 	Llama3-multilingual 	Llama3.1
Language Confusion Performance of Llama Models on the LCB
Benchmark
Acc.
Language
(c)
Figure 1: Language Confusion in LLMs. (a) An exam-
ple of the language confusion phenomenon. (b) Visu-
alization of internal model dynamics using TunedLens,
highlighting how the confusion point emerges during
generation. (c) Benchmarking results of three Llama
models on the LCB benchmark across 5 languages.
such as multilingual continued pretraining and in-
struction tuning, have substantially extended these
models’ ability to support multiple languages (Zhu
et al., 2023; Tan and Monz, 2023; Shaham et al.,
2024; Kew et al., 2024; Wang et al., 2025c). As a
result, contemporary English-centric LLMs have
become foundational tools for multilingual com-
munication, multilingual content generation, and
cross-lingual

ended these
models’ ability to support multiple languages (Zhu
et al., 2023; Tan and Monz, 2023; Shaham et al.,
2024; Kew et al., 2024; Wang et al., 2025c). As a
result, contemporary English-centric LLMs have
become foundational tools for multilingual com-
munication, multilingual content generation, and
cross-lingual applications (Nie et al., 2023; Bang
et al., 2023; Ahuja et al., 2023; Li et al., 2023; Asai
arXiv:2505.16538v2 [cs.CL] 17 Sep 2025

-- 1 of 17 --

et al., 2024). However, despite their impressive
capabilities, a persistent and underexplored limi-
tation remains: LLMs can fail to generate text in
the user’s intended language, even when explicitly
instructed—a phenomenon termed language confu-
sion (Marchisio et al., 2024). Language confusion
manifests as full-response, line-level, or word-level
switches into unintended languages, severely un-
dermining user experience and model reliability,
especially for non-English speakers (Figure 1a).
Recent work by Marchisio et al. (2024) provides
the first systematic characterization of language
confusion, introducing the Language Confusion
Benchmark (LCB) and associated metrics to quan-
tify this phenomenon across a diverse set of lan-
guages and models. Their evaluation revealed that
even state-of-the-art LLMs are susceptible to lan-
guage confusion, with English-centric LLMs such
as Llama2, Llama3, and Mistral exhibiting partic-
ularly high rates of unintended language switch-
ing, especially in the absence of targeted multilin-
gual alignment (Figure 1c). While Marchisio et al.
(2024) propose several mitigation strategies, includ-
ing decoding adjustments, prompting techniques,
and multilingual fine-tuning, these approaches re-
main largely surface-level, offering limited insight
into the internal mechanisms that give rise to lan-
guage confusion.
A key observation from prior work is the identi-
fication of confusion points—specific positions in
the generation process where the model abruptly
switches to an

fine-tuning, these approaches re-
main largely surface-level, offering limited insight
into the internal mechanisms that give rise to lan-
guage confusion.
A key observation from prior work is the identi-
fication of confusion points—specific positions in
the generation process where the model abruptly
switches to an unintended language. However, the
model’s internal dynamics leading to these confu-
sion points and their causal role in language con-
fusion remain largely unexplored. This gap is par-
ticularly salient given the parallels to human bilin-
gual code-switching, where switch points between
languages are cognitively significant as extensively
studied in psycholinguistics (Solorio and Liu, 2008;
Bullock and Toribio, 2009). Further discussions on
the relationship between code-switching in human
communication and language confusion in LLM
text generation are provided in Appendix A.
In this work, we move beyond behavioral eval-
uation to open the black box of LLMs, leveraging
mechanistic interpretability (MI) methods (Conmy
et al., 2023; Rai et al., 2024; Saphra and Wiegreffe,
2024; Sharkey et al., 2025) to investigate the in-
ternal representations and neuron-level processes
underlying language confusion. We first empiri-
cally demonstrate that confusion points are critical
drivers of language confusion: targeted replace-
ment at these points can substantially reduce con-
fusion across languages. Building on the CP token,
we employ MI tools such as TunedLens (Belrose
et al., 2023) to trace the evolution of language rep-
resentations through the model’s layers, revealing
that confusion typically arises from transition fail-
ures in the final layers, where latent conceptual
representations are mapped to surface forms in the
target language (Figure 1b). To further elucidate
the mechanism, we conduct a neuron-level analy-
sis, identifying specific neurons in the last layers
whose activity is predictive of successful or failed
language transitions at confusion

ayers, where latent conceptual
representations are mapped to surface forms in the
target language (Figure 1b). To further elucidate
the mechanism, we conduct a neuron-level analy-
sis, identifying specific neurons in the last layers
whose activity is predictive of successful or failed
language transitions at confusion points. Inspired
by recent advances in neuron attribution and edit-
ing, we show that targeted manipulation of only
100 neurons can effectively mitigate language con-
fusion, offering a novel, model-internal approach
to improving multilingual reliability. Our findings
provide the first mechanistic account of language
confusion in LLMs, bridging the gap between be-
havioral benchmarks and internal model dynamics.
By highlighting the central role of confusion points
and their neural mechanisms, we lay the ground-
work for more robust, interpretable, and cognitively
informed multilingual language models.
Our work makes the following contributions:
(1) We provide the first mechanistic interpretabil-
ity study of language confusion in English-centric
LLMs, revealing the central role of confusion
points in unintended language switching; (2) We
employ layer-wise and neuron-level analyses to
trace the internal dynamics leading to language
confusion and identify critical late-layer neurons
responsible for transition failures; (3) We propose
and validate a principled neuron selection and edit-
ing strategy that effectively mitigates language con-
fusion and preserves the model’s general compe-
tence and output quality.
2 Related Work
Mechanistic Interpretability Methods Mech-
anistic interpretability (MI) seeks to reverse-
engineer neural networks by decomposing their
computations into human-understandable compo-
nents (Stolfo et al., 2023; Wang et al., 2024;
Men et al., 2024). A central technique in MI
is the projection of intermediate representations
into the vocabulary space, as implemented by
tools such as LogitLens (Nostalgebraist, 2020) and
TunedLens

orks by decomposing their
computations into human-understandable compo-
nents (Stolfo et al., 2023; Wang et al., 2024;
Men et al., 2024). A central technique in MI
is the projection of intermediate representations
into the vocabulary space, as implemented by
tools such as LogitLens (Nostalgebraist, 2020) and
TunedLens (Belrose et al., 2023), which enable
researchers to track how information and predic-

-- 2 of 17 --

Dataset 	Data Source Language Prompt Example
Aya Human-generated ar, en, pt, tr, zh 请简单介绍诗人李白的背景。
(Singh et al., 2024) 	Briefly introduce the poet Li Bai.
Dolly MT post-edited ar, es, fr, hi, ru Qu’est-ce qui est plus important, l’inné ou l’acquis?
(Singh et al., 2024) 	What is more important, nature or nurture?
Native Human-generated es, fr, ja, ko ᄏ	ᅩ	ᆫᄏ	ᅳᄅ	ᅵᄐ	ᅳᄂ	ᅳ	ᆫ ᄆ	ᅯ	ᆯᄅ	ᅩ ᄆ	ᅡ	ᆫᄃ	ᅳ	ᆫᄀ	ᅥᄋ	ᅣ?
(Marchisio et al., 2024) 	What is concrete made of?
Okapi Synthetic + MT ar, en, pt, zh,it, Schreib einen Aufsatz von 500 Wörtern zum Thema KI.
(Lai et al., 2023) 	fr, de, id, es, vi Write a 500-word essay on AI.
Table 1: Overview and Prompt Example of the LCB Benchmark (monolingual part). The number of examples per
language is 100 in each dataset.
tions evolve across layers (Dar et al., 2023; Pal
et al., 2023). In addition to layer-wise analysis,
recent work has focused on identifying, attribut-
ing, and intervening on important neurons—those
whose activations are strongly correlated with spe-
cific linguistic functions or behaviors (Bau et al.,
2020; Geva et al., 2022; Yu and Ananiadou, 2024b;
Tan et al., 2024). Methods for neuron selection
and editing, as well as circuit-level analysis (El-
hage et al., 2021; Wang et al., 2023), have proven
effective for uncovering the internal structure un-
derlying phenomena such as factual recall (Meng
et al., 2022; Geva et al., 2023), reasoning process-
ing (Yu and Ananiadou, 2024a), and now, as in our
work, language confusion. By leveraging these MI
techniques, we aim to provide a granular, causal
understanding of how

fective for uncovering the internal structure un-
derlying phenomena such as factual recall (Meng
et al., 2022; Geva et al., 2023), reasoning process-
ing (Yu and Ananiadou, 2024a), and now, as in our
work, language confusion. By leveraging these MI
techniques, we aim to provide a granular, causal
understanding of how and why language confu-
sion arises in multilingual LLMs, and to identify
actionable intervention points for mitigation.
Multilingual Interpretability Recent research
has begun to probe the internal representations of
English-centric and multilingual LLMs to under-
stand how they process and transfer information
across languages (He et al., 2024; Zhao et al., 2024;
Kojima et al., 2024; Zhang et al., 2024). Wendler
et al. (2024) show that models like Llama2 often
rely on English as an internal pivot language and
can disentangle language and conceptual represen-
tations in controlled tasks. Fierro et al. (2025)
examine how mechanisms identified in monolin-
gual contexts generalize to multilingual settings.
Wang et al. (2025a) investigate the internal causes
of crosslingual factual inconsistencies, revealing
how multilingual models transition from language-
independent to language-specific processing. How-
ever, prior work has not systematically connected
these internal mechanisms to language generation
errors such as language confusion.
3 Revisiting Language Confusion:
Benchmark Insights
3.1 Recap of Language Confusion Benchmark
The Language Confusion Benchmark
(LCB) (Marchisio et al., 2024) provides a
systematic framework for evaluating the ability
of LLMs to generate text in the user’s intended
language. The benchmark covers 15 typologically
diverse languages and uses a diverse set of prompts
sourced from human-written, post-edited, and
synthetic datasets to evaluate models, ensuring
coverage of a wide range of domains and linguistic
structures (Table 1). In this work, we focus
on the monolingual setting of LCB, where the
prompt and expected response

ally
diverse languages and uses a diverse set of prompts
sourced from human-written, post-edited, and
synthetic datasets to evaluate models, ensuring
coverage of a wide range of domains and linguistic
structures (Table 1). In this work, we focus
on the monolingual setting of LCB, where the
prompt and expected response are in the same
language. This setting is particularly relevant
for mechanistic interpretability research, as it
isolates language confusion phenomena from the
additional complexities of explicit cross-lingual
transfer.
To quantify language confusion, we adopt two
key metrics from LCB: line-level pass rate (LPR)
and line-level language accuracy (Acc). LPR
measures the percentage of model responses in
which every line is in the correct language. Acc
reflects the proportion of individual lines across all
responses that are correctly generated in the target
language. Both metrics rely on automatic language
identification using the fastText classifier (Joulin
et al., 2016, 2017), which efficiently detects the
language of each line in the generated output.
We conducted preliminary benchmarking exper-
iments on LCB with three instruction-tuned LLMs:
Llama3-8B (English-centric, no multilingual in-
struction tuning), Llama3-8B-multilingual (mul-
tilingually instruction-tuned) (Devine, 2024), and
Llama3.1-8B (multilingually optimized). As shown
in Figure 1c, Llama3-8B exhibits substantial lan-
guage confusion, with frequent line-level switches

-- 3 of 17 --

Model 	Metric ar en pt tr 	zh es fr hi ru 	ja ko 	de 	id it 	vi avg
Llama3 	LPR 33.0 99.5 71.0 33.0 19.3 73.0 59.3 8.0 28.0 14.0 23.0 19.0 22.0 34.0 11.0 36.5
(original) 	Acc 33.7 99.8 74.5 37.5 23.4 77.1 64.1 15.1 28.2 17.1 23.6 23.0 27.3 39.8 14.8 39.9
Llama3 	LPR 71.0 99.0 93.0 50.0 57.3 94.3 84.0 37.0 78.6 50.0 45.0 60.0 67.0 86.0 62.0 68.9
(replace) 	Acc 74.8 99.6 95.4 55.5 64.1 95.3 86.5 47.6 83.1 55.3 48.6 62.3 77.7 87.5 66.1 73.3
Llama3 	LPR 98.3 98.5 99.0 95.8 88.8 98.3 95.9 97.0 100.0 93.5 100.0

7.5 23.4 77.1 64.1 15.1 28.2 17.1 23.6 23.0 27.3 39.8 14.8 39.9
Llama3 	LPR 71.0 99.0 93.0 50.0 57.3 94.3 84.0 37.0 78.6 50.0 45.0 60.0 67.0 86.0 62.0 68.9
(replace) 	Acc 74.8 99.6 95.4 55.5 64.1 95.3 86.5 47.6 83.1 55.3 48.6 62.3 77.7 87.5 66.1 73.3
Llama3 	LPR 98.3 98.5 99.0 95.8 88.8 98.3 95.9 97.0 100.0 93.5 100.0 100.0 88.8 100.0 97.9 96.8
(multilingual) Acc 98.7 99.5 99.8 96.9 93.8 99.3 96.9 97.5 100.0 95.8 100.0 100.0 94.2 100.0 97.9 98.0
Table 2: Impact of Confusion Point Replacement on Language Confusion Metrics. Line-level pass rate (LPR)
and line-level accuracy for original Llama3-8B, multilingual Llama3-8B, and Llama3-8B with confusion point
replacement, reported by language.
to unintended languages (mostly English). In con-
trast, both Llama3-8B-multilingual and Llama3.1-
8B achieve near-perfect line-level accuracy, demon-
strating the effectiveness of multilingual instruction
tuning and targeted optimization for multilingual
dialogue.
Given these findings, our work centers on un-
derstanding and mitigating the language confusion
observed in English-centric Llama3-8B. By lever-
aging mechanistic interpretability methods, we aim
to uncover the internal causes of confusion and de-
velop interventions that can bring its performance
closer to that of explicitly multilingually-tuned
models. In the following subsection, we delve
deeper into the significance of confusion points
as critical junctures in the generation process.
3.2 Significance of Confusion Points
A confusion point (CP) is the position in a model’s
output where the first token of an unintended lan-
guage abruptly appears, marking the onset of lan-
guage confusion (Marchisio et al., 2024). This
concept is inspired by psycho- and neurolinguistic
research on code-switching, where the precise lo-
cation of a language switch—known as a switch
point—is central to understanding bilingual lan-
guage production and processing (Blanco-Elorrieta
and Pylkkänen, 2017; Suurmeijer et al., 2020).

chisio et al., 2024). This
concept is inspired by psycho- and neurolinguistic
research on code-switching, where the precise lo-
cation of a language switch—known as a switch
point—is central to understanding bilingual lan-
guage production and processing (Blanco-Elorrieta
and Pylkkänen, 2017; Suurmeijer et al., 2020). To
empirically assess the role of CPs in LLM lan-
guage confusion, we conduct a replacement ex-
periment on Llama3-8B. For each instance of lan-
guage confusion, we identify the CP using the
fastText language detector. We then replace the
token at the subword-level CP with the correspond-
ing token generated by Llama3-8B-multilingual,
which achieves near-perfect language accuracy, un-
der the same prompt. This approach is motivated
by the psycholinguistic observation that, in hu-
man code-switching, the choice at the switch point
strongly influences the subsequent language trajec-
tory (Moreno et al., 2002; Lai and O’Brien, 2020).
Our results, summarized in Table 2, show a sub-
stantial reduction in language confusion after CP
replacement, even though our method does not rep-
resent an oracle upper bound. These findings high-
light the centrality of confusion points in the emer-
gence of language confusion and motivate us to
base the subsequent mechanistic analysis and tar-
geted interventions on the generation of the token
at the CP position.
4 Mechanistic Analysis of Language
Confusion Points
4.1 Analyzing Layer-wise Language
Transition
A central question in understanding language con-
fusion is where and how the model’s internal repre-
sentations fail to transition from a shared concep-
tual space to the intended target language. Moti-
vated by recent findings that English-centric LLMs
process information in a latent, often English-
biased, conceptual space before converting it to
the target language in the final layers (Wendler
et al., 2024; Wang et al., 2025a), we conduct a de-
tailed layer-wise analysis of this transition using
TunedLens

e. Moti-
vated by recent findings that English-centric LLMs
process information in a latent, often English-
biased, conceptual space before converting it to
the target language in the final layers (Wendler
et al., 2024; Wang et al., 2025a), we conduct a de-
tailed layer-wise analysis of this transition using
TunedLens (Belrose et al., 2023).
We employ TunedLens, the more reliable variant
of LogitLens (Nostalgebraist, 2020), to unembed
the hidden states of Llama3-8B at each layer into
the vocabulary space. With this, we inspect every
layer of the model and extract the top 10 predicted
tokens with the largest logits at the position im-
mediately preceding the confusion point (CP) (for
confusion cases) or the output token (for correct
cases). For each layer, we compute the average
number and summed probabilities of English and
target language tokens among the top-10 predic-
tions, using fastText for language identification.
Our analysis focuses on four typologically diverse
languages (Arabic, Portuguese, Turkish, Chinese)
from the LCB benchmark. We separate samples
into two groups: (1) Correct—where the model

-- 4 of 17 --

(a) Average Token Count
(b) Average Probability
Figure 2: Average token counts and probabilities for English and target language tokens among the top-10 predictions
at each layer, shown for both correct and confusion samples across four languages from Aya.
generates the intended language throughout, and
(2) Confusion—where the model switches to an un-
intended language at a CP. For confusion samples,
we analyze the model’s state up to the token before
the CP.
Figure 2 presents the evolution of language to-
ken counts and probabilities across layers for both
groups. In early and middle layers, English to-
kens dominate the top-10 predictions for all lan-
guages, reflecting the English-centric latent con-
ceptual space of Llama3-8B. This is consistent with
prior work showing that LLMs encode information
in a shared, language-agnostic space in

ilities across layers for both
groups. In early and middle layers, English to-
kens dominate the top-10 predictions for all lan-
guages, reflecting the English-centric latent con-
ceptual space of Llama3-8B. This is consistent with
prior work showing that LLMs encode information
in a shared, language-agnostic space in interme-
diate layers. In the final layers, a sharp transition
emerges. For correct samples, the number and
probability of target language tokens rise steeply,
overtaking English tokens in the last few layers—
indicating a successful transition to the target lan-
guage surface form. In contrast, for confusion sam-
ples, this transition fails: English tokens remain
dominant or even increase, while target language
tokens lag behind. This failure to shift from the
latent conceptual space to the target language at the
critical moment leads to CPs and erroneous output.
Our layer-wise analysis with TunedLens reveals
that the transition to the target language occurs in
the final layers, and that failures in this process are
tightly linked to language confusion. These find-
ings provide direct evidence that language confu-
sion in Llama3-8B is primarily caused by transition
failures in the last few layers, motivating our sub-
sequent neuron-level investigation to pinpoint and
intervene on the specific components responsible
for these failures.
4.2 Localizing Critical Neurons at Confusion
Points
A key step toward understanding and mitigating
language confusion is to identify which neurons are
most responsible for the emergence of confusion
points. Building on recent advances in neuron-level
attribution (Geva et al., 2022; Yu and Ananiadou,
2024b), we adopt a static, efficient method to lo-
cate and analyze the most influential feed-forward
network (FFN) neurons in Llama3-8B.
Methodology In the inference pass in decoder-
only LLMs, for a given input sequence, each layer
output hl
i (layer l, token position i) is a sum of the
previous layer’s output hl−1
i , the

we adopt a static, efficient method to lo-
cate and analyze the most influential feed-forward
network (FFN) neurons in Llama3-8B.
Methodology In the inference pass in decoder-
only LLMs, for a given input sequence, each layer
output hl
i (layer l, token position i) is a sum of the
previous layer’s output hl−1
i , the attention output
Al
i, and the FFN output F l
i :
hl
i = hl−1
i + Al
i + F l
i (1)
The FFN output F l
i is calculated by a non-linear σ
on two MLPs W l
f c1 ∈ RN ×d and W l
f c2 ∈ Rd×N :
F l
i = W l
f c2σ(W l
f c1(hl−1
i + Al
i)) (2)
Following Geva et al. (2021), the FFN layer out-
put F l
i can be represented as a weighted sum over
neuron subvalues:
F l
i =
N	X
k=1
ml
i,k · f c2l
k (3)
ml
i,k = σ(f c1l
k · (hl−1
i + Al
i)) (4)

-- 5 of 17 --

where f c2l
k is the k-th column of W l
f c2, and ml
i,k is
derived from the inner product between the residual
output (hl−1
i + Al
i) and f c1l
k, the k-th row of W l
f c1.
Geva et al. (2022) and Dar et al. (2023) project
FFN neuron subvalues with unembedding matrices
to compute the token probability distribution. To
quantify the importance of each neuron for gener-
ating a specific token (e.g., at a confusion point),
we adopt the log probability increase method of Yu
and Ananiadou (2024b). For a neuron in the l-th
FFN layer vl, its importance score is defined as
the increase in log probability of the target token
when vl is added to the residual stream Al + hl−1,
compared to the baseline without vl:
Imp(vl) = log(p(w|vl + Al + hl−1)−
log(p(w|Al + hl−1) (5)
This approach efficiently identifies neurons whose
activations most strongly influence the model’s pre-
diction at a given position.
(a) Individual Case
(b) Aggregated Neuron Scores
Figure 3: Distribution of Important Neurons Associated
with Confusion Points in Llama3-8B. (a) Distribution of
the top 300 most important FFN neurons across layers
for an individual Chinese prompt “请解释拆东墙补西
墙的意思。(Please explain ‘拆东墙补西墙.’)” from
Aya. (b) Aggregated distribution of

dual Case
(b) Aggregated Neuron Scores
Figure 3: Distribution of Important Neurons Associated
with Confusion Points in Llama3-8B. (a) Distribution of
the top 300 most important FFN neurons across layers
for an individual Chinese prompt “请解释拆东墙补西
墙的意思。(Please explain ‘拆东墙补西墙.’)” from
Aya. (b) Aggregated distribution of important neuron
scores across all Chinese test samples in Aya.
Experimental Observations We apply this
method to Llama3-8B on confusion samples from
the LCB benchmark, focusing on the token posi-
tion immediately preceding each confusion point.
For each sample and language, we compute the im-
portance scores for all 14,336 FFN neurons in each
layer of Llama3-8B, rank them, and select the top
300 most important neurons per sample. We then
analyze the distribution of these critical neurons
across layers, both for individual samples and ag-
gregated over all samples in a language. Our anal-
ysis reveals a striking concentration of important
neurons in the final layers, as visualized in Figure 3.
This pattern holds both at the single-sample level
and when aggregating across samples, indicating
that the emergence of confusion points is primarily
driven by late-layer FFN activity. We further rank
neurons by their frequency of appearance in the top
300 sets across samples, finding that a subset of
neurons consistently recurs as highly influential for
confusion points.
To understand the effect of multilingual align-
ment, we repeat the analysis on Llama3-8B-
multilingual using the same set of prompts. Af-
ter multilingual instruction tuning, language con-
fusion is nearly eliminated. Comparing neuron
importance scores between the two models (Fig-
ure 4), we observe that most neurons critical for
confusion in the Llama3-8B become much less im-
portant in its multilingual counterpart, suggesting
that multilingual alignment suppresses the activity
of confusion-inducing neurons. However, a small
number of neurons remain important or even in-
crease in

models (Fig-
ure 4), we observe that most neurons critical for
confusion in the Llama3-8B become much less im-
portant in its multilingual counterpart, suggesting
that multilingual alignment suppresses the activity
of confusion-inducing neurons. However, a small
number of neurons remain important or even in-
crease in importance, likely reflecting their role in
encoding general semantic information rather than
language-specific transitions.
Figure 4: Neuron rank comparison between original
Llama3 and multilingual Llama3. Results of Chinese
test samples in Aya.
These findings reinforce the conclusion from our

-- 6 of 17 --

ar 	pt 	tr 	zh 	es 	fr 	hi 	ru 	ja 	ko 	de 	id 	it 	vi 	Avg.
original 	33.44 74.26 37.55 24.04 77.15 63.16 16.47 28.20 17.44 23.50 23.00 27.33 39.83 14.79 35.73
freq 	31.75 75.10 36.51 22.09 76.29 66.98 18.66 27.70 19.29 23.08 22.25 27.83 39.45 13.58 35.75
score 	76.97 93.41 67.61 80.63 91.22 74.77 60.00 50.32 53.50 33.25 40.27 53.58 96.00 67.56 67.08
comparative 85.45 97.12 57.27 89.39 92.20 83.17 82.74 89.43 49.95 40.33 80.82 78.94 95.25 66.50 77.75
Table 3: Confusion mitigation performance of different selection strategies. Line-level accuracy is reported.
layer-wise analysis: language confusion is tightly
linked to the activity of specific FFN neurons in
the final layers. The suppression of these neurons
through multilingual alignment provides a mech-
anistic explanation for the effectiveness of such
tuning. Moreover, the identification of a small set
of persistent, semantically important neurons sug-
gests that targeted neuron-level interventions could
mitigate confusion without harming overall model
performance. These insights directly inform our
subsequent strategies for neuron-based mitigation
of language confusion.
5 Mitigating Language Confusion via
Neuron Editing
A central challenge in mitigating language confu-
sion via neuron editing is to identify a set of neu-
rons whose intervention effectively reduces confu-
sion without

insights directly inform our
subsequent strategies for neuron-based mitigation
of language confusion.
5 Mitigating Language Confusion via
Neuron Editing
A central challenge in mitigating language confu-
sion via neuron editing is to identify a set of neu-
rons whose intervention effectively reduces confu-
sion without degrading the model’s general compe-
tence or fluency. Insights from our previous mech-
anistic analysis indicate that language confusion is
primarily driven by a subset of late-layer FFN neu-
rons. However, indiscriminate deactivation of im-
portant neurons risks harming the model’s overall
performance. Thus, a principled neuron selection
strategy is essential.
token_num token_prob fluency acc_ood xnli senti
Original 	1.96 	24.5 	25.8 	39.9 	46.4 98.4
Edited 	3.43 	36.8 	21.8 	74.25 44.9 98.2
Diff 	1.47 	12.3 	-4.0 	34.4 	-1.5 -0.2
Table 4: Results of generalization and robustness of
neuron editing. Average performance across languages
is reported. Detailed results in Appendix B.
5.1 Neuron Selection and Intervention
We compare three neuron selection strategies: (1)
Frequency-Based Selection: Selects the neurons
most frequently identified as important across all
confusion samples for a given language. (2) Ag-
gregate Importance Selection: Ranks neurons by
the sum of their importance scores across all con-
fusion samples, selecting those with the highest
cumulative influence. While this method captures
the overall impact, it may still include neurons es-
sential for general language competence. (3) Com-
parative Importance Selection: Inspired by Yu and
Ananiadou (2024a), this strategy identifies neurons
whose importance scores for confusion points de-
crease most substantially after multilingual align-
ment. Specifically, for each neuron, we compute
the difference in importance score between orig-
inal Llama3-8B and Llama3-8B-multilingual on
the same input. Neurons with the largest drop are
prioritized for intervention, as they are likely to
be

confusion points de-
crease most substantially after multilingual align-
ment. Specifically, for each neuron, we compute
the difference in importance score between orig-
inal Llama3-8B and Llama3-8B-multilingual on
the same input. Neurons with the largest drop are
prioritized for intervention, as they are likely to
be specifically implicated in language confusion
rather than general semantic processing.
For each strategy, we select the top 100 neurons
and intervene by setting their activations to zero
during generation. We evaluate the impact of each
method on the LCB benchmark. Our results (Ta-
ble 3) demonstrate that Comparative Importance
Selection achieves the most effective reduction in
language confusion, substantially outperforming
both frequency-based and aggregate importance
methods. Frequency-based selection yields mini-
mal benefit, while aggregate importance provides
moderate improvement but still lags behind our
proposed comparative approach. Notably, the com-
parative strategy selectively targets neurons impli-
cated in confusion, minimizing collateral impact
on general model competence.
5.2 Generalization and Robustness of Neuron
Editing
To further validate the effectiveness and safety of
our Comparative Importance Selection strategy, we
conduct a comprehensive evaluation across mul-
tiple metrics and experimental setups. Our goal
is to ensure that neuron editing not only mitigates
language confusion but also preserves the model’s
general competence, fluency, and robustness across
domains (Table 4). Details of the experimental
setup for the generalization and robustness experi-
ments are provided in Appendix C.
Language Confusion Mitigation We first assess
the impact of neuron editing on language confu-
sion using the LCB benchmark. In addition to

-- 7 of 17 --

standard metrics (line-level pass rate and line-level
accuracy), we analyze the internal output distribu-
tions by reporting (1) the number of target language
tokens among the top-10

Mitigation We first assess
the impact of neuron editing on language confu-
sion using the LCB benchmark. In addition to

-- 7 of 17 --

standard metrics (line-level pass rate and line-level
accuracy), we analyze the internal output distribu-
tions by reporting (1) the number of target language
tokens among the top-10 candidates in the final out-
put token logit, and (2) the total probability mass
assigned to target language tokens in the top-10.
These metrics provide a deeper view of how neu-
ron editing shifts the model’s internal preference
toward the intended language, beyond surface-level
accuracy.
Robustness on General Tasks To evaluate
whether neuron editing affects the model’s gen-
eral capabilities, we test the edited model on
widely used multilingual benchmarks, including
XNLI (Conneau et al., 2018) and multilingual sen-
timent analysis (Keung et al., 2020). We also as-
sess output fluency by measuring the perplexity of
generated responses using the multilingual model
facebook/xglm-564M (Lin et al., 2022). Across
all these metrics, the edited model maintains perfor-
mance comparable to the original Llama3-8B, in-
dicating that our neuron intervention method does
not degrade general language understanding or gen-
eration quality.
Out-of-Domain Generalization We further ex-
amine the generalization of neuron editing by ap-
plying neurons selected from one data source (e.g.,
Aya) to out-of-domain test sets (e.g., Okapi) for
the same language. The edited model continues to
demonstrate strong mitigation of language confu-
sion, suggesting that the identified neurons capture
robust, domain-independent mechanisms underly-
ing confusion points.
5.3 Comparison with Multilingual Alignment
To contextualize the effectiveness of neuron edit-
ing, we compare the performance of the edited
Llama3-8B model with that of the multilingual-
tuned Llama3-8B. Quantitative results show that
neuron editing yields confusion mitigation compa-
rable to multilingual instruction

5.3 Comparison with Multilingual Alignment
To contextualize the effectiveness of neuron edit-
ing, we compare the performance of the edited
Llama3-8B model with that of the multilingual-
tuned Llama3-8B. Quantitative results show that
neuron editing yields confusion mitigation compa-
rable to multilingual instruction tuning for many
languages, despite requiring no additional data or
retraining. This demonstrates that targeted neuron
intervention can match the benefits of extensive
multilingual instruction tuning for confusion reduc-
tion. However, for languages with limited pretrain-
ing exposure, multilingual supervised fine-tuning
can achieve stronger performance.
Beyond aggregate metrics, qualitative analysis
reveals further advantages of the neuron editing
approach. Through case studies, we observe that
the edited Llama3-8B not only generates fluent
and accurate responses in the intended target lan-
guage, but also avoids certain artifacts introduced
by multilingual alignment. For example, in several
instances (Figure 5), the multilingual Llama3-8B
produces outputs containing HTML tags or format-
ting patterns reminiscent of its instruction tuning
data, reflecting the influence of imperfect or noisy
multilingual datasets. In contrast, the neuron-edited
model consistently produces clean, well-structured,
and contextually appropriate responses, free from
such extraneous formatting.
Case 2 (okapi-zh, id5)
Input: 撰写一篇有关健康饮食重要性的争论性文章。Write
an argumentative essay on the importance of healthy eating.
Before Neuron Editing: English Translation: "To tear down the
eastern wall to repair the western wall" - Meaning: To solve a
problem by creating a new one, or to fix one issue by making
another one worse.
After Neuron Editing: 英语作文\n健康饮食的重要性：是或
不是？\n在当今的生活中，健康饮食已经成为人们日常生
活中的一部分。然而，是否健康饮食真的那么重要？
Multilingual Llama3:
](https://www.zhihu.com/question/34614445) \n[如何
在日常生活中更好地保持健康饮食习
惯？](https://www.zhihu.com/question/34614445)
Case 1 (aya-zh, id1)
Input:

creating a new one, or to fix one issue by making
another one worse.
After Neuron Editing: 英语作文\n健康饮食的重要性：是或
不是？\n在当今的生活中，健康饮食已经成为人们日常生
活中的一部分。然而，是否健康饮食真的那么重要？
Multilingual Llama3:
](https://www.zhihu.com/question/34614445) \n[如何
在日常生活中更好地保持健康饮食习
惯？](https://www.zhihu.com/question/34614445)
Case 1 (aya-zh, id1)
Input: 请解释“拆东墙补西墙”的意思。Please explain “
拆东墙
补西墙”
Before Neuron Editing: English Translation: "To tear down the
eastern wall to repair the western wall" - Meaning: To solve a
problem by creating a new one, or to fix one issue by making
another one worse.
After Neuron Editing: “拆东墙补西墙”是中国古代的一句成语
，它的意思是“解决问题的方法是解决问题的根源，而不是
简单地解决表面现象”。\n …
Multilingual Llama3: 2023-03-23 14:34:23 拆东墙补西墙的
意思是为了解决某个问题或问题所在的某个方面，而不是
直接解决问题本身，…
Figure 5: Case study of neuron editing.
These findings highlight a key strength of mecha-
nistic neuron editing: it directly addresses the inter-
nal causes of language confusion without introduc-
ing side effects from large-scale data-driven align-
ment. By preserving the original model’s seman-
tic competence and output quality, neuron editing
offers a more targeted and interpretable solution.
This suggests that, beyond traditional multilingual
instruction tuning, mechanistic interpretability-
driven interventions can provide a promising path
toward high-quality, robust multilingual language
models.

-- 8 of 17 --

6 Conclusions
This work provides the first mechanistic inter-
pretability account of language confusion in
English-centric LLMs. By tracing confusion points
to failures in late-layer transitions and localizing
the critical neurons responsible, we demonstrate
that targeted neuron editing can robustly mitigate
language confusion without sacrificing general
competence or fluency. Our approach achieves
results comparable with multilingual-tuned mod-
els for many languages, while preserving cleaner
output quality, though multilingual SFT remains
stronger for some low-resource cases. These find-
ings highlight the promise of

te
language confusion without sacrificing general
competence or fluency. Our approach achieves
results comparable with multilingual-tuned mod-
els for many languages, while preserving cleaner
output quality, though multilingual SFT remains
stronger for some low-resource cases. These find-
ings highlight the promise of neuron-level interven-
tions for more reliable and interpretable multilin-
gual language modeling.
Limitations
While this work provides the first mechanistic in-
terpretability account of language confusion in
English-centric LLMs, several limitations remain.
Our analysis primarily focuses on the monolingual
setting; cross-lingual contexts, which may involve
distinct mechanisms and challenges, are left for
future research. Additionally, neuron editing inter-
ventions are evaluated on selected benchmark tasks
and may require further validation across broader
domains and model architectures. Lastly, while our
approach identifies and mitigates language confu-
sion, fully understanding how these mechanisms
interact with other multilingual phenomena war-
rants further investigation.
Ethic Statement
This research was conducted in accordance with
the ACM Code of Ethics. The datasets that we use
are publicly available. We have not intended or do
not intend to share any Personally Identifiable Data
with this paper. Regarding the usage of AI tools,
we only use AI models for language refining.
Acknowledgments
We want to thank the anonymous reviewers for
their valuable feedback. This work was supported
by the German Research Foundation (DFG) under
grant SCHU 2246/14-1, Munich Center for Ma-
chine Learning (MCML), and China Scholarship
Council (CSC).
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama
Ahmad, Ilge Akkaya, Florencia Leoni Aleman,
Diogo Almeida, Janko Altenschmidt, Sam Altman,
Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni-
cal report. arXiv preprint arXiv:2303.08774.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Milli-
cent Ochieng,

(CSC).
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama
Ahmad, Ilge Akkaya, Florencia Leoni Aleman,
Diogo Almeida, Janko Altenschmidt, Sam Altman,
Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni-
cal report. arXiv preprint arXiv:2303.08774.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Milli-
cent Ochieng, Krithika Ramesh, Prachi Jain, Ak-
shay Nambi, Tanuja Ganu, Sameer Segal, Mohamed
Ahmed, Kalika Bali, and Sunayana Sitaram. 2023.
MEGA: Multilingual evaluation of generative AI.
In Proceedings of the 2023 Conference on Empir-
ical Methods in Natural Language Processing, pages
4232–4267, Singapore. Association for Computa-
tional Linguistics.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John-
son, Dmitry Lepikhin, Alexandre Passos, Siamak
Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng
Chen, Eric Chu, Jonathan H. Clark, Laurent El
Shafey, Yanping Huang, Kathy Meier-Hellstern, Gau-
rav Mishra, Erica Moreira, Mark Omernick, Kevin
Robinson, and 109 others. 2023. Palm 2 technical
report. arXiv preprint arXiv:2305.10403.
Akari Asai, Sneha Kudugunta, Xinyan Yu, Terra
Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov,
Sebastian Ruder, and Hannaneh Hajishirzi. 2024.
BUFFET: Benchmarking large language models for
few-shot cross-lingual transfer. In Proceedings of
the 2024 Conference of the North American Chap-
ter of the Association for Computational Linguistics:
Human Language Technologies (Volume 1: Long
Papers), pages 1771–1800, Mexico City, Mexico. As-
sociation for Computational Linguistics.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen-
liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei
Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu,
and Pascale Fung. 2023. A multitask, multilingual,
multimodal evaluation of ChatGPT on reasoning, hal-
lucination, and interactivity. In Proceedings of the
13th International Joint Conference on Natural Lan-
guage Processing and the 3rd Conference of the Asia-
Pacific Chapter of the Association for

uyet V. Do, Yan Xu,
and Pascale Fung. 2023. A multitask, multilingual,
multimodal evaluation of ChatGPT on reasoning, hal-
lucination, and interactivity. In Proceedings of the
13th International Joint Conference on Natural Lan-
guage Processing and the 3rd Conference of the Asia-
Pacific Chapter of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 675–718,
Nusa Dua, Bali. Association for Computational Lin-
guistics.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata
Lapedriza, Bolei Zhou, and Antonio Torralba. 2020.
Understanding the role of individual units in a
deep neural network. Proceedings of the National
Academy of Sciences, 117(48):30071–30078.
Nora Belrose, Zach Furman, Logan Smith, Danny Ha-
lawi, Igor Ostrovsky, Lev McKinney, Stella Bider-
man, and Jacob Steinhardt. 2023. Eliciting latent
predictions from transformers with the tuned lens.
arXiv preprint arXiv:2303.08112.
Esti Blanco-Elorrieta and Liina Pylkkänen. 2017. Bilin-
gual language switching in the laboratory versus

-- 9 of 17 --

in the wild: The spatiotemporal dynamics of adap-
tive language control. Journal of Neuroscience,
37(37):9022–9036.
Barbara E Bullock and Almeida Jacqueline Ed Toribio.
2009. The Cambridge handbook of linguistic code-
switching. Cambridge university press.
Nadezhda Chirkova and Vassilina Nikoulina. 2024.
Key ingredients for effective zero-shot cross-lingual
knowledge transfer in generative tasks. In Proceed-
ings of the 2024 Conference of the North American
Chapter of the Association for Computational Lin-
guistics: Human Language Technologies (Volume
1: Long Papers), pages 7222–7238, Mexico City,
Mexico. Association for Computational Linguistics.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch,
Stefan Heimersheim, and Adrià Garriga-Alonso.
2023. Towards automated circuit discovery for mech-
anistic interpretability. In Advances in Neural Infor-
mation Processing Systems, volume 36, pages 16318–
16352. Curran Associates, Inc.
Alexis Conneau,

nal Linguistics.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch,
Stefan Heimersheim, and Adrià Garriga-Alonso.
2023. Towards automated circuit discovery for mech-
anistic interpretability. In Advances in Neural Infor-
mation Processing Systems, volume 36, pages 16318–
16352. Curran Associates, Inc.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina
Williams, Samuel Bowman, Holger Schwenk, and
Veselin Stoyanov. 2018. XNLI: Evaluating cross-
lingual sentence representations. In Proceedings of
the 2018 Conference on Empirical Methods in Nat-
ural Language Processing, pages 2475–2485, Brus-
sels, Belgium. Association for Computational Lin-
guistics.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant.
2023. Analyzing transformers in embedding space.
In Proceedings of the 61st Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), pages 16124–16170, Toronto, Canada.
Association for Computational Linguistics.
Peter Devine. 2024. Tagengo: A multilingual chat
dataset. In Proceedings of the Fourth Workshop on
Multilingual Representation Learning (MRL 2024),
pages 106–113, Miami, Florida, USA. Association
for Computational Linguistics.
A. Seza Do˘gruöz, Sunayana Sitaram, Barbara E. Bul-
lock, and Almeida Jacqueline Toribio. 2021. A sur-
vey of code-switching: Linguistic and social per-
spectives for language technologies. In Proceedings
of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing
(Volume 1: Long Papers), pages 1654–1666, Online.
Association for Computational Linguistics.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan
Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu,
Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui.
2024. A survey on in-context learning. In Proceed-
ings of the 2024 Conference on Empirical Methods
in Natural Language Processing, pages 1107–1128,
Miami, Florida, USA. Association for

Lei Li, Damai Dai, Ce Zheng, Jingyuan
Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu,
Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui.
2024. A survey on in-context learning. In Proceed-
ings of the 2024 Conference on Empirical Methods
in Natural Language Processing, pages 1107–1128,
Miami, Florida, USA. Association for Computational
Linguistics.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom
Henighan, Nicholas Joseph, Ben Mann, Amanda
Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova
DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-
Dodds, Danny Hernandez, Andy Jones, Jackson
Kernion, Liane Lovitt, Kamal Ndousse, and 6 others.
2021. A mathematical framework for transformer
circuits. Transformer Circuits Thread.
Constanza Fierro, Negar Foroutan, Desmond Elliott,
and Anders Søgaard. 2025. How do multilingual
language models remember facts? arXiv preprint
arXiv:2410.14387.
Penelope Gardner-Chloros. 2009. Code-switching.
Cambridge university press.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir
Globerson. 2023. Dissecting recall of factual associa-
tions in auto-regressive language models. In Proceed-
ings of the 2023 Conference on Empirical Methods in
Natural Language Processing, pages 12216–12235,
Singapore. Association for Computational Linguis-
tics.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold-
berg. 2022. Transformer feed-forward layers build
predictions by promoting concepts in the vocabulary
space. In Proceedings of the 2022 Conference on
Empirical Methods in Natural Language Process-
ing, pages 30–45, Abu Dhabi, United Arab Emirates.
Association for Computational Linguistics.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer
Levy. 2021. Transformer feed-forward layers are key-
value memories. In Proceedings of the 2021 Confer-
ence on Empirical Methods in Natural Language Pro-
cessing, pages 5484–5495, Online and Punta Cana,
Dominican Republic. Association for Computational
Linguistics.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri,
Abhinav

021. Transformer feed-forward layers are key-
value memories. In Proceedings of the 2021 Confer-
ence on Empirical Methods in Natural Language Pro-
cessing, pages 5484–5495, Online and Punta Cana,
Dominican Republic. Association for Computational
Linguistics.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri,
Abhinav Pandey, Abhishek Kadian, Ahmad Al-
Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten,
Alex Vaughan, and 1 others. 2024. The llama 3 herd
of models. arXiv preprint arXiv:2407.21783.
Rosa E. Guzzardo Tamargo, Jorge R. Valdés Kroff, and
Paola E. Dussias. 2016. Examining the relationship
between comprehension and production processes
in code-switched language. Journal of Memory and
Language, 89:138–161. Speaking and Listening: Re-
lationships Between Language Production and Com-
prehension.
Linyang He, Ercong Nie, Helmut Schmid, Hinrich
Schütze, Nima Mesgarani, and Jonathan Brennan.
2024. Large language models as neurolinguistic sub-
jects: Identifying internal representations for form
and meaning. arXiv preprint arXiv:2411.07533.
Armand Joulin, Edouard Grave, Piotr Bojanowski,
Matthijs Douze, Hérve Jégou, and Tomas Mikolov.
2016. Fasttext.zip: Compressing text classification
models. arXiv preprint arXiv:1612.03651.

-- 10 of 17 --

Armand Joulin, Edouard Grave, Piotr Bojanowski, and
Tomas Mikolov. 2017. Bag of tricks for efficient
text classification. In Proceedings of the 15th Con-
ference of the European Chapter of the Association
for Computational Linguistics: Volume 2, Short Pa-
pers, pages 427–431, Valencia, Spain. Association
for Computational Linguistics.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A.
Smith. 2020. The multilingual Amazon reviews cor-
pus. In Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing
(EMNLP), pages 4563–4568, Online. Association for
Computational Linguistics.
Tannon Kew, Florian Schottmann, and Rico Sennrich.
2024. Turning English-centric LLMs into polyglots:
How much

The multilingual Amazon reviews cor-
pus. In Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing
(EMNLP), pages 4563–4568, Online. Association for
Computational Linguistics.
Tannon Kew, Florian Schottmann, and Rico Sennrich.
2024. Turning English-centric LLMs into polyglots:
How much multilinguality is needed? In Findings
of the Association for Computational Linguistics:
EMNLP 2024, pages 13097–13124, Miami, Florida,
USA. Association for Computational Linguistics.
Simran Khanuja, Sandipan Dandapat, Anirudh Srini-
vasan, Sunayana Sitaram, and Monojit Choudhury.
2020. GLUECoS: An evaluation benchmark for
code-switched NLP. In Proceedings of the 58th An-
nual Meeting of the Association for Computational
Linguistics, pages 3575–3585, Online. Association
for Computational Linguistics.
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hit-
omi Yanaka, and Yutaka Matsuo. 2024. On the multi-
lingual ability of decoder-based pre-trained language
models: Finding and controlling language-specific
neurons. In Proceedings of the 2024 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Tech-
nologies (Volume 1: Long Papers), pages 6919–6971,
Mexico City, Mexico. Association for Computational
Linguistics.
Gabrielle Lai and Beth A O’Brien. 2020. Examining
language switching and cognitive control through the
adaptive control hypothesis. Frontiers in Psychology,
11:1171.
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen,
Franck Dernoncourt, Ryan Rossi, and Thien Nguyen.
2023. Okapi: Instruction-tuned large language mod-
els in multiple languages with reinforcement learning
from human feedback. In Proceedings of the 2023
Conference on Empirical Methods in Natural Lan-
guage Processing: System Demonstrations, pages
318–327, Singapore. Association for Computational
Linguistics.
Tianjian Li and Kenton Murray. 2023. Why does zero-
shot cross-lingual generation fail? an explanation and
a solution. In

an feedback. In Proceedings of the 2023
Conference on Empirical Methods in Natural Lan-
guage Processing: System Demonstrations, pages
318–327, Singapore. Association for Computational
Linguistics.
Tianjian Li and Kenton Murray. 2023. Why does zero-
shot cross-lingual generation fail? an explanation and
a solution. In Findings of the Association for Compu-
tational Linguistics: ACL 2023, pages 12461–12476,
Toronto, Canada. Association for Computational Lin-
guistics.
Xiaoqian Li, Ercong Nie, and Sheng Liang. 2023.
Crosslingual retrieval augmented in-context learn-
ing for Bangla. In Proceedings of the First Workshop
on Bangla Language Processing (BLP-2023), pages
136–151, Singapore. Association for Computational
Linguistics.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu
Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na-
man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth
Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav
Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettle-
moyer, Zornitsa Kozareva, Mona Diab, and 2 others.
2022. Few-shot learning with multilingual generative
language models. In Proceedings of the 2022 Con-
ference on Empirical Methods in Natural Language
Processing, pages 9019–9052, Abu Dhabi, United
Arab Emirates. Association for Computational Lin-
guistics.
Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo
Dehaze, and Sebastian Ruder. 2024. Understanding
and mitigating language confusion in LLMs. In Pro-
ceedings of the 2024 Conference on Empirical Meth-
ods in Natural Language Processing, pages 6653–
6677, Miami, Florida, USA. Association for Compu-
tational Linguistics.
Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen,
Kang Liu, and Jun Zhao. 2024. Unlocking the fu-
ture: Exploring look-ahead planning mechanistic in-
terpretability in large language models. In Proceed-
ings of the 2024 Conference on Empirical Methods
in Natural Language Processing, pages 7713–7724,
Miami, Florida, USA. Association for Computational
Linguistics.
Kevin Meng,

u, and Jun Zhao. 2024. Unlocking the fu-
ture: Exploring look-ahead planning mechanistic in-
terpretability in large language models. In Proceed-
ings of the 2024 Conference on Empirical Methods
in Natural Language Processing, pages 7713–7724,
Miami, Florida, USA. Association for Computational
Linguistics.
Kevin Meng, David Bau, Alex J Andonian, and Yonatan
Belinkov. 2022. Locating and editing factual associ-
ations in GPT. In Advances in Neural Information
Processing Systems.
Eva M Moreno, Kara D Federmeier, and Marta Ku-
tas. 2002. Switching languages, switching pal-
abras (words): An electrophysiological study of code
switching. Brain and language, 80(2):188–207.
Ercong Nie, Sheng Liang, Helmut Schmid, and Hinrich
Schütze. 2023. Cross-lingual retrieval augmented
prompt for low-resource languages. In Findings of
the Association for Computational Linguistics: ACL
2023, pages 8320–8340, Toronto, Canada. Associa-
tion for Computational Linguistics.
Ercong Nie, Bo Shao, Mingyang Wang, Zifeng
Ding, Helmut Schmid, and Hinrich Schuetze. 2025.
BMIKE-53: Investigating cross-lingual knowledge
editing with in-context learning. In Proceedings
of the 63rd Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
pages 16357–16374, Vienna, Austria. Association
for Computational Linguistics.
Nostalgebraist. 2020. interpreting gpt: the logit lens.
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wal-
lace, and David Bau. 2023. Future lens: Anticipating
subsequent tokens from a single hidden state. In

-- 11 of 17 --

Proceedings of the 27th Conference on Computa-
tional Natural Language Learning (CoNLL), pages
548–560, Singapore. Association for Computational
Linguistics.
Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia,
Xinyi Wang, Machel Reid, and Sebastian Ruder.
2023. mmT5: Modular multilingual pre-training
solves source language hallucinations. In Findings
of the Association for Computational Linguistics:
EMNLP 2023, pages 1978–2008, Singapore.

on for Computational
Linguistics.
Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia,
Xinyi Wang, Machel Reid, and Sebastian Ruder.
2023. mmT5: Modular multilingual pre-training
solves source language hallucinations. In Findings
of the Association for Computational Linguistics:
EMNLP 2023, pages 1978–2008, Singapore. Associ-
ation for Computational Linguistics.
Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov,
and Ziyu Yao. 2024. A practical review of mecha-
nistic interpretability for transformer-based language
models. arXiv preprint arXiv:2407.02646.
Naomi Saphra and Sarah Wiegreffe. 2024. Mechanistic?
In Proceedings of the 7th BlackboxNLP Workshop:
Analyzing and Interpreting Neural Networks for NLP,
pages 480–498, Miami, Florida, US. Association for
Computational Linguistics.
Rico Sennrich, Jannis Vamvas, and Alireza Moham-
madshahi. 2024. Mitigating hallucinations and off-
target machine translation with source-contrastive
and language-contrastive decoding. In Proceedings
of the 18th Conference of the European Chapter of
the Association for Computational Linguistics (Vol-
ume 2: Short Papers), pages 21–33, St. Julian’s,
Malta. Association for Computational Linguistics.
Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan
Szpektor, Reut Tsarfaty, and Matan Eyal. 2024. Mul-
tilingual instruction tuning with just a pinch of mul-
tilinguality. In Findings of the Association for Com-
putational Linguistics: ACL 2024, pages 2304–2317,
Bangkok, Thailand. Association for Computational
Linguistics.
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lind-
sey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-
Dill, Stefan Heimersheim, Alejandro Ortega, Joseph
Bloom, Stella Biderman, Adria Garriga-Alonso,
Arthur Conmy, Neel Nanda, Jessica Rumbelow,
Martin Wattenberg, Nandi Schoots, Joseph Miller,
Eric J. Michaud, and 10 others. 2025. Open prob-
lems in mechanistic interpretability. arXiv preprint
arXiv:2501.16496.
Shivalika Singh, Freddie Vargus, Daniel D’souza,
Börje Karlsson,

m, Stella Biderman, Adria Garriga-Alonso,
Arthur Conmy, Neel Nanda, Jessica Rumbelow,
Martin Wattenberg, Nandi Schoots, Joseph Miller,
Eric J. Michaud, and 10 others. 2025. Open prob-
lems in mechanistic interpretability. arXiv preprint
arXiv:2501.16496.
Shivalika Singh, Freddie Vargus, Daniel D’souza,
Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko,
Herumb Shandilya, Jay Patel, Deividas Mataciu-
nas, Laura O’Mahony, Mike Zhang, Ramith Het-
tiarachchi, Joseph Wilson, Marina Machado, Luisa
Moura, Dominik Krzemi´nski, Hakimeh Fadaei, Irem
Ergun, Ifeoma Okoh, and 14 others. 2024. Aya
dataset: An open-access collection for multilingual
instruction tuning. In Proceedings of the 62nd An-
nual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 11521–
11567, Bangkok, Thailand. Association for Compu-
tational Linguistics.
Thamar Solorio and Yang Liu. 2008. Learning to predict
code-switching points. In Proceedings of the 2008
Conference on Empirical Methods in Natural Lan-
guage Processing, pages 973–981, Honolulu, Hawaii.
Association for Computational Linguistics.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya
Sachan. 2023. A mechanistic interpretation of arith-
metic reasoning in language models using causal
mediation analysis. In Proceedings of the 2023 Con-
ference on Empirical Methods in Natural Language
Processing, pages 7035–7052, Singapore. Associa-
tion for Computational Linguistics.
Luuk Suurmeijer, M Carmen Parafita Couto, and
Marianne Gullberg. 2020. Structural and extralin-
guistic aspects of code-switching: Evidence from
papiamentu-dutch auditory sentence matching. Fron-
tiers in Psychology, 11:592266.
Shaomu Tan and Christof Monz. 2023. Towards a better
understanding of variations in zero-shot neural ma-
chine translation performance. In Proceedings of the
2023 Conference on Empirical Methods in Natural
Language Processing, pages 13553–13568, Singa-
pore. Association for Computational Linguistics.
Shaomu Tan, Di Wu, and

u Tan and Christof Monz. 2023. Towards a better
understanding of variations in zero-shot neural ma-
chine translation performance. In Proceedings of the
2023 Conference on Empirical Methods in Natural
Language Processing, pages 13553–13568, Singa-
pore. Association for Computational Linguistics.
Shaomu Tan, Di Wu, and Christof Monz. 2024. Neuron
specialization: Leveraging intrinsic task modularity
for multilingual machine translation. In Proceed-
ings of the 2024 Conference on Empirical Methods
in Natural Language Processing, pages 6506–6527,
Miami, Florida, USA. Association for Computational
Linguistics.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-
bert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti
Bhosale, and 1 others. 2023. Llama 2: Open foun-
dation and fine-tuned chat models. arXiv preprint
arXiv:2307.09288.
Jeanine Treffers-Daller. 2009. Code-switching and
transfer: an exploration of similarities and differ-
ences. Cambridge University Press.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy,
Buck Shlegeris, and Jacob Steinhardt. 2023. Inter-
pretability in the wild: a circuit for indirect object
identification in GPT-2 small. In The Eleventh Inter-
national Conference on Learning Representations.
Mingyang Wang, Heike Adel, Lukas Lange, Yihong Liu,
Ercong Nie, Jannik Strötgen, and Hinrich Schuetze.
2025a. Lost in multilinguality: Dissecting cross-
lingual factual inconsistency in transformer language
models. In Proceedings of the 63rd Annual Meet-
ing of the Association for Computational Linguistics
(Volume 1: Long Papers), pages 5075–5094, Vienna,
Austria. Association for Computational Linguistics.
Mingyang Wang, Lukas Lange, Heike Adel, Yunpu
Ma, Jannik Strötgen, and Hinrich Schütze. 2025b.
Language mixing in reasoning language models: Pat-
terns, impact, and internal causes. arXiv preprint
arXiv:2505.14815.

-- 12 of 17 --

Shumin Wang, Yuexiang Xie, Bolin Ding, Jinyang Gao,
and Yanyong Zhang. 2025c.

guistics.
Mingyang Wang, Lukas Lange, Heike Adel, Yunpu
Ma, Jannik Strötgen, and Hinrich Schütze. 2025b.
Language mixing in reasoning language models: Pat-
terns, impact, and internal causes. arXiv preprint
arXiv:2505.14815.

-- 12 of 17 --

Shumin Wang, Yuexiang Xie, Bolin Ding, Jinyang Gao,
and Yanyong Zhang. 2025c. Language adaptation
of large language models: An empirical study on
LLaMA2. In Proceedings of the 31st International
Conference on Computational Linguistics, pages
7195–7208, Abu Dhabi, UAE. Association for Com-
putational Linguistics.
Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng,
Linjing Li, and Daniel Dajun Zeng. 2024. Unveil-
ing factual recall behaviors of large language mod-
els through knowledge neurons. In Proceedings of
the 2024 Conference on Empirical Methods in Natu-
ral Language Processing, pages 7388–7402, Miami,
Florida, USA. Association for Computational Lin-
guistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou,
and 1 others. 2022. Chain-of-thought prompting elic-
its reasoning in large language models. Advances
in neural information processing systems, 35:24824–
24837.
Chris Wendler, Veniamin Veselovsky, Giovanni Monea,
and Robert West. 2024. Do llamas work in English?
on the latent language of multilingual transformers.
In Proceedings of the 62nd Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), pages 15366–15394, Bangkok, Thai-
land. Association for Computational Linguistics.
Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and
Thamar Solorio. 2023. The decades progress on code-
switching research in NLP: A systematic survey on
trends and challenges. In Findings of the Associa-
tion for Computational Linguistics: ACL 2023, pages
2936–2978, Toronto, Canada. Association for Com-
putational Linguistics.
Odilia Yim and Richard Clément. 2021. Accultura-
tion and attitudes toward code-switching: A bidimen-
sional framework. International Journal of

rends and challenges. In Findings of the Associa-
tion for Computational Linguistics: ACL 2023, pages
2936–2978, Toronto, Canada. Association for Com-
putational Linguistics.
Odilia Yim and Richard Clément. 2021. Accultura-
tion and attitudes toward code-switching: A bidimen-
sional framework. International Journal of Bilingual-
ism, 25(5):1369–1388.
Zeping Yu and Sophia Ananiadou. 2024a. Interpret-
ing arithmetic mechanism in large language models
through comparative neuron analysis. In Proceed-
ings of the 2024 Conference on Empirical Methods
in Natural Language Processing, pages 3293–3306,
Miami, Florida, USA. Association for Computational
Linguistics.
Zeping Yu and Sophia Ananiadou. 2024b. Neuron-
level knowledge attribution in large language models.
In Proceedings of the 2024 Conference on Empiri-
cal Methods in Natural Language Processing, pages
3267–3280, Miami, Florida, USA. Association for
Computational Linguistics.
Liang Zhang, Qin Jin, Haoyang Huang, Dongdong
Zhang, and Furu Wei. 2024. Respond in my lan-
guage: Mitigating language inconsistency in re-
sponse generation based on large language models.
In Proceedings of the 62nd Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), pages 4177–4192, Bangkok, Thailand.
Association for Computational Linguistics.
Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui,
and Xuanjing Huang. 2024. Llama beyond english:
An empirical study on language capability transfer.
arXiv preprint arXiv:2401.01055.
Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng
Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki,
and Sadao Kurohashi. 2024. Beyond english-centric
llms: What language do multilingual language mod-
els think in? arXiv preprint arXiv:2408.10811.
Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan,
Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun
Chen, and Lei Li. 2023. Extrapolating large language
models to non-english by aligning languages. arXiv
preprint arXiv:2308.04948.

-- 13 of 17 --

A

ultilingual language mod-
els think in? arXiv preprint arXiv:2408.10811.
Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan,
Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun
Chen, and Lei Li. 2023. Extrapolating large language
models to non-english by aligning languages. arXiv
preprint arXiv:2308.04948.

-- 13 of 17 --

A Further Discussion on Code-Switching
and Language Confusion
Code-switching as a Linguistic Phenomenon
Code-switching, the practice of alternating between
languages within a single conversation or utterance,
is a well-studied natural phenomenon in bilingual-
ism and psycholinguistics (Gardner-Chloros, 2009).
Code-switching is typically intentional, often re-
flecting speakers’ identities, social relationships,
and contextual adaptation (Treffers-Daller, 2009;
Yim and Clément, 2021). In NLP, code-switching
has been explored through evaluating model per-
formance on code-switched data for tasks such as
sentiment analysis, machine translation, summa-
rization, and language identification (Khanuja et al.,
2020; Do˘gruöz et al., 2021; Winata et al., 2023).
Code-switching is a natural, contextually appro-
priate strategy in human communication, whereas
language confusion, on which our work focuses, is
an unintended and erroneous switch to an incorrect
language in LLMs (Marchisio et al., 2024). Though
related to code-switching, language confusion is
an unnatural phenomenon that arises from model
failures rather than communicative intent.
Language Confusion and Confusion Points in
LLMs Language confusion has been observed in
various multilingual NLP settings, such as “source
language hallucinations” in zero-shot cross-lingual
transfer (Li and Murray, 2023; Pfeiffer et al., 2023;
Chirkova and Nikoulina, 2024) and “off-target
translation” in machine translation (Sennrich et al.,
2024). In LLMs, this manifests as abrupt, unex-
pected switches to the wrong language during gen-
eration, even under explicit instructions. This issue
is particularly prevalent in

d Murray, 2023; Pfeiffer et al., 2023;
Chirkova and Nikoulina, 2024) and “off-target
translation” in machine translation (Sennrich et al.,
2024). In LLMs, this manifests as abrupt, unex-
pected switches to the wrong language during gen-
eration, even under explicit instructions. This issue
is particularly prevalent in English-centric mod-
els lacking robust multilingual alignment (Zhong
et al., 2024). A key concept in recent work is the
confusion point—the specific position in genera-
tion where the model transitions to an unintended
language. Inspired by the importance of code-
switching points in human bilingualism, confusion
points are central to understanding and diagnosing
language confusion in LLMs (Guzzardo Tamargo
et al., 2016). Unlike natural code-switching, these
points reflect internal model failures. Recent bench-
marks (Marchisio et al., 2024) systematically char-
acterize confusion points at response, line, and
word levels, revealing their widespread impact and
motivating deeper mechanistic investigation, as pur-
sued in this work.
B Full Experimental Results
Table 5 presents the full benchmarking results. Ta-
ble 6 shows the full results of the CP replacement
experiment. Tables 7 and 8 present the full results
of robustness and generalization experiments.
C Detailed Experimental Setup
C.1 Models
We primarily use three variants of the Llama3 fam-
ily for our experiments:
• Llama3-8B: The baseline English-centric
model without multilingual instruction tuning.
• Llama3-8B-multilingual: The multilingual
instruction-tuned version, as described in
(Devine, 2024).
• Llama3.1-8B: An improved model optimized
for multilingual dialogue.
All models are used in their publicly released forms
unless otherwise stated. For neuron editing experi-
ments, we intervene on Llama3-8B using the strate-
gies described in Section 5.
C.2 Datasets and Tasks
Language Confusion Benchmarking and Re-
placement Experiments We use the Language
Confusion Benchmark (LCB) (Marchisio et

are used in their publicly released forms
unless otherwise stated. For neuron editing experi-
ments, we intervene on Llama3-8B using the strate-
gies described in Section 5.
C.2 Datasets and Tasks
Language Confusion Benchmarking and Re-
placement Experiments We use the Language
Confusion Benchmark (LCB) (Marchisio et al.,
2024) for all language confusion detection and mit-
igation experiments. LCB covers 15 typologically
diverse languages and comprises several monolin-
gual and cross-lingual datasets:
• Monolingual sources: Aya (human-
generated), Dolly (post-edited), Native
(human-generated), and Okapi (synthetic +
machine translated).
• Languages: Arabic, English, Portuguese,
Turkish, Chinese, Spanish, French, Hindi,
Russian, Japanese, Korean, German, Indone-
sian, Italian, Vietnamese.
All main benchmarking and confusion point re-
placement experiments are run on the monolingual
portions of LCB, using 100 prompts per language
per dataset as described in Table 1.

-- 14 of 17 --

Robustness and Generalization Experiments
To assess the robustness and generalization of neu-
ron editing, we evaluate on:
• XNLI (Conneau et al., 2018): Cross-lingual
natural language inference in 15 languages.
• Multilingual Sentiment Analysis: Standard
multilingual sentiment datasets (including
German, Spanish, French, Japanese, and Chi-
nese). It is a binary classification task de-
rived from the multilingual Amazon review
dataset (Keung et al., 2020).
• Out-of-domain LCB evaluation: For each
language, neurons are selected from one LCB
source (e.g., Aya), then tested on a different
source (e.g., Okapi) to assess generalization.
C.3 Metrics
Language Confusion Metrics We adopt two pri-
mary metrics from LCB:
• Line-level Pass Rate (LPR): Percentage of
responses where every line is in the correct
language.
• Line-level Accuracy: Proportion of lines gen-
erated in the correct language.
Language identification for these metrics is per-
formed using the fastText classifier (Joulin et

s We adopt two pri-
mary metrics from LCB:
• Line-level Pass Rate (LPR): Percentage of
responses where every line is in the correct
language.
• Line-level Accuracy: Proportion of lines gen-
erated in the correct language.
Language identification for these metrics is per-
formed using the fastText classifier (Joulin et al.,
2016).
Internal Model Metrics We further report:
• Target Language Token Count: Number of
target language tokens among the top-10 out-
put logits in the final layer.
• Target Language Token Probability: Total
probability mass assigned to target language
tokens in the top-10 output logits.
Generalization and Fluency Metrics
• XNLI and Sentiment Accuracy: Standard
classification accuracy on XNLI and multilin-
gual sentiment analysis tasks.
• Fluency (Perplexity): Perplexity of gener-
ated outputs, measured using the multilin-
gual facebook/xglm-564M model (Lin et al.,
2022).
C.4 Implementation Details
All experiments are run on NVIDIA A100 GPUs.
Prompt formatting and decoding settings follow
the LCB benchmark defaults. Neuron interven-
tions are implemented at inference time via custom
hooks in PyTorch, zeroing out selected neuron acti-
vations layer-wise as described in Section 5.1. For
TunedLens analysis, we use the public implemen-
tation from Belrose et al. (2023).

-- 15 of 17 --

metrics: acc
Monolingual
source 	ar 	en 	pt 	tr 	zh 	es 	fr 	hi 	ru 	ja 	ko 	de 	id 	it 	vi 	avg
Llama3
aya 	55.55 100.00 86.90 37.69 42.23 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	64.47
dolly 	33.00 - 	- 	- 	- 	75.77 60.49 19.05 34.45 - 	- 	- 	- 	- 	- 	44.55
native 	- 	- 	- 	- 	- 	91.47 79.17 - 	- 	18.05 25.92 - 	- 	- 	- 	53.65
okapi 	22.00 	99.67 63.12 - 	9.08 67.75 55.03 - 	- 	- 	- 	25.25 27.83 39.83 15.41 42.50
avg 	36.85 	99.83 75.01 37.69 25.65 78.33 64.90 19.05 34.45 18.05 25.92 25.25 27.83 39.83 15.41 41.60
Llama3-
multilingual
aya 	98 	98.93 99.83 96.93 92.35 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	97.21
dolly 	98.99 - 	- 	- 	- 	98.15 93.03 97.50 100.00- 	- 	- 	- 	- 	-

- 	9.08 67.75 55.03 - 	- 	- 	- 	25.25 27.83 39.83 15.41 42.50
avg 	36.85 	99.83 75.01 37.69 25.65 78.33 64.90 19.05 34.45 18.05 25.92 25.25 27.83 39.83 15.41 41.60
Llama3-
multilingual
aya 	98 	98.93 99.83 96.93 92.35 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	97.21
dolly 	98.99 - 	- 	- 	- 	98.15 93.03 97.50 100.00- 	- 	- 	- 	- 	- 	97.53
native 	- 	- 	- 	- 	- 	99.75 97.87 - 	- 	95.83 100.00- 	- 	- 	- 	98.36
okapi 	98.97 100.00 99.83 - 	95.20 100.00 99.80 - 	- 	- 	- 	100.00 94.23 100.00 97.87 98.65
avg 	98.65 	99.47 99.83 96.93 93.78 99.30 96.90 97.50 100.00 95.83 100.00100.00 94.23 100.00 97.87 98.02
Llama3.1
aya 	93.35 	99.50 97.82 98.98 96.21 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	97.17
dolly 	97.94 - 	- 	- 	- 	98.00 97.84 99.50 98.99 - 	- 	- 	- 	- 	- 	98.45
native 	- 	- 	- 	- 	- 	98.8 99.75 - 	- 	97.82 	100 - 	- 	- 	- 	99.09
okapi 	97.31 100.00 99.50 - 	97.28 100.00100.00- 	- 	- 	- 	100.00 97.08 100.00 99.67 99.08
avg 	96.20 	99.75 98.66 98.98 96.75 98.93 99.20 99.50 98.99 97.82 100.00100.00 97.08 100.00 99.67 98.77
Crosslingual
source 	ar 	en 	pt 	tr 	zh 	es 	fr 	hi 	ru 	ja 	ko 	de 	id 	it 	vi 	avg
Llama3
complex 	32.84 - 	47.04 29.13 	9.68 48.98 47.47 35.56 41.22 	6.80 12.42 41.54 37.43 43.16 38.12 33.67
okapi 	23.94 - 	56.34 27.38 	6.88 63.48 59.88 51.83 29.62 	2.23 	5.50 45.35 45.34 52.51 32.50 35.91
sharegpt 	27.75 - 	56.37 32.18 17.19 54.78 59.95 47.83 28.78 13.67 17.19 46.37 52.17 53.90 44.90 39.50
avg 	28.18 - 	53.25 29.56 11.25 55.75 55.77 45.07 33.21 	7.57 11.70 44.42 44.98 49.86 38.51 36.36
Llama3.1
complex 	36.43 - 	63.16 56.34 38.02 64.34 68.31 53.54 47.93 38.30 38.71 65.14 56.56 61.19 63.60 53.68
okapi 	23.74 - 	49.38 48.46 18.63 55.57 56.71 50.92 41.42 21.12 27.12 56.92 54.54 61.47 48.03 43.86
sharegpt 	40.28 - 	62.57 60.69 41.98 65.50 74.17 63.50 51.00 42.58 48.40 69.75 72.67 66.70 66.68 59.03
avg 	33.48 - 	58.37 55.16 32.88 61.80 66.39 55.99 46.78 34.00 38.08 63.94 61.26 63.12 59.44 52.19
Table 5: Full benchmarking results on LCB.
metrics: lpr
Monolingual
source 	ar

.12 27.12 56.92 54.54 61.47 48.03 43.86
sharegpt 	40.28 - 	62.57 60.69 41.98 65.50 74.17 63.50 51.00 42.58 48.40 69.75 72.67 66.70 66.68 59.03
avg 	33.48 - 	58.37 55.16 32.88 61.80 66.39 55.99 46.78 34.00 38.08 63.94 61.26 63.12 59.44 52.19
Table 5: Full benchmarking results on LCB.
metrics: lpr
Monolingual
source 	ar 	en 	pt 	tr 	zh 	es 	fr 	hi 	ru 	ja 	ko 	de 	id 	it 	vi 	avg
Llama3-ori
aya 	53 	100 	83 	33 31.63 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	64.47
dolly 	30 - 	- 	- 	- 	68 	54 	8 	28 - 	- 	- 	- 	- 	- 	44.55
native 	- 	- 	- 	- 	- 	88 	72 - 	- 	14 	23 - 	- 	- 	- 	53.65
okapi 	16 	99 	59 - 	7 	63 	52 - 	- 	- 	- 	19 	22 	34 	11 42.50
avg 	33.00 	99.50 71.00 33.00 19.32 73.00 59.33 	8.00 28.00 14.00 23.00 19.00 22.00 34.00 11.00 36.48
Llama3-re
aya 	83.67 	98 	91 	50 65.66 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	77.67
dolly 	65.66 - 	- 	- 	- 	94 	76 	37 78.57 - 	- 	- 	- 	- 	- 	70.25
native 	- 	- 	- 	- 	- 	97 	86 - 	- 	50 	45 - 	- 	- 	- 	69.50
okapi 	63.54 	100 	95 - 	49 	92 	90 - 	- 	- 	- 	60 	67 	86 	62 76.17
avg 	70.96 	99.00 93.00 50.00 57.33 94.33 84.00 37.00 78.57 50.00 45.00 60.00 67.00 86.00 62.00 68.95
Llama3-multi
aya 	98 	96.97 	99 95.83 84.69 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	97.17
dolly 	97.98 - 	- 	- 	- 	95.96 91.84 	97 	100 - 	- 	- 	- 	- 	- 	98.45
native 	- 	- 	- 	- 	- 	99 96.81 - 	- 	93.48 	100 - 	- 	- 	- 	99.09
okapi 	98.97 	100 	99 - 	92.93 	100 	99 - 	- 	- 	- 	100 88.78 	100 97.87 99.08
avg 	98.32 	98.49 99.00 95.83 88.81 98.32 95.88 97.00 100.00 93.48 100.00100.00 88.78 100.00 97.87 96.79
metrics: acc
Monolingual
source 	ar 	en 	pt 	tr 	zh 	es 	fr 	hi 	ru 	ja 	ko 	de 	id 	it 	vi 	avg
Llama3-ori
aya 	53.75 	100 	86.4 	37.5 39.46 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	64.47
dolly 	30.75 - 	- 	- 	- 	73.45 59.99 15.05 	28.2 - 	- 	- 	- 	- 	- 	44.55
native 	- 	- 	- 	- 	- 	91.05 77.75 - 	- 	17.13 23.58 - 	- 	- 	- 	53.65
okapi 	16.5 	99.67 62.62 - 	7.33 66.83 	54.7 - 	- 	- 	- 	23 27.33 39.83 14.79 42.50
avg 	33.67 	99.84 74.51 37.50 23.40 77.11 64.15 15.05 28.20 17.13 23.58 23.00

- 	- 	64.47
dolly 	30.75 - 	- 	- 	- 	73.45 59.99 15.05 	28.2 - 	- 	- 	- 	- 	- 	44.55
native 	- 	- 	- 	- 	- 	91.05 77.75 - 	- 	17.13 23.58 - 	- 	- 	- 	53.65
okapi 	16.5 	99.67 62.62 - 	7.33 66.83 	54.7 - 	- 	- 	- 	23 27.33 39.83 14.79 42.50
avg 	33.67 	99.84 74.51 37.50 23.40 77.11 64.15 15.05 28.20 17.13 23.58 23.00 27.33 39.83 14.79 39.94
Llama3-re
aya 	86.9 	99.17 94.97 55.53 71.12 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	81.54
dolly 	68.48 - 	- 	- 	- 	94.25 80.66 47.62 	83.1 - 	- 	- 	- 	- 	- 	74.82
native 	- 	- 	- 	- 	- 	97 87.92 - 	- 	55.27 48.58 - 	- 	- 	- 	72.19
okapi 	68.92 	100 95.79 - 	57.13 94.67 	91 - 	- 	- 	- 	62.33 77.67 	87.5 66.08 79.88
avg 	74.77 	99.59 95.38 55.53 64.13 95.31 86.53 47.62 83.10 55.27 48.58 62.33 77.67 87.50 66.08 73.29
Llama3-multi
aya 	98 	98.93 99.83 96.93 92.35 - 	- 	- 	- 	- 	- 	- 	- 	- 	- 	97.17
dolly 	98.99 - 	- 	- 	- 	98.15 93.03 	97.5 	100 - 	- 	- 	- 	- 	- 	98.45
native 	- 	- 	- 	- 	- 	99.75 97.87 - 	- 	95.83 	100 - 	- 	- 	- 	99.09
okapi 	98.97 	100 99.83 - 	95.2 	100 	99.8 - 	- 	- 	- 	100 94.23 	100 97.87 99.08
avg 	98.65 	99.47 99.83 96.93 93.78 99.30 96.90 97.50 100.00 95.83 100.00100.00 94.23 100.00 97.87 98.02
avg
ar 	en 	pt 	tr 	zh 	es 	fr 	hi 	ru 	ja 	ko 	de 	id 	it 	vi 	avg
Llama3 	LPR 	33.0 	99.5 	71.0 	33.0 	19.3 	73.0 	59.3 	8.0 	28.0 	14.0 	23.0 	19.0 	22.0 	34.0 	11.0 	36.5
(original) 	Acc 	33.7 	99.8 	74.5 	37.5 	23.4 	77.1 	64.1 	15.1 	28.2 	17.1 	23.6 	23.0 	27.3 	39.8 	14.8 	39.9
Llama3 	LPR 	71.0 	99.0 	93.0 	50.0 	57.3 	94.3 	84.0 	37.0 	78.6 	50.0 	45.0 	60.0 	67.0 	86.0 	62.0 	68.9
(replace) 	Acc 	74.8 	99.6 	95.4 	55.5 	64.1 	95.3 	86.5 	47.6 	83.1 	55.3 	48.6 	62.3 	77.7 	87.5 	66.1 	73.3
Llama3 	LPR 	98.3 	98.5 	99.0 	95.8 	88.8 	98.3 	95.9 	97.0 100.0 	93.5 100.0 100.0 	88.8 100.0 	97.9 	96.8
(multilingual) Acc 	98.7 	99.5 	99.8 	96.9 	93.8 	99.3 	96.9 	97.5 100.0 	95.8 100.0 100.0 	94.2 100.0 	97.9 	98.0
Table 6: Full results of CP replacement experiments

-- 16 of 17 --

num_ori prob_ori num_edit prob_edit

LPR 	98.3 	98.5 	99.0 	95.8 	88.8 	98.3 	95.9 	97.0 100.0 	93.5 100.0 100.0 	88.8 100.0 	97.9 	96.8
(multilingual) Acc 	98.7 	99.5 	99.8 	96.9 	93.8 	99.3 	96.9 	97.5 100.0 	95.8 100.0 100.0 	94.2 100.0 	97.9 	98.0
Table 6: Full results of CP replacement experiments

-- 16 of 17 --

num_ori prob_ori num_edit prob_edit num_diff prob_diff fluency_ori fluency_cna diff
ar 2.83 25.8 5.37 30.3 2.55 4.5 30.1 24.7 -5.4
pt 2.86 49.5 3.41 56.0 0.56 6.5 25.7 23.3 -2.3
tr 2.05 29.5 2.42 23.5 0.37 -6.0 21.2 18.8 -2.5
zh 1.33 8.6 5.10 37.3 3.78 28.7 33.1 26.0 -7.0
es 1.67 26.5 3.28 50.3 1.61 23.8 25.4 23.2 -2.2
fr 2.48 43.0 2.91 49.2 0.43 6.2 21.2 21.1 -0.1
hi 1.25 12.0 1.64 13.7 0.39 1.8 28.5 22.9 -5.6
ru 1.09 18.0 3.21 31.0 2.12 13.0 23.7 19.5 -4.2
de 2.73 23.7 4.45 37.1 1.72 13.4 23.8 18.5 -5.3
it 1.33 8.4 2.50 39.3 1.17 31.0 25.7 20.2 -5.5
avg 1.96 24.5 3.43 36.8 1.47 12.3 25.8 21.8 -4.0
Table 7: Full results of robustness experiments. Perplexity is calculated to measure fluency.
xnli
language acc_ori acc_edit
ar 0.42 0.37
de 0.54 0.54
es 0.46 0.5
fr 0.49 0.5
hi 0.47 0.48
ru 0.37 0.3
tr 0.46 0.52
vi 0.46 0.37
zh 0.51 0.46
avg 0.464 0.449
sentiment analysis
language acc_ori acc_edit
de 0.98 0.98
es 0.98 0.98
fr 0.98 0.97
ja 0.99 0.99
zh 0.99 0.99
avg 0.984 0.982
Table 8: Full results of generalization experiments.

-- 17 of 17 --