Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages

Utilizing Multilingual Encoders to Improve Large
Language Models for Low-Resource Languages
Imalsha Puranegedara∗, Themira Chathumina∗, Nisal Ranathunga∗,
Nisansa de Silva∗, Surangika Ranathunga†, Mokanarangan Thayaparan‡
∗Dept. of Computer Science & Engineering, University of Moratuwa, Sri Lanka.
{imalsha.20, themira.20, nisal.20, NisansaDds}@cse.mrt.ac.lk
†School of Mathematical and Computational Sciences, Massey University, Auckland, New Zealand.
s.ranathunga@massey.ac.nz
‡Department of Computer Science, The Open University, Milton Keynes, United Kingdom.
mokanarangan.thayaparan@open.ac.uk
Abstract—Large Language Models (LLMs) excel in English,
but their performance degrades significantly on low-resource
languages (LRLs) due to English-centric training. While there are
methods to align LLMs with multilingual encoders such as the
Massively Multilingual Text-to-Text Transfer Transformer (mT5),
they typically use only the final encoder layer. We propose a
novel architecture that fuses all intermediate layers, enriching the
linguistic information passed to the LLM. Our approach features
two strategies: (1) a Global Softmax weighting for overall layer
importance, and (2) a Transformer Softmax model that learns
token-specific weights. The fused representations are mapped into
the LLM’s embedding space, enabling it to process multilingual
inputs. The model is trained only on English data, without
using any parallel or multilingual data. Evaluated on XNLI,
IndicXNLI, Sinhala News Classification, and Amazon Reviews,
our Transformer Softmax model significantly outperforms the
existing baseline. We observe strong performance gains in LRLs,
improving Sinhala classification accuracy from 71.66% to 75.86%
and achieving clear improvements across Indic languages such as
Tamil, Bengali, and Malayalam. These specific gains contribute
to an overall boost in average XNLI accuracy from 70.36%
to 71.50%. This approach offers a scalable, data-efficient path
toward more capable and

a classification accuracy from 71.66% to 75.86%
and achieving clear improvements across Indic languages such as
Tamil, Bengali, and Malayalam. These specific gains contribute
to an overall boost in average XNLI accuracy from 70.36%
to 71.50%. This approach offers a scalable, data-efficient path
toward more capable and equitable multilingual LLMs1.
Index Terms—Large Language Models, Multilingual Under-
standing, Zero-Shot Transfer, Layer Fusion, Low-Resource Lan-
guages
I. INTRODUCTION
Large Language Models (LLMs) are neural networks trained
on large corpora to generate and understand human lan-
guages. Recent advances favor decoder-only transformer ar-
chitectures (e.g., GPT-3 [1], PaLM [2], DeepSeek-V3 [3])
over earlier encoder-based (BERT [4]) and encoder-decoder
models (BART [5], T5 [6]), due to their ability to scale
effectively [7], excel in next-token generation, and general-
ize well in unseen tasks [8]. Despite their strengths, LLMs
remain heavily English-centric, yielding strong performance
on reasoning and instruction tasks in English and other high-
resource languages [9, 10], but performing poorly in low
resource languages (LRLs) [11].
In contrast, multilingual encoders such as XLM-R [12],
mBERT [13], and mT5 encoder [14] are pretrained on mul-
1Our code is available at https://github.com/ImalshaD/MultiLingualImprove
tilingual corpora and are designed for cross-lingual general-
ization. They generate contextualized token embeddings and
are widely used in multilingual Natural Language Processing
(NLP) tasks [15]. While downstream applications often use
only the final encoder layer, recent studies show that inter-
mediate layers also encode valuable syntactic and semantic
features [16], which can enrich cross-lingual transfer when
properly utilized. However, these models lack the robust
reasoning abilities of English-centric LLMs due to limited
exposure to multilingual instruction data.
This mismatch results in a performance gap in

ate layers also encode valuable syntactic and semantic
features [16], which can enrich cross-lingual transfer when
properly utilized. However, these models lack the robust
reasoning abilities of English-centric LLMs due to limited
exposure to multilingual instruction data.
This mismatch results in a performance gap in multilingual
instruction-following [17]. Attempts to bridge this include: (1)
fine-tuning LLMs on translated instruction data [18, 19], (2)
using translation at inference time [20], or (3) aligning mul-
tilingual encoder outputs (typically final-layer only) with En-
glish LLMs [21, 22]. However, translation-based data creation
suffers from noise and semantic drift in LRLs [23, 24], while
multi-stage translation inference introduces latency and com-
pounding errors [24, 25]. Methods such as LangBridge [21]
and MindMerger [22] align multilingual encoders with LLMs
using only the final-layer hidden states to understand multilin-
gual input through language agnostic representations provided
by multilingual encoders. Such approaches overlook the rich,
language-agnostic representations embedded in the intermedi-
ate layers of multilingual encoders [26].
To address these limitations, we propose a method that
feeds all multilingual encoder layers into the LLM using a
learned, weighted fusion by extending LangBridge architec-
ture. Unlike prior work that uses static final-layer outputs [21],
our model learns to adaptively combine representations across
all layers, capturing diverse linguistic features and improving
interpretability. Our methodology targets multilingual under-
standing and reasoning tasks where the input is multilingual
(English and non-English) and the LLM generated output is in
English. We train the model exclusively on English instruction
data and evaluate it in a zero-shot2 setting on other languages
without any multilingual fine-tuning. This setup reflects real-
2In this context, zero-shot means that the model is trained only on

(English and non-English) and the LLM generated output is in
English. We train the model exclusively on English instruction
data and evaluate it in a zero-shot2 setting on other languages
without any multilingual fine-tuning. This setup reflects real-
2In this context, zero-shot means that the model is trained only on English-
language data but is evaluated directly on the same task in languages it has
not encountered during training.
arXiv:2508.09091v3 [cs.CL] 8 Nov 2025

-- 1 of 6 --

world scenarios where task-specific instruction data is not
available in low-resource languages.
In our experiments, we evaluate the proposed method
on a diverse set of datasets covering multiple languages,
including XNLI [27] (15 languages), IndicXNLI [28] (12
Indic languages), Sinhala News Classification [29, 30], and
Amazon Multilingual Reviews [31] (covering languages such
as English, Spanish, French, German, Chinese, and Japanese).
Empirically, we observe that our approach yields consistent
improvements across languages, with notably higher gains for
languages whose scripts are similar to English. The ability to
leverage shared script and structural similarities enables more
effective transfer, offering a practical benefit for expanding
LLM capabilities to low-resource, script-similar languages
without requiring additional annotated data.
II. RELATED WORK
A. Aligning Pretrained Representations
Aligning pretrained encoder outputs with LLMs has gained
traction in multi-modal tasks (e.g., vision-language mod-
els [32]) and is now being extended to multilingual settings.
Approaches such as LangBridge [21] and MindMerger [22]
aim to equip English-centric LLMs with multilingual capa-
bilities by incorporating outputs from multilingual encoders.
LangBridge uses a soft replacement strategy, projecting the
multilingual encoder’s final-layer hidden states into the LLM’s
input space via a learned transformation. MindMerger intro-
duces a two-stage process: (1) a mapping stage using

h multilingual capa-
bilities by incorporating outputs from multilingual encoders.
LangBridge uses a soft replacement strategy, projecting the
multilingual encoder’s final-layer hidden states into the LLM’s
input space via a learned transformation. MindMerger intro-
duces a two-stage process: (1) a mapping stage using bilingual
pairs to align multilingual representations with the LLM, and
(2) an augmentation stage to integrate them with the LLM’s
inputs. However, both methods rely solely on the final-layer
encoder output, neglecting rich linguistic information present
in intermediate layers [26]. Moreover, MindMerger depends on
parallel instruction or bilingual data, which is scarce for many
low-resource languages, limiting its applicability in zero-shot
scenarios.
B. Using All Layers of the Encoder
Prior work shows that intermediate encoder layers capture
diverse linguistic features useful for downstream tasks. Meth-
ods such as weighted summation or attention-based fusion [16]
have been applied in English classification and Neural Ma-
chine Translation. For example, Bapna et al.[33] use a trainable
softmax over encoder layers for neural machine translation,
while Liu et al.[34] introduce fixed, learned scaling vectors per
layer. DWAtt [35] further improves layer selection through an
attention mechanism. These strategies have proven effective in
monolingual or bilingual contexts, but have not been extended
to multilingual instruction-following LLMs, especially where
input is non-English and output is English.
III. METHODOLOGY
We propose an architecture that enhances multilingual
understanding in English-centric LLMs by extending the
LangBridge [21] architecture. Unlike LangBridge, which uses
only the final-layer hidden state from a multilingual encoder,
our method integrates all intermediate layer representations
through a learned fusion mechanism. This allows the model
to dynamically capture both low-level and high-level linguistic
features, improving its ability

chitecture. Unlike LangBridge, which uses
only the final-layer hidden state from a multilingual encoder,
our method integrates all intermediate layer representations
through a learned fusion mechanism. This allows the model
to dynamically capture both low-level and high-level linguistic
features, improving its ability to understand and reason over
inputs in diverse languages, particularly in zero-shot, low-
resource scenarios.
Given a multilingual instruction prompt, we first pass it
through a frozen multilingual encoder, which outputs token-
level hidden states from all transformer layers. These multi-
layered representations encode different linguistic abstractions,
from syntactic signals in early layers to semantic and task-
relevant features in deeper ones. To effectively leverage this
full depth, we introduce a trainable depth-wise weighting
mechanism that learns to assign importance scores to each
layer per token, generating a single fused representation per to-
ken. These fused token embeddings are then projected into the
input space of an English-centric LLM using a learned linear
transformation. The encoder and LLM remain frozen during
training; however, the fusion and projection components are
trained end-to-end using only English instruction–response
data. This setup enables the model to align multilingual inputs
with English LLM reasoning capabilities without the need for
multilingual fine-tuning or parallel instruction data.
A. Fusion Module
The Fusion Module combines all intermediate hidden states
from the multilingual encoder into a single embedding for
each token. For each token, the multilingual encoder produces
a sequence of hidden states h1, h2, . . . , hL ∈ Rd, where d
is the embedding dimension. The Fusion Module computes a
weighted sum of these representations:
hfused =
L	X
i=1
αi · hi (1)
where αi denotes the normalized weight assigned to the i-th
layer, and the sum is performed independently for each token.
We propose two strategies for

en states h1, h2, . . . , hL ∈ Rd, where d
is the embedding dimension. The Fusion Module computes a
weighted sum of these representations:
hfused =
L	X
i=1
αi · hi (1)
where αi denotes the normalized weight assigned to the i-th
layer, and the sum is performed independently for each token.
We propose two strategies for computing these weights:
1) Global Softmax Weighting: Here, each αl is a scalar
weight that determines the contribution of the l-th layer to
the final embedding. These weights are shared across all
tokens and samples in the batch, and are normalized using
a temperature-scaled softmax:
αl = exp(τ · wl)
PL
k=1 exp(τ · wk) (2)
where w ∈ RL is a trainable parameter vector initialized
with small values. The effective temperature τ is not directly
learned but computed as:
τ = base_temp + temp × factor (3)
where base_temp is a fixed constant (e.g., 102), factor
is a large scaling constant (e.g., 105), and temp is a learnable
parameter.

-- 2 of 6 --

(a) LangBridge (b) Proposed (c) Training (d) Inferencing
Fig. 1: Figure 1: Model comparison and usage. (a) LangBridge uses only the final encoder layer. (b) Our approach fuses all hidden layers via a fusion module.
(c) Training with English inputs. (d) Zero-shot inference on a low-resource language.
2) Token-Wise Softmax Weighting: Rather than applying a
uniform weighting across all tokens, we compute a unique
layer attention distribution for each token individually. The
central hypothesis underlying this approach is that different to-
kens may benefit from information residing in different layers
of the multilingual encoder. By leveraging the Encoder-based
attention mechanism in the Transformer architecture [36] over
the set of a token’s layer-wise representations, the model can
learn to attend to the most informative layers per token.
This fine-grained, token-specific layer fusion enables the
model to dynamically extract semantic features best suited for
each input position. Importantly, such a mechanism can

rchitecture [36] over
the set of a token’s layer-wise representations, the model can
learn to attend to the most informative layers per token.
This fine-grained, token-specific layer fusion enables the
model to dynamically extract semantic features best suited for
each input position. Importantly, such a mechanism can be
particularly beneficial for languages that share similar scripts
with English. Since attention weights are computed at the
token level, the model can more effectively align similar
subword patterns across languages, thereby enhancing transfer
and generalization in multilingual instruction following.
Let E be a multilingual encoder with L layers. Given
an input sequence X = {x1, x2, . . . , xT } in a non-English
language, the encoder produces for each token xt a set of
hidden states from all layers:
Ht = {h(1)
t , h(2)
t , . . . , h(L)
t }, h(l)
t ∈ Rd (4)
where d is the hidden dimension of the encoder.
To learn token-specific importance over layers, we augment
Ht with a learnable query vector c ∈ Rd, and form the
following sequence:
St = [c; h(1)
t ; . . . ; h(L)
t ] ∈ R(L+1)×d (5)
We then add positional embeddings P ∈ R(L+1)×d to
St and pass the result through a Transformer encoder block
TransEnc(·):
Ut = TransEncSt + P
 (6)
Let u(0)
t ∈ Rd be the transformed output corresponding
to the query vector c. We then project this vector to a layer
attention score vector αt ∈ RL:
αt = softmaxWa u(0)
t + ba

 (7)
where Wa ∈ RL×d and ba ∈ RL are trainable parameters.
The final token representation ˜ht ∈ Rd is computed as the
weighted sum over all encoder layers:
˜ht =
L	X
l=1
α(l)
t · h(l)
t (8)
B. Mapping to LLM Embedding Space
Since the multilingual encoder and LLM may operate in dif-
ferent vector spaces, we apply a linear projection Wp ∈ Rd′×d
to map the fused representations into the LLM’s embedding
space Rd′
:
zt = Wp ˜ht + bp (9)
The transformed sequence Z = {z1, . . . , zT } is then used
as input to the frozen LLM decoder.
C. Training Objective
We

tilingual encoder and LLM may operate in dif-
ferent vector spaces, we apply a linear projection Wp ∈ Rd′×d
to map the fused representations into the LLM’s embedding
space Rd′
:
zt = Wp ˜ht + bp (9)
The transformed sequence Z = {z1, . . . , zT } is then used
as input to the frozen LLM decoder.
C. Training Objective
We train the fusion module and the projection layer while
keeping the encoder and LLM weights frozen. The training is
conducted using a prefix language modeling objective: given
input X in a non-English language, the model is trained
to generate the corresponding output Y = {y1, y2, . . . , yK }

-- 3 of 6 --

in English. The objective is the cross-entropy loss over the
predicted token distribution:
LCE = −
K	X
k=1
log pyk | y<k, Z
 (10)
This setup enables multilingual instruction following a zero-
shot setting by leveraging pretrained reasoning capabilities
of the LLM and language representations of the multilingual
encoder.
IV. EXPERIMENTAL SETUP
A. Datasets
To evaluate the effectiveness of our proposed method, we
conduct experiments on a diverse set of multilingual bench-
marks that assess natural language understanding, reasoning,
and generalization to low-resource languages. We use the
XNLI dataset [27]. To further evaluate performance on low-
resource languages, we utilize the IndicXNLI dataset [28],
an extension of XNLI designed for Indian languages with
limited resources. Additionally, we include the Sinhala News
Classification dataset [30], which involves categorizing news
headlines written in Sinhala. We also employ the multilingual
Amazon Reviews sentiment analysis dataset [31] to test sen-
timent classification across multiple languages.
B. Models and Baseline
For our experiments, we use Llama 3.2 1B Instruct [37]
as the base large language model (LLM). This model has
strong instruction-following capabilities in English but lacks
robust multilingual understanding due to its English-centric
pretraining. As the multilingual encoder, we employ

languages.
B. Models and Baseline
For our experiments, we use Llama 3.2 1B Instruct [37]
as the base large language model (LLM). This model has
strong instruction-following capabilities in English but lacks
robust multilingual understanding due to its English-centric
pretraining. As the multilingual encoder, we employ mT5-
Large, which is pretrained on a wide range of multilingual data
and serves as a strong source of cross-lingual representations.
Given the high computational cost associated with large-scale
LLMs, we restrict our experiments to smaller-scale models to
ensure feasibility and reproducibility.
Our primary baseline is LangBridge [21], a recent and
competitive architecture designed to enhance the multilingual
capabilities of LLMs by aligning a multilingual encoder with
an English-centric decoder. This was enhanced by Mind-
merger [22] with the use of parallel instruction data. In our
experiments, we do not consider using parallel instruction
data (non-English instruction and English response) because
it does not represent real world scenarios where parallel
instruction data are limited in non-English languages. Thus
Mindmerger is not considered a baseline. We compare our
proposed method against LangBridge using two variants of
our model: Global Softmax Weighting (Global Softmax) and
Token-Wise Softmax Weighting (Transformer SM).
C. Training details
For each dataset, we train the model for 3 epochs using a
learning rate of 3e-5 and a cosine learning rate scheduler. We
use only the English subset for training and evaluate the model
on both English and other languages. For Global Softmax
Weighting, we set the temperature to 1e-2. For Token-Wise
Softmax Weighting, we use a single transformer encoder layer
with 4 attention heads. All models are trained on a single RTX
4090 GPU with 24 GB of VRAM.
V. RESULTS AND ANALYSIS
Our evaluation across multiple multilingual benchmarks
reveals that integrating intermediate encoder layers through
learned fusion

Token-Wise
Softmax Weighting, we use a single transformer encoder layer
with 4 attention heads. All models are trained on a single RTX
4090 GPU with 24 GB of VRAM.
V. RESULTS AND ANALYSIS
Our evaluation across multiple multilingual benchmarks
reveals that integrating intermediate encoder layers through
learned fusion significantly enhances the performance of
English-centric LLMs, especially in zero-shot scenarios in-
volving low-resource languages.
A. Overall Performance
Both of our proposed models, Global Softmax and Trans-
former SM, consistently outperform the LangBridge baseline
in all evaluated data sets. As shown in Table I, Transformer
SM achieves the highest average accuracy on the XNLI
benchmark, with improvements evident in both high-resource
(e.g., English, French) and non-English languages. This trend
persists across IndicXNLI (Table II), Sinhala News Classifi-
cation (Table III), and Amazon Review Sentiment Analysis
(Table IV), demonstrating strong generalization and transfer
capabilities.
B. Improvements in Low-Resource Languages
The most significant gains are observed in LRLs, which
are traditionally underserved by English-centric LLMs. In
the IndicXNLI benchmark (Table II), both fusion strategies
outperform LangBridge in nearly all languages, with marked
improvements in Assamese, Gujarati, and Malayalam. Sim-
ilarly, on the Sinhala News Classification task (Table III),
Transformer SM notably boosts performance over LangBridge,
indicating its ability to capture meaningful linguistic represen-
tations even in limited-resource settings. These results validate
our core hypothesis that incorporating representations from all
encoder layers provides richer linguistic grounding, particu-
larly crucial for LRLs where single-layer representations may
not be sufficient to convey complex linguistic structure.
C. Token-Wise Fusion vs. Global Weighting
While Global Softmax improves over LangBridge in most
settings, Transformer SM often yields the best results,

provides richer linguistic grounding, particu-
larly crucial for LRLs where single-layer representations may
not be sufficient to convey complex linguistic structure.
C. Token-Wise Fusion vs. Global Weighting
While Global Softmax improves over LangBridge in most
settings, Transformer SM often yields the best results, par-
ticularly in morphologically rich and structurally diverse lan-
guages such as Tamil, Malayalam, Arabic, and Russian. This
suggests that the token-wise fusion mechanism offers finer
control over the alignment process, enabling the model to
dynamically select the most relevant layers on a per-token
basis. The advantage is especially visible in the XNLI (Table I)
and Indic XNLI (Table II) datasets, where Transformer SM
either matches or exceeds the performance of Global Softmax
across languages.
D. Cross-Task Generalization
Our architecture generalizes well beyond natural language
inference. In sentiment analysis (Table IV), Transformer SM
again demonstrates consistent improvements over LangBridge,
particularly in non-English languages such as Spanish, French,

-- 4 of 6 --

TABLE I: XNLI Results (accuracy)
Experiment avg en ar bg de el es fr hi ru sw th tr ur vi zh
LangBridge 70.36 77.31 69.62 73.35 71.54 71.76 74.01 73.43 67.35 71.16 66.09 67.49 68.54 65.37 69.80 68.58
Global Softmax 70.63 79.30 69.24 74.75 71.74 72.75 75.51 74.49 68.14 71.32 62.81 66.69 68.20 64.45 70.86 69.24
Transformer SM 71.50 79.42 70.34 74.99 73.87 73.47 75.43 75.55 68.02 72.22 67.27 67.01 69.56 64.95 70.60 69.80
TABLE II: INDIC XNLI Results (accuracy)
Experiment avg as bn gu hi kn ml mr or pa ta te
LangBridge 65.75 62.69 68.70 67.21 70.54 67.29 66.17 64.07 57.05 66.41 66.63 66.49
Global Softmax 66.96 65.03 69.08 68.94 71.38 68.44 66.59 65.27 58.40 69.34 66.73 67.37
Transformer SM 66.94 64.17 69.70 68.66 71.26 69.48 68.06 64.95 55.11 68.96 67.50 68.52
TABLE III: News Results (accuracy)
Experiment en si
LangBridge 88.23 71.66
Global Softmax 88.97 73.24
Transformer SM 88.79

7.05 66.41 66.63 66.49
Global Softmax 66.96 65.03 69.08 68.94 71.38 68.44 66.59 65.27 58.40 69.34 66.73 67.37
Transformer SM 66.94 64.17 69.70 68.66 71.26 69.48 68.06 64.95 55.11 68.96 67.50 68.52
TABLE III: News Results (accuracy)
Experiment en si
LangBridge 88.23 71.66
Global Softmax 88.97 73.24
Transformer SM 88.79 75.86
TABLE IV: Amazon Review Sentiment Analysis (accuracy)
Experiment en es fr de zh ja avg
LangBridge 64.62 54.84 55.40 61.04 50.64 51.60 56.36
Global Softmax 64.92 55.22 56.60 60.9 51.26 51.66 56.76
Transformer SM 65.06 55.72 56.00 60.80 50.90 52.46 56.82
TABLE V: Inference Latency and GPU Memory Usage.
Model/Method Latency ms per token Peak VRAM MB
Translation pipeline 234.68 5000.03
Langbridge 24.23 11453.35
Global Softmax 23.74 11453.51
Transformer SM 25.41 11535.03
and Japanese. This illustrates the effectiveness of our approach
across diverse NLP task types, confirming that the enriched
encoder fusion improves not just language understanding but
also task-specific reasoning.
E. Layer Weight Distribution Analysis
Fig. 2: Comparison of layer-wise contributions and weight distribution in the
global softmax mechanism during the XNLI experiment.
To better understand the model’s behavior, we visualize
the learned layer weights in Figure 2. The plot reveals that
lower layers contribute minimally, while middle and upper
layers receive significantly higher weights, peaking around
layers 15–19. This pattern is consistent with prior findings that
intermediate encoder layers often encode critical syntactic and
semantic features [38].
F. Inference Efficiency Analysis
Our efficiency analysis on the Sinhala News Classification
dataset (Table V) reveals a key trade-off. A baseline Trans-
lation pipeline using NLLB 1.3B translation model has low
memory usage but suffers from high latency. Conversely, the
fusion-based methods (‘LangBridge‘, ‘Global Softmax‘, and
‘Transformer SM‘) show comparably low latencies, making
them significantly more time-efficient for

) reveals a key trade-off. A baseline Trans-
lation pipeline using NLLB 1.3B translation model has low
memory usage but suffers from high latency. Conversely, the
fusion-based methods (‘LangBridge‘, ‘Global Softmax‘, and
‘Transformer SM‘) show comparably low latencies, making
them significantly more time-efficient for real-time applica-
tions despite their higher VRAM requirements.
VI. CONCLUSION
This work introduces a fusion-based method to enhance
the multilingual capabilities of English-centric Large Lan-
guage Models (LLMs) by incorporating all hidden layers of
a frozen multilingual encoder. Unlike prior approaches that
rely only on the final layer, our method uses either global
or token-wise fusion to generate richer token embeddings,
trained using a prefix language modeling objective without
requiring parallel multilingual data. Experiments on different
datasets show consistent improvements, especially in LRLs.
The Transformer Softmax variant achieved the highest gains.
These results confirm the value of deep encoder fusion in zero-
shot multilingual settings. Thus, our findings highlight the
value of multilayer encoder fusion in bridging performance
gaps for LRLs and offer a scalable path to more inclusive
multilingual NLP. However, due to computational constraints,
the evaluations were limited to an LLM with 1B parameters.
It remains to be tested whether our method provides similar
gains when applied to larger or already multilingual-tuned
LLMs such as Mistral-MoE. This is a key direction for future
research.
REFERENCES
[1] T. Brown, B. Mann et al., “Language models are few-
shot learners,” NeurIPS, vol. 33, pp. 1877–1901, 2020.
[2] A. Chowdhery, S. Narang et al., “Palm: Scaling language
modeling with pathways,” Journal of Machine Learning
Research, vol. 24, no. 240, pp. 1–113, 2023.

-- 5 of 6 --

[3] F. D. Puspitasari, C. Zhang et al., “Deepseek models:
A comprehensive survey of methods and applications,”
Authorea Preprints, 2025.
[4] J. Devlin, M.-W.

dhery, S. Narang et al., “Palm: Scaling language
modeling with pathways,” Journal of Machine Learning
Research, vol. 24, no. 240, pp. 1–113, 2023.

-- 5 of 6 --

[3] F. D. Puspitasari, C. Zhang et al., “Deepseek models:
A comprehensive survey of methods and applications,”
Authorea Preprints, 2025.
[4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova,
“Bert: Pre-training of deep bidirectional transformers for
language understanding,” in NAACL, 2019, pp. 4171–
4186.
[5] M. Lewis, Y. Liu et al., “BART: Denoising sequence-
to-sequence pre-training for natural language genera-
tion, translation, and comprehension,” arXiv preprint
arXiv:1910.13461, 2019.
[6] C. Raffel, N. Shazeer et al., “Exploring the limits of
transfer learning with a unified text-to-text transformer,”
Journal of machine learning research, vol. 21, no. 140,
pp. 1–67, 2020.
[7] J. Kaplan, S. McCandlish et al., “Scaling laws for neu-
ral language models,” arXiv preprint arXiv:2001.08361,
2020.
[8] J. Wei, X. Wang et al., “Chain-of-thought prompting
elicits reasoning in large language models,” NeurIPS,
vol. 35, pp. 24 824–24 837, 2022.
[9] L. Yu, W. Jiang et al., “Metamath: Bootstrap your
own mathematical questions for large language models,”
arXiv preprint arXiv:2309.12284, 2023.
[10] S. Bai, K. Chen et al., “Qwen2. 5-vl technical report,”
arXiv preprint arXiv:2502.13923, 2025.
[11] L. Qin, Q. Chen et al., “Cross-lingual prompting: Im-
proving zero-shot chain-of-thought reasoning across lan-
guages,” arXiv preprint arXiv:2310.14799, 2023.
[12] A. Conneau, K. Khandelwal et al., “Unsupervised cross-
lingual representation learning at scale,” arXiv preprint
arXiv:1911.02116, 2019.
[13] J. Libovick`y, R. Rosa, and A. Fraser, “How
language-neutral is multilingual bert?” arXiv preprint
arXiv:1911.03310, 2019.
[14] L. Xue, N. Constant et al., “mt5: A massively multilin-
gual pre-trained text-to-text transformer,” arXiv preprint
arXiv:2010.11934, 2020.
[15] H. Choi, J. Kim et al., “Analyzing zero-shot

3] J. Libovick`y, R. Rosa, and A. Fraser, “How
language-neutral is multilingual bert?” arXiv preprint
arXiv:1911.03310, 2019.
[14] L. Xue, N. Constant et al., “mt5: A massively multilin-
gual pre-trained text-to-text transformer,” arXiv preprint
arXiv:2010.11934, 2020.
[15] H. Choi, J. Kim et al., “Analyzing zero-shot cross-lingual
transfer in supervised nlp tasks,” in ICPR. IEEE, 2021,
pp. 9608–9613.
[16] Q. Wang, F. Li et al., “Multi-layer representation fu-
sion for neural machine translation,” arXiv preprint
arXiv:2002.06714, 2020.
[17] Z. Li, K. Chen et al., “Xifbench: Evaluating large
language models on multilingual instruction following,”
arXiv preprint arXiv:2503.07539, 2025.
[18] W. Zhu, S. Huang et al., “Question translation train-
ing for better multilingual reasoning,” arXiv preprint
arXiv:2401.07817, 2024.
[19] A. Ebrahimi and K. von der Wense, “Zero-shot vs.
translation-based cross-lingual transfer: The case of lex-
ical gaps,” in NAACL, 2024, pp. 443–458.
[20] F. Shi, M. Suzgun et al., “Language models are
multilingual chain-of-thought reasoners,” arXiv preprint
arXiv:2210.03057, 2022.
[21] D. Yoon, J. Jang et al., “Langbridge: Multilingual rea-
soning without multilingual supervision,” arXiv preprint
arXiv:2401.10695, 2024.
[22] Z. Huang, W. Zhu et al., “Mindmerger: Efficiently boost-
ing llm reasoning in non-english languages,” NeurIPS,
vol. 37, pp. 34 161–34 187, 2024.
[23] N. Chen, Z. Zheng, N. Wu, M. Gong, D. Zhang, and J. Li,
“Breaking language barriers in multilingual mathematical
reasoning: Insights and observations,” arXiv preprint
arXiv:2310.20246, 2023.
[24] N. Goyal, C. Gao et al., “The flores-101 evaluation
benchmark for low-resource and multilingual machine
translation,” TACL, vol. 10, pp. 522–538, 05 2022.
[25] M. R. Costa-Juss`a, J. Cross et al., “No language left
behind: Scaling human-centered machine translation,”
arXiv preprint arXiv:2207.04672, 2022.
[26] C. Escolano, M. R. Costa-juss`a et al., “Multilingual,
multi-scale and

or low-resource and multilingual machine
translation,” TACL, vol. 10, pp. 522–538, 05 2022.
[25] M. R. Costa-Juss`a, J. Cross et al., “No language left
behind: Scaling human-centered machine translation,”
arXiv preprint arXiv:2207.04672, 2022.
[26] C. Escolano, M. R. Costa-juss`a et al., “Multilingual,
multi-scale and multi-layer visualization of intermediate
representations,” arXiv preprint arXiv:1907.00810, 2019.
[27] A. Conneau, G. Lample et al., “Xnli: Evaluating
cross-lingual sentence representations,” arXiv preprint
arXiv:1809.05053, 2018.
[28] D. Aggarwal, V. Gupta, and A. Kunchukuttan, “Indicxnli:
Evaluating multilingual inference for indian languages,”
arXiv preprint arXiv:2204.08776, 2022.
[29] N. De Silva, “Survey on publicly available sinhala natural
language processing tools and research,” arXiv preprint
arXiv:1906.02358v25, 2025.
[30] N. de Silva, “Sinhala text classification: observations
from the perspective of a resource poor language,” Re-
searchGate, 2015.
[31] Y. Hou, J. Li et al., “Bridging language and items
for retrieval and recommendation,” arXiv preprint
arXiv:2403.03952, 2024.
[32] H. Liu, C. Li et al., “Visual instruction tuning,” NeurIPS,
vol. 36, pp. 34 892–34 916, 2023.
[33] A. Bapna, M. X. Chen et al., “Training deeper neural
machine translation models with transparent attention,”
arXiv preprint arXiv:1808.07561, 2018.
[34] X. Liu, L. Wang et al., “Understanding and improving
encoder layer fusion in sequence-to-sequence learning,”
arXiv preprint arXiv:2012.14768, 2020.
[35] M. ElNokrashy, B. AlKhamissi, and M. Diab, “Depth-
wise attention (dwatt): A layer fusion method
for data-efficient classification,” arXiv preprint
arXiv:2209.15168, 2022.
[36] A. Vaswani, N. Shazeer et al., “Attention is all you need,”
NeurIPS, vol. 30, 2017.
[37] Meta AI, “Introducing llama 3: Advancing open founda-
tion models,” 2023.
[38] R. Choenni, E. Shutova, and D. Garrette, “Examining
modularity in multilingual lms via language-specialized
subnetworks,”

arXiv:2209.15168, 2022.
[36] A. Vaswani, N. Shazeer et al., “Attention is all you need,”
NeurIPS, vol. 30, 2017.
[37] Meta AI, “Introducing llama 3: Advancing open founda-
tion models,” 2023.
[38] R. Choenni, E. Shutova, and D. Garrette, “Examining
modularity in multilingual lms via language-specialized
subnetworks,” arXiv preprint arXiv:2311.08273, 2023.

-- 6 of 6 --