LLM Parameters for Math Across Languages: Shared or Separate?
Summary
This study investigates whether mathematical reasoning in large language models (LLMs) relies on shared parameters across languages or distinct, language-specific circuits. Using the MathNeurosurgery framework, the authors localized math-specific parameters in Llama and Qwen models across English, German, French, and Hindi. They found that math-related behavior is neither fully invariant nor fully specific; instead, it exhibits partial cross-lingual overlap, with the strongest similarity concentrated in intermediate model layers. English consistently possessed the largest set of math-relevant parameters, correlating with its superior reasoning performance. Lower-resource languages like Hindi showed fewer relevant parameters and less overlap with English, potentially due to script and tokenization differences. Surprisingly, non-English language pairs (e.g., German-Hindi) sometimes exhibited higher parameter overlap than English-Hindi pairs, suggesting a shared non-English reasoning subspace. Functional validation via weight interventions confirmed these findings. Pruning identified parameters significantly impaired mathematical performance while leaving general language understanding intact, indicating a collective effect rather than reliance on individual critical weights. Scaling these parameters primarily corrected arithmetic execution errors rather than altering reasoning logic. The results suggest that while a core substrate supports multilingual math reasoning, models allocate dedicated sub-networks for language-specific processing, particularly in deeper layers.
PDF viewer
Chunks(59)
Chunk 0 · 1,987 chars
LLM Parameters for Math Across Languages: Shared or Separate? Behzad Shomali1,2† Luisa Victor2† Tim Selbach2† Ali Hamza Bashir1,3 David Berghaus1,3 Joachim Koehler1,3 Mehdi Ali1,3 Markus Frey1,2,3 1Lamarr Institute 2University of Bonn 3 Fraunhofer IAIS † Equal contribution Abstract Large language models (LLMs) exhibit sub- stantial cross-lingual variation in mathematical reasoning performance, but it remains unclear whether these differences reflect language- specific parameters or a shared mechanism that manifests differently by language. We present a cross-lingual mechanistic analysis of mathe- matical reasoning in LLMs, enabling us to lo- calize and compare model parameters that sup- port mathematical reasoning across languages. We find that the extracted math-associated pa- rameters exhibit partial cross-lingual overlap, with the strongest overlap concentrated in inter- mediate model layers. We further observe that English consistently produces the largest set of math-relevant parameters, whereas lower- resource languages reveal smaller sets of rel- evant parameters. These results suggest that math-related behavior in multilingual LLMs is neither fully language-invariant nor fully language-specific, but instead exhibits partial cross-lingual parameter overlap with system- atic language-dependent differences. Code: https://github.com/luisavictor/math- across-languages Translated datasets: https://huggingface .co/math-across-languages 1 Introduction Large language models (LLMs) have achieved re- markable success across a range of tasks, yet their performance on multilingual mathematical reason- ing remains uneven (Chen et al., 2024). Under- standing how these models process mathematical tasks across languages is critical for building ro- bust multilingual systems. Existing research on LLM capabilities can be broadly categorized into two streams. First, mech- anistic studies localize task-specific skills to sub- sets of model parameters. Recent methods
Chunk 1 · 1,995 chars
r- standing how these models process mathematical tasks across languages is critical for building ro- bust multilingual systems. Existing research on LLM capabilities can be broadly categorized into two streams. First, mech- anistic studies localize task-specific skills to sub- sets of model parameters. Recent methods lever- Correspondence to: behzad.shomali@uni-bonn.de age forward-pass statistics (Sun et al., 2023). No- tably, Christ et al. (2025) introduced MathNeuro- surgery, isolating parameters critical to mathemat- ical reasoning in LLMs. Second, multilingual rea- soning studies primarily evaluate performance or representational properties. Datasets and strate- gies like MathOctopus (Chen et al., 2024) and the UST dataset (Ko et al., 2025) target cross-lingual generalization, while Liu et al. (2026) suggests that English-centric latent pathways may domi- nate multilingual reasoning. However, these ap- proaches rarely identify whether the same param- eters support mathematical reasoning across lan- guages. Despite these advances, several challenges re- main. Mechanistic analyses are mostly limited to English, leaving open questions about cross- lingual parameter sharing. Similarly, performance- focused multilingual studies do not reveal the underlying computational substrates of reasoning. These gaps hinder our understanding of whether mathematical reasoning relies on a universal core or language-specific circuits. To address this gap, we investigate the localization and overlap of math- specific parameters in LLMs. Our contributions are threefold: • Mechanistic localization of multilingual math circuits: We develop a framework to identify and compare model parameters supporting mathematical reasoning across languages. • Language-dependent variation in parame- ter allocation: We show that English contains the largest set of math-specific parameters, while lower-resource languages have fewer. • Functional validation via weight interven- tion: We demonstrate
Chunk 2 · 1,997 chars
compare model parameters supporting mathematical reasoning across languages. • Language-dependent variation in parame- ter allocation: We show that English contains the largest set of math-specific parameters, while lower-resource languages have fewer. • Functional validation via weight interven- tion: We demonstrate that mathematical rea- soning emerges as a collective property of these localized parameters. arXiv:2606.18453v1 [cs.CL] 16 Jun 2026 -- 1 of 24 -- 2 Related Work Mechanistic analysis of LLMs. Several stud- ies have explored localizing task-specific capa- bilities within LLM parameters. Gradient-based attribution techniques identify parameters critical to specific tasks (Panigrahi et al., 2023), while forward-pass statistics offer a computationally ef- ficient alternative (Sun et al., 2023). Christ et al. (2025) proposed MathNeurosurgery, which isolates math-specific parameters by comparing weight–activation products on math and non-math datasets. However, their analysis primarily fo- cuses on English and does not examine cross- lingual parameter behavior. Multilingual LLMs. Research on multilingual LLMs has explored dataset creation and evalua- tion. MathOctopus (Chen et al., 2024) and the UST dataset (Ko et al., 2025) target cross-lingual generalization, while other studies suggest that multilingual models often rely on English-centric latent pathways (Liu et al., 2026). These anal- yses typically evaluate performance or represen- tations, leaving the underlying parameter-level mechanisms unexplored. Cross-lingual mechanistic studies. Some recent work has investigated neuron activation patterns across translated inputs to identify shared func- tional units (Wang et al., 2024). Our study differs by using a parameter-importance framework, ana- lyzing both attention and Feed-Forward Network (FFN) weights, to localize math-specific circuits and quantify their overlap across languages. 3 Methodology We employ the MathNeurosurgery frame- work (Christ et
Chunk 3 · 1,991 chars
red func- tional units (Wang et al., 2024). Our study differs by using a parameter-importance framework, ana- lyzing both attention and Feed-Forward Network (FFN) weights, to localize math-specific circuits and quantify their overlap across languages. 3 Methodology We employ the MathNeurosurgery frame- work (Christ et al., 2025) to identify math-specific parameters and subsequently analyze their overlap, both at the level of the entire model and across individual layers, across multiple languages. 3.1 Parameter Identification Following (Christ et al., 2025), we first compute the task-specific Sij for weight j in neuron i: Sij = N ∑ k=1 |Wij | · ∥ ∥ ∥Xk j ∥ ∥ ∥2 for X ∈ D (1) where Wij represents the weight, |.| denotes the absolute value operator, and ∥Xk j ∥2 is the L2 norm of the j-th feature aggregated across input tokens to normalize the input X from dataset D for sample k. Using this formulation, we compute scores for each parameter over math (Dmath) and non-math datasets (Dnon-math). Then we separately identify the top-k proportion of parameters with the high- est score for each task in each layer. Finally, we identify the math-specific parameters by taking the subset of parameters that are crucial for the math task but not for the unrelated task. 3.2 Similarity Metric To measure similarity between two sets of ex- tracted parameters, we first separately run the pre- vious step for each language, and then measure their pairwise overlaps using the Jaccard coeffi- cient. Let T1 and T2 denote the sets of important parameters representing task-specific parameters for the same task in two languages. The overlap between the two parameter sets is computed as: J(T1, T2) = |T1 ∩ T2| |T1 ∪ T2| (2) 4 Results We begin with the experimental setup, followed by an analysis of the results. 4.1 Experimental Setup We evaluate Llama 3.2 1B IT (Llama 1B) (Meta AI, 2024b), Qwen3 4B IT-2507 (Qwen3 4B) (Qwen Team, 2025), and Llama 3.1 8B IT (Llama 8B) (Meta AI, 2024a) across
Chunk 4 · 1,992 chars
computed as: J(T1, T2) = |T1 ∩ T2| |T1 ∪ T2| (2) 4 Results We begin with the experimental setup, followed by an analysis of the results. 4.1 Experimental Setup We evaluate Llama 3.2 1B IT (Llama 1B) (Meta AI, 2024b), Qwen3 4B IT-2507 (Qwen3 4B) (Qwen Team, 2025), and Llama 3.1 8B IT (Llama 8B) (Meta AI, 2024a) across English, German, French, and Hindi on GSM8K (Cobbe et al., 2021) for math reasoning (Dmath), MMLU (Hendrycks et al., 2020) and RACE (Lai et al., 2017) as general language understanding datasets (Dnon-math). We mainly focus on Llama 1B in the main text, while results for larger models are in the Appendices. 4.2 Does math performance correlate with the model’s capacity for each language? Figure 1 (left) illustrates that, for each top-k value, English consistently exhibits the highest number of math-specific parameters, followed by German and French, and then Hindi. This pattern aligns with the observed reasoning performance across languages, as shown in Table 1. More specifically, the trend in the number of math-specific parame- ters aligns with the relative performance ranking across languages. For example, in Figure 1, at top-k = 0.01, English has the highest parame- ter count, French and German have approximately equal counts, and Hindi has the lowest. Moreover, -- 2 of 24 -- Model English German Hindi French GSM8K RACE GSM8K RACE GSM8K RACE GSM8K RACE Llama 1B 0.340 0.379 0.235 0.321 0.145 0.299 0.185 0.348 Qwen3 4B 0.735 0.414 0.685 0.356 0.385 0.353 0.690 0.403 Llama 8B 0.765 0.448 0.585 0.396 0.415 0.388 0.580 0.413 Table 1: Pretraining accuracy (%) on GSM8K and RACE across English, German, Hindi, and French. 0 0.05 0.1 0.15 Top-k 0.0 0.5 1.0 1.5 2.0 # Math-specific Params ×107 Parameter Count 0.0001 0.001 0.01 0.05 0.1 0.15 Top-k En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair 0.221 0.240 0.222 0.211 0.194 0.199 0.251 0.256 0.233 0.222 0.204 0.209 0.120 0.140 0.162 0.159 0.151
Chunk 5 · 1,992 chars
nd French. 0 0.05 0.1 0.15 Top-k 0.0 0.5 1.0 1.5 2.0 # Math-specific Params ×107 Parameter Count 0.0001 0.001 0.01 0.05 0.1 0.15 Top-k En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair 0.221 0.240 0.222 0.211 0.194 0.199 0.251 0.256 0.233 0.222 0.204 0.209 0.120 0.140 0.162 0.159 0.151 0.145 0.195 0.217 0.220 0.193 0.177 0.186 0.187 0.206 0.212 0.190 0.173 0.181 0.383 0.415 0.379 0.336 0.322 0.316 0.000 0.000 0.005 0.024 0.047 0.068 Global Jaccard Similarity 0.1 0.2 0.3 0.4 En De Fr Hi Figure 1: Math-specific parameters in Llama 1B. Left: Number of isolated parameters per language. Right: Jaccard similarity between language-specific parameter sets. the gap between Hindi and French is smaller than that between Hindi and English. This pattern be- comes even more pronounced with larger models (Figures F.4, F.5). Finding: The distribution of math-specific pa- rameters across languages tends to align with their relative reasoning performance in LLMs. 4.3 Is there a shared cross-lingual core for mathematical reasoning? Figure 1 highlights the number of math-associated parameters in English (En), German (De), Hindi (Hi) and French (Fr), as well as the Jaccard simi- larity across language pairs for Llama 1B. Across all language pairs, the Jaccard similarity primarily peaks in the intermediate layers and decreases in deeper layers. The same behavior is also observed in larger models (Figures F.1-F.3). This pattern is consistent with prior findings that earlier lay- ers capture more surface-level linguistic features, while later layers can become more specialized and language-dependent (Tang et al., 2024). Figures 1 and 2 demonstrate that English con- sistently possesses the highest number of math- associated parameters, both in the entire model and in nearly every layer. This implies that the model possesses a more distributed and resilient internal circuit for mathematical computations in English. Notably, the
Chunk 6 · 1,993 chars
2024). Figures 1 and 2 demonstrate that English con- sistently possesses the highest number of math- associated parameters, both in the entire model and in nearly every layer. This implies that the model possesses a more distributed and resilient internal circuit for mathematical computations in English. Notably, the Jaccard similarity remains substan- tially above the random baseline across all pairings, except in the initial and final few layers. This sug- gests the presence of a partially shared parameter substrate associated with mathematical problem solving, especially in intermediate layers. Lower similarity in the initial and final layers, especially in larger models (Figures F.2, F.3), likely reflects two distinct phenomena. In early layers, low over- lap is consistent with the surface-level processing discussed above. In later layers, it instead points to increased language-specific specialization: with larger parameter budgets, models can mitigate the curse of multilinguality (Conneau et al., 2020) by allocating dedicated sub-networks for language- specific reasoning, rather than relying on shared parameters, consistent with other findings on layer specialization in LLMs (Frey et al., 2026). While Figures 1 and 3 confirm that lexically sim- ilar, high-resource languages like French and Ger- man exhibit the highest overlap, the lower simi- larity observed in pairs involving Hindi, particu- larly En-Hi, suggests a structural barrier imposed by script and tokenization. Because English and Hindi share neither an alphabet nor substantial lex- ical overlap, the model appears to rely on more dis- joint parameter subsets to map divergent input en- codings to a shared internal representation. This pattern is also observed in Llama 8B, independent of model scale (Figure F.3). In contrast, the effect is less pronounced in Qwen3 4B (Figure F.2), po- tentially reflecting differences in multilingual pre- training coverage, vocabulary design, or tokeniza- tion. We
Chunk 7 · 1,998 chars
en- codings to a shared internal representation. This pattern is also observed in Llama 8B, independent of model scale (Figure F.3). In contrast, the effect is less pronounced in Qwen3 4B (Figure F.2), po- tentially reflecting differences in multilingual pre- training coverage, vocabulary design, or tokeniza- tion. We emphasize that this interpretation is spec- ulative, as our experiments do not isolate these fac- tors. Surprisingly, the De-Hi and Fr-Hi overlaps in many layers are notably higher than that of En- Hi. We consider two possible explanations. First, French and Hindi as well as German and Hindi have a comparable number of parameters com- pared to English and Hindi, which explains the lower Jaccard scores for En-Hi. The larger denom- inator in the similarity computation using Equa- tion 2 is due to English’s larger set of math- associated parameters, which increases the size of the union and reduces the Jaccard similarity. Sec- ond, an alternative interpretation is the existence of a shared non-English subspace: a multilingual reasoning manifold that the model exploits when operating outside its primary English-centric path- ways. Notably, the pattern in which En-Hi similar- ity is lower than De-Hi, Fr-Hi, and De-Fr persists across almost all layers and scales to larger mod- els (Figures F.2, F.3). This consistent, surprising observation further motivates the hypothesis that non-English languages may rely on partially over- lapping parameter subsets distinct from those used -- 3 of 24 -- for English. Finding: There exists a partially shared parame- ter substrate for math problem-solving, which is most prominent in the intermediate layers. 4.4 How do weights scaling & pruning change the model’s behavior? To validate and investigate the effectiveness of iso- lated parameters, following Christ et al. (2025), we apply two interventions: pruning and scaling (full results in Appendix D). In pruning, weights are set to zero, which ideally impairs mathematical
Chunk 8 · 1,999 chars
4.4 How do weights scaling & pruning change the model’s behavior? To validate and investigate the effectiveness of iso- lated parameters, following Christ et al. (2025), we apply two interventions: pruning and scaling (full results in Appendix D). In pruning, weights are set to zero, which ideally impairs mathematical perfor- mance more than non-mathematical performance. In scaling, weights are multiplied by a positive factor (1.1 for Llama 1B, 1.01 otherwise). While scaling may improve performance, the effect is less predictable than pruning, as outcomes depend sensitively on the factor choice. In both settings, the intended trend is observed: non-mathematical performance remains largely unchanged, whereas mathematical reasoning is either significantly re- duced by pruning or enhanced by scaling. Analyzing examples where scaling guides the model to the correct answer, we find that the reasoning logic remains largely unchanged, even though the reasoning may not be entirely accurate and may include some missing steps (Figure E.2). However, for all four languages, the gains mainly come from correcting errors in arithmetic compu- tations, as shown in Figures E.1-E.4. This obser- vation explains the trivial improvements observed in larger models, which already have strong arith- metic capabilities and are therefore less responsive to small parameter perturbations. Tables D.9 and D.10 demonstrate an emerging pattern when pruning Qwen3 4B model with top- k=1e-5. Instead of dropping performance, perfor- mance on the English GSM8K task improved re- markably. Manual inspection revealed that sup- pressing such parameters mainly improved out- put formatting and better in-context learning (Fig- ure E.9). More details in Appendix D. Finding: Scaling math-specific parameters pri- marily corrects arithmetic execution rather than reasoning logic. In some cases, pruning may im- prove GSM8K accuracy by changing output for- mat or few-shot imitation behavior, rather than by improving
Chunk 9 · 1,989 chars
etter in-context learning (Fig- ure E.9). More details in Appendix D. Finding: Scaling math-specific parameters pri- marily corrects arithmetic execution rather than reasoning logic. In some cases, pruning may im- prove GSM8K accuracy by changing output for- mat or few-shot imitation behavior, rather than by improving reasoning itself. 0 2 4 6 8 10 12 14 Layer Index 0.60 0.90 1.20 # Isolated Params ×106 Top-k=0.1 En De Fr Hi Figure 2: Layer-wise distribution of math-reasoning pa- rameters per language for Llama 1B. 0 2 4 6 8 10 12 14 Layer Index En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .136 .146 .191 .189 .217 .231 .237 .255 .235 .238 .203 .221 .173 .148 .159 .120 .139 .148 .195 .194 .226 .246 .248 .271 .250 .258 .218 .233 .204 .140 .167 .131 .095 .109 .151 .157 .159 .196 .191 .191 .177 .167 .146 .177 .126 .117 .140 .110 .115 .159 .189 .176 .205 .210 .208 .191 .172 .204 .183 .199 .173 .150 .167 .116 .113 .150 .177 .166 .196 .202 .204 .192 .175 .205 .176 .193 .174 .140 .163 .121 .192 .297 .358 .393 .379 .391 .425 .424 .354 .400 .325 .331 .273 .205 .245 .198 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 Top-k=0.1 0.1 0.2 0.3 0.4 Figure 3: Layer-wise Jaccard similarity of math- specific parameters across language pairs for Llama 1B. 4.5 Math-specific parameters effect: collective or individual? We investigate whether isolated math-specific pa- rameters exhibit a collective effect, i.e. their im- pact arises from their aggregation, or an individual effect, where performance changes are driven by a small number of critical parameters. To this end, for the selected top-k parameters, we sample ran- dom subsets at several intermediate proportions. For each proportion, we repeat the sampling pro- cess three times and prune the selected parameters. Figure 4 depicts the results for Llama 1B
Chunk 10 · 1,999 chars
nce changes are driven by a small number of critical parameters. To this end, for the selected top-k parameters, we sample ran- dom subsets at several intermediate proportions. For each proportion, we repeat the sampling pro- cess three times and prune the selected parameters. Figure 4 depicts the results for Llama 1B across three languages. The observed trend is approx- imately linear, suggesting that the isolated pa- rameters effect is predominantly collective rather than driven by individual parameters. Specifically, the linear relationship indicates that performance degradation scales with the proportion of impor- tant parameters removed. Moreover, the low stan- dard deviation supports this conclusion, indicating that the results are not sensitive to the specific sub- set selected; instead, performance degradation is primarily determined by the proportion of parame- ters ablated. We observe similar trends for larger models (Figures D.1, D.2). Finding: Identified math-specific parameters have a collective influence on the outcome. 5 Conclusion Our mechanistic analysis suggests that math- associated parameters in multilingual LLMs ex- hibit partial cross-lingual overlap, with the high- est overlap concentrated in intermediate layers. At the same time, deeper layers show more language- specific patterns. We find that English yields the -- 4 of 24 -- 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Pruned Parameter 100 75 50 25 0 Performance Change from Baseline (%) Top-k = 0.001 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Pruned Parameter Top-k = 0.01 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Pruned Parameter Top-k = 0.1 0 51K 102K 152K 203K 254K Parameters Pruned 0 451K 903K 1.4M 1.8M 2.3M Parameters Pruned 0 2.8M 5.7M 8.5M 11.3M 14.2M Parameters Pruned English German Hindi Figure 4: Effect of pruning isolated math-specific parameters on performance across languages for Llama 1B. largest set of extracted math-associated parame- ters, consistent with its
Chunk 11 · 1,983 chars
eters Pruned 0 451K 903K 1.4M 1.8M 2.3M Parameters Pruned 0 2.8M 5.7M 8.5M 11.3M 14.2M Parameters Pruned English German Hindi Figure 4: Effect of pruning isolated math-specific parameters on performance across languages for Llama 1B. largest set of extracted math-associated parame- ters, consistent with its stronger performance in our experiments. Lower-resource languages and languages with different scripts, such as Hindi, show less overlap with English, while still sharing some overlap with other non-English languages. These results suggest a distinct pattern of over- lap among non-English math-associated parame- ters. Finally, intervention experiments indicate that the effect of the identified parameters is dis- tributed across the selected set, i.e. exhibiting a collective effect, rather than dominated by a small number of individual parameters. 6 Acknowledgment This research has been funded by the Federal Min- istry of Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr Institute for Machine Learning and Artifi- cial Intelligence. Limitations Our work has several limitations. First, despite manual verification, the translated datasets may still lack full fluency and could benefit from stronger translation models. Second, we focus exclusively on mathematical reasoning, leaving open whether similar shared–core and language- dependent patterns extend to other reasoning forms. Third, our analysis considers a limited set of languages and model families, which may restrict the generality of our conclusions across scripts and tokenization schemes. We leave these directions for future work. References Ameen Ali, Shahar Katz, Lior Wolf, and Ivan Titov. 2025. Detecting and pruning prominent but detri- mental neurons in large language models. arXiv preprint arXiv:2507.09185. Ameen Ali, Lior Wolf, and Ivan Titov. 2024. Mitigat- ing copy bias in in-context learning through neuron pruning. arXiv preprint
Chunk 12 · 1,990 chars
work. References Ameen Ali, Shahar Katz, Lior Wolf, and Ivan Titov. 2025. Detecting and pruning prominent but detri- mental neurons in large language models. arXiv preprint arXiv:2507.09185. Ameen Ali, Lior Wolf, and Ivan Titov. 2024. Mitigat- ing copy bias in in-context learning through neuron pruning. arXiv preprint arXiv:2410.01288. Robert SP Beekes. 2011. Comparative Indo-European Linguistics: An Introduction, 2nd edition. John Ben- jamins Publishing Company, Amsterdam/Philadel- phia. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Aza- lia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787. Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dong- mei Zhang, and Jia Li. 2024. Breaking language barriers in multilingual mathematical reasoning: In- sights and observations. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 7001–7016. Bryan R Christ, Zachary Gottesman, Jonathan Kropko, and Thomas Hartvigsen. 2025. Math neurosurgery: Isolating language models’ math reasoning abilities using only forward passes. In Proceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 24803–24840. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th annual meeting of the associa- tion for computational linguistics, pages 8440–8451. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin
Chunk 13 · 1,998 chars
e, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th annual meeting of the associa- tion for computational linguistics, pages 8440–8451. Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. Fangan Dong, Zuming Yan, Xuri Ge, Zhiwei Xu, Mengqi Zhang, Xuanang Chen, Ben He, Xin Xin, Zhumin Chen, and Ying Zhou. 2026. Identifying and transferring reasoning-critical neurons: Improv- ing llm inference reliability via activation steering. arXiv preprint arXiv:2601.19847. Markus Frey, Behzad Shomali, Ali Hamza Bashir, David Berghaus, Joachim Koehler, and Mehdi Ali. 2026. Adaptive loops and memory in transform- ers: Think harder or know more? arXiv preprint arXiv:2603.08391. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Bider- man, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey -- 5 of 24 -- Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. The language model evaluation har- ness. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Hyunwoo Ko, Guijin Son, and Dasol Choi. 2025. Un- derstand, solve and translate: Bridging the multilin- gual mathematical reasoning gap. arXiv preprint arXiv:2501.02448. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. Yihong Liu, Raoyuan Zhao, Hinrich Schütze, and Michael A Hedderich. 2026. Large reasoning mod- els are
Chunk 14 · 1,989 chars
soning gap. arXiv preprint arXiv:2501.02448. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. Yihong Liu, Raoyuan Zhao, Hinrich Schütze, and Michael A Hedderich. 2026. Large reasoning mod- els are (not yet) multilingual latent reasoners. arXiv preprint arXiv:2601.02996. Meta AI. 2024a. Introducing llama 3.1: Our most capa- ble models to date. Accessed: 2026-01-26. Meta AI. 2024b. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Ac- cessed: 2026-01-26. Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. 2023. Task-specific skill localization in fine-tuned language models. In International Con- ference on Machine Learning, pages 27011–27033. PMLR. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning ap- proach for large language models. arXiv preprint arXiv:2306.11695. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dong- dong Zhang, Xiaolei Wang, Wayne Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. In Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5701– 5715. Weixuan Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. 2024. Sharing mat- ters: Analysing neurons across languages and tasks in llms. arXiv preprint arXiv:2406.09265. -- 6 of 24 -- A Experimental Setup In this section, we describe the languages, the dataset curation, and the translation process em- ployed throughout this study. Languages To evaluate cross-lingual transferabil- ity of mathematical reasoning, we consider En- glish (En), German (De), French (Fr), and Hindi (Hi). This selection enables comparisons across both closely related and more distantly related
Chunk 15 · 1,996 chars
ataset curation, and the translation process em- ployed throughout this study. Languages To evaluate cross-lingual transferabil- ity of mathematical reasoning, we consider En- glish (En), German (De), French (Fr), and Hindi (Hi). This selection enables comparisons across both closely related and more distantly related lan- guages within the Indo-European family. English and German belong to the Germanic branch, while French is a Romance language, and Hindi belongs to the Indo-Aryan branch, following the classifica- tion outlined by Beekes (2011). Dataset curation: Since MMLU includes a sub- stantial number of math-oriented questions, unlike Christ et al. (2025), who include all available subjects, we apply category-based filtering and remove subject areas that are strongly quantitative, as these can confound our parameter extraction by making parameters associated with math- ematical reasoning appear as general language parameters. Concretely, we exclude mathematics- related categories (abstract_algebra, elemen- tary_mathematics, college_mathematics, high_school_mathematics, high_school_statistics), physics cat- egories (conceptual_physics, col- lege_physics, high_school_physics, as- tronomy), chemistry categories (col- lege_chemistry, high_school_chemistry), biology (high_school_biology), com- puter science (college_computer_science, high_school_computer_science), and quanti- tatively oriented fields (econometrics, profes- sional_accounting). Translations: We translate all datasets into Ger- man, Hindi, and French using the pretrained multi- lingual machine translation model facebook/nllb- 200-3.3B (Costa-Jussà et al., 2022). For GSM8K, our mathematical reasoning benchmark, we employ a placeholder-based preprocessing step to preserve the integrity of mathematical expressions during translation. Specifically, dataset-specific markers such as ⟨⟨·⟩⟩ and #### are temporarily replaced with placeholders prior to translation and subsequently restored to their original
Chunk 16 · 1,990 chars
tical reasoning benchmark, we employ a placeholder-based preprocessing step to preserve the integrity of mathematical expressions during translation. Specifically, dataset-specific markers such as ⟨⟨·⟩⟩ and #### are temporarily replaced with placeholders prior to translation and subsequently restored to their original form. For RACE, we adopt a sentence-aware chunk- ing strategy to accommodate the substantially longer reading passages compared to GSM8K and MMLU. The articles, questions, and answer op- tions are translated independently, after which we reconstruct the hierarchical reading comprehen- sion structure while strictly preserving the original evaluation schema. To ensure translation reliability, we manually in- spected a random sample of approximately 10% of the translated instances. B Evaluation To assess model performance before and after weight manipulation, we use the EleutherAI Evalu- ation Harness framework Gao et al. (2024). In ad- dition to English, we evaluate on German, Hindi, and French by implementing custom lm_eval tasks for GSM8K, RACE, and MMLU that mirror the original English task specifications (i.e. match- ing dataset splits, preprocessing, and scoring). For GSM8K, we adopt the gsm8k_cot configuration, which uses an 8-shot chain-of-thought prompt- ing format and following prior work Christ et al. (2025); Brown et al. (2024), we use a subset of 200 samples of GSM8K test set for efficacy. Across languages, we evaluate on the same fixed set of examples per task to ensure comparability of re- sults. All evaluations are repeated for three differ- ent random seeds, and the mean and standard de- viations are reported accordingly. To further facil- itate a fair comparison across settings, we increase max_new_tokens from its default of 150 to 300 for evaluations on Hindi and French. This adjust- ment accounts for the possibly higher token counts needed to represent Hindi and French texts com- pared to math responses in other languages due
Chunk 17 · 1,996 chars
ingly. To further facil- itate a fair comparison across settings, we increase max_new_tokens from its default of 150 to 300 for evaluations on Hindi and French. This adjust- ment accounts for the possibly higher token counts needed to represent Hindi and French texts com- pared to math responses in other languages due to script differences. It is worth noting that employ- ing varying values of max_new_tokens for each language does not undermine the objectives of our experiments. Our primary focus is on assessing the relative improvements or deteriorations observed in the intervention experiments for each language independently. C Jaccard Random Baseline Setup (fixed-size, independent masks) Let U be a universe of N parameters. Each set Si ⊆ U is sampled independently and uniformly from all sub- sets of size |Si| = m, m = pN, -- 7 of 24 -- where p ∈ (0, 1) and m is assumed to be an integer. For any fixed parameter θ ∈ U , Pr(θ ∈ Si) = m N = p, Pr(θ /∈ Si) = 1 − p. Difference sets Define the one-sided difference sets A = S1 \ S2, B = S3 \ S4, where all Si are independent. A parameter θ belongs to A if and only if θ ∈ S1 and θ /∈ S2. Since S1 and S2 are independent, Pr(θ ∈ A) = Pr(θ ∈ S1) Pr(θ /∈ S2) = p(1 − p). The same holds for B: Pr(θ ∈ B) = p(1 − p). Expected Jaccard similarity The Jaccard simi- larity between A and B is J(A, B) = |A ∩ B| |A ∪ B| . For any fixed θ ∈ U , membership in A and B depends on disjoint pairs of sets and is therefore independent. Hence, Pr(θ ∈ A ∩ B) = [p(1 − p)]2, and Pr(θ ∈ A ∪ B) = Pr(θ ∈ A) + Pr(θ ∈ B) − Pr(θ ∈ A ∩ B) = 2p(1 − p) − [p(1 − p)]2. For large N , the random set sizes concentrate around their expectations, and the expected Jac- card similarity is well approximated by the ratio of expectations: E[J(A, B)] ≈ [p(1 − p)]2 2p(1 − p) − [p(1 − p)]2 . Factoring out p(1−p) yields the baseline expres- sion: E[J(A, B)] ≈ p(1 − p) 2 − p(1 − p) = p(1 − p) p2 − p + 2 . It is worth noting that, the approximation arises from replacing
Chunk 18 · 1,996 chars
and the expected Jac- card similarity is well approximated by the ratio of expectations: E[J(A, B)] ≈ [p(1 − p)]2 2p(1 − p) − [p(1 − p)]2 . Factoring out p(1−p) yields the baseline expres- sion: E[J(A, B)] ≈ p(1 − p) 2 − p(1 − p) = p(1 − p) p2 − p + 2 . It is worth noting that, the approximation arises from replacing E[|A ∩ B|/|A ∪ B|] by E|A ∩ B|/E|A ∪ B|. This becomes accurate when N is large due to concentration of measure. D Intervention Results In the following, we present results for scaling (Tables D.1-D.6) and pruning (Tables D.7-D.12) math-specific parameters across different models and multiple languages. These results serve as proof that the parameter identification for the task and language was performed correctly. In addi- tion to accuracies on GSM8K, MMLU, and RACE, we also report the flexible-extract GSM8K score from the lm_eval (lm-evaluation-harness) frame- work. The harness computes GSM8K exact-match accuracy after applying an answer-extraction step: a strict extractor that follows the dataset’s canon- ical output format (e.g., The answer is <an- swer>) and a more permissive flexible extractor that searches the generated text for a number-like substring. We include the flexible metric to reduce sensitivity to output formatting, as our analysis is not focused on instruction-following behavior. D.1 Robustness of parameter identification Across all settings both in scaling and pruning ex- periments, the standard deviation across three ran- dom seeds remains low. We take this as evidence that the identified parameter subset captures math- reasoning capacity in a stable manner, rather than reflecting idiosyncrasies of a particular sample set. Moreover, all scaling experiments yield highly stable accuracies on MMLU and RACE, indicat- ing that scaling the identified parameters does not measurably affect general language understanding. For pruning, the within-dataset variability is more sensitive to the choice of top-k, but remains within a bounded and
Chunk 19 · 1,999 chars
e set. Moreover, all scaling experiments yield highly stable accuracies on MMLU and RACE, indicat- ing that scaling the identified parameters does not measurably affect general language understanding. For pruning, the within-dataset variability is more sensitive to the choice of top-k, but remains within a bounded and reasonably stable range. This con- firms that we isolate parameter subsets that drive math reasoning while being largely irrelevant to general language understanding, which is of high importance to our parameter comparison and Jac- card similarity interpretations. D.2 Pruning results Across all settings, increasing top-k leads to a substantial decline in math-reasoning performance, while non-math performance remains compara- tively stable on MMLU and RACE (Tables D.7- D.12). This selective degradation is the intended effect and indicates that our parameter identifica- tion is successful also across model sizes. However, surprisingly, when pruning the Qwen3 model with a top-k=1e-5 threshold, cor- responding to suppressing approximately 16K parameters, the performance on English GSM8K improves substantially, whereas only a marginal improvement is observed on English GSM8K- Flex shown in Table D.9, which may be a strong signal of wrong output formatting. This counter- intuitive gain from parameter reduction aligns with the framework proposed in (Ali et al., 2025), which identifies a class of prominent but detrimen- -- 8 of 24 -- tal neurons. These are highly activated units that, while capturing frequent training patterns, often enforce brittle heuristics that hinder the model’s ability to generalize to nuanced reasoning tasks. Manual inspection reveals that this improvement is only partly due to corrections in arithmetic computation; more importantly, it stems from enhanced in-context instruction following. This manifests as shorter reasoning traces that are less likely to be truncated by the maximum generation length, and it also outputs the required
Chunk 20 · 1,991 chars
ection reveals that this improvement is only partly due to corrections in arithmetic computation; more importantly, it stems from enhanced in-context instruction following. This manifests as shorter reasoning traces that are less likely to be truncated by the maximum generation length, and it also outputs the required output formats. This stabilization of the output path is consistent with the AdaRAS by Dong et al. (2026), which suggests that over-active reasoning circuits can lead to logical redundancy and verbosity. By pruning these circuits, we effectively guide the model toward more direct and stable reasoning paths, thereby better mimicking the reasoning style demonstrated in the provided examples. Figure E.9 illustrates a representative example. Prior to pruning, the model persistently follows its own internal reasoning style, presumably ac- quired during post-training, rather than adhering to the few-shot exemplars. After suppressing approx- imately 16K of the most highly activated param- eters, the model more consistently aligns its rea- soning format with that of the provided demonstra- tions. This behavior provides empirical support for the theory of Copy Bias in In-Context Learning (Ali et al., 2024). According to this work, specific parameters act as stylistic anchors that prioritize internal priors over contextual cues; pruning them releases the model from these rigid post-training biases, allowing for greater behavioral flexibility. D.3 Scaling results In contrast to our smallest model (Llama 1B), scal- ing does not yield significant consistent gains on larger models. Given the strong pruning results, we attribute this mainly to an insufficient scaling factor (1.01) rather than to low-quality parameter identification. Since optimizing this hyperparame- ter is beyond the scope of this work and does not affect our cross-language comparisons, we leave a systematic search over scaling factors to future work. However, one can observe a relationship
Chunk 21 · 1,991 chars
n insufficient scaling factor (1.01) rather than to low-quality parameter identification. Since optimizing this hyperparame- ter is beyond the scope of this work and does not affect our cross-language comparisons, we leave a systematic search over scaling factors to future work. However, one can observe a relationship be- tween the model’s general language understanding ability and the effect of scaling on performance, including indications of a saturation effect. For the small Llama 1B model, scaling yields perfor- mance gains (in English) of up to 9.7%. For larger models, performance also improves in some cases, but the changes are much smaller, ranging only from -1.4% to 2.4% compared to the base- line. This suggests that scaling is most effective when the model has not yet reached a level of sat- urated performance. In cases where performance is already high, scaling may be limited because the task-relevant parameters have already been suf- ficiently optimized, rather than because the task- specific weights were poorly identified. For Ger- man, Hindi, and French, we frequently observe that baseline general language understanding per- formance (e.g., measured by MMLU or RACE) in- fluences the gains obtained from scaling. When language understanding performance is too low, as is often the case for Hindi, scaling tends to pro- duce only small positive or even negative effects. One possible explanation is that insufficient lan- guage knowledge prevents the model from identi- fying the optimal language-specific parameters re- quired for mathematical reasoning. For instance, Qwen3 4B shows consistent improvements from scaling for both German and Hindi; however, the gains are larger for German, which has higher base- line language understanding scores on RACE and MMLU. Overall, these results suggest that a suf- ficient level of language understanding is benefi- cial for the scaling approach, while very high base- line performance can lead to saturation effects
Chunk 22 · 1,991 chars
Hindi; however, the gains are larger for German, which has higher base- line language understanding scores on RACE and MMLU. Overall, these results suggest that a suf- ficient level of language understanding is benefi- cial for the scaling approach, while very high base- line performance can lead to saturation effects that limit further improvements through scaling. D.4 Collective vs. Individual Effects We examine whether the identified math-specific parameters operate primarily through a collective mechanism or exert independent (individual) ef- fects. For all experiments in this section, we set max_new_tokens to 300 across all languages and use RACE as the non-math dataset. As discussed in Section 4.5 and shown in Fig- ure 4, the Llama 1B model exhibits collective be- havior. This trend persists in larger models: Fig- ures D.1 and D.2 show that both Qwen3 4B and Llama 8B demonstrate similar collective dynam- ics, suggesting that this phenomenon is consistent across model scales. E Qualitative Examples for Scaling & Pruning In this section, we present several qualitative ex- amples where scaling improves the model’s ini- tial answers for mathematical reasoning across lan- -- 9 of 24 -- Top-k English German Hindi French GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE 0.0 (Pre-train) 0.340 0.345 0.379 0.235 0.230 0.321 0.145 0.160 0.299 0.185 0.195 0.348 0.000001 +0.02.9% +0.02.9% +0.50.0% +0.90.0% +1.34.3% +0.00.0% -9.06.9% -7.56.2% -0.30.0% +1.10.0% +0.05.1% +0.00.0% 0.00001 +0.65.9% +0.65.8% +0.80.0% -2.14.3% -2.24.3% +0.30.0% -1.46.9% -1.96.2% -0.30.0% +6.510.8% +5.115.4% -0.30.0% 0.0001 +2.48.8% +2.38.7% +0.30.0% -2.14.3% -0.94.3% -0.30.0% +1.46.9% +0.06.2% +0.30.0% +6.55.4% +5.15.1% +0.00.0% 0.001 +6.82.9% +6.72.9% +1.10.0% -5.58.5% -4.313.0% +0.00.0% +4.86.9% +3.16.2% +0.30.0% +1.15.4% -1.55.1% +0.00.0% 0.005 +6.52.9% +6.42.9% +0.30.0% -3.00.0% -2.20.0% +0.60.0% +0.013.8%
Chunk 23 · 1,990 chars
-0.30.0% 0.0001 +2.48.8% +2.38.7% +0.30.0% -2.14.3% -0.94.3% -0.30.0% +1.46.9% +0.06.2% +0.30.0% +6.55.4% +5.15.1% +0.00.0% 0.001 +6.82.9% +6.72.9% +1.10.0% -5.58.5% -4.313.0% +0.00.0% +4.86.9% +3.16.2% +0.30.0% +1.15.4% -1.55.1% +0.00.0% 0.005 +6.52.9% +6.42.9% +0.30.0% -3.00.0% -2.20.0% +0.60.0% +0.013.8% -4.412.5% +0.70.0% +8.10.0% +6.75.1% -0.30.0% 0.01 +2.90.0% +3.50.0% +0.50.0% +1.34.3% +3.54.3% +0.00.0% +2.16.9% +3.16.2% +0.70.0% +10.80.0% +8.75.1% -0.30.0% 0.025 +5.02.9% +4.92.9% +0.30.0% +3.04.3% +5.24.3% +0.30.0% -4.86.9% -4.46.2% +0.30.0% +2.75.4% +0.05.1% +0.00.0% 0.05 +5.30.0% +4.90.0% +0.50.0% +3.44.3% +5.20.0% +0.30.0% -3.40.0% -6.36.2% +0.30.0% +7.05.4% +6.75.1% +0.00.0% 0.1 +3.52.9% +3.52.9% +0.30.0% +0.94.3% +2.20.0% +0.90.0% -5.56.9% -3.16.2% +0.30.0% +8.15.4% +8.75.1% -0.30.0% 0.15 +3.52.9% +3.52.9% +0.00.0% -1.38.5% -1.38.7% +0.30.0% -4.80.0% -3.16.2% +0.70.0% +5.45.4% +3.65.1% +0.00.0% Table D.1: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for scaling with Llama 1B and RACE. Top-k English German Hindi French GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU 0.0 (Pre-train) 0.340 0.345 0.458 0.235 0.230 0.363 0.145 0.160 0.322 0.185 0.195 0.360 0.000001 +3.52.9% +3.52.9% +0.20.0% +3.04.3% +3.54.3% +0.30.0% -1.46.9% -3.16.2% +0.30.0% +4.35.4% +1.55.1% +0.00.0% 0.00001 +2.42.9% +2.32.9% +0.20.0% -5.54.3% -4.34.3% +0.30.0% +4.80.0% +3.16.2% +0.30.0% +3.85.4% +2.60.0% +0.00.0% 0.0001 +1.52.9% +1.42.9% +0.40.0% +3.04.3% +3.04.3% +0.00.0% -4.80.0% -3.16.2% +0.30.0% +5.45.4% +3.65.1% +0.00.0% 0.001 +1.55.9% +1.45.8% +0.70.0% -5.18.5% -5.28.7% +0.00.0% -1.40.0% -1.30.0% +0.30.0% +0.05.4% -1.05.1% +0.00.0% 0.005 +3.55.9% +3.55.8% +0.20.0% +1.30.0% +2.20.0% +0.00.0% -2.16.9% +0.06.2% +0.30.0% +8.10.0% +6.25.1% +0.00.0% 0.01 +6.52.9% +6.42.9% +0.40.0% +4.34.3% +4.34.3% +0.30.0% -4.86.9% -3.16.2% +0.60.0% +8.10.0%
Chunk 24 · 1,987 chars
+1.45.8% +0.70.0% -5.18.5% -5.28.7% +0.00.0% -1.40.0% -1.30.0% +0.30.0% +0.05.4% -1.05.1% +0.00.0%
0.005 +3.55.9% +3.55.8% +0.20.0% +1.30.0% +2.20.0% +0.00.0% -2.16.9% +0.06.2% +0.30.0% +8.10.0% +6.25.1% +0.00.0%
0.01 +6.52.9% +6.42.9% +0.40.0% +4.34.3% +4.34.3% +0.30.0% -4.86.9% -3.16.2% +0.60.0% +8.10.0% +6.75.1% -0.30.0%
0.025 +5.32.9% +5.82.9% +0.20.0% -5.14.3% -3.04.3% +0.60.0% -11.76.9% -10.60.0% +0.30.0% +3.85.4% +2.610.3% +0.00.0%
0.05 +3.52.9% +3.52.9% +0.40.0% -0.94.3% +0.04.3% +0.30.0% -1.40.0% -3.16.2% +0.30.0% +3.85.4% +4.110.3% -0.60.0%
0.1 +6.82.9% +6.72.9% +0.20.0% +1.30.0% +2.20.0% +0.30.0% -9.06.9% -9.46.2% +0.30.0% +16.25.4% +15.40.0% -0.30.0%
0.15 +9.75.9% +9.35.8% +0.20.0% +1.34.3% +1.34.3% +0.30.0% -4.86.9% -4.46.2% +0.60.0% +9.25.4% +7.75.1% +0.00.0%
Table D.2: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for scaling with Llama 1B and
MMLU.
0.0 0.2 0.4 0.6 0.8 1.0
Fraction of Pruned Parameter
100
50
0
Performance Change
from Baseline (%)
Top-k = 0.001
0.0 0.2 0.4 0.6 0.8 1.0
Fraction of Pruned Parameter
Top-k = 0.01
0.0 0.2 0.4 0.6 0.8 1.0
Fraction of Pruned Parameter
Top-k = 0.1
0 117K 234K 351K 468K 585K
Parameters Pruned
0 1.1M 2.2M 3.3M 4.3M 5.4M
Parameters Pruned
0 7.3M 14.5M 21.8M 29.0M 36.3M
Parameters Pruned
English German Hindi
Figure D.1: Effect of pruning isolated math-specific pa-
rameters on performance across languages for Qwen3
4B at top-k ∈ {0.001, 0.01, 0.1}.
0.00 0.25 0.50 0.75 1.00
Fraction of Pruned Parameter
100
75
50
25
0
Performance Change
from Baseline (%)
Top-k = 0.001
0.00 0.25 0.50 0.75 1.00
Fraction of Pruned Parameter
Top-k = 0.01
0.00 0.25 0.50 0.75 1.00
Fraction of Pruned Parameter
Top-k = 0.1
0 302K 604K 906K 1.2M 1.5M
Parameters Pruned
0 2.8M 5.5M 8.3M 11.0M 13.8M
Parameters Pruned
0 16.1M 32.1M 48.2M 64.2M 80.3M
Parameters Pruned
English German Hindi
Figure D.2: Effect of pruning isolatedChunk 25 · 1,990 chars
n of Pruned Parameter
Top-k = 0.01
0.00 0.25 0.50 0.75 1.00
Fraction of Pruned Parameter
Top-k = 0.1
0 302K 604K 906K 1.2M 1.5M
Parameters Pruned
0 2.8M 5.5M 8.3M 11.0M 13.8M
Parameters Pruned
0 16.1M 32.1M 48.2M 64.2M 80.3M
Parameters Pruned
English German Hindi
Figure D.2: Effect of pruning isolated math-specific pa-
rameters on performance across languages for Llama
8B at top-k ∈ {0.001, 0.01, 0.1}.
guages (Figures E.1-E.4). Furthermore, we show
examples where pruning steers the model toward
incorrect answers for mathematical reasoning (Fig-
ures E.5-E.8), alongside a notable exception in
which pruning instead improves performance (Fig-
ure E.9), as discussed in Appendix D.
F Similarities across Languages & Tasks
In this section, we present plots showing the num-
ber of isolated parameters across languages (En-
glish (En), German (De), Hindi (Hi), and French
(Fr)), as well as Jaccard similarity plots for each
language pair, across all top-k values and models.
-- 10 of 24 --
Top-k English German Hindi French
GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE
0.0 (Pre-train) 0.735 0.815 0.414 0.685 0.400 0.356 0.385 0.460 0.353 0.690 0.615 0.403
0.000001 +0.30.0% +0.00.0% -0.20.0% +3.91.5% +5.52.5% +0.30.0% -0.50.0% -1.52.2% -0.60.0% +2.21.4% +3.63.3% +0.20.0%
0.00001 -0.31.4% +2.10.0% +0.00.0% +3.91.5% +13.75.0% +0.80.0% +2.60.0% +1.52.2% -0.80.0% +1.71.4% +2.91.6% +0.70.0%
0.0001 +0.41.4% +0.21.2% -0.70.0% +4.41.5% +7.02.5% +1.10.0% +0.50.0% -0.74.3% -1.10.0% +2.50.0% +1.30.0% +0.50.0%
0.001 +1.00.0% +1.82.5% -0.20.0% +2.51.5% +5.75.0% +0.60.0% +0.80.0% -1.12.2% -0.30.0% +3.61.4% +3.71.6% +0.50.0%
0.005 +0.41.4% +1.61.2% +0.00.0% +2.61.5% +8.07.5% +0.60.0% +0.52.6% -0.74.3% -1.40.0% +2.51.4% +5.43.3% +0.50.0%
0.01 +0.31.4% +1.21.2% +0.20.0% +5.50.0% +16.80.0% +0.30.0% +0.82.6% -2.62.2% -0.60.0% +2.51.4% +2.91.6% +0.50.0%
0.025 +0.41.4% -0.42.5% +0.00.0% +3.91.5%Chunk 26 · 1,990 chars
2.2% -0.30.0% +3.61.4% +3.71.6% +0.50.0% 0.005 +0.41.4% +1.61.2% +0.00.0% +2.61.5% +8.07.5% +0.60.0% +0.52.6% -0.74.3% -1.40.0% +2.51.4% +5.43.3% +0.50.0% 0.01 +0.31.4% +1.21.2% +0.20.0% +5.50.0% +16.80.0% +0.30.0% +0.82.6% -2.62.2% -0.60.0% +2.51.4% +2.91.6% +0.50.0% 0.025 +0.41.4% -0.42.5% +0.00.0% +3.91.5% +13.20.0% +0.30.0% +0.02.6% -1.72.2% -0.60.0% +3.61.4% +3.31.6% +0.20.0% 0.05 -0.41.4% +0.01.2% -0.70.0% +4.81.5% +8.02.5% +0.80.0% +1.82.6% +1.70.0% -0.60.0% +1.91.4% +2.01.6% +0.20.0% 0.1 +0.31.4% +0.42.5% -0.50.0% +2.91.5% +8.27.5% -0.30.0% -0.80.0% -1.52.2% -0.80.0% +2.91.4% +3.61.6% +0.70.0% 0.15 -1.41.4% +0.42.5% -0.50.0% +4.11.5% +8.07.5% +0.30.0% +0.02.6% +0.44.3% +0.00.0% +1.91.4% +1.31.6% +0.70.0% Table D.3: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for scaling with Qwen3 4B and RACE. Top-k English German Hindi French GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU 0.0 (Pre-train) 0.735 0.815 0.710 0.685 0.400 0.585 0.385 0.460 0.471 0.690 0.615 0.548 0.000001 +1.81.4% +1.21.2% +0.00.0% +2.61.5% +9.27.5% -0.20.0% +0.00.0% -2.22.2% +0.20.0% +1.70.0% +2.01.6% +0.20.0% 0.00001 +0.31.4% +0.00.0% +0.00.0% +3.41.5% +2.05.0% -0.20.0% +1.32.6% +1.12.2% +0.20.0% +2.91.4% +3.31.6% +0.40.0% 0.0001 +1.11.4% -1.80.0% +0.00.0% +4.11.5% +10.05.0% -0.20.0% +0.50.0% -1.72.2% +0.20.0% +2.21.4% +0.01.6% +0.20.0% 0.001 +2.01.4% +0.90.0% +0.00.0% +2.91.5% +6.72.5% -0.20.0% +0.50.0% +0.02.2% +0.20.0% +2.50.0% +2.41.6% +0.20.0% 0.005 +0.71.4% -1.82.5% +0.10.0% +3.41.5% +8.05.0% -0.20.0% +1.32.6% +2.24.3% +0.20.0% +1.21.4% +2.91.6% +0.20.0% 0.01 +0.31.4% +1.50.0% +0.00.0% +4.81.5% +8.22.5% -0.20.0% +1.32.6% -0.44.3% +0.20.0% +3.21.4% +2.90.0% +0.20.0% 0.025 +1.02.7% -1.61.2% +0.00.0% +4.41.5% +13.05.0% -0.20.0% +0.52.6% -1.72.2% +0.20.0% +2.21.4% +2.41.6% +0.20.0% 0.05 -1.01.4% -1.00.0% +0.00.0% +4.11.5% +7.55.0% -0.20.0% +0.55.2% -1.12.2%
Chunk 27 · 1,998 chars
0.0% 0.01 +0.31.4% +1.50.0% +0.00.0% +4.81.5% +8.22.5% -0.20.0% +1.32.6% -0.44.3% +0.20.0% +3.21.4% +2.90.0% +0.20.0% 0.025 +1.02.7% -1.61.2% +0.00.0% +4.41.5% +13.05.0% -0.20.0% +0.52.6% -1.72.2% +0.20.0% +2.21.4% +2.41.6% +0.20.0% 0.05 -1.01.4% -1.00.0% +0.00.0% +4.11.5% +7.55.0% -0.20.0% +0.55.2% -1.12.2% +0.40.0% +2.21.4% +1.61.6% +0.20.0% 0.1 -0.41.4% -0.91.2% +0.00.0% +3.21.5% +11.25.0% -0.20.0% +0.50.0% -0.44.3% +0.20.0% +1.41.4% -0.30.0% +0.20.0% 0.15 -0.31.4% -0.92.5% +0.00.0% +4.11.5% +8.25.0% -0.20.0% -0.52.6% +0.02.2% +0.20.0% +1.91.4% +0.81.6% +0.00.0% Table D.4: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for scaling with Qwen3 4B and MMLU. Top-k English German Hindi French GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE 0.0 (Pre-train) 0.765 0.775 0.448 0.585 0.590 0.396 0.415 0.490 0.388 0.580 0.585 0.413 0.000001 +1.00.0% +0.01.3% +0.00.0% +0.31.7% +0.81.7% -0.50.0% +3.62.4% +1.60.0% +0.00.0% +2.11.7% +2.11.7% +0.20.0% 0.00001 +2.40.0% +1.90.0% +0.20.0% +1.21.7% +2.21.7% -0.30.0% +3.10.0% +0.02.0% +0.00.0% +0.51.7% +0.33.4% +0.20.0% 0.0001 +0.01.3% -0.31.3% +0.00.0% +2.90.0% +3.70.0% -0.30.0% +0.50.0% -2.70.0% +0.30.0% +0.33.4% +0.33.4% +0.50.0% 0.001 +1.61.3% +1.01.3% -0.40.0% +0.91.7% +2.51.7% -0.30.0% -0.54.8% -2.02.0% +0.00.0% +1.71.7% +1.71.7% +0.20.0% 0.005 +1.61.3% +0.61.3% +0.20.0% +0.51.7% +0.50.0% +0.00.0% +5.30.0% +1.60.0% +0.30.0% +0.33.4% -0.33.4% +0.50.0% 0.01 +0.71.3% +0.02.6% +0.20.0% +0.51.7% +0.81.7% -0.30.0% +0.74.8% -1.44.1% +0.00.0% +0.51.7% +0.31.7% +0.20.0% 0.025 +1.60.0% +0.60.0% -0.40.0% +1.71.7% +2.91.7% -0.50.0% +1.92.4% +0.60.0% +0.30.0% +1.21.7% +1.21.7% +0.20.0% 0.05 +2.21.3% +1.50.0% -0.40.0% +2.60.0% +3.90.0% +0.00.0% +2.92.4% +0.42.0% +0.30.0% +0.50.0% +0.50.0% +0.50.0% 0.1 +0.31.3% -0.41.3% +0.00.0% -0.31.7% +0.31.7% +0.00.0% +3.10.0% +0.62.0% +0.30.0% +0.01.7% -0.31.7% +0.00.0% 0.15 +0.71.3%
Chunk 28 · 1,998 chars
% +2.91.7% -0.50.0% +1.92.4% +0.60.0% +0.30.0% +1.21.7% +1.21.7% +0.20.0% 0.05 +2.21.3% +1.50.0% -0.40.0% +2.60.0% +3.90.0% +0.00.0% +2.92.4% +0.42.0% +0.30.0% +0.50.0% +0.50.0% +0.50.0% 0.1 +0.31.3% -0.41.3% +0.00.0% -0.31.7% +0.31.7% +0.00.0% +3.10.0% +0.62.0% +0.30.0% +0.01.7% -0.31.7% +0.00.0% 0.15 +0.71.3% +0.32.6% -0.20.0% +0.01.7% +0.50.0% +0.00.0% +1.70.0% -2.02.0% +0.00.0% +0.30.0% +0.30.0% +0.20.0% Table D.5: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for scaling with Llama 8B and RACE. Top-k English German Hindi French GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU 0.0 (Pre-train) 0.765 0.775 0.679 0.585 0.590 0.538 0.415 0.490 0.450 0.580 0.585 0.512 0.000001 +1.60.0% +1.31.3% -0.10.0% +1.21.7% +1.71.7% +0.20.0% -0.52.4% -3.12.0% +1.10.0% +0.50.0% +0.30.0% +0.20.0% 0.00001 +0.71.3% +0.41.3% -0.10.0% +0.01.7% +0.81.7% +0.20.0% +0.52.4% +0.02.0% +1.10.0% -0.31.7% -0.31.7% +0.20.0% 0.0001 +1.00.0% +1.00.0% -0.30.0% +2.20.0% +2.91.7% +0.00.0% +1.92.4% +1.02.0% +1.10.0% -0.31.7% -0.31.7% +0.20.0% 0.001 +1.61.3% +0.41.3% -0.10.0% +0.91.7% +2.51.7% +0.20.0% -0.72.4% -3.12.0% +1.10.0% +0.00.0% +0.00.0% +0.20.0% 0.005 +2.01.3% +1.91.3% -0.10.0% +0.01.7% +1.71.7% +0.20.0% +1.22.4% -0.42.0% +0.90.0% -1.73.4% -2.23.4% +0.20.0% 0.01 +2.21.3% +1.31.3% -0.10.0% +0.51.7% +0.81.7% +0.00.0% +2.42.4% -0.44.1% +0.90.0% +0.03.4% -0.33.4% +0.00.0% 0.025 +0.71.3% -0.41.3% -0.10.0% +0.51.7% +1.41.7% +0.20.0% +1.72.4% -1.02.0% +1.10.0% -1.21.7% -1.21.7% +0.00.0% 0.05 +1.02.6% +0.91.3% -0.10.0% +0.91.7% +1.21.7% +0.20.0% +3.60.0% +0.40.0% +1.10.0% -2.61.7% -2.61.7% +0.00.0% 0.1 -0.31.3% -0.91.3% +0.00.0% +1.71.7% +2.91.7% +0.20.0% +1.94.8% +0.02.0% +0.90.0% +0.30.0% +0.30.0% +0.20.0% 0.15 +1.01.3% +0.62.6% -0.10.0% +0.01.7% +0.51.7% +0.20.0% -0.52.4% -3.12.0% +0.90.0% -0.53.4% -1.21.7% +0.20.0% Table D.6: Relative change in accuracy (%), averaged
Chunk 29 · 1,997 chars
0.0% -2.61.7% -2.61.7% +0.00.0% 0.1 -0.31.3% -0.91.3% +0.00.0% +1.71.7% +2.91.7% +0.20.0% +1.94.8% +0.02.0% +0.90.0% +0.30.0% +0.30.0% +0.20.0% 0.15 +1.01.3% +0.62.6% -0.10.0% +0.01.7% +0.51.7% +0.20.0% -0.52.4% -3.12.0% +0.90.0% -0.53.4% -1.21.7% +0.20.0% Table D.6: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for scaling with Llama 8B and MMLU. -- 11 of 24 -- Top-k English German Hindi French GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE 0.0 (Pre-train) 0.340 0.345 0.379 0.235 0.230 0.321 0.145 0.160 0.299 0.185 0.195 0.348 0.000001 +9.72.9% +8.72.9% +0.30.0% -9.44.3% -6.54.3% +1.20.0% -9.06.9% -8.112.5% +0.70.0% -3.85.4% -4.15.1% -0.30.0% 0.00001 -7.92.9% -9.32.9% +1.62.6% -18.312.8% -16.113.0% +0.00.0% -31.020.7% -29.418.8% +2.00.0% -7.05.4% -8.70.0% -1.42.9% 0.0001 -25.02.9% -26.12.9% -0.50.0% -31.14.3% -29.64.3% +2.83.1% -43.46.9% -31.218.8% -1.00.0% -17.35.4% -19.05.1% -0.90.0% 0.001 -81.55.9% -80.65.8% -0.80.0% -66.88.5% -65.28.7% +0.30.0% -75.96.9% -70.612.5% -3.70.0% -59.510.8% -60.010.3% -5.20.0% 0.005 -94.72.9% -94.22.9% -2.12.6% -89.44.3% -91.38.7% -2.83.1% -87.66.9% -84.46.2% -5.73.3% -90.35.4% -92.35.1% -6.00.0% 0.01 -96.22.9% -96.22.9% -2.12.6% -92.30.0% -90.44.3% -1.20.0% -94.56.9% -92.56.2% -6.40.0% -90.80.0% -90.80.0% -6.30.0% 0.025 -92.60.0% -92.20.0% -9.00.0% -90.60.0% -90.44.3% -3.13.1% -96.66.9% -90.66.2% -8.00.0% -88.15.4% -88.25.1% -6.60.0% 0.05 -93.55.9% -93.65.8% -20.15.3% -89.44.3% -91.34.3% -7.20.0% -95.26.9% -91.90.0% -8.03.3% -88.15.4% -88.75.1% -8.60.0% 0.1 -93.50.0% -93.60.0% -12.42.6% -93.64.3% -90.44.3% -10.33.1% -93.16.9% -92.50.0% -9.03.3% -94.65.4% -91.30.0% -12.92.9% 0.15 -96.52.9% -95.70.0% -13.72.6% -92.38.5% -90.44.3% -10.33.1% -97.96.9% -88.86.2% -8.03.3% -88.15.4% -88.710.3% -10.10.0% Table D.7: Relative change in accuracy (%), averaged over 3 seeds (mean ± std),
Chunk 30 · 1,987 chars
3.60.0% -12.42.6% -93.64.3% -90.44.3% -10.33.1% -93.16.9% -92.50.0% -9.03.3% -94.65.4% -91.30.0% -12.92.9% 0.15 -96.52.9% -95.70.0% -13.72.6% -92.38.5% -90.44.3% -10.33.1% -97.96.9% -88.86.2% -8.03.3% -88.15.4% -88.710.3% -10.10.0% Table D.7: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Llama 1B and RACE. Top-k English German Hindi French GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU 0.0 (Pre-train) 0.340 0.345 0.458 0.235 0.230 0.363 0.145 0.160 0.322 0.185 0.195 0.360 0.000001 +3.88.8% +2.98.7% -0.20.0% -9.44.3% -8.74.3% -0.60.0% -15.96.9% -18.86.2% -1.20.0% +0.05.4% -3.65.1% -1.10.0% 0.00001 -14.72.9% -15.40.0% -3.70.0% -18.312.8% -15.213.0% -0.80.0% -35.96.9% -30.06.2% -0.30.0% -18.95.4% -21.55.1% -4.72.8% 0.0001 -19.78.8% -20.98.7% -2.40.0% -34.04.3% -33.54.3% -0.60.0% -35.913.8% -29.46.2% -1.90.0% -11.95.4% -15.45.1% -0.60.0% 0.001 -80.32.9% -80.32.9% -7.62.2% -67.212.8% -67.48.7% -5.52.8% -46.213.8% -48.812.5% -3.40.0% -55.710.8% -57.410.3% -4.70.0% 0.005 -90.62.9% -89.92.9% -6.62.2% -91.54.3% -91.34.3% -8.30.0% -88.36.9% -83.16.2% -4.73.1% -87.65.4% -88.75.1% -9.20.0% 0.01 -92.62.9% -92.22.9% -12.22.2% -86.44.3% -85.70.0% -11.80.0% -91.06.9% -91.96.2% -9.30.0% -90.30.0% -88.70.0% -15.32.8% 0.025 -92.15.9% -92.25.8% -25.36.6% -95.74.3% -92.24.3% -14.92.8% -98.60.0% -93.80.0% -14.93.1% -88.15.4% -88.75.1% -13.12.8% 0.05 -91.82.9% -91.32.9% -27.54.4% -92.84.3% -90.48.7% -14.32.8% -96.66.9% -93.86.2% -15.53.1% -90.30.0% -90.80.0% -17.25.6% 0.1 -89.12.9% -88.42.9% -15.90.0% -94.54.3% -92.64.3% -17.40.0% -95.20.0% -92.56.2% -14.36.2% -91.90.0% -90.80.0% -11.90.0% 0.15 -91.82.9% -89.92.9% -13.82.2% -90.64.3% -90.40.0% -15.22.8% -94.56.9% -92.56.2% -16.83.1% -90.35.4% -84.65.1% -10.82.8% Table D.8: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Llama 1B
Chunk 31 · 1,997 chars
% -92.64.3% -17.40.0% -95.20.0% -92.56.2% -14.36.2% -91.90.0% -90.80.0% -11.90.0% 0.15 -91.82.9% -89.92.9% -13.82.2% -90.64.3% -90.40.0% -15.22.8% -94.56.9% -92.56.2% -16.83.1% -90.35.4% -84.65.1% -10.82.8% Table D.8: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Llama 1B and MMLU. Top-k English German Hindi French GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE 0.0 (Pre-train) 0.735 0.815 0.414 0.685 0.400 0.356 0.385 0.460 0.353 0.690 0.615 0.403 0.000001 -21.89.5% -6.13.7% +1.00.0% -0.32.9% +4.52.5% -0.30.0% +8.62.6% -7.019.6% -3.70.0% +4.12.9% +11.43.3% -0.52.5% 0.00001 +19.34.1% +4.72.5% -0.70.0% -3.21.5% +24.227.5% -1.10.0% -1.87.8% -24.310.9% -3.70.0% +2.91.4% +3.69.8% -5.20.0% 0.0001 +1.46.8% -15.78.6% -2.70.0% -22.64.4% -28.027.5% -0.30.0% -19.07.8% -31.523.9% -7.42.8% -16.45.8% -32.826.0% -3.72.5% 0.001 -12.28.2% -55.58.6% -3.62.4% -23.45.8% +1.35.0% -0.62.8% -40.32.6% -13.54.3% -3.42.8% -28.01.4% -59.717.9% -3.72.5% 0.005 -77.16.8% -90.46.1% -1.00.0% -62.51.5% -45.87.5% -0.80.0% -50.17.8% -46.38.7% -5.10.0% -67.41.4% -93.04.9% -4.00.0% 0.01 -97.31.4% -97.31.2% -2.92.4% -94.91.5% -93.22.5% +3.12.8% -77.97.8% -75.74.3% -5.40.0% -95.71.4% -94.81.6% -0.70.0% 0.025 -97.60.0% -97.51.2% -4.30.0% -97.51.5% -93.02.5% -1.72.8% -97.42.6% -95.22.2% -5.92.8% -98.31.4% -96.40.0% -3.50.0% 0.05 -97.60.0% -97.80.0% -3.62.4% -96.81.5% -94.52.5% -1.12.8% -97.40.0% -96.12.2% -7.40.0% -98.61.4% -95.61.6% -3.20.0% 0.1 -98.01.4% -96.72.5% -4.12.4% -96.11.5% -93.20.0% +0.30.0% -98.72.6% -96.12.2% -7.12.8% -98.61.4% -97.91.6% -4.20.0% 0.15 -97.71.4% -97.31.2% -3.12.4% -95.31.5% -94.52.5% -1.40.0% -97.42.6% -96.72.2% -7.90.0% -97.41.4% -97.91.6% -6.52.5% Table D.9: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Qwen3 4B and RACE. Top-k English German Hindi French GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K
Chunk 32 · 1,994 chars
97.71.4% -97.31.2% -3.12.4% -95.31.5% -94.52.5% -1.40.0% -97.42.6% -96.72.2% -7.90.0% -97.41.4% -97.91.6% -6.52.5% Table D.9: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Qwen3 4B and RACE. Top-k English German Hindi French GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU 0.0 (Pre-train) 0.735 0.815 0.710 0.685 0.400 0.585 0.385 0.460 0.471 0.690 0.615 0.548 0.000001 -6.85.4% +0.22.5% -0.70.0% +3.92.9% +38.812.5% -0.70.0% +7.32.6% -5.44.3% -3.40.0% +2.61.4% +7.03.3% +0.00.0% 0.00001 +8.84.1% -13.38.6% -0.80.0% +0.71.5% -0.515.0% -2.41.7% +7.05.2% -53.92.2% -2.50.0% +6.11.4% +10.63.3% -0.20.0% 0.0001 -14.76.8% -39.03.7% -1.00.0% -11.40.0% -18.315.0% -2.20.0% -15.12.6% +5.04.3% -9.32.1% -7.51.4% -17.416.3% -0.40.0% 0.001 -15.41.4% -27.07.4% -2.50.0% -18.52.9% -32.512.5% -6.30.0% -21.35.2% -7.02.2% -5.12.1% -13.32.9% -17.98.1% -3.60.0% 0.005 -79.22.7% -90.61.2% -3.90.0% -68.21.5% -64.27.5% -6.00.0% -61.85.2% -52.24.3% -8.72.1% -64.52.9% -75.91.6% -2.20.0% 0.01 -96.61.4% -97.30.0% -6.80.0% -94.61.5% -89.52.5% -6.70.0% -78.75.2% -76.72.2% -8.92.1% -92.02.9% -89.93.3% -2.90.0% 0.025 -97.71.4% -97.81.2% -8.20.0% -97.51.5% -97.02.5% -8.20.0% -94.32.6% -93.02.2% -9.10.0% -95.72.9% -94.31.6% -9.31.8% 0.05 -98.91.4% -98.41.2% -9.40.0% -95.61.5% -93.80.0% -10.10.0% -98.72.6% -96.72.2% -10.82.1% -98.80.0% -97.11.6% -10.01.8% 0.1 -96.91.4% -96.61.2% -8.30.0% -97.51.5% -95.82.5% -8.00.0% -97.92.6% -96.14.3% -7.90.0% -97.81.4% -97.11.6% -6.91.8% 0.15 -95.61.4% -95.31.2% -7.90.0% -96.40.0% -94.20.0% -7.41.7% -96.12.6% -95.02.2% -8.10.0% -96.72.9% -96.41.6% -5.30.0% Table D.10: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Qwen3 4B and MMLU. -- 12 of 24 -- Top-k English German Hindi French GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE 0.0 (Pre-train) 0.765
Chunk 33 · 1,994 chars
.0% -96.72.9% -96.41.6% -5.30.0% Table D.10: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Qwen3 4B and MMLU. -- 12 of 24 -- Top-k English German Hindi French GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE GSM8K GSM8K flex RACE 0.0 (Pre-train) 0.765 0.775 0.448 0.585 0.590 0.396 0.415 0.490 0.388 0.580 0.585 0.413 0.000001 -2.43.9% -3.23.9% +0.20.0% +2.11.7% +1.71.7% +0.30.0% -8.914.5% -2.46.1% +1.00.0% -7.46.9% -10.38.5% +0.20.0% 0.00001 +0.31.3% -0.62.6% +1.80.0% -4.81.7% -5.91.7% +0.80.0% -10.821.7% -9.82.0% -2.30.0% -14.18.6% -19.712.0% +0.50.0% 0.0001 -15.42.6% -15.52.6% -0.40.0% -22.61.7% -22.41.7% +0.30.0% -22.49.6% -24.14.1% -2.62.6% -22.98.6% -30.418.8% -3.10.0% 0.001 -66.33.9% -66.52.6% -2.92.2% -55.65.1% -54.63.4% -3.52.5% -62.77.2% -60.86.1% -5.20.0% -60.05.2% -59.85.1% -3.42.4% 0.005 -94.81.3% -93.82.6% -4.00.0% -93.21.7% -92.71.7% -4.52.5% -94.72.4% -94.32.0% -8.02.6% -92.81.7% -92.31.7% -3.92.4% 0.01 -98.01.3% -97.71.3% -4.70.0% -97.11.7% -97.11.7% -4.00.0% -95.92.4% -94.94.1% -6.70.0% -97.90.0% -97.11.7% -7.52.4% 0.025 -97.81.3% -96.80.0% -6.50.0% -98.31.7% -97.11.7% -5.60.0% -99.50.0% -99.02.0% -12.60.0% -97.11.7% -96.10.0% -5.82.4% 0.05 -98.41.3% -98.71.3% -2.00.0% -97.91.7% -97.81.7% -9.10.0% -98.32.4% -94.90.0% -12.90.0% -96.90.0% -96.61.7% -8.02.4% 0.1 -99.60.0% -98.11.3% -7.60.0% -96.60.0% -94.91.7% -10.92.5% -97.12.4% -96.32.0% -12.92.6% -97.91.7% -96.91.7% -7.04.8% 0.15 -96.70.0% -94.81.3% -6.02.2% -96.11.7% -95.41.7% -7.10.0% -97.62.4% -95.94.1% -12.12.6% -98.31.7% -97.13.4% -7.52.4% Table D.11: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Llama 8B and RACE. Top-k English German Hindi French GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU 0.0 (Pre-train) 0.765 0.775 0.679 0.585 0.590 0.538 0.415 0.490 0.450 0.580 0.585
Chunk 34 · 1,994 chars
e change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Llama 8B and RACE. Top-k English German Hindi French GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU GSM8K GSM8K flex MMLU 0.0 (Pre-train) 0.765 0.775 0.679 0.585 0.590 0.538 0.415 0.490 0.450 0.580 0.585 0.512 0.000001 -12.06.5% -13.25.2% -0.90.0% +1.20.0% +0.81.7% +0.00.0% -27.74.8% -6.74.1% +0.90.0% -11.65.2% -14.08.5% -0.20.0% 0.00001 -4.82.6% -5.42.6% -0.40.0% +0.01.7% -0.50.0% -0.40.0% -15.719.3% -7.10.0% +0.40.0% -11.20.0% -11.61.7% -0.40.0% 0.0001 -14.01.3% -14.81.3% -2.51.5% -14.23.4% -14.45.1% -0.60.0% -18.62.4% -20.44.1% -0.70.0% -26.43.4% -32.010.3% -1.40.0% 0.001 -81.45.2% -79.02.6% -6.61.5% -60.71.7% -60.50.0% -3.20.0% -56.17.2% -61.24.1% -8.70.0% -63.33.4% -62.93.4% -2.70.0% 0.005 -94.41.3% -94.21.3% -6.50.0% -94.01.7% -92.71.7% -5.80.0% -89.22.4% -89.22.0% -4.90.0% -91.41.7% -91.11.7% -5.72.0% 0.01 -99.10.0% -97.81.3% -8.11.5% -95.43.4% -94.93.4% -8.91.9% -96.92.4% -94.92.0% -9.30.0% -97.90.0% -97.40.0% -4.90.0% 0.025 -98.41.3% -98.11.3% -9.41.5% -97.90.0% -98.00.0% -11.50.0% -99.50.0% -98.02.0% -6.90.0% -96.61.7% -95.21.7% -8.60.0% 0.05 -98.30.0% -96.41.3% -10.91.5% -96.61.7% -96.31.7% -15.21.9% -98.10.0% -96.92.0% -15.34.4% -95.33.4% -94.51.7% -7.02.0% 0.1 -97.61.3% -96.42.6% -17.15.9% -96.10.0% -95.30.0% -20.81.9% -96.92.4% -95.30.0% -21.64.4% -97.81.7% -97.10.0% -10.42.0% 0.15 -92.82.6% -91.92.6% -9.60.0% -94.01.7% -94.11.7% -13.01.9% -93.52.4% -92.22.0% -11.62.2% -94.01.7% -93.71.7% -9.02.0% Table D.12: Relative change in accuracy (%), averaged over 3 seeds (mean ± std), for pruning with Llama 8B and MMLU. Input Q: Toula went to the bakery and bought various types of pastries. She bought 3 dozen donuts which cost $68 per dozen, 2 dozen mini cupcakes which cost $80 per dozen, and 6 dozen mini cheesecakes for $55 per dozen. How much was the total cost? A: Output before scaling
Chunk 35 · 1,998 chars
n ± std), for pruning with Llama 8B and MMLU. Input Q: Toula went to the bakery and bought various types of pastries. She bought 3 dozen donuts which cost $68 per dozen, 2 dozen mini cupcakes which cost $80 per dozen, and 6 dozen mini cheesecakes for $55 per dozen. How much was the total cost? A: Output before scaling (wrong) 3 dozen donuts cost 3 × 68 = 204 dollars. 2 dozen mini cupcakes cost 2 × 80 = 160 dollars. 6 dozen mini cheesecakes cost 6 × 55 = 330 dollars. 204 + 160 + 330 is 734. The answer is 734. Output after scaling (correct) 3 dozen donuts cost 3 × 68 = 204 dollars. 2 dozen mini cupcakes cost 2 × 80 = 160 dollars. 6 dozen mini cheesecakes cost 6 × 55 = 330 dollars. 204 + 160 + 330 is 694. The answer is 694. Figure E.1: Illustration of arithmetic inconsistency before scaling and corrected output after scaling for English reasoning using Llama 1B. -- 13 of 24 -- Input Q: Gunter versucht, die Marmeladebohnen in einem Glas zu zählen. Er fragt seine Freunde, wie viele in dem Glas sind. Einer sagt 80. Ein anderer sagt 20 mehr als die Hälfte des ersten. Ein Drittel sagt 25% mehr als der erste. Wie hoch ist ihre durchschnittliche Vermutung? A: (English translation: Gunter is trying to count the jam beans in a jar. He asks his friends how many are in the jar. One says 80. Another says 20, more than half the first. A third say 25% more than the first. What is their average guess?) Output before scaling (wrong) Der erste sagt 80. Ein anderer sagt 20 mehr als die Hälfte des ersten, also 80 + 20 = 100. Ein Drittel sagt 25% mehr als der erste, also 80 + (80 * 0,25) = 100. Die durchschnit- tliche Vermutung ist 100. Die Antwort ist 100. (English translation: The first person says 80. Another says 20 more than half of the first, so 80 + 20 = 100. One third says 25% more than the first, so 80 + (80 * 0.25) = 100. The average guess is 100. The answer is 100.) Output after scaling (correct) Der erste sagt 80. Ein anderer sagt 20 mehr als die Hälfte des ersten, also 80
Chunk 36 · 1,998 chars
lation: The first person says 80. Another says 20 more than half of the first, so 80 + 20 = 100. One third says 25% more than the first, so 80 + (80 * 0.25) = 100. The average guess is 100. The answer is 100.) Output after scaling (correct) Der erste sagt 80. Ein anderer sagt 20 mehr als die Hälfte des ersten, also 80 + 20 = 100. Ein Drittel sagt 25% mehr als der erste, also 80 + (80 * 0,25) = 100 + 20 = 120. Die durchschnittliche Vermutung ist 80. Die Antwort ist 80. (English translation: The first person says 80. Another says 20 more than half of the first, so 80 + 20 = 100. One third says 25% more than the first, so 80 + (80 * 0.25) = 100 + 20 = 120. The average guess is 80. The answer is 80.) Figure E.2: Illustration of arithmetic inconsistency before scaling and corrected output after scaling for German reasoning using Llama 1B. Input Q: मैरी ने एक िचकन भोजन का ऑडर्र िदया िजसकी कीमत 12 डॉलर है, 5 पैकेटदू ध की कीमत 3 डॉलर है, 4 सेबाें की कीमत 1.50 डॉलर है, और िपज्जा के कुछ िडब्बे हैं। मैरी ने कुल 50 डॉलर का भुगतान िकया। अगरप्रत् येक बॉक्स की कीमत 8.50 डॉलर है तो मैरी ने िपज्जा के िकतने िडब्बे मंगवाए? A: (English translation: Mary ordered a chicken meal that costs $12, 5 packets of milk that cost $3, 4 apples that cost $1.50, and some boxes of pizza. Mary paid a total of $50. How many boxes of pizza did Mary order if each box costs $8.50?) Output before scaling (wrong) मैरी ने एक िचकन की कीमत 12 डॉलर के िलए 1 बॉक्स खरीदा। उसके पास 50 - 12 = 38 डॉलर बचे हैं। 5 पैकेटदू ध की कीमत 3 डॉलर है, िजसकी कुल कीमत 5 × 3 = 15 डॉलर है। उसके पास 38 - 15 = 23 डॉलर बचे हैं। 4 सेबाें की कीमत 1.50 डॉलर है, िजसकी कुल कीमत 4 × 1.50 = 6 डॉलर है। उसके पास 23 - 6 = 17 डॉलर बचे हैं। िपज्जा के कुछ िडब्बे की कीमत 8.50 डॉलर है, िजसकी कुल कीमत 17 / 8.50 = 1.96 डॉलर है। उसके पास 1.96 डॉलर बचे हैं। 1.96 / 8.50 = 0.23। उसके पास 0.23 िडब्बे हैं। उत्तर 0.23 है। (English translation: Mary bought a box of chicken for $12. She has $50 - 12 = $38 left. 5 packets of milk cost $3, for a total cost of
Chunk 37 · 1,999 chars
र बचे हैं। िपज्जा के कुछ िडब्बे की कीमत 8.50 डॉलर है, िजसकी कुल कीमत 17 / 8.50 = 1.96 डॉलर है। उसके पास 1.96 डॉलर बचे हैं। 1.96 / 8.50 = 0.23। उसके पास 0.23 िडब्बे हैं। उत्तर 0.23 है। (English translation: Mary bought a box of chicken for $12. She has $50 - 12 = $38 left. 5 packets of milk cost $3, for a total cost of $5 × 3 = $15. She has $38 - 15 = $23 left. 4 apples cost $1.50, for a total cost of 4 × 1.50 = $6. She has $23 - 6 = $17 left. Some boxes of pizza cost $8.50, for a total cost of $17 / 8.50 = $1.96. She has $1.96 left. 1.96 / 8.50 = 0.23. She has 0.23 boxes. The answer is 0.23.) Output after scaling (correct) मैरी ने एक िचकन की कीमत 12 डॉलर के िलए 1 बॉक्स खरीदा। उसके पास 50 - 12 = 38 डॉलर बचे हैं। 5 पैकेटदू ध की कीमत 3 डॉलर है, िजसकी कुल कीमत 5 × 3 = 15 डॉलर है। उसके पास 38 - 15 = 23 डॉलर बचे हैं। 4 सेबाें की कीमत 1.50 डॉलर है, िजसकी कुल कीमत 4 × 1.50 = 6 डॉलर है। उसके पास 23 - 6 = 17 डॉलर बचे हैं। िपज्जा के कुछ िडब्बे की कीमत 8.50 डॉलर है, िजसकी कुल कीमत 17 / 8.50 = 2 िडब्बे हैं। उत्तर 2 है। (English translation: Mary bought a box of chicken for $12. She has $50 - 12 = $38 left. 5 packets of milk cost $3, for a total cost of 5 × 3 = $15. She has $38 - 15 = $23 left. 4 apples cost $1.50, for a total cost of 4 × 1.50 = $6. She has $23 - 6 = $17 left. Some boxes of pizza cost $8.50, for a total cost of 17 / 8.50 = 2 boxes. The answer is 2.) Figure E.3: Illustration of arithmetic inconsistency before scaling and corrected output after scaling for Hindi reasoning using Llama 1B. -- 14 of 24 -- Input Q: Peter prévoit d’aller au cinéma cette semaine. Il a toujours un billet à 7 $ et du pop-corn à 7 $. S’il a 42 dollars par semaine, combien de fois peut-il aller au cinéma ? A: (English translation: Peter plans to go to the movies this week. He always has a $7 ticket and $7 popcorn. If he has $42 a week, how many times can he go to the movies?) Output before scaling (wrong) Peter a 42 dollars. Il a 7 $ pour le billet et 7 $ pour le pop-corn. Donc 42 - 7 - 7
Chunk 38 · 1,990 chars
fois peut-il aller au cinéma ? A: (English translation: Peter plans to go to the movies this week. He always has a $7 ticket and $7 popcorn. If he has $42 a week, how many times can he go to the movies?) Output before scaling (wrong) Peter a 42 dollars. Il a 7 $ pour le billet et 7 $ pour le pop-corn. Donc 42 - 7 - 7 = 28. Il peut aller au cinéma 28 fois. La réponse est 28. (English translation: Peter has $42. He has $7 for the ticket and $7 for the popcorn. So 42 - 7 - 7 = 28. He can go to the movies 28 times. The answer is 28.) Output after scaling (correct) Peter a 42 dollars. 7 dollars pour le billet et 7 dollars pour le pop-corn font 7 + 7 = 14 dollars. Il peut donc aller au cinéma 42 / 14 = 3 fois. La réponse est 3. (English translation: Peter has $42. $7 for the ticket and $7 for the popcorn make $7 + $7 = $14. He can therefore go to the movies 42 / 14 = 3 times. The answer is 3.) Figure E.4: Illustration of arithmetic inconsistency before scaling and corrected output after scaling for French reasoning using Llama 1B. Input Q: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market? A: Output before pruning (correct) Janet has 16 eggs per day. She eats 3 for breakfast. So she has 16 - 3 = 13 eggs left. She bakes muffins for 4 eggs. So she has 13 - 4 = 9 eggs left. She sells 9 eggs for $2 each. 9 * 2 = 18 dollars. The answer is 18. Output after pruning (wrong) Janet started with 16 eggs. She eats 3 for breakfast. She bakes 4 muffins. She sells 4 - 3 = 2 eggs at the market. 16 - 3 is 8. The answer is 8. Figure E.5: Illustration of correct output before pruning and wrong output after pruning for English reasoning (top- k=0.01) using Llama 1B. -- 15 of 24 -- Input Q: Janets Enten legen 16 Eier am Tag. Sie isst jeden Morgen drei zum
Chunk 39 · 1,999 chars
bakes 4 muffins. She sells 4 - 3 = 2 eggs at the market. 16 - 3 is 8. The answer is 8. Figure E.5: Illustration of correct output before pruning and wrong output after pruning for English reasoning (top- k=0.01) using Llama 1B. -- 15 of 24 -- Input Q: Janets Enten legen 16 Eier am Tag. Sie isst jeden Morgen drei zum Frühstück und backt mit vier täglich Muffins für ihre Freunde. Sie verkauft den Rest auf dem Bauernmarkt täglich für 2 Dollar pro frisches Enten-Ei. Wie viel Dollar verdient sie jeden Tag auf dem Bauernmarkt? A: (English translation: Janet’s ducks lay 16 eggs a day. She eats three for breakfast every morning and bakes muffins with four of them daily for her friends. She sells the rest at the farmers market every day for $2 per fresh duck egg. How many dollars does she earn each day at the farmers market?) Output before pruning (correct) Janets Enten legen 16 Eier am Tag. Sie isst 3 zum Früh- stück und backt 4 Muffins. Das sind 3 + 4 = 7 Eier pro Tag. Sie verkauft 16 - 7 = 9 Eier pro Tag. Sie verdient 2 Dollar pro Ei. 9 * 2 = 18. Die Antwort ist 18. (English translation: Janet’s ducks lay 16 eggs a day. She eats 3 for breakfast and bakes 4 muffins. That’s 3 + 4 = 7 eggs per day. She sells 16 - 7 = 9 eggs per day. She earns $2 per egg. 9 * 2 = 18. The answer is 18.) Output after pruning (wrong) Janets Enten legten 16 Eier am Tag. Sie isst jeden Morgen drei zum Frühstück und backt mit vier täglich Muffins für ihre Freunde. Sie verkauft den Rest auf dem Bauernmarkt täglich für 2 Dollar pro frisches Ei. 16 - 2 = 4. Die Antwort ist 4. Die Antwort ist 4. Die Antwort ist 4. Die Antwort ist 4. Die Antwort ist 4. (English translation: Janet’s ducks laid 16 eggs a day. She eats three for breakfast every morning and bakes muffins for her friends with four of them daily. She sells the rest at the farmers market every day for $2 a fresh egg. 16 - 2 = 4. The answer is 4. The answer is 4. The answer is 4. The answer is 4. The answer is 4.) Figure E.6: Illustration of
Chunk 40 · 1,998 chars
laid 16 eggs a day. She eats three for breakfast every morning and bakes muffins for her friends with four of them daily. She sells the rest at the farmers market every day for $2 a fresh egg. 16 - 2 = 4. The answer is 4. The answer is 4. The answer is 4. The answer is 4. The answer is 4.) Figure E.6: Illustration of correct output before pruning and wrong output after pruning for German reasoning (top-k=0.01) using Llama 1B. Input Q: हेनरी ने अपनी 60 मील की साइिकल यात्रा के दौरान दो बाररु का। वह पहली बार 20 मील के बादरु का। यात्रा के अंत से 15 मील पहले उनकादू सरा पड़ाव था। उसने अपने पहले औरदू सरेस् टॉप के बीच िकतने मील की यात्रा की? A: (English translation: Henry stopped twice during his 60-mile bicycle trip. He first stopped after 20 miles. His second stop was 15 miles before the end of the trip. How many miles did he travel between his first and second stops?) Output before pruning (correct) हेनरी ने अपनी यात्रा के दौरान 60 मील की यात्रा की और दो बाररु का। पहली बार 20 मील की यात्रा के बाद, उसकादू सरा पड़ाव 15 मील की यात्रा के बाद था। 60 - 20 - 15 = 25 मील। उत्तर 25 है। (English translation: Henry traveled 60 miles during his journey and stopped twice. The first time was after trav- eling 20 miles, and his second stop was after traveling 15 miles. 60 - 20 - 15 = 25 miles. The answer is 25.) Output after pruning (wrong) हेनरी ने अपनी 60 मील की साइिकल यात्रा के दौरान दो बाररु का। पहली बार 20 मील के बाद, उसकादू सरा पड़ाव 15 मील के बादहु आ । 60 - 20 = 40। 60 - 20 = 40। 60 - 20 = 40। उत्तर 40 है (English translation: Henry stopped twice during his 60- mile bicycle ride. The first time was after 20 miles, and his second stop was after 15 miles. 60 - 20 = 40. 60 - 20 = 40. 60 - 20 = 40. The answer is 40.) Figure E.7: Illustration of correct output before pruning and wrong output after pruning for Hindi reasoning (top- k=0.01) using Llama 1B. -- 16 of 24 -- Input Q: James décide de courir 3 sprints 3 fois par semaine. Il court 60 mètres à chaque sprint. Combien de mètres
Chunk 41 · 1,998 chars
60 - 20 = 40. 60 - 20 = 40. The answer is 40.) Figure E.7: Illustration of correct output before pruning and wrong output after pruning for Hindi reasoning (top- k=0.01) using Llama 1B. -- 16 of 24 -- Input Q: James décide de courir 3 sprints 3 fois par semaine. Il court 60 mètres à chaque sprint. Combien de mètres au total par semaine ? A: (English translation: James decides to run 3 sprints 3 times a week. He runs 60 meters in each sprint. How many meters in total per week?) Output before pruning (correct) James court 3 sprints 3 fois par semaine. Cela fait 3 × 3 = 9 sprints. Chaque sprint fait 60 mètres. Donc 9 × 60 = 540 mètres. La réponse est 540. (English translation: James runs 3 sprints 3 times a week. That makes 3 × 3 = 9 sprints. Each sprint is 60 meters. So 9 × 60 = 540 meters. The answer is 540.) Output after pruning (wrong) James a 3 sprints 3 fois par semaine. Il court 60 mètres à chaque sprint. 3 + 3 = 5 sprints. 3 + 3 = 5. La réponse est 5. (English translation: James has 3 sprints 3 times a week. He runs 60 meters in each sprint. 3 + 3 = 5 sprints. 3 + 3 = 5. The answer is 5.) Figure E.8: Illustration of correct output before pruning and wrong output after pruning for French reasoning (top- k=0.01) using Llama 1B. Input Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today? A: There are 15 trees originally. Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6. The answer is 6. [...] Q: A treasure hunter found a buried treasure chest filled with gems. There were 175 diamonds, 35 fewer rubies than diamonds, and twice the number of emeralds than the rubies. How many of the gems were there in the chest? A: Output before pruning (wrong format) Let’s break down the problem step by step: 1. **Number of diamonds**: The problem states there are 175 diamonds. So, diamonds = 175. 2. **Number of
Chunk 42 · 1,995 chars
nds, 35 fewer rubies than diamonds, and twice the number of emeralds than the rubies. How many of the gems were there in the chest? A: Output before pruning (wrong format) Let’s break down the problem step by step: 1. **Number of diamonds**: The problem states there are 175 diamonds. So, diamonds = 175. 2. **Number of rubies**: There are 35 fewer rubies than diamonds. So, rubies = diamonds - 35 = 175 - 35 = 140. 3. **Number of emeralds**: There are twice as many emeralds as rubies. So, emeralds = 2 × rubies = 2 × 140 = 280. 4. **Total number of gems**: Total gems = diamonds + rubies + emeralds Total gems = 175 + 140 + 280 = 595. **Answer: 595**. ✅ The total number of gems in the chest is **595**. ✅ Final Answer: **595**. ✅ Output after pruning (correct format) The number of rubies is 175 - 35 = 140. The number of emeralds is 2 * 140 = 280. The total number of gems is 175 + 140 + 280 = 595. The answer is 595. Figure E.9: Illustration of wrong output formatting and poor in-context learning before pruning and correct output after pruning for English reasoning (top-k=1e-5) using Qwen3 4B. -- 17 of 24 -- En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .208 .294 .251 .394 .422 .302 .321 .273 .226 .226 .243 .234 .119 .067 .052 .088 .196 .276 .276 .423 .455 .327 .377 .282 .292 .298 .299 .244 .161 .081 .053 .090 .065 .118 .081 .146 .224 .139 .175 .155 .127 .118 .148 .110 .095 .106 .052 .044 .169 .186 .160 .209 .245 .266 .340 .252 .212 .225 .221 .166 .128 .072 .092 .168 .163 .194 .155 .213 .273 .241 .261 .214 .196 .202 .230 .145 .114 .081 .111 .173 .473 .431 .409 .499 .506 .447 .522 .491 .386 .401 .345 .331 .214 .202 .214 .369 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Top-k=0.0001 0.1 0.2 0.3 0.4 0.5 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .154 .251 .299 .369 .397 .330 .287 .246 .266 .273 .236
Chunk 43 · 1,994 chars
506 .447 .522 .491 .386 .401 .345 .331 .214 .202 .214 .369 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Top-k=0.0001 0.1 0.2 0.3 0.4 0.5 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .154 .251 .299 .369 .397 .330 .287 .246 .266 .273 .236 .232 .181 .084 .095 .193 .155 .253 .286 .368 .426 .375 .313 .264 .302 .326 .269 .265 .190 .072 .096 .192 .078 .182 .158 .162 .227 .199 .194 .165 .156 .160 .130 .100 .125 .106 .065 .042 .173 .300 .251 .216 .268 .250 .239 .192 .257 .314 .250 .183 .203 .113 .139 .100 .179 .302 .234 .203 .250 .242 .222 .189 .244 .280 .247 .162 .212 .094 .122 .090 .449 .444 .493 .525 .516 .499 .440 .393 .448 .491 .402 .386 .346 .211 .223 .335 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Top-k=0.001 0.1 0.2 0.3 0.4 0.5 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .195 .210 .207 .256 .271 .287 .259 .225 .249 .263 .234 .283 .210 .136 .151 .122 .198 .214 .202 .264 .276 .310 .274 .239 .267 .294 .248 .299 .230 .129 .157 .130 .143 .145 .180 .165 .191 .221 .203 .196 .179 .164 .128 .159 .145 .128 .151 .091 .165 .208 .230 .235 .250 .258 .235 .230 .261 .258 .218 .235 .216 .165 .187 .141 .169 .208 .215 .223 .234 .250 .231 .229 .252 .253 .208 .222 .214 .147 .176 .140 .347 .359 .371 .436 .433 .451 .444 .411 .418 .447 .393 .424 .358 .227 .263 .242 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 Top-k=0.01 0.1 0.2 0.3 0.4 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .156 .177 .179 .253 .257 .285 .271 .257 .222 .228 .211 .248 .200 .160 .155 .138 .159 .180 .187 .263 .271 .302 .279 .272 .232 .245 .230 .262 .226 .159 .163 .144 .099 .159 .167 .159 .178 .207 .204 .209
Chunk 44 · 1,993 chars
op-k=0.01 0.1 0.2 0.3 0.4 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .156 .177 .179 .253 .257 .285 .271 .257 .222 .228 .211 .248 .200 .160 .155 .138 .159 .180 .187 .263 .271 .302 .279 .272 .232 .245 .230 .262 .226 .159 .163 .144 .099 .159 .167 .159 .178 .207 .204 .209 .171 .160 .149 .172 .151 .124 .135 .109 .143 .156 .154 .184 .206 .250 .246 .242 .231 .234 .188 .212 .197 .148 .156 .133 .139 .155 .148 .174 .198 .250 .245 .242 .222 .231 .181 .205 .198 .147 .156 .132 .307 .320 .312 .406 .376 .421 .423 .368 .356 .360 .346 .380 .300 .251 .232 .218 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 Top-k=0.05 0.1 0.2 0.3 0.4 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .136 .146 .191 .189 .217 .231 .237 .255 .235 .238 .203 .221 .173 .148 .159 .120 .139 .148 .195 .194 .226 .246 .248 .271 .250 .258 .218 .233 .204 .140 .167 .131 .095 .109 .151 .157 .159 .196 .191 .191 .177 .167 .146 .177 .126 .117 .140 .110 .115 .159 .189 .176 .205 .210 .208 .191 .172 .204 .183 .199 .173 .150 .167 .116 .113 .150 .177 .166 .196 .202 .204 .192 .175 .205 .176 .193 .174 .140 .163 .121 .192 .297 .358 .393 .379 .391 .425 .424 .354 .400 .325 .331 .273 .205 .245 .198 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 Top-k=0.1 0.1 0.2 0.3 0.4 0 2 4 6 8 10 12 14 Layer Index En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .144 .136 .164 .232 .250 .277 .275 .262 .253 .252 .202 .232 .163 .126 .147 .132 .150 .142 .159 .236 .273 .293 .288 .272 .268 .277 .215 .236 .183 .127 .148 .144 .114 .106 .158 .136 .173 .174 .163 .154 .161 .177 .148 .176 .136 .110 .124 .091 .116 .154 .161 .201 .232 .261 .270 .270 .229 .197 .152 .192 .162 .122 .152
Chunk 45 · 1,995 chars
.252 .202 .232 .163 .126 .147 .132 .150 .142 .159 .236 .273 .293 .288 .272 .268 .277 .215 .236 .183 .127 .148 .144 .114 .106 .158 .136 .173 .174 .163 .154 .161 .177 .148 .176 .136 .110 .124 .091 .116 .154 .161 .201 .232 .261 .270 .270 .229 .197 .152 .192 .162 .122 .152 .127 .117 .141 .150 .188 .232 .250 .258 .260 .224 .201 .148 .186 .160 .118 .148 .131 .217 .256 .262 .424 .402 .486 .518 .520 .473 .379 .267 .310 .232 .183 .207 .203 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 Top-k=0.15 0.1 0.2 0.3 0.4 0.5 Figure F.1: Layer-wise Jaccard similarity of math-reasoning parameters across top-k values in Llama 1B, computed for all language pairs (English (En), German (De), French (Fr), and Hindi (Hi)) and the random baseline (Chance) on RACE. -- 18 of 24 -- En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .130 .070 .205 .156 .217 .337 .279 .231 .272 .398 .504 .456 .335 .422 .452 .414 .469 .454 .425 .419 .456 .365 .315 .332 .231 .314 .248 .282 .246 .204 .171 .074 .042 .109 .183 .192 .240 .147 .169 .094 .264 .395 .335 .435 .262 .491 .563 .572 .476 .459 .471 .447 .550 .510 .512 .487 .543 .420 .421 .435 .298 .377 .272 .299 .239 .219 .181 .059 .048 .106 .214 .311 .077 .026 .060 .017 .017 .054 .065 .056 .086 .215 .198 .232 .152 .205 .204 .210 .258 .224 .247 .246 .229 .167 .142 .112 .070 .082 .095 .093 .055 .061 .095 .044 .040 .050 .051 .076 .046 .048 .061 .044 .068 .190 .111 .086 .131 .261 .273 .332 .188 .260 .250 .297 .346 .325 .319 .320 .309 .213 .197 .168 .161 .136 .196 .195 .143 .158 .164 .051 .032 .042 .053 .057 .044 .039 .064 .044 .060 .155 .098 .055 .126 .267 .285 .278 .211 .264 .290 .311 .351 .323 .321 .330 .312 .234 .189 .160 .130 .126 .159 .150 .099 .105 .176 .047 .038 .050 .054 .102 .330 .195 .235 .169 .399 .635 .378 .304 .410 .560 .642 .638 .482 .594 .598 .680 .654 .687 .617 .623 .659 .601 .507 .473 .401 .447 .456 .463 .465
Chunk 46 · 1,996 chars
.039 .064 .044 .060 .155 .098 .055 .126 .267 .285 .278 .211 .264 .290 .311 .351 .323 .321 .330 .312 .234 .189 .160 .130 .126 .159 .150 .099 .105 .176 .047 .038 .050 .054 .102 .330 .195 .235 .169 .399 .635 .378 .304 .410 .560 .642 .638 .482 .594 .598 .680 .654 .687 .617 .623 .659 .601 .507 .473 .401 .447 .456 .463 .465 .383 .305 .277 .133 .281 .366 .344 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Top-k=0.0001 0.1 0.2 0.3 0.4 0.5 0.6 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .171 .085 .190 .238 .222 .319 .249 .344 .403 .415 .444 .487 .461 .416 .431 .467 .363 .470 .429 .410 .451 .444 .389 .394 .364 .420 .408 .362 .327 .295 .228 .132 .067 .112 .170 .176 .234 .097 .233 .216 .256 .336 .338 .381 .421 .469 .488 .547 .544 .457 .464 .505 .418 .514 .464 .456 .509 .482 .453 .477 .434 .471 .462 .391 .362 .318 .246 .122 .060 .106 .184 .231 .059 .032 .065 .034 .038 .090 .062 .076 .149 .218 .236 .266 .227 .186 .198 .205 .197 .231 .212 .155 .154 .134 .119 .111 .103 .122 .130 .127 .107 .108 .126 .070 .054 .086 .105 .058 .079 .116 .101 .098 .109 .118 .073 .106 .181 .262 .284 .325 .256 .230 .238 .253 .250 .291 .286 .248 .253 .199 .195 .191 .167 .183 .194 .192 .171 .184 .202 .113 .075 .102 .098 .113 .080 .111 .087 .093 .100 .105 .108 .111 .186 .254 .289 .333 .270 .229 .240 .251 .239 .289 .277 .241 .235 .192 .180 .167 .156 .171 .187 .189 .183 .182 .220 .117 .076 .095 .109 .122 .374 .264 .372 .340 .462 .510 .373 .502 .575 .583 .608 .655 .617 .584 .585 .649 .542 .624 .581 .599 .635 .668 .617 .615 .571 .592 .637 .577 .556 .561 .500 .354 .249 .381 .458 .379 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Top-k=0.001 0.1 0.2 0.3 0.4 0.5 0.6 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .198 .152 .237
Chunk 47 · 1,996 chars
1 .500 .354 .249 .381 .458 .379 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Top-k=0.001 0.1 0.2 0.3 0.4 0.5 0.6 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .198 .152 .237 .247 .236 .277 .247 .293 .323 .344 .355 .381 .375 .376 .337 .318 .330 .374 .396 .404 .442 .421 .389 .367 .351 .373 .337 .344 .339 .319 .280 .233 .207 .197 .196 .209 .244 .165 .279 .265 .262 .282 .291 .320 .351 .375 .391 .417 .413 .414 .375 .349 .361 .397 .421 .430 .463 .454 .426 .408 .386 .404 .374 .365 .353 .321 .308 .228 .196 .184 .201 .218 .053 .045 .075 .066 .058 .087 .089 .130 .168 .198 .201 .242 .238 .218 .224 .216 .241 .248 .249 .227 .238 .204 .163 .132 .092 .108 .107 .108 .108 .114 .116 .085 .074 .081 .101 .073 .062 .090 .129 .122 .105 .120 .125 .161 .215 .249 .258 .295 .282 .273 .267 .278 .283 .304 .297 .260 .278 .260 .241 .209 .146 .164 .172 .178 .174 .181 .195 .158 .144 .141 .122 .101 .065 .074 .115 .113 .091 .109 .115 .150 .200 .238 .250 .292 .277 .262 .261 .278 .276 .307 .290 .251 .262 .240 .225 .195 .133 .151 .161 .166 .164 .174 .180 .152 .144 .134 .125 .096 .357 .293 .373 .390 .405 .394 .373 .429 .486 .535 .564 .598 .586 .599 .579 .570 .590 .609 .608 .620 .650 .611 .558 .554 .499 .522 .515 .521 .507 .467 .476 .417 .415 .403 .364 .344 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 Top-k=0.01 0.1 0.2 0.3 0.4 0.5 0.6 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .168 .129 .204 .189 .208 .231 .233 .283 .286 .287 .297 .334 .373 .441 .370 .355 .375 .412 .437 .407 .405 .304 .280 .368 .365 .349 .307 .321 .313 .247 .215 .186 .193 .192 .192 .169 .190 .142 .236 .200 .234 .249 .260 .307 .311 .305 .317 .360 .398 .472 .401 .373 .395 .427 .455 .411 .402 .298 .299 .407 .395 .370 .336 .351 .332 .253 .228 .192
Chunk 48 · 1,996 chars
.208 .231 .233 .283 .286 .287 .297 .334 .373 .441 .370 .355 .375 .412 .437 .407 .405 .304 .280 .368 .365 .349 .307 .321 .313 .247 .215 .186 .193 .192 .192 .169 .190 .142 .236 .200 .234 .249 .260 .307 .311 .305 .317 .360 .398 .472 .401 .373 .395 .427 .455 .411 .402 .298 .299 .407 .395 .370 .336 .351 .332 .253 .228 .192 .200 .182 .194 .176 .061 .050 .081 .067 .072 .091 .104 .139 .162 .177 .214 .243 .239 .237 .223 .231 .264 .265 .291 .247 .257 .187 .186 .174 .146 .149 .135 .138 .127 .113 .111 .105 .082 .075 .078 .061 .100 .074 .118 .111 .111 .119 .132 .160 .183 .200 .228 .244 .282 .279 .279 .291 .305 .324 .333 .291 .287 .217 .230 .227 .184 .199 .200 .199 .189 .153 .171 .150 .122 .121 .107 .094 .094 .065 .111 .102 .103 .109 .125 .167 .175 .196 .218 .244 .270 .259 .265 .282 .290 .308 .313 .262 .261 .205 .226 .214 .175 .185 .182 .186 .170 .145 .147 .148 .117 .118 .106 .085 .215 .205 .334 .308 .343 .364 .337 .344 .386 .441 .449 .434 .504 .584 .561 .541 .579 .596 .627 .520 .476 .353 .432 .545 .507 .517 .463 .460 .427 .344 .306 .322 .312 .326 .330 .287 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 Top-k=0.05 0.1 0.2 0.3 0.4 0.5 0.6 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .162 .127 .179 .195 .191 .224 .228 .246 .270 .303 .281 .334 .341 .306 .304 .298 .270 .304 .312 .348 .385 .423 .367 .359 .294 .278 .299 .306 .283 .271 .252 .201 .193 .171 .170 .153 .201 .142 .208 .213 .211 .238 .257 .274 .295 .326 .308 .363 .367 .322 .318 .315 .284 .315 .320 .361 .392 .447 .397 .380 .325 .303 .326 .329 .297 .289 .271 .209 .197 .164 .174 .155 .075 .044 .081 .066 .078 .091 .111 .133 .157 .223 .215 .249 .239 .244 .228 .265 .245 .258 .268 .275 .297 .248 .204 .198 .166 .135 .130 .138 .129 .123 .114 .088 .079 .080 .090 .064 .079 .078 .116 .110 .109 .132 .130 .192 .231 .254 .235 .266 .259 .224 .202 .223 .226 .239 .246 .274 .291 .305 .263 .235 .195
Chunk 49 · 1,997 chars
.164 .174 .155 .075 .044 .081 .066 .078 .091 .111 .133 .157 .223 .215 .249 .239 .244 .228 .265 .245 .258 .268 .275 .297 .248 .204 .198 .166 .135 .130 .138 .129 .123 .114 .088 .079 .080 .090 .064 .079 .078 .116 .110 .109 .132 .130 .192 .231 .254 .235 .266 .259 .224 .202 .223 .226 .239 .246 .274 .291 .305 .263 .235 .195 .198 .175 .187 .188 .190 .186 .146 .117 .108 .144 .089 .083 .058 .111 .104 .103 .120 .125 .184 .223 .243 .230 .259 .249 .212 .192 .215 .220 .235 .240 .261 .274 .282 .248 .222 .189 .180 .163 .176 .173 .184 .175 .140 .120 .094 .121 .081 .249 .212 .301 .322 .363 .371 .371 .446 .476 .476 .462 .586 .608 .590 .569 .584 .543 .551 .576 .606 .619 .615 .506 .467 .384 .394 .432 .444 .447 .425 .415 .340 .310 .278 .258 .253 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 Top-k=0.1 0.1 0.2 0.3 0.4 0.5 0.6 0 3 6 9 12 15 18 21 24 27 30 33 Layer Index En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .145 .105 .174 .157 .179 .202 .203 .220 .240 .302 .289 .315 .317 .313 .298 .255 .279 .309 .365 .335 .332 .309 .339 .391 .359 .329 .235 .251 .306 .251 .246 .191 .180 .165 .161 .127 .161 .124 .195 .174 .195 .220 .216 .234 .262 .322 .318 .338 .343 .344 .329 .282 .307 .334 .396 .361 .344 .326 .350 .426 .388 .342 .272 .274 .320 .262 .257 .203 .188 .165 .160 .133 .060 .040 .068 .065 .080 .086 .084 .128 .148 .205 .192 .203 .196 .232 .211 .196 .197 .233 .279 .223 .256 .264 .215 .198 .182 .149 .126 .124 .126 .118 .113 .089 .076 .067 .077 .078 .086 .062 .120 .108 .105 .118 .128 .154 .204 .249 .234 .230 .241 .258 .253 .229 .252 .295 .326 .270 .257 .221 .222 .261 .227 .182 .182 .191 .161 .176 .168 .148 .135 .120 .100 .108 .083 .056 .110 .105 .100 .109 .113 .148 .194 .240 .225 .228 .234 .251 .249 .238 .253 .299 .323 .262 .247 .210 .207 .242 .214 .173 .161 .188 .154 .168 .164 .143 .132 .116 .095 .098 .281 .197 .317 .273 .281 .277 .308
Chunk 50 · 1,994 chars
.229 .252 .295 .326 .270 .257 .221 .222 .261 .227 .182 .182 .191 .161 .176 .168 .148 .135 .120 .100 .108 .083 .056 .110 .105 .100 .109 .113 .148 .194 .240 .225 .228 .234 .251 .249 .238 .253 .299 .323 .262 .247 .210 .207 .242 .214 .173 .161 .188 .154 .168 .164 .143 .132 .116 .095 .098 .281 .197 .317 .273 .281 .277 .308 .383 .489 .552 .512 .572 .581 .562 .545 .497 .522 .573 .545 .566 .552 .577 .542 .557 .468 .394 .356 .387 .430 .383 .409 .341 .328 .305 .269 .198 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 Top-k=0.15 0.1 0.2 0.3 0.4 0.5 Figure F.2: Layer-wise Jaccard similarity of math-reasoning parameters across top-k values in Qwen3 4B, com- puted for all language pairs (English (En), German (De), French (Fr), and Hindi (Hi)) and the random baseline (Chance) on RACE. -- 19 of 24 -- En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .220 .173 .195 .327 .417 .309 .344 .423 .330 .320 .274 .281 .340 .370 .309 .365 .288 .268 .293 .286 .377 .243 .340 .224 .194 .181 .154 .157 .186 .119 .141 .217 .201 .162 .167 .361 .422 .359 .358 .434 .344 .459 .361 .323 .401 .433 .395 .466 .356 .342 .331 .325 .379 .273 .284 .221 .159 .190 .162 .124 .168 .115 .134 .150 .112 .144 .100 .190 .172 .185 .164 .281 .186 .264 .204 .164 .169 .210 .149 .208 .182 .173 .146 .182 .222 .160 .205 .188 .149 .144 .135 .133 .167 .169 .097 .091 .183 .242 .198 .269 .241 .252 .261 .413 .381 .409 .367 .383 .312 .363 .257 .362 .329 .294 .283 .388 .388 .319 .304 .278 .240 .178 .204 .148 .192 .115 .164 .180 .188 .270 .184 .256 .224 .258 .235 .417 .387 .423 .352 .375 .307 .376 .260 .307 .290 .283 .249 .326 .324 .252 .272 .224 .181 .202 .238 .113 .166 .092 .159 .181 .364 .417 .422 .533 .558 .529 .537 .609 .620
Chunk 51 · 1,995 chars
388 .388 .319 .304 .278 .240 .178 .204 .148 .192 .115 .164 .180 .188 .270 .184 .256 .224 .258 .235 .417 .387 .423 .352 .375 .307 .376 .260 .307 .290 .283 .249 .326 .324 .252 .272 .224 .181 .202 .238 .113 .166 .092 .159 .181 .364 .417 .422 .533 .558 .529 .537 .609 .620 .553 .550 .609 .639 .626 .558 .613 .506 .504 .477 .492 .502 .427 .390 .342 .303 .329 .371 .268 .317 .250 .303 .383 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Top-k=0.0001 0.1 0.2 0.3 0.4 0.5 0.6 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .179 .213 .241 .286 .365 .327 .382 .396 .376 .386 .368 .372 .350 .420 .334 .339 .305 .265 .267 .272 .255 .229 .245 .214 .167 .148 .132 .119 .148 .104 .100 .144 .172 .214 .258 .308 .392 .351 .389 .401 .387 .447 .421 .401 .393 .464 .406 .424 .373 .341 .316 .327 .276 .285 .256 .216 .176 .172 .159 .109 .146 .109 .108 .141 .127 .186 .157 .146 .187 .182 .233 .280 .284 .304 .271 .243 .232 .248 .164 .166 .163 .163 .146 .167 .145 .139 .127 .111 .087 .102 .096 .092 .148 .093 .055 .062 .202 .279 .210 .226 .257 .294 .327 .389 .421 .443 .374 .380 .348 .384 .273 .281 .269 .274 .219 .277 .244 .273 .244 .219 .167 .187 .187 .162 .190 .142 .104 .156 .202 .296 .198 .214 .253 .271 .302 .383 .417 .451 .376 .387 .351 .380 .261 .251 .249 .255 .216 .266 .229 .254 .230 .221 .174 .204 .188 .153 .164 .134 .119 .162 .337 .513 .503 .510 .536 .578 .568 .626 .611 .622 .553 .591 .625 .703 .605 .594 .538 .474 .454 .454 .416 .423 .412 .380 .302 .309 .326 .245 .268 .216 .187 .279 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000
Chunk 52 · 1,998 chars
164 .134 .119 .162 .337 .513 .503 .510 .536 .578 .568 .626 .611 .622 .553 .591 .625 .703 .605 .594 .538 .474 .454 .454 .416 .423 .412 .380 .302 .309 .326 .245 .268 .216 .187 .279 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Top-k=0.001 0.2 0.4 0.6 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .195 .258 .210 .234 .301 .275 .301 .305 .304 .329 .335 .322 .303 .340 .287 .285 .273 .269 .265 .258 .245 .248 .246 .223 .203 .185 .151 .138 .168 .125 .104 .134 .195 .250 .227 .241 .317 .284 .310 .308 .315 .348 .358 .340 .322 .357 .305 .309 .297 .297 .296 .288 .262 .259 .256 .216 .205 .186 .157 .135 .168 .135 .110 .139 .088 .167 .185 .174 .189 .221 .231 .239 .237 .250 .253 .224 .228 .256 .209 .203 .189 .206 .179 .187 .159 .167 .155 .133 .099 .113 .104 .107 .142 .096 .070 .070 .129 .181 .253 .237 .272 .259 .265 .265 .299 .331 .330 .315 .295 .335 .267 .261 .251 .268 .237 .252 .210 .234 .223 .206 .165 .183 .148 .142 .179 .127 .091 .080 .129 .183 .249 .225 .262 .250 .257 .259 .291 .324 .322 .308 .286 .321 .255 .248 .239 .262 .232 .242 .206 .220 .209 .203 .164 .179 .140 .142 .171 .131 .091 .089 .380 .369 .380 .431 .502 .474 .477 .499 .546 .569 .585 .576 .542 .587 .485 .512 .460 .428 .415 .406 .390 .369 .366 .349 .328 .297 .251 .227 .273 .214 .160 .170 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 Top-k=0.01 0.1 0.2 0.3 0.4 0.5 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .156 .163 .208 .217 .223 .228 .257 .261 .283 .325 .333 .294 .279 .301 .272
Chunk 53 · 1,999 chars
.005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 .005 Top-k=0.01 0.1 0.2 0.3 0.4 0.5 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .156 .163 .208 .217 .223 .228 .257 .261 .283 .325 .333 .294 .279 .301 .272 .289 .269 .270 .255 .249 .246 .233 .233 .204 .175 .156 .152 .138 .153 .122 .108 .112 .155 .162 .218 .229 .232 .237 .274 .270 .295 .344 .355 .309 .291 .312 .284 .303 .284 .285 .276 .267 .251 .244 .236 .211 .181 .164 .158 .134 .156 .131 .117 .122 .093 .115 .194 .156 .191 .192 .206 .201 .210 .225 .225 .201 .197 .226 .206 .213 .209 .206 .187 .185 .154 .156 .152 .144 .108 .100 .109 .096 .116 .100 .091 .078 .114 .209 .176 .176 .198 .256 .279 .310 .326 .321 .339 .311 .303 .315 .270 .262 .234 .227 .193 .196 .181 .198 .200 .187 .144 .136 .139 .131 .163 .118 .100 .081 .111 .200 .171 .176 .192 .255 .271 .298 .312 .308 .326 .300 .292 .310 .265 .255 .230 .222 .190 .190 .178 .188 .188 .187 .141 .134 .134 .132 .162 .120 .098 .083 .223 .342 .354 .315 .358 .380 .486 .515 .558 .567 .594 .584 .568 .616 .534 .465 .384 .365 .306 .315 .317 .334 .329 .282 .248 .224 .203 .214 .256 .210 .170 .155 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 .024 Top-k=0.05 0.1 0.2 0.3 0.4 0.5 0.6 En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .140 .138 .185 .212 .226 .242 .255 .243 .247 .263 .273 .240 .239 .293 .257 .269 .272 .280 .273 .251 .230 .214 .212 .190 .174 .155 .159 .132 .137 .112 .107 .110 .141 .136 .198 .216 .239 .254 .262 .252 .252 .269 .292 .262 .254 .311 .274 .284 .291 .301 .296 .268 .240 .227 .221 .204 .179
Chunk 54 · 1,999 chars
12 .226 .242 .255 .243 .247 .263 .273 .240 .239 .293 .257 .269 .272 .280 .273 .251 .230 .214 .212 .190 .174 .155 .159 .132 .137 .112 .107 .110 .141 .136 .198 .216 .239 .254 .262 .252 .252 .269 .292 .262 .254 .311 .274 .284 .291 .301 .296 .268 .240 .227 .221 .204 .179 .160 .167 .129 .139 .120 .118 .118 .085 .113 .181 .165 .157 .191 .225 .234 .252 .261 .267 .231 .217 .238 .178 .181 .190 .192 .162 .156 .147 .175 .165 .150 .128 .116 .104 .090 .108 .092 .080 .071 .123 .140 .189 .168 .161 .214 .240 .254 .267 .289 .296 .282 .270 .339 .302 .286 .234 .232 .205 .205 .196 .181 .174 .153 .130 .130 .131 .117 .143 .119 .105 .086 .119 .133 .183 .162 .157 .209 .230 .248 .257 .277 .294 .280 .257 .337 .293 .275 .226 .224 .197 .195 .186 .173 .167 .155 .127 .127 .126 .113 .141 .123 .107 .085 .317 .202 .385 .349 .352 .410 .392 .406 .415 .442 .417 .451 .458 .528 .498 .512 .459 .424 .391 .372 .349 .295 .273 .238 .231 .207 .222 .189 .207 .172 .162 .134 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 .047 Top-k=0.1 0.1 0.2 0.3 0.4 0.5 0 3 6 9 12 15 18 21 24 27 30 Layer Index En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair .122 .149 .189 .187 .217 .229 .254 .263 .277 .268 .268 .259 .241 .257 .244 .250 .246 .236 .226 .215 .213 .215 .220 .200 .176 .151 .154 .128 .142 .111 .106 .107 .119 .152 .200 .191 .230 .240 .263 .269 .277 .292 .287 .264 .250 .267 .263 .262 .262 .249 .248 .236 .228 .225 .229 .215 .184 .157 .159 .128 .149 .117 .118 .114 .082 .109 .140 .146 .158 .181 .195 .209 .222 .232 .244 .230 .229 .268 .223 .212 .189 .167 .151 .156 .143 .149 .147 .139 .120 .113 .120
Chunk 55 · 1,993 chars
40 .263 .269 .277 .292 .287 .264 .250 .267 .263 .262 .262 .249 .248 .236 .228 .225 .229 .215 .184 .157 .159 .128 .149 .117 .118 .114 .082 .109 .140 .146 .158 .181 .195 .209 .222 .232 .244 .230 .229 .268 .223 .212 .189 .167 .151 .156 .143 .149 .147 .139 .120 .113 .120 .105 .104 .094 .083 .075 .104 .127 .208 .160 .219 .194 .184 .191 .208 .240 .261 .247 .226 .266 .245 .238 .236 .238 .218 .199 .174 .168 .165 .160 .137 .123 .120 .106 .119 .103 .099 .080 .102 .117 .199 .152 .216 .187 .179 .184 .204 .234 .255 .243 .224 .266 .245 .233 .230 .230 .210 .193 .168 .160 .158 .157 .133 .121 .118 .102 .118 .103 .100 .077 .279 .217 .324 .366 .433 .389 .394 .429 .424 .417 .467 .397 .426 .450 .387 .384 .415 .381 .349 .333 .314 .308 .303 .276 .232 .199 .212 .171 .182 .169 .155 .144 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 .068 Top-k=0.15 0.1 0.2 0.3 0.4 Figure F.3: Layer-wise Jaccard similarity of math-reasoning parameters across top-k values in Llama 8B, computed for all language pairs (English (En), German (De), French (Fr), and Hindi (Hi)) and the random baseline (Chance) on RACE. -- 20 of 24 -- 0 0.05 0.1 0.15 Top-k 0.0 0.2 0.4 0.6 0.8 1.0 # Math-specific Params ×108 Parameter Count 0.0001 0.001 0.01 0.05 0.1 0.15 Top-k En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair 0.292 0.325 0.312 0.285 0.267 0.254 0.342 0.362 0.336 0.302 0.285 0.272 0.125 0.131 0.146 0.152 0.164 0.150 0.184 0.188 0.202 0.195 0.193 0.192 0.179 0.187 0.192 0.184 0.182 0.184 0.466 0.530 0.498 0.412 0.432 0.421 0.000 0.000 0.005 0.024 0.047 0.068 Global Jaccard Similarity 0.1 0.2 0.3 0.4 0.5 En De Fr Hi Figure F.4: Math-specific parameters in Qwen3 4B. Left:
Chunk 56 · 1,995 chars
125 0.131 0.146 0.152 0.164 0.150 0.184 0.188 0.202 0.195 0.193 0.192 0.179 0.187 0.192 0.184 0.182 0.184 0.466 0.530 0.498 0.412 0.432 0.421 0.000 0.000 0.005 0.024 0.047 0.068 Global Jaccard Similarity 0.1 0.2 0.3 0.4 0.5 En De Fr Hi Figure F.4: Math-specific parameters in Qwen3 4B. Left: Number of isolated parameters per language (English (En), German (De), French (Fr), and Hindi (Hi)). Right: Jaccard similarity between language-specific parameter sets across all language pairs, compared to a random baseline using RACE as comparison dataset. 0 0.05 0.1 0.15 Top-k 0.0 0.5 1.0 1.5 # Math-specific Params ×108 Parameter Count 0.0001 0.001 0.01 0.05 0.1 0.15 Top-k En-De En-Fr En-Hi De-Hi Fr-Hi De-Fr Chance Language Pair 0.256 0.256 0.240 0.217 0.202 0.197 0.277 0.279 0.252 0.227 0.212 0.205 0.160 0.159 0.169 0.160 0.159 0.155 0.272 0.258 0.226 0.205 0.190 0.171 0.246 0.247 0.220 0.200 0.186 0.167 0.443 0.443 0.401 0.345 0.316 0.303 0.000 0.000 0.005 0.024 0.047 0.068 Global Jaccard Similarity 0.1 0.2 0.3 0.4 En De Fr Hi Figure F.5: Math-specific parameters in Llama 8B. Left: Number of isolated parameters per language (English (En), German (De), French (Fr), and Hindi (Hi)). Right: Jaccard similarity between language-specific parameter sets across all language pairs, compared to a random baseline using RACE as comparison dataset. -- 21 of 24 -- 0 2 4 6 8 10 12 14 0.80 1.60 2.40 3.20 # Isolated Params ×103 Top-k=0.0001 En De Fr Hi 0 2 4 6 8 10 12 14 1.60 2.40 3.20 ×104 Top-k=0.001 En De Fr Hi 0 2 4 6 8 10 12 14 1.00 1.50 2.00 # Isolated Params ×105 Top-k=0.01 En De Fr Hi 0 2 4 6 8 10 12 14 4.00 6.00 8.00 ×105 Top-k=0.05 En De Fr Hi 0 2 4 6 8 10 12 14 Layer Index 0.60 0.90 1.20 # Isolated Params ×106 Top-k=0.1 En De Fr Hi 0 2 4 6 8 10 12 14 Layer Index 0.80 1.20 1.60 ×106 Top-k=0.15 En De Fr Hi Figure F.6: Layer-wise
Chunk 57 · 1,996 chars
×105 Top-k=0.01 En De Fr Hi 0 2 4 6 8 10 12 14 4.00 6.00 8.00 ×105 Top-k=0.05 En De Fr Hi 0 2 4 6 8 10 12 14 Layer Index 0.60 0.90 1.20 # Isolated Params ×106 Top-k=0.1 En De Fr Hi 0 2 4 6 8 10 12 14 Layer Index 0.80 1.20 1.60 ×106 Top-k=0.15 En De Fr Hi Figure F.6: Layer-wise distribution of math-reasoning parameters across top-k values in Llama 1B, using RACE as comparison dataset. -- 22 of 24 -- 0 3 6 9 12 15 18 21 24 27 30 33 2.00 4.00 6.00 # Isolated Params ×103 Top-k=0.0001 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 33 1.50 3.00 4.50 6.00 ×104 Top-k=0.001 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 33 1.50 3.00 4.50 # Isolated Params ×105 Top-k=0.01 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 33 0.40 0.80 1.20 1.60 ×106 Top-k=0.05 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 33 Layer Index 0.80 1.60 2.40 3.20 # Isolated Params ×106 Top-k=0.1 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 33 Layer Index 0.80 1.60 2.40 3.20 ×106 Top-k=0.15 En De Fr Hi Figure F.7: Layer-wise distribution of math-reasoning parameters across top-k values in Qwen3 4B, using RACE as comparison dataset. -- 23 of 24 -- 0 3 6 9 12 15 18 21 24 27 30 0.30 0.60 0.90 1.20 # Isolated Params ×104 Top-k=0.0001 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 0.50 0.75 1.00 ×105 Top-k=0.001 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 0.40 0.60 0.80 1.00 # Isolated Params ×106 Top-k=0.01 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 0.80 1.60 2.40 3.20 ×106 Top-k=0.05 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 Layer Index 1.50 3.00 4.50 # Isolated Params ×106 Top-k=0.1 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 Layer Index 3.00 4.50 6.00 ×106 Top-k=0.15 En De Fr Hi Figure F.8: Layer-wise distribution of math-reasoning parameters across top-k values in Llama 8B, using RACE as comparison dataset. -- 24 of
Chunk 58 · 325 chars
0 Layer Index 1.50 3.00 4.50 # Isolated Params ×106 Top-k=0.1 En De Fr Hi 0 3 6 9 12 15 18 21 24 27 30 Layer Index 3.00 4.50 6.00 ×106 Top-k=0.15 En De Fr Hi Figure F.8: Layer-wise distribution of math-reasoning parameters across top-k values in Llama 8B, using RACE as comparison dataset. -- 24 of 24 --