SEATauBench: Adapting Tool-Agent-User Evaluation Into Low-Resource Southeast Asian Languages

SEATauBench: Adapting Tool-Agent-User Evaluation
Into Low-Resource Southeast Asian Languages
My Chiffon Nguyen1, Aulia Adila1, Saksorn Ruangtanusak1,2, Kittiphat Leesombatwathana1,3,
Vissuta Gunawan Lim1, Patomporn Payoungkhamdee1,4, Samuel Cahyawijaya1,5
1SEACrowd, 2SCB DataX, SCBX Group, 3Chulalongkorn University, 4VISTEC, 5Cohere
{chiffonng136,auliaadila036,vglim3653}@gmail.com
saksorn.ruangtanusak@data-x.ai, 6534404823@student.chula.ac.th
patomporn.p_s21@vistec.ac.th,samuelcahyawijaya@cohere.com
Abstract
While AI development and evaluation for
Southeast Asia (SEA) has grown rapidly,
agent capabilities in regional languages are
still poorly understood despite its importance
to sovereign AI. To fill this gap, we intro-
duce SEATauBench 1, the first agent-focused
evaluation framework for SEA sovereign
AI. SEATauBench adapts τ 2-Bench to five
languages—Mandarin, Vietnamese, Thai, In-
donesian, and Filipino—and evaluates agents
across progressively localized settings that vary
the language of user-agent interaction, tool
specifications, and task domains. Across three
recent models, we find that English agent ca-
pabilities transfer reasonably well when only
the conversation language changes, but qual-
ity and robustness degrade sharply as more
task contexts are localized, with the largest
losses in full domain adaptation. We also
the limits of English-only agent assessment
for measuring agent capabilities in SEA lan-
guages. More broadly, SEATauBench pro-
vides a diagnostic benchmark and reusable
adaptation pipeline for building reliable mul-
tilingual agents for linguistically diverse re-
gions. Data and code can be accessed at
github.com/SEACrowd/SEATauBench.
1 Introduction
Sovereign artificial intelligence (AI) has become
critical for nations seeking to maintain autonomy
in their digital futures (Chae et al., 2025), includ-
ing Southeast Asia. As articulated in Mushkani
et al. (2025); Barasa et al. (2026), sovereign AI
encompasses not only technological

Crowd/SEATauBench.
1 Introduction
Sovereign artificial intelligence (AI) has become
critical for nations seeking to maintain autonomy
in their digital futures (Chae et al., 2025), includ-
ing Southeast Asia. As articulated in Mushkani
et al. (2025); Barasa et al. (2026), sovereign AI
encompasses not only technological self-reliance
but also cultural and linguistic relevance, a vital
dimension for the more than 700 million people
whose linguistic diversity is poorly represented by
English-centric development and evaluation (Bhan-
dari and Modi, 2026; Putra, 2024).
1SEATauBench is pronounced "si-tau-bench", similar to
the Filipino word for string beans, "sitaw".
Figure 1: SEATauBench exposes a critical English-SEA
agentic capability gap in existing proprietary and open-
source LLMs across progressively localized evaluation
scenarios. This evidence exposes the unreliability of
existing English-centric benchmark to reflect the actual
capabilities of LLMs for sovereign AI adoption.
SEA-specific language evaluation has expanded
rapidly, with SEA-Exam and SEA-Bench (Liu
et al., 2025; Zhang et al., 2024; Nguyen et al.,
2024), SEA-VL (Cahyawijaya et al., 2025a, 2026),
SEACrowd (Lovenia et al., 2024), SEA-HELM
(Susanto et al., 2025), NusaCrowd (Cahyawijaya
et al., 2023a), NusaWrites (Cahyawijaya et al.,
2023b), and NusaX (Winata et al., 2023) estab-
lishing valuable foundations for measuring re-
gional language understanding, cultural knowl-
edge, reasoning, safety, and multimodal capabil-
ities. Nevertheless, these benchmarks primarily
1
arXiv:2606.28715v1 [cs.CL] 27 Jun 2026

-- 1 of 23 --

assess static model behaviors, with limited work
evaluating whether agents can complete multi-turn,
tool-mediated tasks in regional languages, the ca-
pabilities required when more and more AI sys-
tems operate in real-world deployments like ser-
vice, commerce and travel (Budzianowski et al.,
2018; Eric et al., 2020; Zang et al., 2020; Han et al.,
2021; Ye et al., 2022).
To address

r agents can complete multi-turn,
tool-mediated tasks in regional languages, the ca-
pabilities required when more and more AI sys-
tems operate in real-world deployments like ser-
vice, commerce and travel (Budzianowski et al.,
2018; Eric et al., 2020; Zang et al., 2020; Han et al.,
2021; Ye et al., 2022).
To address this gap, we introduce SEATauBench,
the first agent-focused evaluation framework for
SEA. SEATauBench adapts τ 2-Bench (Yao et al.,
2024; Barres et al., 2025; Shi et al., 2026; Ray et al.,
2026) into five target languages L2 (Mandarin Chi-
nese, Vietnamese, Thai, Indonesian, Filipino) and
evaluates agents across three progressively local-
ized settings: (1) L2 Interaction, isolating lin-
guistic capability in user-agent conversation; (2)
L2 Tool, testing the ability to use tools with non-
English tool specifications; and (3) L2 Domain,
evaluating performance when all task contexts are
in L2 (Section 3.3). To translate different inter-
faces an AI agent interacts with, without breaking
the execution of τ 2-Bench, we develop a structured,
non-breaking translation pipeline (Section 3.2).
Across three recent agent models, we find that
English agent capabilities transfer reasonably well
when agents only need to respond in target lan-
guages, but quality and robustness degrade sharply
once tools, policies, and task contexts are progres-
sively provided in SEA languages (Section 5.1).
These results expose a gap between the growth of
SEA evaluation resources and the readiness of cur-
rent agents for sovereign AI deployment, establish-
ing SEATauBench as a diagnostic benchmark for
building reliable multilingual agents for the region.
2 Related Works
2.1 Agent Evaluation
Task-oriented dialogue benchmarks such as Multi-
WOZ (Budzianowski et al., 2018; Eric et al., 2020;
Zang et al., 2020; Han et al., 2021; Ye et al., 2022)
and MASSIVE (FitzGerald et al., 2023) established
evaluation for goal-directed dialogue and multi-
lingual intent-slot understanding. Recent

rks
2.1 Agent Evaluation
Task-oriented dialogue benchmarks such as Multi-
WOZ (Budzianowski et al., 2018; Eric et al., 2020;
Zang et al., 2020; Han et al., 2021; Ye et al., 2022)
and MASSIVE (FitzGerald et al., 2023) established
evaluation for goal-directed dialogue and multi-
lingual intent-slot understanding. Recent tool-use
benchmarks, including ToolEval (Qin et al., 2024)
and BFCL (Patil et al., 2025), extend this direc-
tion to function calling and API use. However,
deployed service agents must further sustain multi-
turn interactions, follow domain policies, and up-
date external states. τ 2-Bench (Yao et al., 2024;
Barres et al., 2025; Shi et al., 2026; Ray et al., 2026)
address this through simulated user-agent-tool envi-
ronments, scored by task completion metrics pass^1
and pass^3. Related benchmarks further assess re-
alistic agent behavior in business and professional
settings (Huang et al., 2025; Drouin et al., 2024;
Boisvert et al., 2024; Xu et al., 2026; Patwardhan
et al., 2026).
Despite these advances, existing agent bench-
marks remain largely centered on high-resource
languages, limiting their applicability to sovereign
AI settings. SEATauBench addresses this gap by ex-
tending τ 2-Bench to SEA languages through local-
ized conversational and tool-use scenarios. To our
knowledge, it is the first multilingual benchmark to
preserve the task-oriented evaluation framework of
τ 2-Bench, contributing towards reliable assessment
of agent capabilities in multilingual contexts.
2.2 Multilingual Evaluation
Multilingual evaluation has mainly targeted un-
derstanding, reasoning, translation, and multi-
modal comprehension through benchmarks such as
mMMLU (Hendrycks et al., 2021; OpenAI, 2024),
GlobalMMLU (Singh et al., 2025), GlotEval (Luo
et al., 2025), CVQA (Romero et al., 2024), and
AyaVisionBench (CohereLabs, 2025). For South-
east Asian languages, resources such as NusaX
(Winata et al., 2023), NusaWrites (Cahyawijaya
et al., 2023b), NusaDialogue

rks such as
mMMLU (Hendrycks et al., 2021; OpenAI, 2024),
GlobalMMLU (Singh et al., 2025), GlotEval (Luo
et al., 2025), CVQA (Romero et al., 2024), and
AyaVisionBench (CohereLabs, 2025). For South-
east Asian languages, resources such as NusaX
(Winata et al., 2023), NusaWrites (Cahyawijaya
et al., 2023b), NusaDialogue (Purwarianti et al.,
2025), IndoTOD (Kautsar et al., 2023), SEACrowd
(Lovenia et al., 2024), StingrayBench (Cahyaw-
ijaya et al., 2025b), SEAExam/SEABench (Liu
et al., 2025), SEA-HELM (Susanto et al., 2025),
SEA-VL (Cahyawijaya et al., 2025a), and SEA-
VQA (Urailertprasert et al., 2024) expand evalua-
tion to local linguistic, cultural, reasoning, safety,
and multimodal contexts. However, these bench-
marks largely focus on static evaluation rather than
interactive tool use or multi-turn task completion.
The closest work is MASSIVE-Agents (Kulka-
rni et al., 2025), which evaluates function calling
across 52 languages using BFCL, but remains lim-
ited to function selection and argument prediction.
SEATauBench goes further by evaluating full user-
agent-tool interaction, requiring agents to interact
over multiple turns, follow instructions, use tools,
and complete realistic tasks in SEA languages. It
therefore bridges multilingual function calling and
interactive agent evaluation, addressing a gap left
by existing SEA multilingual and agent evaluation.
2

-- 2 of 23 --

Discover source artifacts	1
Phase 1. Offline translation
2 	Extract segments for translation
3 Mask IDs, enum literals,
tool names, placeholders.
Mask tokens
Reformat translated text & record
manifest
5
Environment
canonical runtime
Localized tool
schema
Translated desc.
+ enum choices
Agent
Translated
tool schema
Execute tool
canonical
arguments
Localized
tool output
canonical payload
+ messages
Build runtime maps
Map canonical ↔ localized literals
Return localizations
Normalize
arguments
localized → canonical
Phase 2. Runtime localization
During evaluation
4 Translate

anslated desc.
+ enum choices
Agent
Translated
tool schema
Execute tool
canonical
arguments
Localized
tool output
canonical payload
+ messages
Build runtime maps
Map canonical ↔ localized literals
Return localizations
Normalize
arguments
localized → canonical
Phase 2. Runtime localization
During evaluation
4 Translate 	gemini-3.1-flash
schema and literal values to build the
glossary, segment and unmask
content, and apply glossary terms.
EN
Tool
Docstring
Data Model
Schema/Literals
Task
Query/Content
Prompt
Policy/Workflow
Database
Content
ID · TH · TL · VI · ZH
7
6
8a
8b	Canonicalize
translated
values
9
get_tools
manifest
Translated
artifacts
/{lang_id}/
Figure 2: Overview of the automated translation pipeline for generating multilingual τ 2-Bench artifacts (Section 3.2).
We provide more details about the pipeline and the resulting translated artifacts in Appendix A.
3 Adapting τ 2-Bench to SEATauBench
3.1 Background
We construct SEATauBench by extending the En-
glish tool-agent-user benchmark τ 2-Bench (Bar-
res et al., 2025) to multilingual settings. This
adaptation must account for two coupled agent-
facing surfaces: interaction content (task defini-
tions, domain policies and workflows, and struc-
tured databases) and executable interfaces (tool
schemas and, in telecom, return messages).
At runtime, the agent must read policies, reason
about tasks, call tools, inspect outputs, and talk
to a simulated user in the target language. The
benchmark therefore needs a localized interface
whose visible text is translated, while the execution
layer still preserves canonical English values. Our
pipeline aligns these representations to preserve
task semantics and metric comparability.
Section 3.2 describes how we translate and con-
struct L2 artifacts, and Section 3.3 uses those arti-
facts to define controlled evaluation scenarios with
increasing levels of L2 adaptation.
3.2 SEATauBench L2 Adaptation Pipeline
Figure 2 summarizes our two-phase adaptation
pipeline: offline

and metric comparability.
Section 3.2 describes how we translate and con-
struct L2 artifacts, and Section 3.3 uses those arti-
facts to define controlled evaluation scenarios with
increasing levels of L2 adaptation.
3.2 SEATauBench L2 Adaptation Pipeline
Figure 2 summarizes our two-phase adaptation
pipeline: offline translation, which materializes
language-specific assets, and runtime localization,
which patches the environment observed by the
agent. The full details are provided in Appendix A.
We adapt three domains from τ 2-Bench (retail,
airline, and telecom) into five L2: Vietnamese (vi),
Indonesian (id), Thai (th), Filipino (tl), and Man-
darin Chinese (zh). 2
Offline Translation. We translate static domain
assets to construct L2 artifacts. It extracts natural-
language spans from tasks, policies, databases,
tool docstrings, schemas, and tool-return templates,
while masking executable tokens such as IDs, sta-
tus values, tool names, and structural markers. It
first translates schema literals to establish a glos-
sary, preventing the same executable value from
being rendered inconsistently. The pipeline then
writes outputs with format-specific writers and
records per-language manifests with model meta-
data and source-file SHA-256 fingerprints. Ap-
pendix A.5 reports the resulting artifact statistics.
Runtime Localization. The runtime localization
phase handles dynamic content exposed during in-
ference. First, it localizes the tool schema shown to
the agent: descriptions, enum choices, and exam-
ples are rendered in L2 so the scenario tests target-
language tool use rather than English schema read-
ing, while the underlying implementation remains
unchanged. Second, it preserves executability by
2We do not include other SEA languages such as Malay,
Lao, or Cambodian due to limited annotator capacity.
3

-- 3 of 23 --

Scenario L2
Tools
L2
Convo
L2 DB
& Policy
(S1) English Only ✗ ✗ ✗
(S2) L2 Interaction ✗ ✓ ✗
(S3) L2 Tool ✓ ✗ ✗
(S4) L2 Domain ✓ ✓ ✓
Table 1:

ains
unchanged. Second, it preserves executability by
2We do not include other SEA languages such as Malay,
Lao, or Cambodian due to limited annotator capacity.
3

-- 3 of 23 --

Scenario L2
Tools
L2
Convo
L2 DB
& Policy
(S1) English Only ✗ ✗ ✗
(S2) L2 Interaction ✗ ✓ ✗
(S3) L2 Tool ✓ ✗ ✗
(S4) L2 Domain ✓ ✓ ✓
Table 1: SEATauBench evaluation scenarios across En-
glish and five SEA languages. S2 and S3 isolate L2
dialogue and L2 tool docstring respectively, while S4
L2 domain uses all translated contexts.
normalizing localized arguments back to canon-
ical English values before tool calls; otherwise,
correct L2 arguments could fail only because the
original tools expect English literals. After exe-
cution, tool responses are localized back into L2,
and final payloads are canonicalized again before
scoring, keeping the interaction monolingual for
the agent while making metrics comparable across
languages and with the English benchmark. To-
gether, offline translation and runtime localization
let us vary which benchmark surfaces are exposed
in L2, forming the scenario design.
Human Manual Review. For each target lan-
guage, a native speaker—either an author or a re-
cruited reviewer—reviews the machine-translated
artifacts. They read every translated prose docu-
ment in full (domain policy and agent/user instruc-
tions) and a sample of 100 segments each from
the database and the task contexts. Reviews are
carried out in a per-language Excel workbook with
one sheet per artifact (see details in Appendix ??),
where annotators add a corrected translation and
notes (both optional). If human corrected values
are present, we use them in experiments; otherwise,
we consider machine translated versions usable.
We find that our pipeline ingests 91% of machine-
translated artifacts, showing that they can be trusted
even if used by themselves. Since we do not em-
ploy more than one reviewer per language, we do
not calculate inter-annotator agreement scores.
3.3 Evaluation

erwise,
we consider machine translated versions usable.
We find that our pipeline ingests 91% of machine-
translated artifacts, showing that they can be trusted
even if used by themselves. Since we do not em-
ploy more than one reviewer per language, we do
not calculate inter-annotator agreement scores.
3.3 Evaluation Scenarios
SEATauBench has four scenarios with increasing
level of L2 adaptation (summary in Table 1).
(S1) English Only. The original English bench-
mark, to establish the baseline performance for
tested agents.
(S2) L2 Interaction. The simulated user and agent
converse in L2, while tools, policies, databases,
and task contexts remain in English.
(S3) L2 Tool. Tool schemas are rewritten in L2
while conversation and domain context remain in
English. We present translated tool schemas in
two formats: single-L2, where each schema uses
one target language, and mixed-L2, where schemas
combine an increasing number of target languages.
(S4) L2 Domain. Agents operate in a complete
L2 setting where dialogue, tool schemas, policies,
task descriptions, and agent-visible database are
translated while task semantics remain fixed. This
scenario uses all translated artifacts and runtime
localization described in Section 3.2.
4 Experimental Setup
Metrics. Following τ 2-Bench (Barres et al.,
2025), we report two task metrics. First, pass@1
measures quality as the mean success rate of inde-
pendent trials reaching the expected final state, i.e.,
pass@1=pass1=E[r]. Building on existing metrics,
we also derive a robustness measure, ρ ∈ [0, 1],
defined as
ρq = passq
pass@1 (1)
where ρq denotes the probability that all q in-
dependent trials succeed, averaged across tasks.
ρq = 1 indicates that the agent performs consis-
tently across multiple runs, while lower ρq indi-
cates that the agent may solve a task once but fail
to do so reliably across repeated trials.
Task and Domain. For high-quality evaluation,
we use latest versions of retail and airline domains
(Cuadron et

d across tasks.
ρq = 1 indicates that the agent performs consis-
tently across multiple runs, while lower ρq indi-
cates that the agent may solve a task once but fail
to do so reliably across repeated trials.
Task and Domain. For high-quality evaluation,
we use latest versions of retail and airline domains
(Cuadron et al., 2026), which incorporate several
corrections and refinements over prior versions
(Yao et al., 2024; Barres et al., 2025). For the
telecom domain, we follow the original τ 2-Bench,
as its tasks were not updated in the most recent
release. We leave exploration of the banking do-
main (Shi et al., 2026) and the voice modality (Ray
et al., 2026) introduced in τ 3-bench to future work.
Hyperparameters and Models. For all evalua-
tions, we use q = 3, resulting in the robustness met-
ric ρ3, and Qwen3-235B-A22B-Inst3 (Qwen-Team,
2025) as the user LLM. For the natural-language
assertion judge, we use GPT-4.1, following the
implementations provided in τ 2-Bench. We evalu-
ate three recent LLM agents spanning three model
families: GPT-5-Mini4 (Singh et al., 2026) as a pro-
prietary model, Qwen3-235B-A22B-Instruct-2507
3https://huggingface.co/Qwen/Qwen3-235B-A22B-Inst-A22B-Instruct-2507
4https://developers.openai.com/api/docs/models/gpt-5-mini
4

-- 4 of 23 --

En Baseline
L2 Interaction
L2 Tools
L2 Domain
0
0.2
0.4
0.6
0.8
1
Score
TH (Thai)
En Baseline
L2 Interaction
L2 Tools
L2 Domain
0
0.2
0.4
0.6
0.8
1
Score
VI (Vietnamese)
En Baseline
L2 Interaction
L2 Tools
L2 Domain
0
0.2
0.4
0.6
0.8
1
Score
ID (Indonesian)
En Baseline
L2 Interaction
L2 Tools
L2 Domain
0
0.2
0.4
0.6
0.8
1
Score
ZH (Chinese)
En Baseline
L2 Interaction
L2 Tools
L2 Domain
0
0.2
0.4
0.6
0.8
1
Score
TL (Filipino)
pass@1 scenario avg pass@1 model avg ρ3 scenario avg ρ3 model avg
Figure 3: Evaluation results on SEATauBench. SEATauBench reveals consistent degradation as the setting becomes
less English-centric on all languages in both pass@1 and ρ3 metrics, with most severe drops in

Domain
0
0.2
0.4
0.6
0.8
1
Score
TL (Filipino)
pass@1 scenario avg pass@1 model avg ρ3 scenario avg ρ3 model avg
Figure 3: Evaluation results on SEATauBench. SEATauBench reveals consistent degradation as the setting becomes
less English-centric on all languages in both pass@1 and ρ3 metrics, with most severe drops in low-resource
languages with non-Latin scripts. All scenario results are reported with three models, except for L2 Tool, where
only GPT-5-mini and Qwen3-235B-A22B-Inst are reported.
as a representative open-source model, and Kimi-
K2.5 (Kimi Team et al., 2026) as a larger-scale
open-source model.
For each model, we use the default hyperparame-
ters defined in each provider (See Appendix C). We
avoid any model-specific hyperparameter tuning to
better reflect the out-of-the-box tool-agent-user in-
teraction capability when adopted to the specified
scenario and L2.
5 Results
We present two main results. First, we examine
how quality and robustness change as more com-
ponents of the English benchmark are converted to
L2 (Section Section 5.1). Second, we test whether
these trends are consistent across agent models and
task domains (Section Section 5.2). Together, the
results show that English-centric evaluations over-
estimate multilingual agent reliability, and that the
size of this gap depends on both the target language
and the structure of the task environment.
5.1 Performance Degrades in L2 Languages
Figure 3 demonstrates a consistent decline in model
performance as the evaluation setting becomes pro-
gressively less English-centric. Both pass@1 and
ρ3 are strongest in English settings, but perfor-
mance drops when the user–agent conversation is
conducted in an L2, drops further when tool inter-
faces are localized, and typically reaches its lowest
levels when the domain context is also translated.
This progression indicates that multilingual agentic
competence is not only a matter of understanding
non-English user utterances. Agents must also

conversation is
conducted in an L2, drops further when tool inter-
faces are localized, and typically reaches its lowest
levels when the domain context is also translated.
This progression indicates that multilingual agentic
competence is not only a matter of understanding
non-English user utterances. Agents must also use
localized tool schemas, entities, and ground deci-
sions in domain policies expressed in L2.
Although all L2 settings degrade relative to En-
glish, the severity and form of degradation differ
by language. In L2 Domain, Thai and Filipino
show the largest pass@1 declines, falling to roughly
the low-0.4 range. Vietnamese and Chinese re-
main closer to the mid-0.5 range, while Indone-
sian shows a partial robustness recovery despite
lower pass@1. The sharper drop in Thai may re-
flect challenges associated with a distinct writing
system and lower-resource coverage. However, the
comparable drop in Filipino shows that localiza-
tion failures are not limited to non-Latin scripts:
even Latin-script languages can expose weaknesses
when agents must reason over fully localized dia-
logue, tools, schemas, and domain context.
Quality and robustness diverge. Figure 3 also
shows that pass@1 and ρ3 do not always move to-
gether. In Indonesian, Filipino, Chinese, and to a
5

-- 5 of 23 --

EN
TH	VI
ID
ZH 	TL
20%
40%
60%
80%
pass@1
EN
TH	VI
ID
ZH 	TL
20%
40%
60%
80%
ρ3
GPT 5 Mini 	Kimi K2.5 	Qwen3 235B IT
Retail
Airline
Telecom
pass
@1
0% 	20% 	40% 	60% 	80% 	100%
Retail
Airline
Telecom
ρ3
Non-English mean 	English mean
Figure 4: Quality and robustness of varying agent models across languages and domains. (left) All agents yield
consistently higher pass@1 in English and Chinese indicating a quality bias toward high-resource languages. (center)
A similar trend is observed on ρ3, with negative correlation of ρ3 between English and L2 languages especially Thai.
(right) The trend of pass@1 holds across most domains and models with some exceptions as described in

her pass@1 in English and Chinese indicating a quality bias toward high-resource languages. (center)
A similar trend is observed on ρ3, with negative correlation of ρ3 between English and L2 languages especially Thai.
(right) The trend of pass@1 holds across most domains and models with some exceptions as described in §5.2.
lesser extent Vietnamese, pass@1 drops in L2 Do-
main while ρ3 remains comparatively stable or re-
bounds relative to L2 Tool. This means that agents
may solve fewer tasks overall, but when they do
solve a task, their successful behavior is relatively
reproducible in some L2 settings. Thai is the clear-
est exception: both pass@1 and ρ3 decline in L2
Domain. This suggests that some L2 conditions
weaken not only average task success but also con-
sistency across repeated trials.
5.2 Does the Trend Hold across Agents?
Overall, model performance differs across metrics,
as shown in Fig. 4. For pass@1, Kimi K2.5 per-
forms best in most languages, including English,
Vietnamese, Filipino, and Chinese, while GPT-5
Mini leads in Indonesian and Thai, indicating that
model advantages do not transfer uniformly across
SEA languages. Qwen3-235B-A22B-Inst consis-
tently has the lowest pass@1, suggesting weaker
single-run task completion. However, ρ3 reveals a
different pattern: despite its lower pass@1, Qwen3-
235B-A22B-Inst often shows stronger robustness
in lower-resource L2 settings. This robustness ad-
vantage weakens in higher-resource languages such
as English and Chinese, where Kimi K2.5 and GPT-
5 Mini remain competitive or stronger.
The domain-level results show that the English–
non-English gap varies across domains and models.
For GPT-5 Mini and Kimi K2.5, pass@1 gener-
ally follows Telecom > Airline > Retail in both
English and non-English settings, suggesting that
Retail is the most challenging domain due to its
policies, item-level constraints, and state updates.
Qwen3-235B-A22B-Inst Instruct shows a differ-
ent pattern: Retail performs best in

-5 Mini and Kimi K2.5, pass@1 gener-
ally follows Telecom > Airline > Retail in both
English and non-English settings, suggesting that
Retail is the most challenging domain due to its
policies, item-level constraints, and state updates.
Qwen3-235B-A22B-Inst Instruct shows a differ-
ent pattern: Retail performs best in English but
becomes weakest in non-English settings, indi-
cating that domain-specific strengths in English
may not transfer under localization. The ρ3 results
further show that localization reduces robustness,
with English settings generally achieving higher
consistency than non-English settings across do-
mains. However, this degradation is not uniform.
For Qwen3-235B-A22B-Inst, agent performance
follows the expected non-English decline in Tele-
com and Retail domains, while shows a relative
robustness spike for Airline. This suggests that
robustness under localization depends on both task
domain and the agent model behavior.
6 Analysis and Discussion
In this section, we examine error patterns arising
from conversational user–agent interactions in L2
Interaction and L2 Domain scenarios (Section 6.1)
and the effects of having tools in multiple lan-
guages (Section 6.2). Furthermore, we show that
agent performance has little to no association with
language correctness (Section 6.3) and that English
performance is an unreliable proxy for L2 perfor-
mance with SEA languages (Section 6.4).
6.1 Error Analysis
To further examine the downstream performance
metrics reported in the previous section, we con-
duct a qualitative error analysis based on the LLM-
as-a-Judge framework proposed in Shi et al. (2026).
Following Barres et al. (2025), we categorize task
simulation errors into two types: Critical error, in
which the agent deviates from the user’s intent or
commits irrecoverable mistakes, and Benign error,
which do not prevent successful task completion.
For early results, we analyze only scenarios that af-
6

-- 6 of 23 --

0% 20% 40% 60% 80%

2025), we categorize task
simulation errors into two types: Critical error, in
which the agent deviates from the user’s intent or
commits irrecoverable mistakes, and Benign error,
which do not prevent successful task completion.
For early results, we analyze only scenarios that af-
6

-- 6 of 23 --

0% 20% 40% 60% 80% 100%
EN
VI
ZH
ID
TL
TH
38.3%
L2 Interaction - Agent
0% 20% 40% 60% 80% 100%
21.0%
L2 Interaction - User
0% 20% 40% 60% 80% 100%
38.3%
L2 Domain - Agent
0% 20% 40% 60% 80% 100%
21.0%
L2 Domain - User
Critical 	Benign 	Correct
Figure 5: Error categorization for simulated user and agent, across SEA language in the L2 Interaction and L2
Domain scenarios. Both agents and users tend to have higher critical errors in L2 than English, while having a
similar benign error in comparison to English. The errors are reported by judge DeepSeek-V4-Flash, for simulated
user Qwen3-235B-A22B-Inst and agent GPT-5-mini.
fect both behavioral changes on both simulated user
and agent, i.e., L2 Interaction and L2 Domain sce-
narios, since L2 Tool only translates tool specifica-
tions while keeping all tool-agent-user interactions
in English. Specifically, we employ DeepSeek-V4-
Flash as the oracle LLM on simulations done by
GPT-5-mini as the agent and Qwen3-235B-A22B-
Inst as the simulated user.
L2 Interaction. Agent performance degrades
when interacting with users in L2. As shown in
Figure 5, critical errors appear in around 40% of
total simulations across SEA languages, with Thai
being the most severe (nearly 50%). Moreover,
when the user model must communicate only in
SEA languages, its behavior becomes less stable
and less consistent with the intended simulation
setup, with the most severe degradation in Filipino.
L2 Domain. Agent performance degradation is
more pronounced in the L2 domain adaptation set-
ting. There are slightly more critical errors for
agents here than L2 interaction, with clearest trends
in Thai, Filipino, and Indonesian (Figure 5). This
finding

simulation
setup, with the most severe degradation in Filipino.
L2 Domain. Agent performance degradation is
more pronounced in the L2 domain adaptation set-
ting. There are slightly more critical errors for
agents here than L2 interaction, with clearest trends
in Thai, Filipino, and Indonesian (Figure 5). This
finding suggests that operating in complete L2 en-
vironments further impairs the agent’s ability to
complete tasks effectively.
Simulated User. A similar trend is observed on
the simulated user, where critical errors consis-
tently account for around 20% of all errors in
1 	2 	3 	4 	5
No. languages for translated tool
0.0
0.2
0.4
0.6
0.8
Score
Performance when tool descriptions are multilingual
pass@1 ρ3
Figure 6: Pass@1 and ρ3 as the number of languages
used for translated tools increase from 1 (English) to 5.
Performance for two agents (GPT-5-Mini & Qwen3-
235B-A22B-Inst) slightly drops when a second lan-
guage is added, but it plateaus for more languages.
both analyzed scenarios. Our main finding that
agent performance drops as L2 adaptation increases
across four scenarios (Figure 3) might be compro-
mised by the low reliability of the simulated user.
This account previously appeared in Barres et al.
(2025) for English and Seshadri et al. (2026) for
dialectical English.
6.2 Mixed language tool causes modest drop
in task performance
Beyond adaptation using a single L2, we also in-
vestigate how agents handle tools in multiple lan-
guages, which is increasingly relevant as tools for
7

-- 7 of 23 --

0.5 0.6 0.7 0.8 0.9 1.0
Mean agent language correctness
0.0
0.2
0.4
0.6
0.8
1.0
pass
3
R²=0.014
Language correctness vs. pass3
En Baseline
L2 Tools
L2 Interaction
L2 Domain
Figure 7: Only 1.4% of the variation in task performance
metric pass3 is explained by language use capability of
tested agents, suggesting a very weak linear relationship.
Each dot represents a run for one scenario x domain x
language setting.
agents are developed in different world regions.

s
L2 Interaction
L2 Domain
Figure 7: Only 1.4% of the variation in task performance
metric pass3 is explained by language use capability of
tested agents, suggesting a very weak linear relationship.
Each dot represents a run for one scenario x domain x
language setting.
agents are developed in different world regions. We
do so by extending L2 Tool experiment and explore
five different mixed language settings by gradually
adding more languages to the mix for tool specifi-
cation: Mix-2 uses English and Thai, Mix-3 adds
Vietnamese, Mix-4 adds Indonesian, and Mix-5
adds Chinese (for more details, see Appendix B).
Figure 6 shows that mixed language tool causes
only a modest drop. Pass@1 drops from 0.68
for English-only average to 0.55 across all mixed-
language settings (0.13 decrease). ρ3 shows a sim-
ilar trend of degradation (0.11 decrease). These
results indicate that agents with strong tool-using
performance in English might use tools slightly less
effectively when tools are multilingual; however,
increasing the language diversity in tool specifica-
tion has little to no impact to the overall agentic
tool-using capability.
6.3 Language use is uncorrelated with
performance
We also measure whether the simulated user and
agent operated across tasks in the correct L2, or lan-
guage correctness. It is calculated as the fraction
of eligible turns whose detected language (by fast-
Text) matches the expected L2 for a single run. We
exclude trajectories with system errors and other
errors unrelated to task execution.
Figure 7 shows that run-level language correct-
ness for agent has little association with task per-
formance: a linear fit explains only R2 = 0.014 of
the variance in pass3. Several runs achieve near-
perfect language correctness while spanning a wide
range of pass3 scores, especially in L2 Tool and
L2 Domain. Conversely, L2 Interaction contains
runs with visibly lower language correctness but
still moderate or high task success. This suggests
that, at the

0.014 of
the variance in pass3. Several runs achieve near-
perfect language correctness while spanning a wide
range of pass3 scores, especially in L2 Tool and
L2 Domain. Conversely, L2 Interaction contains
runs with visibly lower language correctness but
still moderate or high task success. This suggests
that, at the aggregate run level, task failures are
not primarily explained by whether the agent re-
mains in the target language. For language drift,
we also measure how often the agent produces text
in the expected L2 (Figure 9), where in a conver-
sation off-target text appears (Figure 10), and how
much of the off-target text is specifically English
(Figure 11).
6.4 Is English a reliable proxy for L2 in SEA?
The τ 2-Bench evaluation framework, while effec-
tive for measuring agent capabilities, is resource-
intensive and computationally expensive. Extend-
ing to SEATauBenchwith an additional language
dimension adds to these costs. To address this chal-
lenge, we examine correlation between quality and
robustness metrics across SEATauBench languages
to determine whether performance in one language
could serve as a proxy for other languages.
As illustrated in Figure 8, agent performance
in English has high predictive power of L2 per-
formance for pass@1, with correlation above 0.9.
However, there is more variability in ρ3, where
the highest correlation is in Chinese (0.88) and
the lowest in Thai (0.49). Therefore, English per-
formance is not a reliable representation for SEA
language performance when we factor in consis-
tency. Filipino (TL), however, shows the strongest
correlation with other SEA languages, with the
lowest score being 0.91 for pass@1 and 0.85 for
ρ3. Based on these findings, we recommend using
the Filipino subset of SEATauBench as a proxy
to estimate performance across the remaining four
languages, for a more efficient yet representative
evaluations of AI agents in the SEA region.
7 Conclusion
SEATauBenchestablishes the first

being 0.91 for pass@1 and 0.85 for
ρ3. Based on these findings, we recommend using
the Filipino subset of SEATauBench as a proxy
to estimate performance across the remaining four
languages, for a more efficient yet representative
evaluations of AI agents in the SEA region.
7 Conclusion
SEATauBenchestablishes the first agent-focused
framework for SEA linguistic diversity, revealing
that English agentic capabilities transfer effectively
to a superficial L2 interaction but degrade signifi-
cantly with the inclusion of non-English contexts,
due to both agent limitations and reduced user ca-
pacity. By exposing fundamental gaps in English-
only evaluation methodologies as a proxy of agents
capability in SEA languages, SEATauBenchpro-
8

-- 8 of 23 --

EN	
TH	
VI	
ID	
ZH	
TL
EN
TH
VI
ID
ZH
TL
0.92 	0.90 	0.96 	0.90 	0.91
0.49 	0.91 	0.96 	0.94 	0.98
0.60 	0.89 	0.96 	0.98 	0.94
0.72 	0.87 	0.91 	0.95 	0.95
0.88 	0.72 	0.82 	0.85 	0.99
0.63 	0.88 	0.93 	0.94 	0.85
0.0
0.2
0.4
0.6
0.8
1.0
pass@1 corr (upper)
0.0 	0.2 	0.4 	0.6 	0.8 	1.0
ρ3 corr (lower)
Figure 8: Correlation of pass@1 (upper triangule) and
ρ3 (lower triangle) across English and SEATauBench
languages. Correlation scores are aggregated by four
scenarios, three agent models, and three task domains.
vides essential diagnostic tools for sovereign AI
development, establishing a rigorous foundation
for future research in sovereign agentic AI solu-
tions, ultimately supporting autonomous AI across
SEA diverse linguistic communities.
8 Limitation
This work presents a systematic framework for ex-
tending agentic benchmarks to evaluate multilin-
gual conversational capabilities with tool access
in Southeast Asian (SEA) languages, specifically
Vietnamese, Indonesian, Thai, Filipino, and Chi-
nese, across the retail, airline, and telecom domains.
Although the proposed framework can, in principle,
be extended to additional languages, the empirical
findings and their implications are confined to the
SEA language setting

east Asian (SEA) languages, specifically
Vietnamese, Indonesian, Thai, Filipino, and Chi-
nese, across the retail, airline, and telecom domains.
Although the proposed framework can, in principle,
be extended to additional languages, the empirical
findings and their implications are confined to the
SEA language setting explored in this study.
Another limitation is the benchmark adapta-
tion methodology, which is specifically designed
around the τ -Bench evaluation paradigm (Yao et al.,
2024; Barres et al., 2025). While the proposed eval-
uation scenarios may be applicable more broadly,
the underlying design constraints may limit the gen-
eralizability of the methodology beyond τ -Bench
frameworks. In addition, the current analysis is
limited to English and five Southeast Asian lan-
guages, consequently, the observed findings may
not generalize to other SEA languages that are not
covered in this work.
9 Ethical Statement
We conducted this research with careful attention
to ethical considerations throughout the develop-
ment process. In developing of the translation
pipeline and the evaluation framework, we ensure
that there is no external sensitive information is
exposed throughout the process and the resulting
artifacts. We recognize our responsibility to en-
sure these systems are fair, respectful of different
cultures, and beneficial to all users. Our approach
focuses on being transparent about what the agents
can and cannot do, actively working to identify and
address potential biases, and taking steps to prevent
misuse. We handle all research data responsibly,
protecting privacy while respecting linguistic di-
versity. This work aims to contribute to AI that
serves people equitably across languages and cul-
tures, and we are committed to being accountable
for how these technologies impact society.
Acknowledgment
This work was supported by the ThaiLLM collabo-
ration, funded by the Digital Economy and Society
(DE) Development Fund of the Ministry of

o contribute to AI that
serves people equitably across languages and cul-
tures, and we are committed to being accountable
for how these technologies impact society.
Acknowledgment
This work was supported by the ThaiLLM collabo-
ration, funded by the Digital Economy and Society
(DE) Development Fund of the Ministry of Digital
Economy and Society, Thailand. We thank Kian
Kyars for providing OpenAI API access for our
experiments, and SCBX R&D for providing the
necessary Google Cloud Vertex AI resources.
References
Hilda Barasa, PeiChin Tay, Keegan McBride, Alexander
Iosad, and Jakob Mökander. 2026. Sovereignty in
the Age of AI: Strategic Choices, Structural Depen-
dencies. Accessed: 22 May 2026.
Victor Barres, Honghua Dong, Soham Ray, Xujie Si,
and Karthik Narasimhan. 2025. τ 2-bench: Evaluat-
ing conversational agents in a dual-control environ-
ment. Preprint, arXiv:2506.07982.
Manik Bhandari and Gaurav Modi. 2026. Why
sovereign artificial intelligence is imperative in south-
east asia. Accessed: 22 May 2026.
Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo
Caccia, Thibault Le Sellier De Chezelles, Quentin
Cappart, Nicolas Chapados, Alexandre Lacoste, and
Alexandre Drouin. 2024. Workarena++: Towards
compositional planning and reasoning-based com-
mon knowledge work tasks. In Advances in Neural
Information Processing Systems, volume 37, pages
5996–6051. Curran Associates, Inc.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang
Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ra-
9

-- 9 of 23 --

madan, and Milica Gaši´c. 2018. MultiWOZ - a large-
scale multi-domain Wizard-of-Oz dataset for task-
oriented dialogue modelling. In Proceedings of the
2018 Conference on Empirical Methods in Natural
Language Processing, pages 5016–5026, Brussels,
Belgium. Association for Computational Linguistics.
Samuel Cahyawijaya, Peerat Limkonchotiwat,
Tack Hwa Wong, Hitesh Laxmichand Patel, Amit
Agarwal, Manuel Antonio Rufino, Carlos Rafael
Catalan, Muhammad Reza Qorib, Vicky Feliren,
Holy

nce on Empirical Methods in Natural
Language Processing, pages 5016–5026, Brussels,
Belgium. Association for Computational Linguistics.
Samuel Cahyawijaya, Peerat Limkonchotiwat,
Tack Hwa Wong, Hitesh Laxmichand Patel, Amit
Agarwal, Manuel Antonio Rufino, Carlos Rafael
Catalan, Muhammad Reza Qorib, Vicky Feliren,
Holy Lovenia, Aye Hninn Khine, Frederikus Hudi,
David Anugraha, Alham Fikri Aji, Romrawin
Chumpu, Viet-Thanh Pham, Minghan Wang,
Mohamed Fazli Imam, Ruochen Zhang, and 29
others. 2026. Anthropogenic regional adaptation
in multimodal vision-language model. Preprint,
arXiv:2604.11490.
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji,
Genta Winata, Bryan Wilie, Fajri Koto, Rahmad
Mahendra, Christian Wibisono, Ade Romadhony,
Karissa Vincentio, Jennifer Santoso, David Moel-
jadi, Cahya Wirawan, Frederikus Hudi, Muham-
mad Satrio Wicaksono, Ivan Parmonangan, Ika Al-
fina, Ilham Firdausi Putra, Samsul Rahmadani, and
29 others. 2023a. NusaCrowd: Open source initiative
for Indonesian NLP resources. In Findings of the As-
sociation for Computational Linguistics: ACL 2023,
pages 13745–13818, Toronto, Canada. Association
for Computational Linguistics.
Samuel Cahyawijaya, Holy Lovenia, Fajri Koto, Dea
Adhista, Emmanuel Dave, Sarah Oktavianti, Salsabil
Akbar, Jhonson Lee, Nuur Shadieq, Tjeng Wawan
Cenggoro, Hanung Linuwih, Bryan Wilie, Galih
Muridan, Genta Winata, David Moeljadi, Al-
ham Fikri Aji, Ayu Purwarianti, and Pascale Fung.
2023b. NusaWrites: Constructing high-quality
corpora for underrepresented and extremely low-
resource languages. In Proceedings of the 13th In-
ternational Joint Conference on Natural Language
Processing and the 3rd Conference of the Asia-Pacific
Chapter of the Association for Computational Lin-
guistics (Volume 1: Long Papers), pages 921–945,
Nusa Dua, Bali. Association for Computational Lin-
guistics.
Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony
Moniz, Tack Hwa Wong, Mohammad Rifqi Farhan-
syah, Thant Thiri Maung, Frederikus Hudi,

of the Asia-Pacific
Chapter of the Association for Computational Lin-
guistics (Volume 1: Long Papers), pages 921–945,
Nusa Dua, Bali. Association for Computational Lin-
guistics.
Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony
Moniz, Tack Hwa Wong, Mohammad Rifqi Farhan-
syah, Thant Thiri Maung, Frederikus Hudi, David
Anugraha, Muhammad Ravi Shulthan Habibi,
Muhammad Reza Qorib, Amit Agarwal, Joseph Mar-
vin Imperial, Hitesh Laxmichand Patel, Vicky Fe-
liren, Bahrul Ilmi Nasution, Manuel Antonio Rufino,
Genta Indra Winata, Rian Adam Rajagede, Car-
los Rafael Catalan, and 73 others. 2025a. Crowd-
source, crawl, or generate? creating SEA-VL, a mul-
ticultural vision-language dataset for Southeast Asia.
In Proceedings of the 63rd Annual Meeting of the
Association for Computational Linguistics (Volume 1:
Long Papers), pages 18685–18717, Vienna, Austria.
Association for Computational Linguistics.
Samuel Cahyawijaya, Ruochen Zhang, Jan Chris-
tian Blaise Cruz, Holy Lovenia, Elisa Gilbert, Hi-
roki Nomoto, and Alham Fikri Aji. 2025b. Thank
you, stingray: Multilingual large language models
can not (yet) disambiguate cross-lingual word senses.
In Findings of the Association for Computational
Linguistics: NAACL 2025, pages 3228–3250, Al-
buquerque, New Mexico. Association for Computa-
tional Linguistics.
Kyubyung Chae, Gihoon Kim, Gyuseong Lee, Taesup
Kim, Jaejin Lee, and Heejin Kim. 2025. Assess-
ing socio-cultural alignment and technical safety of
sovereign LLMs. In Findings of the Association
for Computational Linguistics: EMNLP 2025, pages
10579–10600, Suzhou, China. Association for Com-
putational Linguistics.
CohereLabs. 2025. AyaVisionBench. https:
//huggingface.co/datasets/CohereLabs/
AyaVisionBench. Hugging Face dataset.
Alejandro Cuadron, Pengfei Yu, Yang Liu, and Arpit
Gupta. 2026. SABER: Small actions, big errors —
safeguarding mutating steps in LLM agents. In ICLR
2026 Workshop on Memory for LLM-Based Agentic
Systems.
Alexandre Drouin, Maxime Gasse,

https:
//huggingface.co/datasets/CohereLabs/
AyaVisionBench. Hugging Face dataset.
Alejandro Cuadron, Pengfei Yu, Yang Liu, and Arpit
Gupta. 2026. SABER: Small actions, big errors —
safeguarding mutating steps in LLM agents. In ICLR
2026 Workshop on Memory for LLM-Based Agentic
Systems.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Is-
sam H. Laradji, Manuel Del Verme, Tom Marty, Léo
Boisvert, Megh Thakkar, Quentin Cappart, David
Vazquez, Nicolas Chapados, and Alexandre Lacoste.
2024. Workarena: How capable are web agents at
solving common knowledge work tasks? Preprint,
arXiv:2403.07718.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi,
Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj
Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. Mul-
tiWOZ 2.1: A consolidated multi-domain dialogue
dataset with state corrections and state tracking base-
lines. In Proceedings of the Twelfth Language Re-
sources and Evaluation Conference, pages 422–428,
Marseille, France. European Language Resources
Association.
Jack FitzGerald, Christopher Hench, Charith Peris,
Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron
Nash, Liam Urbach, Vishesh Kakarala, Richa Singh,
Swetha Ranganath, Laurie Crist, Misha Britan,
Wouter Leeuwis, Gokhan Tur, and Prem Natara-
jan. 2023. MASSIVE: A 1M-example multilin-
gual natural language understanding dataset with
51 typologically-diverse languages. In Proceedings
of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
pages 4277–4302, Toronto, Canada. Association for
Computational Linguistics.
Ting Han, Ximing Liu, Ryuichi Takanabu, Yixin Lian,
Chongxuan Huang, Dazhen Wan, Wei Peng, and Min-
lie Huang. 2021. Multiwoz 2.3: A multi-domain task-
oriented dialogue dataset enhanced with annotation
corrections and co-reference annotation. In Natural
Language Processing and Chinese Computing, pages
206–218, Cham. Springer International Publishing.
10

-- 10 of 23 --

Dan Hendrycks, Collin Burns, Steven Basart, Andy

Huang. 2021. Multiwoz 2.3: A multi-domain task-
oriented dialogue dataset enhanced with annotation
corrections and co-reference annotation. In Natural
Language Processing and Chinese Computing, pages
206–218, Cham. Springer International Publishing.
10

-- 10 of 23 --

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou,
Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
2021. Measuring massive multitask language under-
standing. In International Conference on Learning
Representations.
Kung-Hsiang Huang, Akshara Prabhakar, Sidharth
Dhawan, Yixin Mao, Huan Wang, Silvio Savarese,
Caiming Xiong, Philippe Laban, and Chien-Sheng
Wu. 2025. CRMArena: Understanding the capacity
of LLM agents to perform professional CRM tasks
in realistic environments. In Proceedings of the 2025
Conference of the Nations of the Americas Chap-
ter of the Association for Computational Linguistics:
Human Language Technologies (Volume 1: Long Pa-
pers), pages 3830–3850, Albuquerque, New Mexico.
Association for Computational Linguistics.
Muhammad Kautsar, Rahmah Nurdini, Samuel Cahyaw-
ijaya, Genta Winata, and Ayu Purwarianti. 2023. In-
doToD: A multi-domain Indonesian benchmark for
end-to-end task-oriented dialogue systems. In Pro-
ceedings of the First Workshop in South East Asian
Language Processing, pages 85–99, Nusa Dua, Bali,
Indonesia. Association for Computational Linguis-
tics.
Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao,
S. H. Cai, Yuan Cao, Y. Charles, H. S. Che, Cheng
Chen, Guanduo Chen, Huarong Chen, Jia Chen, Ji-
ahao Chen, Jianlong Chen, Jun Chen, Kefan Chen,
Liang Chen, Ruijue Chen, Xinhao Chen, and 307
others. 2026. Kimi k2.5: Visual agentic intelligence.
Preprint, arXiv:2602.02276.
Mayank Kulkarni, Vittorio Mazzia, Judith Gaspers,
Chris Hench, and Jack FitzGerald. 2025. MASSIVE-
agents: A benchmark for multilingual function-
calling in 52 languages. In Findings of the Associa-
tion for Computational Linguistics: EMNLP 2025,
pages 20193–20215.
Chaoqun Liu, Wenxuan Zhang,

eprint, arXiv:2602.02276.
Mayank Kulkarni, Vittorio Mazzia, Judith Gaspers,
Chris Hench, and Jack FitzGerald. 2025. MASSIVE-
agents: A benchmark for multilingual function-
calling in 52 languages. In Findings of the Associa-
tion for Computational Linguistics: EMNLP 2025,
pages 20193–20215.
Chaoqun Liu, Wenxuan Zhang, Jiahao Ying, Mahani
Aljunied, Anh Tuan Luu, and Lidong Bing. 2025.
SeaExam and SeaBench: Benchmarking LLMs with
local multilingual questions in Southeast Asia. In
Findings of the Association for Computational Lin-
guistics: NAACL 2025, pages 6134–6151, Albu-
querque, New Mexico. Association for Computa-
tional Linguistics.
Holy Lovenia, Rahmad Mahendra, Salsabil Maulana
Akbar, Lester James V. Miranda, Jennifer San-
toso, Elyanah Aco, Akhdan Fadhilah, Jonibek
Mansurov, Joseph Marvin Imperial, Onno P. Kamp-
man, Joel Ruben Antony Moniz, Muhammad
Ravi Shulthan Habibi, Frederikus Hudi, Railey Mon-
talan, Ryan Ignatius, Joanito Agili Lopo, William
Nixon, Börje F. Karlsson, James Jaya, and 42 others.
2024. SEACrowd: A multilingual multimodal data
hub and benchmark suite for Southeast Asian lan-
guages. In Proceedings of the 2024 Conference on
Empirical Methods in Natural Language Processing,
pages 5155–5203, Miami, Florida, USA. Association
for Computational Linguistics.
Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota,
Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin
Lin, Bhavani Sai Praneeth Varma Mantina, Ananda
Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yu-
sofi, Fei Yuan, and Jörg Tiedemann. 2025. GlotEval:
A test suite for massively multilingual evaluation of
large language models. In Proceedings of the 2025
Conference on Empirical Methods in Natural Lan-
guage Processing: System Demonstrations, pages
602–614, Suzhou, China. Association for Computa-
tional Linguistics.
Rashid Mushkani, Hugo Berard, Allison Cohen, and
Shin Koseki. 2025. Position: The right to AI. In
Forty-second International Conference on Machine
Learning Position Paper

mpirical Methods in Natural Lan-
guage Processing: System Demonstrations, pages
602–614, Suzhou, China. Association for Computa-
tional Linguistics.
Rashid Mushkani, Hugo Berard, Allison Cohen, and
Shin Koseki. 2025. Position: The right to AI. In
Forty-second International Conference on Machine
Learning Position Paper Track.
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani
Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken
Chia, Xingxuan Li, Qingyu Tan Jianyu Wang, Liy-
ing Cheng, Guanzheng Chen, Yue Deng, Sen Yang,
Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024.
Seallms - large language models for southeast asia.
OpenAI. 2024. MMMLU: Multilingual massive
multitask language understanding. https://
huggingface.co/datasets/openai/MMMLU. Hug-
ging Face dataset.
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji,
Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E.
Gonzalez. 2025. The berkeley function calling leader-
board (BFCL): From tool use to agentic evaluation of
large language models. In Proceedings of the 42nd
International Conference on Machine Learning.
Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace
Kim, Michele Wang, Olivia Watkins, Simon Posada
Fishman, Marwan Aljubeh, Phoebe Thacker, Lau-
rance Fauconnet, Natalie S. Kim, Samuel Mis-
erendino, Gildas Chabot, David Li, Patrick Chao,
Michael Sharman, Alexandra Barr, Amelia Glaese,
and Jerry Tworek. 2026. GDPval: Evaluating AI
model performance on real-world economically valu-
able tasks. In The Fourteenth International Confer-
ence on Learning Representations.
Ayu Purwarianti, Dea Adhista, Agung Baptiso, Mif-
tahul Mahfuzh, Yusrina Sabila, Aulia Adila, Samuel
Cahyawijaya, and Alham Fikri Aji. 2025. NusaDia-
logue: Dialogue summarization and generation for
underrepresented and extremely low-resource lan-
guages. In Proceedings of the Second Workshop in
South East Asian Language Processing, pages 82–
100, Online. Association for Computational Linguis-
tics.
Bama Andika Putra. 2024. Governing ai in

Fikri Aji. 2025. NusaDia-
logue: Dialogue summarization and generation for
underrepresented and extremely low-resource lan-
guages. In Proceedings of the Second Workshop in
South East Asian Language Processing, pages 82–
100, Online. Association for Computational Linguis-
tics.
Bama Andika Putra. 2024. Governing ai in southeast
asia: Asean’s way forward. Frontiers in Artificial
Intelligence, Volume 7 - 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan
Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang,
Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian,
Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li,
11

-- 11 of 23 --

Zhiyuan Liu, and Maosong Sun. 2024. Toolllm: Fa-
cilitating large language models to master 16000+
real-world apis. In International Conference on
Learning Representations, volume 2024, pages 9695–
9717.
Qwen-Team. 2025. Qwen3 technical report. Preprint,
arXiv:2505.09388.
Soham Ray, Keshav Dhandhania, Victor Barres, and
Karthik Narasimhan. 2026. τ -voice: Benchmark-
ing full-duplex voice agents on real-world domains.
Preprint, arXiv:2603.13686.
David Romero, Chenyang Lyu, Haryo Akbarianto Wi-
bowo, Teresa Lynn, Injy Hamed, Aditya Nanda
Kishore, Aishik Mandal, Alina Dragonetti, Artem
Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha,
Chenxi Whitehouse, Christian Salamea, Dan John
Velasco, David Ifeoluwa Adelani, David Le Meur,
Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, and
56 others. 2024. Cvqa: Culturally-diverse multilin-
gual visual question answering benchmark. Preprint,
arXiv:2406.05967.
Preethi Seshadri, Samuel Cahyawijaya, Ayomide Odu-
makinde, Sameer Singh, and Seraphina Goldfarb-
Tarrant. 2026. Lost in simulation: Llm-simulated
users are unreliable proxies for human users in agen-
tic evaluations. Preprint, arXiv:2601.17087.
Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik
Narasimhan, and Victor Barres. 2026. τ -knowledge:
Evaluating conversational agents over unstructured
knowledge. Preprint, arXiv:2603.04370.
Aaditya Singh, Adam

simulated
users are unreliable proxies for human users in agen-
tic evaluations. Preprint, arXiv:2601.17087.
Quan Shi, Alexandra Zytek, Pedram Razavi, Karthik
Narasimhan, and Victor Barres. 2026. τ -knowledge:
Evaluating conversational agents over unstructured
knowledge. Preprint, arXiv:2603.04370.
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart,
Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin,
Aiden Low, AJ Ostrow, Akhila Ananthram, Akshay
Nathan, Alan Luo, Alec Helyar, Aleksander Madry,
Aleksandr Efremov, Aleksandra Spyra, Alex Baker-
Whitcomb, Alex Beutel, Alex Karpenko, and 467
others. 2026. Openai gpt-5 system card. Preprint,
arXiv:2601.03267.
Shivalika Singh, Angelika Romanou, Clémentine Four-
rier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel
Vila-Suero, Peerat Limkonchotiwat, Kelly Marchi-
sio, Wei Qi Leong, Yosephine Susanto, Raymond
Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin
Ko, Antoine Bosselut, Alice Oh, Andre Martins,
Leshem Choshen, Daphne Ippolito, and 4 others.
2025. Global MMLU: Understanding and addressing
cultural and linguistic biases in multilingual evalua-
tion. In Proceedings of the 63rd Annual Meeting of
the Association for Computational Linguistics (Vol-
ume 1: Long Papers), pages 18761–18799, Vienna,
Austria. Association for Computational Linguistics.
Yosephine Susanto, Adithya Venkatadri Hulagadri,
Jann Railey Montalan, Jian Gang Ngui, Xian-
bin Yong, Wei Qi Leong, Hamsawardhini Ren-
garajan, Peerat Limkonchotiwat, Yifan Mai, and
William Chandra Tjhi. 2025. SEA-HELM: South-
east Asian holistic evaluation of language models. In
Findings of the Association for Computational Lin-
guistics: ACL 2025, pages 12308–12336, Vienna,
Austria. Association for Computational Linguistics.
Norawit Urailertprasert, Peerat Limkonchotiwat, Su-
pasorn Suwajanakorn, and Sarana Nutanong. 2024.
SEA-VQA: Southeast Asian cultural context dataset
for visual question answering. In Proceedings of the
3rd Workshop on Advances in Language and Vision
Research

Vienna,
Austria. Association for Computational Linguistics.
Norawit Urailertprasert, Peerat Limkonchotiwat, Su-
pasorn Suwajanakorn, and Sarana Nutanong. 2024.
SEA-VQA: Southeast Asian cultural context dataset
for visual question answering. In Proceedings of the
3rd Workshop on Advances in Language and Vision
Research (ALVR), pages 173–185, Bangkok, Thai-
land. Association for Computational Linguistics.
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawi-
jaya, Rahmad Mahendra, Fajri Koto, Ade Romad-
hony, Kemal Kurniawan, David Moeljadi, Radi-
tyo Eko Prasojo, Pascale Fung, Timothy Baldwin,
Jey Han Lau, Rico Sennrich, and Sebastian Ruder.
2023. NusaX: Multilingual parallel sentiment dataset
for 10 Indonesian local languages. In Proceedings
of the 17th Conference of the European Chapter of
the Association for Computational Linguistics, pages
815–834, Dubrovnik, Croatia. Association for Com-
putational Linguistics.
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kri-
tanjali Jain, Mengxue Bao, Zora Zhiruo Wang, Xuhui
Zhou, Zhitong Guo, Murong Cao, Mingyang Yang,
Hao Yang Lu, Amaad Martin, Zhe Su, Leander Mel-
roy Maben, Raj Mehta, Wayne Chi, Lawrence Ke-
unho Jang, Yiqing Xie, and 2 others. 2026. Theagent-
company: Benchmarking LLM agents on consequen-
tial real world tasks. In The Thirty-ninth Annual Con-
ference on Neural Information Processing Systems
Datasets and Benchmarks Track.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik
Narasimhan. 2024. τ -bench: A benchmark for
tool-agent-user interaction in real-world domains.
Preprint, arXiv:2406.12045.
Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz.
2022. MultiWOZ 2.4: A multi-domain task-oriented
dialogue dataset with essential annotation corrections
to improve state tracking evaluation. In Proceedings
of the 23rd Annual Meeting of the Special Interest
Group on Discourse and Dialogue, pages 351–360,
Edinburgh, UK. Association for Computational Lin-
guistics.
Xiaoxue Zang, Abhinav Rastogi, Srinivas

ask-oriented
dialogue dataset with essential annotation corrections
to improve state tracking evaluation. In Proceedings
of the 23rd Annual Meeting of the Special Interest
Group on Discourse and Dialogue, pages 351–360,
Edinburgh, UK. Association for Computational Lin-
guistics.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara,
Raghav Gupta, Jianguo Zhang, and Jindong Chen.
2020. MultiWOZ 2.2 : A dialogue dataset with
additional annotation corrections and state tracking
baselines. In Proceedings of the 2nd Workshop on
Natural Language Processing for Conversational AI,
pages 109–117, Online. Association for Computa-
tional Linguistics.
Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani
Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng,
Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, Xin Li, and
Lidong Bing. 2024. Seallms 3: Open foundation and
chat multilingual large language models for southeast
asian languages.
12

-- 12 of 23 --

A Machine Translation Pipeline Details
A.1 τ 2-BenchContext
Each τ 2-Benchdomain (airline, retail, telecom)
exposes the agent to two coupled surfaces.
First, data/tau2/domains/{domain}/ contains
the benchmark content seen during an inter-
action: task definitions in tasks.json, do-
main policies and workflow documents in mark-
down files *.md, and structured databases in
db.json, db.toml, or user-side DB files. Sec-
ond, src/tau2/domains/{domain}/ contains the
executable tool interface: toolkits in tools.py
and user_tools.py, schema definitions in
data_model.py and user_data_model.py, and
in telecom, tool return string templates.
At runtime, the agent must read policy, reason
about tasks, call tools, inspect tool outputs, and
communicate with a simulated user entirely in the
target language.
A.2 Translation Principles
Adapting τ 2-Bench to non-English languages re-
quires translating the natural-language content
seen by the language model while leaving un-
touched the runtime-canonical tokens consumed
by the tool execution layer: enum literals,

nicate with a simulated user entirely in the
target language.
A.2 Translation Principles
Adapting τ 2-Bench to non-English languages re-
quires translating the natural-language content
seen by the language model while leaving un-
touched the runtime-canonical tokens consumed
by the tool execution layer: enum literals, entity
identifiers, status codes, and tool argument names.
Conflating these two layers would corrupt evalu-
ation: a translated product name that reaches the
execution engine might trigger a lookup failure,
while an untranslated persona description breaks
the monolingual interaction we intend to test. We
therefore build the pipeline around four principles:
Runtime invariance. Canonical tokens remain
English at execution time. Only the LLM-visible
surface is localized.
Terminological consistency. A canonical value
should surface with the same target-language real-
ization everywhere it appears.
Format fidelity. Translation is applied only to
natural-language leaves. Structural keys, numeric
fields, booleans, and program identifiers are never
modified.
Bidirectional transparency. Localized values can
be mapped back to canonical English forms before
execution and before metric computation.
The implementation has two phases: an of-
fline translation phase that materializes language-
specific assets, and a runtime localization phase
that patches the environment presented to the agent.
A.3 Phase 1. Offline Translation
The offline phase consumes a domain’s
static artifacts, translate with Gemini 3.1
Flash Lite, and writes translated outputs to
data/tau2/domains/{domain}/{lang_id}/.
Step 1. Artifact discovery For each se-
lected domain, the pipeline discovers files
from data/tau2/domains/{domain}/ and
src/tau2/domains/{domain}/, then assigns
each file a processing kind (markdown, json, toml,
or python). Refer to Table 2 for the list of artifacts
and their role in translation.
Step 2. Segment extraction with path-sensitive
rules. Each file is reduced to

e pipeline discovers files
from data/tau2/domains/{domain}/ and
src/tau2/domains/{domain}/, then assigns
each file a processing kind (markdown, json, toml,
or python). Refer to Table 2 for the list of artifacts
and their role in translation.
Step 2. Segment extraction with path-sensitive
rules. Each file is reduced to minimal translat-
able segments together with metadata describing
its source path and type.
• Markdown files yield a single full-document
segment.
• Task JSON is walked recursively, but only a
curated allowlist of paths is translated, includ-
ing persona fields, task instructions, reason-
for-call, natural-language assertions, and user-
or assistant-visible message history.
• Database JSON/TOML is translated only at
conservative natural-language leaf keys such
as name, title, description, summary, and
notes. Domain-specific additions may ex-
tend this set: the airline domain also translates
address1, address2, and city (user profile
address text that is natural language and safe
to localize).
• Tool Python files are parsed with ast;
only docstrings attached to @is_tool or
@is_discoverable_tool methods are ex-
tracted.
• Google-style tool docstrings are decomposed
into short description, long description, pa-
rameter descriptions, returns text, and raises
text so that each part can be translated inde-
pendently and later reassembled.
• Schema Python files are converted into JSON
artifacts whose translatable fields include
class descriptions, field descriptions, enum
values, and Literal[...] alternatives.
• Tool return files expose both exact response
strings and template strings as translatable seg-
ments.
13

-- 13 of 23 --

Artifact Class Source Files Translatable Content Runtime Alignment Purpose
Task setup tasks.json Persona text, instructions,
reason-for-call, history
Defines user personas and
evaluation-facing assertions.
Policy & workflow *.md 	Full document prose Governs the agent’s allowed be-
havior and procedures.
Tool descriptions

ifact Class Source Files Translatable Content Runtime Alignment Purpose
Task setup tasks.json Persona text, instructions,
reason-for-call, history
Defines user personas and
evaluation-facing assertions.
Policy & workflow *.md 	Full document prose Governs the agent’s allowed be-
havior and procedures.
Tool descriptions tools.py,
user_tools.py
Docstrings of tool-
decorated methods
Conditions how the agent inter-
prets tool semantics.
Schemas & literals data_model.py,
user_data_model.py
Descriptions, enum values,
and Literal[...]
Defines choices appearing in dy-
namic tool schemas.
Database content db.json, db.toml Natural-language leaves
(e.g., names, notes)
Supplies textual fields surfaced
through tool results.
Tool return messages tool_returns.json Exact messages and pa-
rameterized templates
Ensures dynamic strings returned
by tools remain localized.
Table 2: Artifact classes processed during offline translation and their role in maintaining runtime evaluation
alignment.
Step 3. Canonical-token masking. Before trans-
lation, the pipeline collects and masks strings that
must remain canonical: IDs such as order_* or
booking_*, status values, structural task markers,
tool names, and docstring section headers such as
Args or Returns. Protection is partly global and
partly contextual. For example, airline literals such
as basic_economy or round_trip are protected
when they occur in cabin-class or trip-type contexts,
but not when similar surface strings appear as ordi-
nary prose. The masking layer replaces protected
strings with opaque placeholders and restores them
after translation.
Step 4. Schema-first literal translation.
Schema enum labels and Literal[...] values
are translated first in a dedicated literal mode. This
produces language-specific schema artifacts from
which the pipeline builds a domain literal map,
including alias forms such as underscore, space-
separated, or hyphenated variants of the same
canonical value. This step fixes the localized

s and Literal[...] values
are translated first in a dedicated literal mode. This
produces language-specific schema artifacts from
which the pipeline builds a domain literal map,
including alias forms such as underscore, space-
separated, or hyphenated variants of the same
canonical value. This step fixes the localized termi-
nology before any longer-form prose is translated.
Step 5. Standard translation with glossary
injection. All remaining segments, including
task prose, policy text, tool docstrings, database
leaves, and tool-return templates, are translated in
standard mode. The model receives the schema-
derived literal map as a glossary, and segments are
pre-masked so that localized forms are restored
consistently. Requests are deduplicated when
possible, batched as structured JSON, and exe-
cuted concurrently through LiteLLM. The current
implementation requires the exact Vertex route
vertex_ai/gemini-3.1-flash-lite-preview.
Batch failures can be retried, recursively split, or
rerun individually if placeholder restoration fails.
Step 6. Format-aware writing and man-
ifest recording. The translated text is writ-
ten back using format-specific writers: mark-
down is emitted directly; JSON/TOML files are
patched only at extracted addresses; tool docstrings
are reconstructed and saved as tools.json or
user_tools.json; schema artifacts are written as
data_model.json and user_data_model.json;
and DB files with no translated leaves are still
copied through so the translated directory remains
complete. Each language directory also receives
translation_manifest.json, which records the
output file, component, source language, target lan-
guage, model, translation timestamp, and SHA-256
fingerprints of the source files. This manifest en-
ables later staleness checks without forcing retrans-
lation.
A.4 Phase 2. Runtime Localization
The offline artifacts cover static content, but the
agent also sees dynamic tool schemas and tool out-
puts at inference time. These are

translation timestamp, and SHA-256
fingerprints of the source files. This manifest en-
ables later staleness checks without forcing retrans-
lation.
A.4 Phase 2. Runtime Localization
The offline artifacts cover static content, but the
agent also sees dynamic tool schemas and tool out-
puts at inference time. These are handled by a run-
time localization layer built from the paired source
and translated schema artifacts.
Step 7. Build runtime localization maps.
A SchemaRuntimeLocalizer constructs four re-
sources from the source and localized schema arti-
facts: a description map, a canonical-to-localized
literal map, a localized-to-canonical inverse map,
and optional maps for exact and templated tool-
return localizations.
Step 8. Localize the tool schema shown to the
agent. The environment’s get_tools() path is
14

-- 14 of 23 --

wrapped so that the agent receives localized tool
schemas: descriptions are translated, enum choices
are shown in the target language, and default or ex-
ample literal values are localized where appropriate.
The tool implementation itself is unchanged.
Step 9. Normalize localized arguments and local-
ize tool outputs. Before tool execution, localized
enum arguments supplied by the agent are mapped
back to their canonical English values. After execu-
tion, structured response payloads and tool-return
messages are localized back into the target lan-
guage so the interaction remains monolingual from
the agent’s perspective.
Step 10. Canonicalize localized values for evalu-
ation. Prior to metric computation, localized pay-
loads are canonicalized back to English, making
pass-rate comparisons directly comparable across
languages and against the original English bench-
mark.
A.5 Translated artifact statistics
Tables 3, 4, 5, and 6 provide summary statistics for
translated artifacts resulting from offline translation
phase (Appendix A.3).
Domain Files / Language Total Translated
Airline 5 25
Retail 5 25
Telecom 13 65
Total 23 115
Table 3: No.

s and against the original English bench-
mark.
A.5 Translated artifact statistics
Tables 3, 4, 5, and 6 provide summary statistics for
translated artifacts resulting from offline translation
phase (Appendix A.3).
Domain Files / Language Total Translated
Airline 5 25
Retail 5 25
Telecom 13 65
Total 23 115
Table 3: No. static artifact files translated per domain.
Each domain is translated into 5 target languages.
Domain Tasks Docstrings Policies Return Msgs
Airline 250 14 	5 	0
Retail 570 16 	5 	0
Telecom 570 43 	25 	60
Table 4: No. instances per type translated across all 5
target languages.
Domain Total Models Value Sets Localized Values
Airline 	23 	15 	21
Retail 	15 	6 	14
Telecom 	18 	12 	51
Table 5: Schema artifacts and literal inventories.
Domain Collections Record Breakdown
Airline 3 flights: 300, users: 500, reser-
vations: 2k
Retail 3 products: 50, users: 500, or-
ders: 1k
Telecom 6 plans: 5, devices: 29, lines: 9,
customers: 4, bills: 6
Table 6: Database artifacts. Structure is preserved; only
designated leaf fields are translated.
B Detail on Mixed Language Tool
Adaptation
For the mixed language tool adaptation, our goal is
to investigate the stability of agents when handling
tools in mixed languages. We conducted a similar
experiment to the (S3) L2 Tool Adaptation where
we modify the language of the tool specification
into L2, but instead of converting to a specific L2,
in this experiment we convert the tool specifica-
tions to a mix of several L2 languages. Specifically,
we explore 5 different mixed language settings by
gradually increasing the number of languages in
the tool specification while maintaining a similar
composition across domains: Mix-2 uses English
and Thai, Mix-3 adds Vietnamese, Mix-4 adds In-
donesian, and Mix-5 adds Chinese. For each run,
the language of a tool is fixed for the entire run to
reduce noise from random sampling. For instance,
if get_item tool is assigned to Thai, it remains in
Thai across all examples. For each

cross domains: Mix-2 uses English
and Thai, Mix-3 adds Vietnamese, Mix-4 adds In-
donesian, and Mix-5 adds Chinese. For each run,
the language of a tool is fixed for the entire run to
reduce noise from random sampling. For instance,
if get_item tool is assigned to Thai, it remains in
Thai across all examples. For each non-English
language added, three tools are localized into that
language, while the remaining tools stay in English;
thus, the English-tool count decreases as more lan-
guages are added.
C Hyperparameters
Hyperparameter Agent LLM User Sim. LLM
Temperature 	0.0 	0.0
Nucleus sampling (p) 	1.0 	1.0
Max. generation tokens — 	—
Table 7: Inference hyperparameters for the L2 Inter-
action Setting. Values follow the τ 2-Bench defaults;
parameters not listed are left at the provider model de-
faults.
All models in the L2 Interaction Setting are
run with the default inference parameters from τ 2-
Bench, as shown in Table 7.
15

-- 15 of 23 --

Scenario Agent LLM Cost (USD)
English Baseline $70.40
L2 Tool Adaptation $340.73
L2 Interaction 	$403.27
L2 Domain Adaptation $440.43
Total 	$1,254.83
Table 8: Approximate agent LLM inference cost (USD)
for each experimental scenario, aggregated across three
evaluation trials and estimated from token usage with
published API pricing.
D Compute Budget
For agent model inference, we use two models
across all experimental settings: GPT-5-mini ac-
cessed via the OpenRouter API, and Kimi-K2.5
accessed via Azure AI Foundry. Table 8 sum-
marizes the approximate total inference cost per
scenario. For user model inference, we self-
host Qwen3-235B-A22B-Instruct-2507 on 8×A100
GPUs (40GB each), resulting in a total of 4,032
GPU hours.
In the translation pipeline section, we used
Gemini 3.1 Flash-Lite Preview for three do-
mains and five target languages. Aggregated across
five languages, the pipeline consumes 8.97M in-
put tokens and 7.21M output tokens. Based on
the Gemini 3.1 Flash-Lite pricing of $0.25/M in-
put tokens and

al of 4,032
GPU hours.
In the translation pipeline section, we used
Gemini 3.1 Flash-Lite Preview for three do-
mains and five target languages. Aggregated across
five languages, the pipeline consumes 8.97M in-
put tokens and 7.21M output tokens. Based on
the Gemini 3.1 Flash-Lite pricing of $0.25/M in-
put tokens and $1.50/M output tokens, the total
translation cost is approximately $13.05.
E Agent language use and drift analysis
We characterize agent language behavior along
three axes: how often the agent produces text in
the expected target language (Figure 9), where in a
conversation any off-target text appears (Figure 10),
and how much of the off-target text is specifically
English (Figure 11).
Correctness is high overall and degrades only
in crosslingual interaction. Figure 9 shows that
L2 Tool is essentially perfect across all tested lan-
guages (mean turn correctness ≈ 1.00 for Thai,
Vietnamese, Indonesian, Chinese, and Filipino),
indicating that translating tool schemas does not
by itself induce agent language drift: with an En-
glish dialogue the agent reliably stays in English.
L2 Domain also remains high (0.92–0.98), with a
gentle ordering in which Thai and Vietnamese are
strongest (0.98) and Filipino is weakest (0.92). The
clearest degradation is confined to L2 Interaction,
TH VI ID ZH TL
L2 Tools
L2 Interaction
L2 Domain
1.00 1.00 1.00 1.00 1.00
0.80 0.82 0.82 0.78 0.75
0.98 0.98 0.96 0.95 0.92
Agent language correctness
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Mean turn correctness
Figure 9: Average turn-level agent language correctness
by setting and L2. Language correctness is calculated
as the fraction of eligible turns whose detected language
(by fastText) matches the expected L2 for one turn. We
exclude simulated user language correctness because it
is consistently above 0.95 across scenarios.
0 	10 	20
Turn index
0.00
0.05
0.10
0.15
0.20
Non-L2 proportion
L2 Tools
0 	10 	20
Turn index
L2 Interaction
0 	10 	20
Turn index
L2 Domain
Agent
User
Language

uage
(by fastText) matches the expected L2 for one turn. We
exclude simulated user language correctness because it
is consistently above 0.95 across scenarios.
0 	10 	20
Turn index
0.00
0.05
0.10
0.15
0.20
Non-L2 proportion
L2 Tools
0 	10 	20
Turn index
L2 Interaction
0 	10 	20
Turn index
L2 Domain
Agent
User
Language drift by agent turn position in a conversation
Figure 10: Proportion of non-L2 turns as a function of
agent turn position, separated by setting and by speaker
role (agent vs. user).
where correctness falls to 0.75–0.82 and is again
lowest for Filipino (0.75) and Chinese (0.78). The
gap between settings is large relative to the gap
between languages: moving from L2 Domain to
L2 Interaction costs roughly 0.15–0.20 in correct-
ness for every language, whereas the spread across
languages within a setting is at most ∼ 0.07. Drift
is therefore primarily a property of the interaction
regime rather than of any single language.
Text of off-target language accumulates over the
course of a conversation. Figure 10 decomposes
this drift by turn position. In L2 Tool, the non-
L2 proportion stays negligible throughout for both
roles, consistent with scenario design where dia-
logue should be in Enlglish. In L2 Interaction and
L2 Domain, the agent’s non-L2 share rises mono-
tonically with turn index (Pearson r ≈ 0.65 and
r ≈ 0.73 between turn position and off-target pro-
portion, respectively), pointing to an accumulation
or self-priming effect rather than a fixed per-turn
error rate. The two settings differ in their dynamics:
in L2 Interaction both the simulated user and the
agent contribute off-target turns and the magnitude
is largest (the agent share peaks near 0.12 in late
turns), whereas in L2 Domain the simulated user
16

-- 16 of 23 --

0.0
0.2
0.4
0.6
0.8
1.0
English share per task
 
L2 Interaction
TH VI ID ZH TL
Target language L2
0.0
0.2
0.4
0.6
0.8
1.0
English share per task
 
L2 Domain
Agent English switching by task and L2
Figure 11: Per-task share of

re peaks near 0.12 in late
turns), whereas in L2 Domain the simulated user
16

-- 16 of 23 --

0.0
0.2
0.4
0.6
0.8
1.0
English share per task
 
L2 Interaction
TH VI ID ZH TL
Target language L2
0.0
0.2
0.4
0.6
0.8
1.0
English share per task
 
L2 Domain
Agent English switching by task and L2
Figure 11: Per-task share of agent turns emitted in En-
glish, by target language, for the two scenarios in which
the dialogue language is L2. Each dot is a task; the
overlaid line tracks the per-language mean.
remains essentially on-target (≈ 0.015 throughout)
and the modest rise to ≈ 0.07 is agent-initiated and
concentrated in the later turns. This is consistent
with Figure 9: L2 Interaction is where both parties’
code-switching reinforces one another.
The off-target text is predominantly English,
and only for Filipino is it non-trivial. Figure 11
shows that when the agent leaves the target lan-
guage it overwhelmingly switches to English, but
that this is rare for most languages. For Thai, Viet-
namese, Indonesian, and Chinese the per-task me-
dian English share is 0 in both settings, with right-
skewed distributions whose means sit at only a few
percent. Filipino is the clear outlier: its English
share is highest in both settings (mean 0.11, me-
dian 0.06 in L2 Interaction; mean 0.08, median
0.04 in L2 Domain), and the per-language mean
line rises toward Filipino in both panels. The right-
skew indicates that English switching is driven by
a minority of tasks rather than being uniform, so
even the elevated Filipino mean reflects a tail of
high-switching tasks rather than pervasive drift.
Drift is real but is not the dominant explanation
for performance. Taken together, language drift
is genuine and scenario-dependent: strongest in L2
Interaction, concentrated in Filipino, accumulating
over turns, and realized mainly as English code-
switching. It is nonetheless too localized and too
weakly associated with task success to serve as the
primary explanation for performance

ken together, language drift
is genuine and scenario-dependent: strongest in L2
Interaction, concentrated in Filipino, accumulating
over turns, and realized mainly as English code-
switching. It is nonetheless too localized and too
weakly associated with task success to serve as the
primary explanation for performance differences.
Across all scenarios, agent language correctness is
essentially uncorrelated with pass3 (overall Pear-
son r = −0.10, R2 ≈ 0.01, n = 166); the only
setting with a strong positive correlation is L2 Tool,
where correctness is saturated near 1.00 and thus
carries no usable variance, while in L2 Interaction
and L2 Domain the scenario-level correlations are
weak (|r| ≲ 0.25). Language drift is therefore best
read as a measurable secondary symptom of the
harder crosslingual regimes (L2 Interaction) rather
than a direct cause of the pass3 gaps reported in the
main text.
F All results
17

-- 17 of 23 --

Domain Model EN
p@1 p2 p3 ρ3
Airline GPT-5-mini 0.693 0.593 0.540 0.779
Qwen3-235B 0.500 0.387 0.340 0.680
Kimi-K2.5 0.707 0.600 0.540 0.764
Retail GPT-5-mini 0.655 0.497 0.412 0.629
Qwen3-235B 0.579 0.415 0.342 0.591
Kimi-K2.5 0.561 0.442 0.395 0.704
Telecom GPT-5-mini 0.713 0.588 0.509 0.714
Qwen3-235B 0.357 0.205 0.132 0.370
Kimi-K2.5 0.997 0.953 0.930 0.933
Table 9: English Baseline (S1) results. Where EN denotes English, p@1 denotes pass@1, p2 denotes pass2, and
p3 denotes pass3. Bold values indicate the best score within each domain–metric group. Qwen3-235B refers to
Qwen3-235B-A22B-Instruct-2507.
Domain Model VI TH ID Zh TL
p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3
Airline GPT-5-mini 0.640 0.500 0.420 0.656 0.593 0.473 0.400 0.674 0.627 0.447 0.360 0.574 0.620 0.460 0.380 0.613 0.620 0.527 0.480 0.774
Qwen3-235B 0.480 0.347 0.280 0.583 0.487 0.373 0.320 0.657 0.493 0.347 0.260 0.527 0.460 0.300 0.220 0.478 0.440 0.313 0.260 0.591
Kimi-K2.5 0.733 0.660 0.600 0.818 0.687 0.580 0.520 0.757 0.740 0.653 0.600 0.811 0.760

.656 0.593 0.473 0.400 0.674 0.627 0.447 0.360 0.574 0.620 0.460 0.380 0.613 0.620 0.527 0.480 0.774
Qwen3-235B 0.480 0.347 0.280 0.583 0.487 0.373 0.320 0.657 0.493 0.347 0.260 0.527 0.460 0.300 0.220 0.478 0.440 0.313 0.260 0.591
Kimi-K2.5 0.733 0.660 0.600 0.818 0.687 0.580 0.520 0.757 0.740 0.653 0.600 0.811 0.760 0.640 0.580 0.763 0.747 0.653 0.600 0.804
Retail GPT-5-mini 0.696 0.570 0.500 0.718 0.699 0.538 0.430 0.615 0.684 0.547 0.456 0.667 0.675 0.506 0.412 0.610 0.655 0.518 0.430 0.656
Qwen3-235B 0.556 0.418 0.333 0.600 0.526 0.383 0.325 0.617 0.608 0.462 0.386 0.635 0.591 0.442 0.351 0.594 0.573 0.444 0.386 0.674
Kimi-K2.5 0.687 0.550 0.465 0.677 0.573 0.430 0.360 0.627 0.640 0.483 0.386 0.603 0.722 0.597 0.526 0.729 0.675 0.541 0.474 0.701
Telecom GPT-5-mini 0.561 0.430 0.368 0.656 0.605 0.436 0.342 0.565 0.582 0.453 0.404 0.693 0.728 0.608 0.518 0.711 0.740 0.617 0.535 0.723
Qwen3-235B 0.260 0.129 0.061 0.236 0.222 0.108 0.061 0.276 0.333 0.170 0.105 0.316 0.228 0.132 0.097 0.423 0.155 0.088 0.061 0.396
Kimi-K2.5 0.912 0.836 0.772 0.846 0.854 0.763 0.702 0.822 0.915 0.839 0.772 0.843 0.930 0.868 0.816 0.877 0.883 0.784 0.702 0.795
Table 10: L2 Interaction (S2) results across Vietnamese (VI), Thai (TH), Indonesian (ID), Chinese (ZH), and
Filipino (TL). Here, p@1 denotes pass@1, p2 denotes pass2, and p3 denotes pass3. Bold values indicate the best
score within each domain–metric group. Qwen3-235B refers to Qwen3-235B-A22B-Instruct-2507.
Domain Model VI TH ID Zh TL
p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3
Airline GPT-5-mini 0.653 0.527 0.460 0.704 0.620 0.473 0.400 0.645 0.687 0.567 0.480 0.699 0.687 0.593 0.520 0.757 0.667 0.580 0.540 0.810
Qwen3-235B 0.447 0.313 0.260 0.582 0.380 0.253 0.220 0.579 0.400 0.233 0.180 0.450 0.480 0.313 0.200 0.417 0.467 0.320 0.240 0.514
Retail GPT-5-mini 0.681 0.532 0.430 0.631 0.646 0.477 0.377 0.584 0.675 0.526 0.430 0.637 0.719 0.579 0.482 0.670 0.661 0.515 0.439 0.664
Qwen3-235B 0.547 0.412

757 0.667 0.580 0.540 0.810
Qwen3-235B 0.447 0.313 0.260 0.582 0.380 0.253 0.220 0.579 0.400 0.233 0.180 0.450 0.480 0.313 0.200 0.417 0.467 0.320 0.240 0.514
Retail GPT-5-mini 0.681 0.532 0.430 0.631 0.646 0.477 0.377 0.584 0.675 0.526 0.430 0.637 0.719 0.579 0.482 0.670 0.661 0.515 0.439 0.664
Qwen3-235B 0.547 0.412 0.351 0.642 0.561 0.421 0.351 0.626 0.550 0.418 0.351 0.638 0.594 0.433 0.360 0.606 0.570 0.421 0.342 0.600
Telecom GPT-5-mini 0.664 0.550 0.482 0.726 0.713 0.588 0.518 0.727 0.713 0.596 0.518 0.727 0.705 0.576 0.500 0.709 0.719 0.594 0.509 0.708
Qwen3-235B 0.254 0.135 0.088 0.346 0.339 0.208 0.140 0.413 0.269 0.155 0.105 0.390 0.222 0.135 0.096 0.432 0.301 0.167 0.105 0.349
Table 11: Tool Adaptation (S3) results in monolingual settings across Vietnamese (VI), Thai (TH), Indonesian (ID),
Chinese (ZH), and Filipino (TL). Here, p@1 denotes pass@1, p2 denotes pass2, and p3 denotes pass3. Bold values in-
dicate the best score within each domain–metric group. Qwen3-235B refers to Qwen3-235B-A22B-Instruct-2507.
Domain Model 	Bi 	Tri 	Quad 	Multi
p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3
Airline GPT-5-mini 0.680 0.560 0.480 0.706 0.660 0.560 0.480 0.727 0.640 0.520 0.480 0.750 0.640 0.513 0.440 0.688
Qwen3-235B 0.440 0.280 0.200 0.455 0.413 0.267 0.200 0.484 0.413 0.287 0.240 0.581 0.413 0.267 0.220 0.533
Retail GPT-5-mini 0.667 0.541 0.465 0.697 0.667 0.538 0.456 0.684 0.652 0.485 0.386 0.592 0.696 0.547 0.447 0.642
Qwen3-235B 0.596 0.456 0.386 0.648 0.535 0.398 0.325 0.607 0.585 0.447 0.377 0.644 0.570 0.401 0.298 0.523
Telecom GPT-5-mini 0.664 0.541 0.482 0.726 0.696 0.558 0.491 0.705 0.670 0.550 0.491 0.733 0.652 0.520 0.465 0.713
Qwen3-235B 0.298 0.155 0.088 0.295 0.392 0.213 0.140 0.357 0.365 0.181 0.105 0.288 0.342 0.181 0.114 0.333
Table 12: Tool Adaptation (S3) results for multilingual settings, including bilingual (Bi), trilingual (Tri), quadlin-
gual (Quad), and multilingual (Multi) configurations. Here, p@1 denotes pass@1, p2 denotes

13
Qwen3-235B 0.298 0.155 0.088 0.295 0.392 0.213 0.140 0.357 0.365 0.181 0.105 0.288 0.342 0.181 0.114 0.333
Table 12: Tool Adaptation (S3) results for multilingual settings, including bilingual (Bi), trilingual (Tri), quadlin-
gual (Quad), and multilingual (Multi) configurations. Here, p@1 denotes pass@1, p2 denotes pass2, and p3
denotes pass3. Bold values indicate the best score within each domain–metric group. Qwen3-235B refers to
Qwen3-235B-A22B-Instruct-2507.
18

-- 18 of 23 --

Domain Model VI TH ID Zh TL
p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3 p@1 p2 p3 ρ3
Airline GPT-5-mini 0.540 0.400 0.320 0.593 0.593 0.453 0.360 0.607 0.600 0.540 0.500 0.833 0.607 0.493 0.440 0.725 0.560 0.433 0.380 0.679
Qwen3-235B 0.487 0.460 0.440 0.903 0.460 0.413 0.380 0.826 0.480 0.440 0.420 0.875 0.500 0.473 0.460 0.920 0.473 0.420 0.400 0.846
Kimi-K2.5 0.560 0.427 0.340 0.607 0.547 0.387 0.300 0.548 0.600 0.487 0.420 0.700 0.660 0.507 0.440 0.667 0.600 0.487 0.400 0.667
Retail GPT-5-mini 0.585 0.427 0.333 0.569 0.325 0.175 0.123 0.378 0.541 0.392 0.325 0.601 0.596 0.450 0.368 0.617 0.488 0.333 0.263 0.539
Qwen3-235B 0.289 0.205 0.149 0.516 0.178 0.143 0.132 0.742 0.243 0.170 0.140 0.576 0.266 0.181 0.149 0.560 0.260 0.187 0.158 0.608
Kimi-K2.5 0.567 0.392 0.268 0.473 0.327 0.190 0.114 0.349 0.433 0.287 0.202 0.467 0.607 0.450 0.366 0.603 0.444 0.292 0.219 0.493
Telecom GPT-5-mini 0.564 0.427 0.360 0.638 0.599 0.447 0.360 0.601 0.497 0.325 0.246 0.495 0.760 0.652 0.588 0.774 0.681 0.547 0.474 0.696
Qwen3-235B 0.383 0.213 0.149 0.389 0.284 0.143 0.079 0.278 0.427 0.295 0.211 0.494 0.330 0.205 0.149 0.452 0.330 0.205 0.149 0.452
Kimi-K2.5 0.699 0.529 0.404 0.578 0.693 0.523 0.43 0.620 0.798 0.681 0.588 0.737 0.863 0.763 0.684 0.793 0.743 0.582 0.474 0.638
Table 13: L2 Domain (S4) results across Vietnamese (VI), Thai (TH), Indonesian (ID), Chinese (ZH), and Filipino
(TL). Here, p@1 denotes pass@1, p2 denotes pass2, and p3 denotes pass3. Bold values indicate the best

.404 0.578 0.693 0.523 0.43 0.620 0.798 0.681 0.588 0.737 0.863 0.763 0.684 0.793 0.743 0.582 0.474 0.638
Table 13: L2 Domain (S4) results across Vietnamese (VI), Thai (TH), Indonesian (ID), Chinese (ZH), and Filipino
(TL). Here, p@1 denotes pass@1, p2 denotes pass2, and p3 denotes pass3. Bold values indicate the best score
within each domain–metric group. Qwen3-235B refers to Qwen3-235B-A22B-Instruct-2507.
19

-- 19 of 23 --

G Detailed Error Analysis Results
Setting Variant Agent Errors User Errors
Critical Minor Correct Critical Minor Correct
English Baseline English 81 22 47 38 48 64
L2 Interaction
Vietnamese 93 25 32 43 48 59
Thai 105 14 31 42 45 63
Indonesian 88 18 44 36 49 65
Chinese 90 18 42 44 51 55
Filipino 94 18 38 53 36 61
Tool
Adaptation
VI Tools 81 23 46 43 51 56
TH Tools 88 28 34 39 45 66
ID Tools 94 20 36 41 41 68
ZH Tools 84 23 43 35 44 71
FIL Tools 92 17 41 45 45 60
Mix-2 (EN+TH) 90 24 36 52 38 60
Mix-3 (+VI) 86 17 47 33 54 63
Mix-4 (+ID) 89 26 35 35 45 70
Mix-5 (+ZH) 102 15 33 40 41 69
L2 Domain
Vietnamese 83 27 40 43 41 66
Thai 97 17 36 55 30 65
Indonesian 92 24 34 42 39 69
Chinese 103 15 32 40 48 62
Filipino 101 17 32 50 47 53
Table 14: Error analysis on the Airline domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=150 per condition). Counts of simulations by maximum severity level.
20

-- 20 of 23 --

Setting Variant Agent Errors User Errors
Critical Minor Correct Critical Minor Correct
English Baseline English 98 70 174 79 135 128
L2 Interaction
Vietnamese 109 78 155 96 103 143
Thai 125 73 144 94 124 124
Indonesian 117 65 160 86 121 135
Chinese 109 73 160 81 126 135
Filipino 103 88 151 105 129 108
Tool
Adaptation
VI Tools 97 75 170 75 130 137
TH Tools 102 79 161 70 123 149
ID Tools 94 86 162 92 130 120
ZH Tools 95 64 183 87 121 134
FIL Tools 88 79 175 79 129 134
Mix-2 (EN+TH) 81 81 180 58 132 152
Mix-3 (+VI) 100 69 173 88 123 131
Mix-4 (+ID) 100 72 170 88 112 142
Mix-5 (+ZH) 87 93 162 82 129 131
L2

105 129 108
Tool
Adaptation
VI Tools 97 75 170 75 130 137
TH Tools 102 79 161 70 123 149
ID Tools 94 86 162 92 130 120
ZH Tools 95 64 183 87 121 134
FIL Tools 88 79 175 79 129 134
Mix-2 (EN+TH) 81 81 180 58 132 152
Mix-3 (+VI) 100 69 173 88 123 131
Mix-4 (+ID) 100 72 170 88 112 142
Mix-5 (+ZH) 87 93 162 82 129 131
L2 Domain
Vietnamese 112 93 137 90 122 130
Thai 180 60 102 127 91 124
Indonesian 124 76 142 93 122 127
Chinese 131 84 127 101 113 128
Filipino 131 74 137 123 118 101
Table 15: Error analysis on the Retail domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=342 per condition). Counts of simulations by maximum severity level.
Setting Variant Agent Errors User Errors
Critical Minor Correct Critical Minor Correct
English Baseline English 110 113 119 50 130 162
L2 Interaction
Vietnamese 101 116 125 52 146 144
Thai 125 90 127 59 133 150
Indonesian 122 100 120 60 136 146
Chinese 95 76 171 60 129 153
Filipino 91 88 163 62 141 139
Tool
Adaptation
VI Tools 103 117 122 45 156 141
TH Tools 87 106 149 46 147 149
ID Tools 89 108 145 48 140 154
ZH Tools 109 91 142 55 135 152
FIL Tools 75 112 155 45 115 182
Mix-2 (EN+TH) 91 106 145 54 138 150
Mix-3 (+VI) 95 117 130 55 144 143
Mix-4 (+ID) 91 102 149 56 128 158
Mix-5 (+ZH) 94 101 147 43 148 151
L2 Domain
Vietnamese 125 71 146 54 112 176
Thai 133 85 124 64 105 173
Indonesian 139 81 122 59 120 163
Chinese 103 86 153 45 128 169
Filipino 131 89 122 64 130 148
Table 16: Error analysis on the Telecom domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=342 per condition). Counts of simulations by maximum severity level.
21

-- 21 of 23 --

Eng. Baseline 	L2 Interaction 	Tool Adaptation 	L2 Domain
Error Tag 	EN 	VI 	TH 	ID 	ZH 	FIL 	VI 	TH 	ID 	ZH 	FIL 	Mix-2 	Mix-3 	Mix-4 	Mix-5 	VI 	TH 	ID 	ZH 	FIL
Guideline Violation 	C:100 M:20 C:116 M:38 C:113 M:23 C:89 M:31 C:85 M:27 C:110 M:28 C:82 M:19 C:105 M:30 C:87 M:32 C:92 M:26 C:89 M:21 C:110 M:22 C:97 M:16 C:96 M:24 C:103 M:26

eraction 	Tool Adaptation 	L2 Domain
Error Tag 	EN 	VI 	TH 	ID 	ZH 	FIL 	VI 	TH 	ID 	ZH 	FIL 	Mix-2 	Mix-3 	Mix-4 	Mix-5 	VI 	TH 	ID 	ZH 	FIL
Guideline Violation 	C:100 M:20 C:116 M:38 C:113 M:23 C:89 M:31 C:85 M:27 C:110 M:28 C:82 M:19 C:105 M:30 C:87 M:32 C:92 M:26 C:89 M:21 C:110 M:22 C:97 M:16 C:96 M:24 C:103 M:26 C:90 M:36 C:130 M:28 C:115 M:18 C:108 M:21 C:114 M:24
	Hallucination 	C:6 M:7 	C:3 M:6 	C:4 M:8 C:6 M:5 C:9 M:1 C:8 M:5 C:0 M:4 C:5 M:2 C:5 M:6 C:2 M:4 C:6 M:5 C:3 M:2 C:3 M:5 C:2 M:5 C:7 M:2 C:8 M:2 C:14 M:10 C:3 M:5 	C:7 M:8 	C:6 M:7
	Inconsistent Behavior 	C:4 M:8 	C:7 M:11 C:11 M:9 C:6 M:15 C:5 M:7 C:5 M:3 C:12 M:5 C:11 M:3 C:5 M:6 C:5 M:10 C:3 M:5 C:3 M:5 C:13 M:4 C:8 M:9 C:6 M:5 C:6 M:13 C:9 M:10 C:6 M:8 C:7 M:10 C:6 M:5
	Incorrect Interpretation 	C:33 M:25 C:67 M:29 C:56 M:30 C:47 M:33 C:46 M:38 C:46 M:36 C:25 M:29 C:46 M:35 C:33 M:40 C:42 M:31 C:39 M:29 C:34 M:29 C:33 M:24 C:36 M:34 C:46 M:21 C:54 M:37 C:71 M:35 C:43 M:25 C:47 M:32 C:58 M:30
	Interruption Error 	C:0 M:0 	C:0 M:0 	C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Irrelevant Tool Call 	C:0 M:0 	C:3 M:4 	C:0 M:4 C:0 M:4 C:0 M:2 C:0 M:0 C:4 M:0 C:0 M:1 C:0 M:0 C:0 M:2 C:0 M:1 C:0 M:1 C:1 M:1 C:1 M:3 C:0 M:2 C:2 M:0 C:1 M:4 	C:1 M:0 	C:0 M:1 	C:0 M:1
	Missed Required Action 	C:71 M:16 C:82 M:28 C:98 M:18 C:74 M:15 C:85 M:20 C:90 M:25 C:79 M:22 C:75 M:22 C:78 M:17 C:75 M:27 C:85 M:22 C:78 M:20 C:78 M:14 C:87 M:33 C:84 M:21 C:77 M:15 C:86 M:16 C:96 M:25 C:94 M:19 C:110 M:23
	Other 	C:0 M:1 	C:2 M:4 	C:0 M:0 C:0 M:1 C:1 M:3 C:0 M:0 C:0 M:2 C:0 M:1 C:2 M:1 C:0 M:0 C:0 M:0 C:0 M:1 C:2 M:0 C:1 M:1 C:0 M:1 C:1 M:0 C:1 M:0 	C:0 M:1 	C:0 M:1 	C:0 M:4
	Premature Termination 	C:28 M:1 	C:20 M:1 C:30 M:2 C:21 M:1 C:18 M:0 C:25 M:1 C:22 M:1 C:21 M:4 C:19 M:1 C:23 M:3 C:34 M:1 C:29 M:1 C:35 M:2 C:26 M:1 C:45 M:2 C:21 M:9 C:25 M:2 C:28 M:0 C:27 M:2 C:30 M:1
	Revealed Info Early 	C:0 M:1 	C:0 M:2 	C:0

M:1 C:2 M:0 C:1 M:1 C:0 M:1 C:1 M:0 C:1 M:0 	C:0 M:1 	C:0 M:1 	C:0 M:4
	Premature Termination 	C:28 M:1 	C:20 M:1 C:30 M:2 C:21 M:1 C:18 M:0 C:25 M:1 C:22 M:1 C:21 M:4 C:19 M:1 C:23 M:3 C:34 M:1 C:29 M:1 C:35 M:2 C:26 M:1 C:45 M:2 C:21 M:9 C:25 M:2 C:28 M:0 C:27 M:2 C:30 M:1
	Revealed Info Early 	C:0 M:1 	C:0 M:2 	C:0 M:1 C:1 M:0 C:1 M:0 C:2 M:2 C:3 M:1 C:1 M:1 C:0 M:0 C:0 M:0 C:1 M:1 C:1 M:1 C:2 M:1 C:0 M:0 C:0 M:0 C:1 M:2 C:0 M:2 	C:3 M:1 	C:1 M:2 	C:0 M:0
	Tool Call Argument Error C:5 M:4 	C:3 M:1 	C:7 M:6 C:4 M:6 C:3 M:7 C:1 M:4 C:2 M:6 C:2 M:6 C:1 M:4 C:1 M:0 C:5 M:0 C:3 M:3 C:2 M:5 C:1 M:4 C:1 M:2 C:5 M:3 C:5 M:3 	C:7 M:8 	C:6 M:4 	C:6 M:2
	Tool Call Schema Error 	C:0 M:0 	C:1 M:0 	C:2 M:2 C:0 M:2 C:0 M:1 C:0 M:4 C:0 M:0 C:0 M:1 C:0 M:0 C:2 M:0 C:0 M:0 C:0 M:1 C:1 M:0 C:0 M:1 C:0 M:2 C:1 M:1 C:0 M:0 	C:0 M:1 	C:0 M:1 	C:0 M:0
	Wrong Sequence 	C:9 M:3 	C:9 M:7 	C:8 M:3 C:3 M:4 C:11 M:4 C:7 M:3 C:12 M:6 C:3 M:3 C:5 M:3 C:12 M:6 C:5 M:2 C:5 M:1 C:11 M:1 C:5 M:1 C:7 M:1 C:8 M:3 C:7 M:3 C:13 M:5 C:12 M:2 C:11 M:5
Table 17: Agent error tag counts — all settings, Airline domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=150 per condition). C: = critical-severity count; M: = minor-severity count. Each
value is the number of simulations in which the tag appeared at that severity level at least once.
Eng. Baseline 	L2 Interaction 	Tool Adaptation 	L2 Domain
Error Tag 	EN 	VI 	TH 	ID 	ZH 	FIL 	VI 	TH 	ID 	ZH 	FIL 	Mix-2 	Mix-3 	Mix-4 	Mix-5 	VI 	TH 	ID 	ZH 	FIL
Guideline Violation 	C:13 M:26 C:18 M:22 C:19 M:25 C:15 M:30 C:28 M:23 C:26 M:31 C:19 M:36 C:9 M:17 C:16 M:13 C:14 M:25 C:22 M:18 C:23 M:19 C:16 M:22 C:19 M:20 C:18 M:14 C:16 M:20 C:29 M:14 C:20 M:13 C:18 M:19 C:21 M:24
	Hallucination 	C:26 M:27 C:22 M:24 C:31 M:24 C:19 M:37 C:23 M:31 C:39 M:23 C:39 M:28 C:39 M:20 C:25 M:21 C:32 M:28 C:35 M:29 C:46 M:19 C:30 M:26 C:25 M:21 C:33 M:20 C:35 M:22 C:48 M:13 C:24 M:13 C:33 M:18 C:29 M:21
	Inconsistent Behavior 	C:15 M:34 C:22 M:34 C:15

M:14 C:16 M:20 C:29 M:14 C:20 M:13 C:18 M:19 C:21 M:24
	Hallucination 	C:26 M:27 C:22 M:24 C:31 M:24 C:19 M:37 C:23 M:31 C:39 M:23 C:39 M:28 C:39 M:20 C:25 M:21 C:32 M:28 C:35 M:29 C:46 M:19 C:30 M:26 C:25 M:21 C:33 M:20 C:35 M:22 C:48 M:13 C:24 M:13 C:33 M:18 C:29 M:21
	Inconsistent Behavior 	C:15 M:34 C:22 M:34 C:15 M:28 C:17 M:42 C:25 M:35 C:25 M:31 C:15 M:45 C:19 M:27 C:13 M:40 C:14 M:37 C:24 M:33 C:21 M:37 C:19 M:36 C:15 M:37 C:25 M:33 C:16 M:23 C:26 M:30 C:15 M:27 C:18 M:37 C:19 M:39
	Incorrect Interpretation 	C:4 M:7 	C:3 M:8 C:2 M:9 C:2 M:4 C:3 M:9 C:11 M:7 C:2 M:11 C:3 M:2 C:3 M:5 C:1 M:6 C:1 M:4 C:6 M:8 C:4 M:9 C:3 M:4 C:1 M:7 C:3 M:8 C:3 M:2 C:3 M:6 C:0 M:13 C:5 M:8
	Interruption Error 	C:0 M:0 	C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0
	Irrelevant Tool Call 	C:0 M:0 	C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0
	Missed Required Action 	C:0 M:1 	C:3 M:1 C:2 M:0 C:0 M:1 C:1 M:2 C:2 M:1 C:1 M:1 C:1 M:0 C:3 M:1 C:1 M:0 C:1 M:2 C:1 M:1 C:0 M:0 C:1 M:3 C:0 M:1 C:2 M:3 C:0 M:1 C:0 M:0 C:0 M:0 C:3 M:3
	Other 	C:0 M:1 	C:0 M:1 C:0 M:0 C:0 M:0 C:0 M:2 C:1 M:0 C:0 M:0 C:0 M:2 C:0 M:0 C:1 M:0 C:0 M:1 C:0 M:0 C:0 M:3 C:1 M:0 C:0 M:0 C:0 M:2 C:0 M:0 C:0 M:1 C:0 M:0 C:2 M:1
	Premature Termination 	C:9 M:10 C:6 M:12 C:12 M:15 C:11 M:9 C:11 M:18 C:8 M:10 C:4 M:7 C:10 M:17 C:10 M:16 C:7 M:10 C:9 M:15 C:9 M:7 C:7 M:9 C:6 M:11 C:3 M:11 C:8 M:14 C:10 M:8 C:9 M:11 C:9 M:12 C:11 M:16
	Revealed Info Early 	C:0 M:2 	C:1 M:0 C:0 M:2 C:0 M:0 C:0 M:1 C:0 M:2 C:0 M:2 C:0 M:0 C:0 M:0 C:1 M:1 C:0 M:1 C:0 M:1 C:0 M:0 C:0 M:0 C:0 M:1 C:0 M:3 C:0 M:0 C:0 M:0 C:0 M:3 C:0 M:1
	Tool Call Argument Error C:0 M:0 	C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0
	Tool Call

1 C:0 M:2 C:0 M:2 C:0 M:0 C:0 M:0 C:1 M:1 C:0 M:1 C:0 M:1 C:0 M:0 C:0 M:0 C:0 M:1 C:0 M:3 C:0 M:0 C:0 M:0 C:0 M:3 C:0 M:1
	Tool Call Argument Error C:0 M:0 	C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0
	Tool Call Schema Error 	C:0 M:0 	C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0
	Wrong Sequence 	C:0 M:0 	C:0 M:0 C:0 M:0 C:0 M:1 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:1 C:0 M:1 C:0 M:0 C:1 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:1 C:0 M:0
Table 18: User error tag counts — all settings, Airline domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=150 per condition). C: = critical-severity count; M: = minor-severity count. Each
value is the number of simulations in which the tag appeared at that severity level at least once.
Eng. Baseline 	L2 Interaction 	Tool Adaptation 	L2 Domain
Error Tag 	EN 	VI 	TH 	ID 	ZH 	FIL 	VI 	TH 	ID 	ZH 	FIL 	Mix-2 	Mix-3 	Mix-4 	Mix-5 	VI 	TH 	ID 	ZH 	FIL
Guideline Violation 	C:70 M:74 C:99 M:80 C:105 M:81 C:88 M:83 C:71 M:59 C:86 M:94 C:68 M:52 C:70 M:66 C:69 M:69 C:62 M:55 C:78 M:56 C:49 M:55 C:67 M:48 C:63 M:53 C:78 M:72 C:73 M:76 C:128 M:54 C:92 M:53 C:112 M:93 C:92 M:59
	Hallucination 	C:7 M:11 C:9 M:12 C:15 M:13 C:17 M:17 C:13 M:11 C:10 M:10 C:9 M:11 C:6 M:13 C:3 M:11 C:16 M:7 C:15 M:10 C:11 M:10 C:11 M:12 C:17 M:18 C:12 M:18 C:18 M:17 C:30 M:10 C:18 M:17 C:9 M:18 C:14 M:13
	Inconsistent Behavior 	C:10 M:15 C:17 M:24 C:13 M:24 C:8 M:15 C:8 M:28 C:8 M:16 C:10 M:15 C:6 M:14 C:7 M:17 C:8 M:10 C:5 M:21 C:10 M:15 C:10 M:13 C:7 M:16 C:8 M:26 C:13 M:27 C:29 M:25 C:12 M:22 C:24 M:27 C:22 M:22
	Incorrect Interpretation 	C:34 M:39 C:32 M:54 C:42 M:59 C:38 M:54 C:26 M:44 C:37 M:39 C:31 M:53 C:40 M:47 C:28 M:39 C:30 M:40 C:36 M:34 C:22 M:33 C:29 M:32 C:38 M:43 C:27 M:65 C:48 M:88 C:142 M:62

6 M:14 C:7 M:17 C:8 M:10 C:5 M:21 C:10 M:15 C:10 M:13 C:7 M:16 C:8 M:26 C:13 M:27 C:29 M:25 C:12 M:22 C:24 M:27 C:22 M:22
	Incorrect Interpretation 	C:34 M:39 C:32 M:54 C:42 M:59 C:38 M:54 C:26 M:44 C:37 M:39 C:31 M:53 C:40 M:47 C:28 M:39 C:30 M:40 C:36 M:34 C:22 M:33 C:29 M:32 C:38 M:43 C:27 M:65 C:48 M:88 C:142 M:62 C:50 M:56 C:63 M:63 C:50 M:55
	Interruption Error 	C:0 M:0 	C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 C:0 M:0 	C:0 M:0
	Irrelevant Tool Call 	C:2 M:3 	C:1 M:3 C:1 M:4 C:5 M:5 C:0 M:5 C:0 M:2 C:1 M:3 C:1 M:5 C:0 M:0 C:0 M:2 C:2 M:2 C:0 M:1 C:1 M:3 C:0 M:3 C:1 M:5 C:0 M:1 C:11 M:1 C:1 M:4 C:1 M:5 	C:1 M:1
	Missed Required Action C:75 M:53 C:78 M:62 C:95 M:51 C:88 M:55 C:93 M:53 C:94 M:63 C:62 M:59 C:79 M:66 C:74 M:64 C:69 M:63 C:69 M:69 C:58 M:56 C:63 M:71 C:73 M:49 C:59 M:67 C:73 M:66 C:160 M:62 C:75 M:65 C:98 M:83 C:101 M:81
	Other 	C:0 M:2 	C:0 M:3 C:1 M:7 C:2 M:1 C:0 M:3 C:0 M:3 C:0 M:3 C:0 M:3 C:0 M:4 C:2 M:1 C:1 M:2 C:1 M:2 C:0 M:4 C:0 M:1 C:0 M:6 C:0 M:3 C:1 M:5 C:1 M:2 C:1 M:5 	C:2 M:1
	Premature Termination 	C:10 M:5 	C:5 M:2 C:7 M:2 C:11 M:0 C:5 M:7 C:9 M:5 C:6 M:3 C:7 M:3 C:8 M:3 C:12 M:3 C:6 M:1 C:12 M:4 C:17 M:2 C:5 M:1 C:6 M:2 C:12 M:4 C:28 M:4 C:15 M:4 C:14 M:1 C:7 M:3
	Revealed Info Early 	C:0 M:0 	C:1 M:5 C:4 M:4 C:4 M:2 C:1 M:2 C:2 M:3 C:2 M:3 C:1 M:2 C:1 M:5 C:0 M:6 C:0 M:0 C:0 M:4 C:3 M:1 C:0 M:5 C:1 M:0 C:1 M:3 C:2 M:3 C:2 M:3 C:0 M:3 	C:2 M:2
	Tool Call Argument Error C:8 M:5 	C:7 M:9 C:7 M:18 C:8 M:6 C:9 M:10 C:5 M:3 C:9 M:7 C:5 M:7 C:8 M:9 C:6 M:3 C:8 M:6 C:5 M:14 C:7 M:4 C:8 M:9 C:5 M:8 C:35 M:49 C:48 M:39 C:39 M:28 C:48 M:25 C:37 M:34
	Tool Call Schema Error 	C:1 M:2 	C:3 M:5 C:4 M:6 C:1 M:5 C:0 M:4 C:4 M:9 C:0 M:6 C:2 M:6 C:1 M:8 C:4 M:2 C:4 M:5 C:1 M:2 C:0 M:9 C:4 M:5 C:2 M:7 C:6 M:1 C:3 M:9 C:5 M:9 C:6 M:5 	C:2 M:4
	Wrong Sequence 	C:20 M:7 C:24 M:24 C:20 M:17 C:19 M:20 C:19 M:16 C:23 M:15 C:16 M:14 C:12 M:21 C:18 M:18 C:8 M:13 C:11

25 C:37 M:34
	Tool Call Schema Error 	C:1 M:2 	C:3 M:5 C:4 M:6 C:1 M:5 C:0 M:4 C:4 M:9 C:0 M:6 C:2 M:6 C:1 M:8 C:4 M:2 C:4 M:5 C:1 M:2 C:0 M:9 C:4 M:5 C:2 M:7 C:6 M:1 C:3 M:9 C:5 M:9 C:6 M:5 	C:2 M:4
	Wrong Sequence 	C:20 M:7 C:24 M:24 C:20 M:17 C:19 M:20 C:19 M:16 C:23 M:15 C:16 M:14 C:12 M:21 C:18 M:18 C:8 M:13 C:11 M:9 C:14 M:14 C:7 M:10 C:11 M:13 C:11 M:18 C:14 M:11 C:24 M:8 C:19 M:21 C:12 M:15 C:13 M:12
Table 19: Agent error tag counts — all settings, Retail domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=342 per condition). C: = critical-severity count; M: = minor-severity count. Each
value is the number of simulations in which the tag appeared at that severity level at least once.
Eng. Baseline 	L2 Interaction 	Tool Adaptation 	L2 Domain
Error Tag 	EN 	VI 	TH 	ID 	ZH 	FIL 	VI 	TH 	ID 	ZH 	FIL 	Mix-2 	Mix-3 	Mix-4 	Mix-5 	VI 	TH 	ID 	ZH 	FIL
Guideline Violation 	C:19 M:50 C:32 M:51 C:41 M:72 C:28 M:36 C:38 M:42 C:31 M:56 C:19 M:46 C:22 M:34 C:41 M:43 C:29 M:57 C:24 M:47 C:17 M:37 C:30 M:39 C:23 M:29 C:27 M:40 C:25 M:31 C:29 M:23 C:29 M:32 C:33 M:31 C:23 M:33
	Hallucination 	C:69 M:105 C:87 M:91 C:71 M:82 C:59 M:92 C:55 M:86 C:95 M:108 C:71 M:115 C:55 M:90 C:61 M:132 C:74 M:104 C:54 M:123 C:53 M:123 C:82 M:106 C:69 M:94 C:64 M:129 C:75 M:76 C:111 M:74 C:76 M:105 C:119 M:110 C:115 M:106
	Inconsistent Behavior 	C:22 M:93 C:38 M:85 C:40 M:101 C:38 M:96 C:32 M:81 C:44 M:101 C:28 M:99 C:28 M:86 C:27 M:86 C:30 M:103 C:37 M:92 C:14 M:66 C:38 M:94 C:31 M:76 C:21 M:85 C:39 M:94 C:39 M:86 C:34 M:102 C:26 M:84 C:51 M:111
	Incorrect Interpretation 	C:4 M:21 C:15 M:8 C:9 M:10 C:6 M:8 C:1 M:13 C:10 M:7 C:2 M:10 C:5 M:16 C:4 M:8 	C:8 M:16 C:7 M:15 C:7 M:12 C:5 M:12 C:5 M:12 C:12 M:14 C:5 M:20 C:11 M:16 C:7 M:12 	C:5 M:17 	C:13 M:15
	Interruption Error 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Irrelevant Tool

C:8 M:16 C:7 M:15 C:7 M:12 C:5 M:12 C:5 M:12 C:12 M:14 C:5 M:20 C:11 M:16 C:7 M:12 	C:5 M:17 	C:13 M:15
	Interruption Error 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Irrelevant Tool Call 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Missed Required Action 	C:0 M:2 	C:1 M:1 	C:2 M:4 	C:4 M:2 C:3 M:1 	C:3 M:3 	C:2 M:1 	C:5 M:4 	C:0 M:4 	C:3 M:4 	C:2 M:3 	C:0 M:0 	C:0 M:6 	C:1 M:2 	C:2 M:2 	C:2 M:0 	C:3 M:2 	C:2 M:5 	C:2 M:3 	C:3 M:2
	Other 	C:0 M:4 	C:2 M:3 C:2 M:10 C:1 M:1 C:0 M:8 	C:1 M:1 	C:0 M:2 	C:0 M:4 	C:0 M:2 	C:0 M:2 	C:1 M:3 	C:1 M:5 	C:0 M:7 	C:0 M:2 	C:0 M:6 	C:0 M:4 	C:0 M:3 	C:0 M:4 	C:1 M:3 	C:0 M:4
	Premature Termination 	C:9 M:19 C:11 M:17 C:17 M:18 C:12 M:19 C:16 M:22 C:10 M:15 C:10 M:16 C:7 M:15 C:15 M:14 C:10 M:16 C:14 M:17 C:6 M:14 C:10 M:15 C:17 M:12 C:11 M:21 C:8 M:18 C:28 M:24 C:16 M:19 C:11 M:12 C:22 M:16
	Revealed Info Early 	C:0 M:1 	C:0 M:0 	C:2 M:0 	C:0 M:0 C:0 M:0 	C:0 M:0 	C:2 M:2 	C:1 M:0 	C:3 M:0 	C:1 M:2 	C:0 M:1 	C:1 M:1 	C:1 M:1 	C:0 M:0 	C:0 M:0 	C:0 M:1 	C:3 M:2 	C:0 M:0 	C:0 M:1 	C:0 M:2
	Tool Call Argument Error C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Tool Call Schema Error 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Wrong Sequence 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:1 	C:0 M:0 	C:0 M:0 	C:0 M:1 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
Table 20: User error tag counts — all settings, Retail domain (gpt-5-mini agent,

C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Wrong Sequence 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:1 	C:0 M:0 	C:0 M:0 	C:0 M:1 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
Table 20: User error tag counts — all settings, Retail domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=342 per condition). C: = critical-severity count; M: = minor-severity count. Each
value is the number of simulations in which the tag appeared at that severity level at least once.
Eng. Baseline 	L2 Interaction 	Tool Adaptation 	L2 Domain
Error Tag 	EN 	VI 	TH 	ID 	ZH 	FIL 	VI 	TH 	ID 	ZH 	FIL 	Mix-2 	Mix-3 	Mix-4 	Mix-5 	VI 	TH 	ID 	ZH 	FIL
Guideline Violation 	C:88 M:81 C:77 M:128 C:99 M:105 C:103 M:116 C:71 M:46 C:61 M:68 C:73 M:90 C:69 M:77 C:57 M:94 C:77 M:61 C:49 M:88 C:80 M:83 C:78 M:81 C:70 M:75 C:72 M:90 C:117 M:104 C:106 M:97 C:98 M:91 C:78 M:81 C:106 M:115
	Hallucination 	C:4 M:8 	C:6 M:16 	C:7 M:15 	C:6 M:14 	C:3 M:5 	C:4 M:11 	C:4 M:18 	C:4 M:11 	C:6 M:8 	C:2 M:14 	C:2 M:11 	C:4 M:15 	C:3 M:14 	C:4 M:17 	C:3 M:14 	C:7 M:16 	C:5 M:11 	C:5 M:19 	C:5 M:9 	C:3 M:11
	Inconsistent Behavior 	C:9 M:28 	C:15 M:49 C:16 M:29 C:12 M:40 	C:4 M:20 	C:6 M:32 	C:6 M:29 	C:6 M:25 	C:3 M:29 	C:3 M:36 	C:7 M:36 	C:8 M:27 	C:9 M:29 	C:3 M:19 	C:7 M:26 	C:10 M:37 	C:9 M:47 	C:8 M:44 	C:8 M:31 	C:12 M:38
	Incorrect Interpretation 	C:39 M:94 	C:38 M:84 C:47 M:100 C:43 M:100 C:36 M:53 C:29 M:60 C:41 M:80 C:17 M:91 C:30 M:80 C:37 M:74 C:26 M:66 C:28 M:71 C:36 M:98 C:26 M:96 C:40 M:80 C:50 M:69 C:54 M:95 C:53 M:77 C:29 M:71 C:54 M:85
	Interruption Error 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Irrelevant Tool Call 	C:0 M:10 	C:4 M:36 	C:7 M:24 	C:5 M:19 	C:1 M:6 	C:3 M:18 	C:1 M:11 	C:2 M:8 	C:2 M:27 	C:5 M:6 	C:2 M:11 	C:2 M:14 	C:1 M:36 	C:0 M:6 	C:1 M:20 	C:3 M:19 	C:2

0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Irrelevant Tool Call 	C:0 M:10 	C:4 M:36 	C:7 M:24 	C:5 M:19 	C:1 M:6 	C:3 M:18 	C:1 M:11 	C:2 M:8 	C:2 M:27 	C:5 M:6 	C:2 M:11 	C:2 M:14 	C:1 M:36 	C:0 M:6 	C:1 M:20 	C:3 M:19 	C:2 M:11 	C:1 M:8 	C:0 M:4 	C:3 M:15
	Missed Required Action C:127 M:144 C:106 M:132 C:142 M:118 C:138 M:132 C:109 M:107 C:107 M:97 C:140 M:154 C:93 M:124 C:104 M:133 C:134 M:142 C:96 M:147 C:107 M:153 C:100 M:189 C:106 M:135 C:118 M:118 C:151 M:96 C:166 M:136 C:178 M:108 C:111 M:119 C:163 M:111
	Other 	C:0 M:6 	C:1 M:6 	C:0 M:10 	C:1 M:8 	C:1 M:3 	C:1 M:8 	C:0 M:5 	C:1 M:14 	C:0 M:17 	C:2 M:5 	C:5 M:2 	C:0 M:11 	C:1 M:27 	C:0 M:2 	C:1 M:6 	C:0 M:15 	C:1 M:9 	C:1 M:7 	C:0 M:7 	C:1 M:2
	Premature Termination 	C:25 M:4 	C:25 M:6 	C:36 M:4 	C:28 M:2 	C:23 M:2 	C:18 M:1 	C:25 M:1 	C:28 M:1 	C:28 M:4 	C:28 M:3 	C:23 M:1 	C:36 M:4 	C:28 M:4 	C:33 M:0 	C:37 M:2 	C:30 M:3 	C:39 M:2 	C:33 M:3 	C:27 M:3 	C:33 M:5
	Revealed Info Early 	C:0 M:2 	C:2 M:2 	C:2 M:2 	C:0 M:1 	C:1 M:1 	C:0 M:0 	C:1 M:1 	C:1 M:3 	C:1 M:3 	C:1 M:1 	C:1 M:0 	C:2 M:1 	C:0 M:0 	C:0 M:1 	C:1 M:4 	C:3 M:4 	C:1 M:4 	C:0 M:5 	C:0 M:0 	C:2 M:2
	Tool Call Argument Error 	C:1 M:0 	C:6 M:7 	C:4 M:19 	C:0 M:4 	C:1 M:3 	C:1 M:4 	C:0 M:2 	C:1 M:3 	C:1 M:4 	C:1 M:1 	C:1 M:7 	C:3 M:1 	C:2 M:3 	C:0 M:0 	C:0 M:6 	C:2 M:7 	C:2 M:7 	C:2 M:3 	C:2 M:5 	C:3 M:11
	Tool Call Schema Error 	C:0 M:1 	C:1 M:3 	C:0 M:6 	C:0 M:1 	C:1 M:0 	C:1 M:2 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:1 	C:2 M:0 	C:1 M:3 	C:0 M:2 	C:0 M:0 	C:0 M:0 	C:2 M:1 	C:0 M:0 	C:1 M:3 	C:0 M:0 	C:5 M:4
	Wrong Sequence 	C:6 M:12 	C:13 M:24 	C:7 M:27 	C:10 M:16 	C:12 M:7 	C:5 M:8 	C:12 M:15 C:7 M:13 C:10 M:14 	C:10 M:8 	C:9 M:9 	C:8 M:13 	C:7 M:16 	C:7 M:9 	C:12 M:18 C:17 M:13 C:16 M:12 C:23 M:21 C:12 M:19 C:13 M:27
Table 21: Agent error tag counts — all settings, Telecom domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=342

C:7 M:27 	C:10 M:16 	C:12 M:7 	C:5 M:8 	C:12 M:15 C:7 M:13 C:10 M:14 	C:10 M:8 	C:9 M:9 	C:8 M:13 	C:7 M:16 	C:7 M:9 	C:12 M:18 C:17 M:13 C:16 M:12 C:23 M:21 C:12 M:19 C:13 M:27
Table 21: Agent error tag counts — all settings, Telecom domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=342 per condition). C: = critical-severity count; M: = minor-severity count. Each
value is the number of simulations in which the tag appeared at that severity level at least once.
22

-- 22 of 23 --

Eng. Baseline 	L2 Interaction 	Tool Adaptation 	L2 Domain
Error Tag 	EN 	VI 	TH 	ID 	ZH 	FIL 	VI 	TH 	ID 	ZH 	FIL 	Mix-2 	Mix-3 	Mix-4 	Mix-5 	VI 	TH 	ID 	ZH 	FIL
Guideline Violation 	C:25 M:42 C:10 M:93 C:30 M:77 C:30 M:53 C:30 M:82 C:54 M:105 C:33 M:52 C:36 M:61 C:16 M:74 C:19 M:64 C:35 M:58 C:15 M:49 C:13 M:52 C:15 M:77 C:12 M:56 C:13 M:42 C:17 M:46 C:11 M:59 C:23 M:85 C:22 M:85
	Hallucination 	C:52 M:85 C:34 M:96 C:47 M:67 C:68 M:74 C:62 M:85 C:29 M:68 C:45 M:91 C:48 M:88 C:51 M:78 C:49 M:55 C:32 M:60 C:48 M:93 C:40 M:95 C:38 M:69 C:74 M:83 C:39 M:48 C:49 M:56 C:57 M:54 C:37 M:61 C:52 M:47
	Inconsistent Behavior 	C:24 M:194 C:19 M:207 C:26 M:186 C:33 M:139 C:40 M:197 C:52 M:194 C:17 M:186 C:47 M:175 C:15 M:180 C:22 M:160 C:23 M:135 C:28 M:158 C:32 M:209 C:23 M:169 C:14 M:184 C:18 M:146 C:16 M:151 C:20 M:140 C:10 M:115 C:20 M:142
	Incorrect Interpretation 	C:1 M:50 	C:6 M:30 C:2 M:36 C:8 M:49 C:7 M:38 C:3 M:31 C:4 M:30 C:6 M:36 C:5 M:49 C:6 M:52 C:4 M:46 C:4 M:38 C:7 M:35 C:4 M:39 C:4 M:29 C:3 M:30 C:7 M:34 C:8 M:25 C:6 M:28 C:7 M:42
	Interruption Error 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Irrelevant Tool Call 	C:0 M:0 	C:0 M:1 	C:0 M:1 	C:0 M:4 	C:0 M:0 	C:0 M:0 	C:0 M:1 	C:0 M:1 	C:0 M:8 	C:0 M:3 	C:0 M:10 	C:0 M:0 	C:0 M:1 	C:0 M:3 	C:0 M:0 	C:0 M:1 	C:0 M:0 	C:0 M:1 	C:0 M:3 	C:0 M:3
	Missed Required Action

:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0 	C:0 M:0
	Irrelevant Tool Call 	C:0 M:0 	C:0 M:1 	C:0 M:1 	C:0 M:4 	C:0 M:0 	C:0 M:0 	C:0 M:1 	C:0 M:1 	C:0 M:8 	C:0 M:3 	C:0 M:10 	C:0 M:0 	C:0 M:1 	C:0 M:3 	C:0 M:0 	C:0 M:1 	C:0 M:0 	C:0 M:1 	C:0 M:3 	C:0 M:3
	Missed Required Action 	C:3 M:24 	C:3 M:9 	C:6 M:16 C:1 M:22 C:3 M:12 C:2 M:29 C:2 M:11 C:2 M:30 C:2 M:21 C:5 M:16 C:9 M:11 C:3 M:22 	C:2 M:8 	C:4 M:17 C:1 M:21 	C:6 M:8 	C:2 M:12 	C:1 M:9 	C:3 M:14 	C:2 M:9
	Other 	C:1 M:10 	C:1 M:9 	C:0 M:13 	C:1 M:9 	C:0 M:28 C:0 M:15 C:0 M:15 	C:1 M:3 	C:0 M:12 C:0 M:32 C:0 M:11 	C:1 M:9 	C:0 M:16 C:1 M:10 C:1 M:10 C:1 M:11 C:0 M:33 C:0 M:20 	C:1 M:8 	C:0 M:27
	Premature Termination 	C:13 M:16 C:9 M:24 C:12 M:20 C:10 M:18 C:14 M:21 C:17 M:16 C:5 M:23 C:3 M:14 C:12 M:20 C:8 M:15 C:11 M:14 C:11 M:15 C:12 M:17 C:15 M:14 C:5 M:16 C:11 M:23 C:17 M:12 C:14 M:29 C:9 M:18 C:11 M:31
	Revealed Info Early 	C:0 M:0 	C:0 M:0 	C:0 M:2 	C:0 M:1 	C:1 M:3 	C:0 M:3 	C:0 M:1 	C:0 M:1 	C:0 M:0 	C:0 M:4 	C:0 M:15 	C:1 M:1 	C:1 M:1 	C:0 M:0 	C:0 M:0 	C:1 M:0 	C:1 M:1 	C:1 M:2 	C:0 M:2 	C:0 M:1
	Tool Call Argument Error C:0 M:3 	C:2 M:7 	C:0 M:2 	C:2 M:1 	C:1 M:0 	C:1 M:2 	C:0 M:10 	C:0 M:5 	C:1 M:9 	C:0 M:1 	C:0 M:2 	C:1 M:4 	C:4 M:8 	C:0 M:1 	C:0 M:1 	C:0 M:5 	C:0 M:2 	C:0 M:1 	C:0 M:3 	C:0 M:2
	Tool Call Schema Error 	C:0 M:1 	C:5 M:6 	C:0 M:0 	C:0 M:2 	C:0 M:8 	C:0 M:8 	C:0 M:3 	C:0 M:6 	C:0 M:5 	C:0 M:2 	C:0 M:4 	C:0 M:3 	C:0 M:1 	C:0 M:5 	C:0 M:1 	C:0 M:0 	C:0 M:1 	C:0 M:22 	C:1 M:1 	C:0 M:2
	Wrong Sequence 	C:0 M:0 	C:0 M:0 	C:0 M:11 	C:4 M:6 	C:2 M:5 	C:0 M:4 	C:0 M:7 	C:1 M:2 	C:0 M:6 	C:2 M:3 	C:0 M:3 	C:0 M:10 	C:0 M:1 	C:0 M:5 	C:0 M:2 	C:0 M:2 	C:1 M:1 	C:0 M:1 	C:0 M:1 	C:0 M:1
Table 22: User error tag counts — all settings, Telecom domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=342 per condition). C: = critical-severity count; M: = minor-severity count. Each
value is the number of simulations in which the tag

M:2 	C:0 M:2 	C:1 M:1 	C:0 M:1 	C:0 M:1 	C:0 M:1
Table 22: User error tag counts — all settings, Telecom domain (gpt-5-mini agent, qwen3-235b user simulator,
DeepSeek-V4-Flash judge, n=342 per condition). C: = critical-severity count; M: = minor-severity count. Each
value is the number of simulations in which the tag appeared at that severity level at least once.
23

-- 23 of 23 --