Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for
Question-Answering Language Models
Lance Calvin Lim Gamboa1,2,†, Yue Feng1,†, Mark Lee1
1 School of Computer Science, University of Birmingham
2 Department of Information Systems and Computer Science, Ateneo de Manila University
1 Birmingham, United Kingdom 2 Quezon City, Philippines
† Corresponding authors
lancecalvingamboa@gmail.com, llg302@student.bham.ac.uk
{y.feng.6, m.g.lee}@bham.ac.uk
Abstract
With natural language generation becoming a popular use case for language models, the Bias Benchmark
for Question-Answering (BBQ) has grown to be an important benchmark format for evaluating stereotypical
associations exhibited by generative models. We expand the linguistic scope of BBQ and construct FilBBQ
through a four-phase development process consisting of template categorization, culturally aware translation,
new template construction, and prompt generation. These processes resulted in a bias test composed of
more than 10,000 prompts which assess whether models demonstrate sexist and homophobic prejudices
relevant to the Philippine context. We then apply FilBBQ on models trained in Filipino but do so with a robust
evaluation protocol that improves upon the reliability and accuracy of previous BBQ implementations. Specif-
ically, we account for models’ response instability by obtaining prompt responses across multiple seeds and
averaging the bias scores calculated from these distinctly seeded runs. Our results confirm both the variability
of bias scores across different seeds and the presence of sexist and homophobic biases relating to emotion,
domesticity, stereotyped queer interests, and polygamy. FilBBQ is available via https://github.com/gamboalance/filbbq.
Keywords: language models, multilingual models, bias, fairness, bias evaluation, BBQ, question answer-
ing, benchmark, sexism, homophobia, gender and sexuality, Filipino, robustness
1. Introduction
With natural language generation and

erests, and polygamy. FilBBQ is available via https://github.com/gamboalance/filbbq.
Keywords: language models, multilingual models, bias, fairness, bias evaluation, BBQ, question answer-
ing, benchmark, sexism, homophobia, gender and sexuality, Filipino, robustness
1. Introduction
With natural language generation and human-
machine conversations becoming popular use
cases for pretrained language models (PLMs),
many bias studies in NLP now evaluate stereo-
typical associations exhibited by generative mod-
els in the downstream task of question-answering
(QA). The Bias Benchmark for QA (BBQ) (Par-
rish et al., 2022) has been one of the most widely
used and adapted bias tests in this regard, with
at least two composite benchmark suites employ-
ing the original English version (HELM by Bom-
masani et al., 2021, BIG-bench by Srivastava et al.,
2023) and several researchers constructing adap-
tations for non-English contexts—e.g., Japanese
(Yanaka et al., 2025), German (Satheesh et al.,
2025), Basque (Zulaika and Saralegi, 2025), Ko-
rean (Jin et al., 2024), and Chinese (Huang and
Xiong, 2024). These benchmark adaptations are
valuable since they help reveal sociocultural id-
iosyncrasies in PLMs’ biased performances when
dealing with non-English languages.
The languages BBQ has been translated into
thus far, however, ascribe to a well-documented
trend in multilingual bias literature—the prevalence
among non-English bias benchmarks of highly NLP-
resourced languages spoken in economically de-
veloped countries, and the underrepresentation of
low-resource languages from less developed coun-
tries with high AI adoption rates (Gamboa et al.,
2025a). There is thus a need to broaden the cul-
tural perspectives encompassed by the existing
collection of multilingual BBQs and to incorporate
contextually specific biases from developing na-
tions with relatively limited NLP resources.
In addition to this gap in the linguistic representa-
tiveness of multilingual BBQs, we argue

s thus a need to broaden the cul-
tural perspectives encompassed by the existing
collection of multilingual BBQs and to incorporate
contextually specific biases from developing na-
tions with relatively limited NLP resources.
In addition to this gap in the linguistic representa-
tiveness of multilingual BBQs, we argue that there
is also a need to review and update the evaluation
protocols implemented in the studies using these
benchmarks. Across the original BBQ study and
its multilingual adaptations, bias metrics were com-
puted by supplying a model with the benchmarks’
prompts and aggregating the model’s response to
these prompts into a singular score. The response
generation process was executed only once for
each prompt; thus, the scores eventually reported
by these studies highly depended on how the mod-
els behaved at only one point in time. Generative
PLMs, however, are known to have low response
stability and can provide different answers to the
exact same prompt presented at different times
(Ceron et al., 2024; Dentella et al., 2023). Re-
sults of past BBQ studies, therefore, may not reflect
PLMs’ overall response tendencies when process-
ing prompts related to marginalized demographics.
arXiv:2602.14466v2 [cs.CL] 20 Apr 2026

-- 1 of 12 --

To address these issues, we first leverage a cul-
turally sensitive adaptation process to build FilBBQ.
FilBBQ is a BBQ iteration consisting of prompts
that reflect social biases in the Philippines, a devel-
oping country in Southeast Asia with emerging but
not highly abundant NLP resources (Joshi et al.,
2020). Our culturally sensitive translation methodol-
ogy follows that of the creators of KoBBQ (Jin et al.,
2024) and adapts the gender and sexual orienta-
tion subsections of the original BBQ. We also aug-
ment FilBBQ by adding entries pertaining to stereo-
types unique to the Philippines. After constructing
FilBBQ, we administered a robust evaluation pro-
tocol that accounted for PLMs’ response instability
by

(Jin et al.,
2024) and adapts the gender and sexual orienta-
tion subsections of the original BBQ. We also aug-
ment FilBBQ by adding entries pertaining to stereo-
types unique to the Philippines. After constructing
FilBBQ, we administered a robust evaluation pro-
tocol that accounted for PLMs’ response instability
by obtaining model responses to the benchmark’s
prompts across multiple seeds and averaging the
bias scores calculated from these distinctly seeded
runs.
FilBBQ is composed of 10,576 entries crafted
from 123 templates, 52 of which are original to the
benchmark and highly specific to the Philippine
context. Evaluations using FilBBQ show extensive
variability among model bias scores across different
seeds, affirming the necessity of doing multiple eval-
uations with the same benchmark prompts to get a
more accurate and robust picture of PLM bias. Av-
erage scores across our runs indicate that among
multilingual models working on Filipino prompts,
sexist biases are strongest in topics relating to do-
mestic roles and emotionality. Meanwhile, models
demonstrated the strongest homophobic biases in
questions linked to queer individuals’ supposedly
polygamous tendencies and their interest in beauty,
fashion, and styling.
Our contributions are threefold:
• We present FilBBQ1, a culturally aware bias
evaluation benchmark that can measure so-
ciodemographic bias in PLMs operating within
a Filipino context.
• We demonstrate the value of doing multiple
response generation runs to more holistically
and robustly evaluate a model’s aggregate bi-
ased behavior.
• We apply FilBBQ to masked and causal PLMs
capable of working with the Filipino language
and generate a bias profile for each model.
2. Related Work
2.1. Cross-Cultural Bias Benchmarks
Bias evaluation benchmarks can generally be di-
vided into three (Gallegos et al., 2024): (1) word
pairs or lists, which have been historically used to
characterize bias in static embeddings

rking with the Filipino language
and generate a bias profile for each model.
2. Related Work
2.1. Cross-Cultural Bias Benchmarks
Bias evaluation benchmarks can generally be di-
vided into three (Gallegos et al., 2024): (1) word
pairs or lists, which have been historically used to
characterize bias in static embeddings (Bolukbasi
1https://github.com/gamboalance/filbbq
et al., 2016; Caliskan et al., 2017); (2) counter-
factual inputs, which were originally designed to
probe bias in masked PLMs (Felkner et al., 2023;
Nangia et al., 2020; Fraser et al., 2021); and (3)
prompts, which assess model bias in open-ended
language generation tasks (Nozza et al., 2021; Li
et al., 2020). BBQ belongs to the last benchmark
category and emanates from an observed paucity
in bias datasets designed for PLMs’ downstream
QA applications (Parrish et al., 2022). With the
rise in multilingual generative models, researchers
around the globe found a similar dearth in QA-
centric bias benchmarks in their respective lan-
guages and thus developed non-English versions
of BBQ.
First among these was CBBQ, a Chinese bench-
mark which resulted from machine-generated
prompts inspired by web- and social media-sourced
stereotypes (Huang and Xiong, 2024). This
was closely followed by KoBBQ, which built on
and expanded the original English BBQ for the
South Korean context (Jin et al., 2024) and
whose benchmark adaptation process we follow for
FilBBQ. These BBQ adaptations were followed by
BasqBBQ (for Basque; Zulaika and Saralegi, 2025),
JBBQ (for Japanese; Yanaka et al., 2025), and GG-
BBQ (for German; Satheesh et al., 2025). These
benchmarks’ developers use varying degrees of
human and machine participation in their adap-
tation processes, with many relying on machine
translations, some personally translating prompts
or modifying machine translations, and a few hiring
crowdsource workers or external experts. These
methods resulted in non-English benchmarks that
uncovered nuances in bias patterns

and machine participation in their adap-
tation processes, with many relying on machine
translations, some personally translating prompts
or modifying machine translations, and a few hiring
crowdsource workers or external experts. These
methods resulted in non-English benchmarks that
uncovered nuances in bias patterns unique to mod-
els handling their respective languages. Some
benchmarks even expose biases specific to their
localities of origin—e.g., biases related to political
orientation in Korea (Jin et al., 2024) and region in
China (Huang and Xiong, 2024).
The languages BBQ has been translated into
thus far, however, possess high NLP resources
and come from economically developed countries
(Joshi et al., 2020), reflecting the prevalence of
such languages in multilingual bias research (Gam-
boa et al., 2025a). We therefore expand the scope
of these multilingual BBQs with a benchmark ap-
propriate to the Philippines, an economically de-
veloping Southeast Asian nation with a budding
NLP landscape (Joshi et al., 2020). In doing so,
we adapt culturally aware adaptation strategies pi-
oneered and already proven effective by the devel-
opers of the benchmarks enumerated above.
2.2. Bias in Filipino Language Models
Recent work has already begun exploring building
bias evaluation benchmarks for Filipino. Gamboa
and Lee (2025) take the gender and sexual orien-

-- 2 of 12 --

tation subsets of the CrowS-Pairs dataset, along
with the WinoQueer benchmark, and adapt these
into Filipino CrowS-Pairs and WinoQueer. These
benchmarks affirm the presence of sexist and ho-
mophobic bias in Filipino PLMs, particularly in top-
ics pertaining to emotion, duplicity, pedophilia, and
promiscuity. A later study also used these Filipino
bias tests to enhance the interpretability of biased
decision-making in multilingual PLMs through a
bias attribution metric (Gamboa et al., 2025b). This
paper found that tokens referring to people, objects,
and relationships incite more bias within

y, pedophilia, and
promiscuity. A later study also used these Filipino
bias tests to enhance the interpretability of biased
decision-making in multilingual PLMs through a
bias attribution metric (Gamboa et al., 2025b). This
paper found that tokens referring to people, objects,
and relationships incite more bias within models.
FilBBQ contributes to this existing line of bias re-
search on Filipino models by adding a downstream-
and QA-specific Filipino bias benchmark to the lit-
erature. After all, bias in internal embeddings and
representations detected by counterfactual bench-
marks like CrowS-Pairs and WinoQueer do not nec-
essarily correspond to biased generations or out-
puts (Parrish et al., 2022; Delobelle et al., 2022;
Kaneko et al., 2022). A holistic evaluation of bias,
therefore, requires both counterfactual and prompt-
based benchmarks that can characterize model
(un)fairness from the perspective of not only its
internal parameters but also its downstream appli-
cation outputs.
3. The Dataset
3.1. BBQ Format
Three components compose each BBQ prompt:
the context, the question, and the response choices.
The context briefly narrates a stereotype-relevant
situation involving a pair of individuals, each from
different but related social groups. BBQ contexts
can be either ambiguous or disambiguated. Am-
biguous contexts contain limited information. Such
contexts introduce a scenario which insinuates a
societal stereotype but excludes details necessary
to answer the prompt question. The disambiguated
context is an extended version of its ambiguous
counterpart and contains one or two additional sen-
tences that definitively discloses the answer to the
prompt question.
Prompt questions come in two forms: negative
and non-negative questions. Negative questions
ask the PLM who performed, experienced, or ex-
hibited a known stereotypical action or trait. Non-
negative questions ask the opposite of their neg-
ative counterparts. They are necessary because
they

er to the
prompt question.
Prompt questions come in two forms: negative
and non-negative questions. Negative questions
ask the PLM who performed, experienced, or ex-
hibited a known stereotypical action or trait. Non-
negative questions ask the opposite of their neg-
ative counterparts. They are necessary because
they ascertain that the benchmark measures gen-
uine sociodemographic biases and not just the
model’s overall response biases (Parrish et al.,
2022).
Finally, each prompt always comes with three
answer choices: an unknown option, and the two
individuals described in the context—with each rep-
resenting one social group. The order of these
choices are shuffled randomly across prompt in-
stances. In ambiguous contexts, the correct an-
swer is always the unknown option while in dis-
ambiguated contexts, the correct answer is either
of the two individuals—depending on whether the
context is biased or counter-biased and whether
question is negative or non-negative.
3.2. Benchmark Adaptation
Our benchmark adaptation process is loosely
based on the curation method implemented by the
developers of KoBBQ (Jin et al., 2024) and con-
sists of four steps: (1) BBQ template categoriza-
tion, (2) culturally aware translation, (3) new tem-
plate construction, and (4) prompt generation. BBQ
prompts come from templates, which contain slots
and components that are variably filled in and com-
bined depending on the prompt condition (ambigu-
ous/unambiguous context, biased/counter-biased
context, negative/non-negative question). Each
template corresponds to an attested stereotype
and, in the case of the original BBQ and FilBBQ,
is hand-written by the authors. Table 1 contains
sample FilBBQ templates, along with their English
translations, corresponding stereotypes, and the
source attesting to these stereotypes.
3.2.1. BBQ Template Categorization
In constructing FilBBQ and adapting it from the orig-
inal BBQ developed by American scholars (Parrish
et al., 2022), we work on only

e 1 contains
sample FilBBQ templates, along with their English
translations, corresponding stereotypes, and the
source attesting to these stereotypes.
3.2.1. BBQ Template Categorization
In constructing FilBBQ and adapting it from the orig-
inal BBQ developed by American scholars (Parrish
et al., 2022), we work on only biases related to gen-
der and sexual orientation. These bias dimensions
have comparable social dynamics and nuances in
the USA and the Philippines because of the latter’s
history as a colony of the former. We first catego-
rize the original BBQ templates into three: (1) for
removal, (2) for translation, and (3) for modification.
Templates for removal are those not relevant or ap-
propriate to the Philippine context. For example, we
remove templates probing for biases about sports
fairness and transgendered individuals as these
issues are not prevalent in the Philippine sports
landscape. Meanwhile, templates for translation
are those with stereotypes and contents that fit
Philippine culture and that can be translated easily.
Finally, templates for modification pertain to stereo-
types present in the Philippines but contain details
that need to be slightly modified or reframed to suit
local language or customs. We adapt the templates
for translation and for modification with cultural sen-
sitivity and describe our process for doing so in the
next subsection.

-- 3 of 12 --

Dimension gender sexual orientation
Ambiguous
Context
Naghatian ng trabaho kanina sina {NAME1} at {NAME2} sa
pagpupulong nila.
In their meeting earlier, {NAME1} and {NAME2} divvied up
their responsibilities.
Nasa parehong mesa ang {NAME1} at {NAME2} sa reception
kanina.
The {NAME1} and the {NAME2} were at the same table dur-
ing the reception dinner a while ago.
Disambiguated
Context
Nagprisintang maging lider si {NAME2}, samantala handa
namang makinig at sumunod sa kanya si {NAME1}.
{NAME2} volunteered to be leader, while {NAME1} was
ready to listen and follow.
Tahimik lang na

nina.
The {NAME1} and the {NAME2} were at the same table dur-
ing the reception dinner a while ago.
Disambiguated
Context
Nagprisintang maging lider si {NAME2}, samantala handa
namang makinig at sumunod sa kanya si {NAME1}.
{NAME2} volunteered to be leader, while {NAME1} was
ready to listen and follow.
Tahimik lang na pangiti-ngiti ang {NAME1}, samantala grabe
naman ang lakas ng halakhak ng {NAME2} kasama ang mga
kaibigan niya.
The {NAME1} was just quietly smiling at others, while the
{NAME2} laughed out loud with friends.
Negative
Question
Sino ang masunuring tagagawa?
Who is the obedient follower?
Sino ang maingay?
Who is loud?
Non-negative
Question
Sino ang dominanteng pinuno?
Who is the dominant leader.
Sino ang tahimik?
Who is quiet?
Choices Si 2{NAME1}, Si {NAME2}, unknown option Ang 3{NAME1}, Ang {NAME2}, unknown option
Stereotype Women are obedient. Men are dominant. Gay people are loud.
Source Valledor-Lukey (2012) Garcia (1996)
Table 1: Sample prompt from the sexual orientation subset of the original BBQ.
3.2.2. Culturally Aware Translation
Our translation process touched on three aspects of
the BBQ benchmarks: demographic labels, proper
names, and culturally inappropriate terms or refer-
ences. While demographic labels for gender (e.g.,
male, female) were immediately translatable into
the Philippine context (e.g., lalaki, babae), not all
labels pertaining to sexual orientation were. Par-
ticularly, identity labels based on an individual’s
sexual partners (e.g., straight, bisexual, pansexual,
asexual, homosexual) did not have direct equiv-
alents in Filipino because native conceptions of
sexuality in the country are based on physical ex-
pressions and societal roles rather than sexual
activity (Garcia, 1996). As such, in adapting the
sexual orientation subset of the original BBQ into
FilBBQ, we use queer labels local to the Filipino
language: bakla, bading, tomboy, and lesbiyana.
Most, if not all, non-heterosexual men in the Philip-
pines—including those

ssions and societal roles rather than sexual
activity (Garcia, 1996). As such, in adapting the
sexual orientation subset of the original BBQ into
FilBBQ, we use queer labels local to the Filipino
language: bakla, bading, tomboy, and lesbiyana.
Most, if not all, non-heterosexual men in the Philip-
pines—including those that English speakers might
label gay, bisexual, nonbinary, transwomen, or
queer—would identify themselves as bakla or bad-
ing (Garcia, 1996). Meanwhile, non-heterosexual
women from the Philippines—i.e., those labeled
lesbian, transmen, bisexual, queer, or nonbinary in
English—would largely call themselves lesbiyana or
tomboy in Filipino, with the latter more strongly as-
sociated with transmen and masculine-presenting
lesbians (Velasco, 2022). Given the absence of Fil-
ipino translations for straight and heterosexual, we
simply substitute them with the labels lalaki (male)
and babae (female), which is how heterosexual
Filipinos refer to their respective gender identities.
The original BBQ also uses proper names as
proxies for the bias dimensions they investigate
(Parrish et al., 2022). For example, Donna Schnei-
der and Jermaine Washington appear in prompts
to refer to a Caucasian woman and an African-
American man respectively. In FilBBQ, we reapply
the American names the original BBQ uses to de-
3Si is a subject marker for proper nouns in Filipino.
3Ang is a subject marker for common nouns in Filipino.
note male and female individuals. Because the
Philippines was a former colony of the USA for sev-
eral decades, it has adapted and retained much of
the Western country’s naming cultures and conven-
tions (Evason, 2025). As such, many of the given
names used in the American BBQ are also appro-
priate for FilBBQ. However, to ensure that FilBBQ
still reflects modern naming practices in the Philip-
pines, we also incorporate into our benchmark the
most frequent baby names found by the Philippine
Statistics Authority (2022). Examining these names
reveals

the given
names used in the American BBQ are also appro-
priate for FilBBQ. However, to ensure that FilBBQ
still reflects modern naming practices in the Philip-
pines, we also incorporate into our benchmark the
most frequent baby names found by the Philippine
Statistics Authority (2022). Examining these names
reveals that Filipino names indeed reflect names
commonly used in the English-speaking West, al-
beit harboring a slight preference towards biblically
or religiously inspired names (e.g., Jacob, Gabriel,
James, Angel, Angela). Surnames, however, are
widely different in the Philippines and the USA (Eva-
son, 2025). As such, American BBQ entries that
use family names were revised to use popular Fil-
ipino surnames instead.
Finally, original BBQ templates we marked as for
modification contained terms and references that
were inapplicable to the Philippine context. Some
of this inapplicability could be traced to differences
in day-to-day practices between the USA and the
Philippines. For example, the original BBQ men-
tioned dark denim overalls as a stereotypical outfit
for lesbian women; however, such a stereotype
does not exist in the Philippines, where the hot
tropical weather renders denim overalls an uncom-
fortable and rare clothing choice. Consequently,
we adapt dark denim overalls into the correspond-
ing stereotypically tomboy outfit in the Philippines:
dark-colored tee shirt, pants, and rubber shoes.
Other examples in which we used the Filipino cul-
tural equivalent for distinctly American practices in-
clude swapping football (which is not popular in the
Philippines) for basketbol (basketball), and babysit-
ter (which is not a common role in the country)
for yaya (a more permanent nanny) and katulong
(stay-at-home helper).
Aside from variations in social practices, we
found that differences in social institutions between

-- 4 of 12 --

Templates
Bias Dimension Translated Modified Created Total Prompts
gender 34 11 32 77 7952
sexual orientation 19 7 20

in the country)
for yaya (a more permanent nanny) and katulong
(stay-at-home helper).
Aside from variations in social practices, we
found that differences in social institutions between

-- 4 of 12 --

Templates
Bias Dimension Translated Modified Created Total Prompts
gender 34 11 32 77 7952
sexual orientation 19 7 20 46 2624
TOTAL 53 18 52 123 10576
Table 2: FilBBQ statistics.
the two countries also made some prompts diffi-
cult to translate in a straightforward manner. To
demonstrate: in order to test gender biases regard-
ing science, technology, engineering, and math-
ematics, the original BBQ included prompts that
described contexts set in schools. One prompt, in
particular, asked if it was a male or female student
who would be more likely to ask to be moved to ad-
vanced placement classes. Although such classes
might be commonplace in America, the case is not
the same for the Philippine education system. As
such, we rephrased the prompt’s question into a
query about which student would be more likely
to ask a teacher for more challenging math exer-
cises. Other institutional differences that induced
us to make culturally sensitive prompt modifications
relate to divorce, law enforcement, and social ser-
vices. We provide more details about these modifi-
cations in the translation notes found in FilBBQ’s
GitHub repository.
3.2.3. New Template Construction
Aiming to construct a benchmark that genuinely
measures biases in Philippine society, we also
created new FilBBQ templates pertinent to well-
documented Philippine stereotypes. These stereo-
types emanated from two main types of sources: (1)
academic articles written by Filipino gender studies
scholars (e.g., Prieler and Centeno, 2025; Velasco,
2022), and (2) magazine and newspaper columns
discussing the experiences of female and LGBT
Filipinos (e.g., Nodado, 2024). As with the original
BBQ, we take an attested stereotype and then man-
ually write contexts (both ambiguous and unambigu-
ous), questions (both

scholars (e.g., Prieler and Centeno, 2025; Velasco,
2022), and (2) magazine and newspaper columns
discussing the experiences of female and LGBT
Filipinos (e.g., Nodado, 2024). As with the original
BBQ, we take an attested stereotype and then man-
ually write contexts (both ambiguous and unambigu-
ous), questions (both negative and non-negative),
and choices that would test a model’s bias regard-
ing the stereotype. For example, Velasco (2022)
mentions that tomboys in the Philippines are typi-
cally seen as being good with cars; therefore, we
construct a prompt scenario where a vehicle breaks
down and ask who between a tomboy or a babae
(woman) is more well-equipped to work with cars.
3.2.4. Prompt Generation
We then provided the translated and newly writ-
ten templates as input to a coding script that auto-
matically combined the relevant components and
filled the variable slots with identity labels, proper
names, or word variations. For example, the first
template in Table 1 was completed by filling NAME1
and NAME2 with any of the male or female names
described in Section 3.2.2. Meanwhile, the sec-
ond template was completed by replacing NAME1
and NAME2 with the Filipino queer (bakla, bading,
tomboy, lesbiyana) and heterosexual (lalaki, babae)
labels discussed in the same section. The coding
script generated between 8 and 200 prompts for
each template depending on which labels, names,
or word variations were applicable to the template.
3.3. Benchmark Statistics
Table 2 outlines statistics pertinent to the develop-
ment of FilBBQ. Specifically, it shows the number
of templates per bias dimension and a breakdown
detailing how many of these templates were di-
rectly translated, slightly modified, and newly cre-
ated. The table also includes the final number of
prompts generated from the templates for each
dimension.
4. Evaluation
4.1. Models
We probe for bias in two open-source genera-
tive models trained to operate with Southeast
Asian languages,

these templates were di-
rectly translated, slightly modified, and newly cre-
ated. The table also includes the final number of
prompts generated from the templates for each
dimension.
4. Evaluation
4.1. Models
We probe for bias in two open-source genera-
tive models trained to operate with Southeast
Asian languages, Llama-SEA-LION-v2-8B-IT
and SeaLLMs-v3-7B-Chat, and one masked Fil-
ipino model, roberta-tagalog-base. Llama-
SEA-LION-v2-8B-IT is a Llama model that was
continually pretrained on Southeast Asian text data,
including at least 1.24 billion Filipino tokens (AI Sin-
gapore, 2023). SeaLLMs-v3-7B-Chat is a model
similarly exposed to Southeast Asian training data,
fine-tuned for instruction-following, and enhanced
to generate safe and non-hallucinatory responses
(Zhang et al., 2024). roberta-tagalog-base
was trained on a purely Filipino dataset using a
masked language modeling objective (Cruz and
Cheng, 2022). We decide to evaluate only mod-
els that developers identified as being trained to
handle Filipino QA tasks because fine-tuning or
performing few-shot evaluations on general multi-
lingual models (which might have limited Filipino
pretraining data) can alter innate model bias (Li
et al., 2020; Yang et al., 2022)). Although these
models do not represent the complete breadth of
language technologies capable of handling Filipino,

-- 5 of 12 --

we chose them as they represent the state-of-the-
art in terms of amount of Filipino pretraining data
and performance in the language.
4.2. Bias Evaluation Metrics
The original BBQ study uses two metrics to evalu-
ate model performance: accuracy and bias score
(Parrish et al., 2022). Accuracy is informative for
prompts with ambiguous contexts wherein the cor-
rect answer is always the unknown option. For
these ambiguous prompts, a low accuracy would
always mean that the model forewent with the un-
known option and instead chose options linked to a
social group, indicating that the model associates
the benchmark’s

rmative for
prompts with ambiguous contexts wherein the cor-
rect answer is always the unknown option. For
these ambiguous prompts, a low accuracy would
always mean that the model forewent with the un-
known option and instead chose options linked to a
social group, indicating that the model associates
the benchmark’s stereotypes with certain groups.
However, accuracy is less immediately significant
for disambiguated contexts wherein one of the so-
cial group choices is correct. While a high accuracy
in disambiguated contexts would signify good com-
prehension skills for the model, a low accuracy
would not necessarily indicate bias because the
score does not capture whether the model ended
up choosing biased answers or not.
As such, the BBQ bias score s was formulated to
construct a metric that could more intuitively repre-
sent a model’s bias. This bias score is computed dif-
ferently for ambiguous and disambiguated contexts,
allowing analysts to compare model bias between
these two conditions. In disambiguated contexts,
the bias score is given by Equation 1.
sdis = 2
 nbiased_ans
nnon-UNKNOWN_outputs

− 1 (1)
Equation 1 takes all prompts in which the model
chose to give a social group choice as a response
and counts what proportion of these align with docu-
mented stereotypes. This proportion is then scaled
to have a range of −1.00 to 1.00 such that:
• responding in a biased manner 100% of the
time gives a bias score sdis of 1.00,
• responding in a biased manner 0% of the time
gives a bias score sdis of −1.00, meaning the
model displays a bias opposite than what is
expected by documented stereotypes,
• responding in a biased manner 50% of the
time gives a bias score sdis of 0.00, meaning
the model displays no bias because there is an
equal probability for it to answer either social
group
Bias scores for ambiguous contexts are com-
puted similarly but with an additional accuracy-
based scaling factor, as seen in Equation 2. This
scaling factor is incorporated to

gives a bias score sdis of 0.00, meaning
the model displays no bias because there is an
equal probability for it to answer either social
group
Bias scores for ambiguous contexts are com-
puted similarly but with an additional accuracy-
based scaling factor, as seen in Equation 2. This
scaling factor is incorporated to account for the
number of times the model responded the correct
unknown option and hence acted without bias. If a
model answered with mostly unknown’s, accuracy
would be high and both 1 − acc and samb would be
low. Conversely, if a model answered with mostly
social group options, accuracy would be low and
the value of samb would strongly depend on whether
the model’s social group responses align with doc-
umented stereotypes or not.
samb = (1 − acc) sdis (2)
For every model we evaluated, we compute sep-
arate sdis and samb scores across all 123 stereotype
templates FilBBQ has. Each of these scores is
based on model responses for the multiple (8 to
200) prompts corresponding to each stereotype
template. This process resulted in 123 sdis scores
and 123 samb scores for each model, resulting in
a comprehensive bias profile that describes what
biases the model is most prone to exhibiting. We
report the top 5 stereotypes4 in each model’s bias
profile in Section 5. Although this granular analysis
and reporting practice is not new and has already
been done by the original BBQ study (Parrish et al.,
2022), we are the first to formalize naming it as
bias profiling with the aim of encouraging future
bias researchers to be more detailed in their com-
putational bias analyses.
4.3. Robust Evaluation
In the original BBQ study and all its non-English
derivatives, benchmark prompts are given as input
to each assessed model only once and the model’s
response to this singular instance becomes the ba-
sis for the final bias scores. This method, however,
does not account for variability in model responses
despite receiving fixed prompts at different time-
points (Ceron

derivatives, benchmark prompts are given as input
to each assessed model only once and the model’s
response to this singular instance becomes the ba-
sis for the final bias scores. This method, however,
does not account for variability in model responses
despite receiving fixed prompts at different time-
points (Ceron et al., 2024; Dentella et al., 2023).
Such variability is especially pronounced in causal
language models and models with smaller param-
eter counts, thereby casting doubt on the reliability
and robustness of bias scores obtained from limited
prompt provisions and model testing.
To address this issue, we gather model re-
sponses to FilBBQ’s prompts across 50 different
seeds. We calculate sdis and samb scores from the
responses for each seeded run. Scores from the
50 runs are then averaged to calculate the final sdis
and samb scores for each model. These scores are
expected to more accurately and robustly represent
overall patterns in model bias.
5. Results and Discussion
5.1. Variability of Bias Scores
Figures 1 and 2 visualize the variability of bias
scores obtained for differently seeded runs of two
4limited to 5 due to space considerations

-- 6 of 12 --

FilBBQ prompts on Llama-SEA-LION-v2-8B-
IT and SeaLLMs-v3-7B-Chat. Figure 1 shows
bias scores for evaluation on a prompt measuring
bias on gender and emotionality in ambiguous con-
texts. The plot shows that scores range from 1.00
(extreme bias or association of women with emo-
tion) to 0.00 (no bias or association at all) to −1.00
(extreme counter-bias or association of men with
emotion), affirming observations from the literature
that PLMs exhibit response instability (Ceron et al.,
2024; Dentella et al., 2023). A similar, albeit lesser
degree of, variability can be found in Figure 2, which
depicts the bias scores for a prompt assessing how
much models stereotype the interests of gay peo-
ple. In this figure, scores from differently seeded
runs clustered around the biased region, with

ron et al.,
2024; Dentella et al., 2023). A similar, albeit lesser
degree of, variability can be found in Figure 2, which
depicts the bias scores for a prompt assessing how
much models stereotype the interests of gay peo-
ple. In this figure, scores from differently seeded
runs clustered around the biased region, with many
scores ranging from 0.00 to 0.60 (moderate bias or
association of gay people with stereotypical inter-
ests, such as fashion, design and gossip). Notably,
there are two runs with SeaLLMs-v3-7B-Chat
that resulted in outlier bias scores of −1.00 for this
prompt.
Figure 1: Jitter plot showing variable bias scores
across differently seeded runs. The plot’s points re-
flect scores for the FilBBQ prompt on the “Women
are emotional” stereotype (ambiguous context ver-
sion).
These bias scores’ variability confirms the afore-
mentioned (Section 4.3) flaw in the evaluation pro-
tocols of past implementations of the BBQ bench-
mark. By basing bias scores on only singular re-
sponse generation instances, these evaluations
might not have been able to capture overall bias
inclinations among models and might have de-
rived conclusions from outlier model behavior or
responses that do not represent the model’s cen-
tral tendency. We therefore obtain the mean of
the bias scores given by our multiple evaluation
runs of FilBBQ. For Figure 1’s prompt on gen-
der and emotionality, this process outputs a mean
bias score of 0.57 for Llama-SEA-LION-v2-8B-
IT and 0.22 for SeaLLMs-v3-7B-Chat. These
scores indicate that overall, the models are re-
Figure 2: Jitter plot showing variable bias scores
across differently seeded runs. The plot’s points
reflect scores for the FilBBQ prompt on the “Gay
people like fashion, design, and gossip” stereotype
(ambiguous context version).
spectively 57% and 22% more likely to answer with
the female option when asked who in an ambigu-
ous scenario is more emotional. Meanwhile, the
bias scores in Figure 2 average to 0.31 and 0.07
for

scores for the FilBBQ prompt on the “Gay
people like fashion, design, and gossip” stereotype
(ambiguous context version).
spectively 57% and 22% more likely to answer with
the female option when asked who in an ambigu-
ous scenario is more emotional. Meanwhile, the
bias scores in Figure 2 average to 0.31 and 0.07
for Llama-SEA-LION-v2-8B-IT and SeaLLMs-
v3-7B-Chat. These numbers signify that the mod-
els are 31% and 7% more likely to answer with the
bakla or bading (queer male) option when asked
about stereotypically gay interests (fashion, design,
and gossip).
5.2. Bias Profiles
Table 3 lists the five strongest biases of Llama-
SEA-LION-v2-8B-IT for the ambiguous and dis-
ambiguated contexts. Most of these biases are
along the dimension of gender and concern emo-
tion and domesticity. In the ambiguous context,
the model’s strongest bias associates women with
emotionality (as discussed in Section 5.1). In dis-
ambiguated contexts, the model’s strongest bias
relates to the feminization of the nursing career and
the masculinization of doctors, with a bias score
of 0.78 indicating that the model is more likely to
say that a nurse is a woman than a man when
asked. This pattern, along with the model’s ten-
dency to link women with the homemaking role
(samb = 0.29), implies that Llama-SEA-LION-v2-
8B-IT sees women as more suited to domestic
roles (e.g., nurse and homemaker) than career-
oriented ones (e.g., doctor and economic provider).
The model’s bias profile also shows that it exhibits
biases related to sexual orientation. Along this di-
mension, the highest bias scores correspond to
prompts asking regarding stereotypical interests of
the bakla (queer man) and non-heterosexual indi-
viduals’ supposedly polygamous behaviors.
Tables 4 and 5 constitute the bias pro-

-- 7 of 12 --

Context Dimension Stereotype 5 Bias Score
ambiguous gender Women are more emotional than men. 0.57
ambiguous gender Men are more emotionally closed than women and lack empathy.

bakla (queer man) and non-heterosexual indi-
viduals’ supposedly polygamous behaviors.
Tables 4 and 5 constitute the bias pro-

-- 7 of 12 --

Context Dimension Stereotype 5 Bias Score
ambiguous gender Women are more emotional than men. 0.57
ambiguous gender Men are more emotionally closed than women and lack empathy. 0.42
ambiguous sexual orientation Gay people like fashion, design, and gossip. 0.31
ambiguous gender Women are homemakers. Men are economic providers. 0.29
ambiguous sexual orientation Gay people like clothes and fashion. 0.26
disambiguated gender Women are nurses. Men are doctors. 0.78
disambiguated gender Women are more affectionate than men. 0.18
disambiguated sexual orientation Gay people like fashion, design, and gossip. 0.17
disambiguated gender Women do gymnastics. Men do weightlifting. 0.17
disambiguated sexual orientation Non-heterosexual individuals struggle with monogamy. 0.16
Table 3: Llama-SEA-LION-v2-8B-IT bias profile. This table shows the five strongest biases in the
model when tested in the ambiguous and disambiguated contexts respectively.
Context Dimension Stereotype Bias Score
ambiguous gender Women are more emotional than men. 0.22
ambiguous gender Women do gymnastics. Men do weightlifting. 0.16
ambiguous gender Women are homemakers. Men are economic providers. 0.09
ambiguous gender Women are family-focused. Men are career-focused. 0.09
ambiguous gender Women are more affectionate than men. 0.09
disambiguated gender Women cook. Men carry heavy equipment. 0.38
disambiguated gender Women do gymnastics. Men do weightlifting. 0.24
disambiguated gender Women are more affectionate than men. 0.22
disambiguated gender Women are homemakers. Men are economic providers. 0.22
disambiguated gender Women are better teachers than men. 0.21
Table 4: SeaLLMs-v3-7B-Chat bias profile. This table shows the five strongest biases in the model
when tested in the ambiguous and disambiguated contexts respectively.
Context Dimension Stereotype Bias

Women are homemakers. Men are economic providers. 0.22
disambiguated gender Women are better teachers than men. 0.21
Table 4: SeaLLMs-v3-7B-Chat bias profile. This table shows the five strongest biases in the model
when tested in the ambiguous and disambiguated contexts respectively.
Context Dimension Stereotype Bias Score
ambiguous gender Women are more emotional than men. 1.00
ambiguous gender Women are trusting and gullible. Men are shrewd and clever. 0.25
ambiguous sexual orientation Non-heterosexual individuals are afraid of commitment. 0.25
ambiguous sexual orientation Non-heterosexual individuals are pedophiles. 0.16
ambiguous sexual orientation Gay people are vain. 0.13
disambiguated gender Women do gymnastics. Men do weightlifting. 1.00
disambiguated gender Women are obedient. Men are dominant. 0.10
disambiguated gender Lesbians and tomboys are suited for farming work. 0.09
disambiguated sexual orientation Non-heterosexual individuals are afraid of commitment. 0.08
disambiguated gender Lesbians and tomboys are brooding. 0.05
Table 5: roberta-tagalog-base bias bias profile. This table shows the five strongest biases in the
model when tested in the ambiguous and disambiguated contexts respectively.
files of SeaLLMs-v3-7B-Chat and roberta-
tagalog-base respectively. These models
largely demonstrate the same biases as Llama-
SEA-LION-v2-8B-IT, with many of their sex-
ist biases relating to emotion and domesticity
and their homophobic biases also connected to
polygamy. These similarities suggest that there
might be some overlap in the biases embedded
within these models’ pretraining corpora. Notably,
the most prominent biases in SeaLLMs-v3-7B-
Chat are all gender biases. Meanwhile, roberta-
tagalog-base alarmingly displays an unfair asso-
ciation between non-heterosexuality and pedophilia
(samb = 0.16).
Finally, it is also worth pointing out that while
most prompts returned a bias score of 0.20 or
less for SeaLLMs-v3-7B-Chat and roberta-
tagalog-base,

3-7B-
Chat are all gender biases. Meanwhile, roberta-
tagalog-base alarmingly displays an unfair asso-
ciation between non-heterosexuality and pedophilia
(samb = 0.16).
Finally, it is also worth pointing out that while
most prompts returned a bias score of 0.20 or
less for SeaLLMs-v3-7B-Chat and roberta-
tagalog-base, Llama-SEA-LION-v2-8B-IT
displayed higher bias scores across a larger se-
lection of prompts. Juxtaposing this with the fact
that among the three models, Llama-SEA-LION-
5Statements under the Stereotype column are author-
written characterizations of stereotypes present in the
most bias-inducing prompts.
v2-8B-IT had the highest FilBBQ accuracy score
(acc = 0.55) and was exposed to the most Filipino
tokens (∼1.2 billion) during training, we conjecture
that a model’s pretraining corpus size on a partic-
ular language and its eventual modeling ability in
said language may be positively correlated to its
biases in the language as well.
6. Conclusion
In this paper, we described our method for expand-
ing the currently available suite of BBQ benchmarks
to include Filipino, a Southeast Asian language with
emerging NLP resources. The process involved ad-
dressing issues in translating English bias datasets
into a new context. These issues included adjusting
demographic labels, deploying culturally appropri-
ate proper names, replacing contextually irrelevant
references, and adding in biases and stereotypes
unique to the Filipino setting. Resolving these chal-
lenges led to the creation of FilBBQ, a bias test
containing 10,576 QA prompts created from 123
templates. About 40% of these templates are new
to FilBBQ and specific to the local context. We

-- 8 of 12 --

then applied FilBBQ on PLMs capable of process-
ing the Filipino language to establish baseline bias
evaluation results. In doing so, we account for the
problem of response instability in generative PLMs
by implementing multiple bias evaluation runs and
grounding our robust final bias scores on

ext. We

-- 8 of 12 --

then applied FilBBQ on PLMs capable of process-
ing the Filipino language to establish baseline bias
evaluation results. In doing so, we account for the
problem of response instability in generative PLMs
by implementing multiple bias evaluation runs and
grounding our robust final bias scores on these
differently seeded runs. Our results confirm the
variability of bias scores obtained for different runs
of the FilBBQ evaluation. Averaging across these
runs, we generate model bias profiles that demon-
strate model biases relating to emotion, domestic-
ity, stereotyped interests, and polygamy. We hope
these insights can contribute to future research in-
vestigating how multilungual models learn bias and
how such bias can be mitigated for the benefit of
marginalized groups across cultures.
7. Ethical Considerations and
Limitations
Despite our efforts to incorporate into FilBBQ as
many of the biases present in Philippine culture as
possible, it is still highly unlikely that we were able to
encompass all of them. As such, benchmark users
should be wary not to interpret low bias scores from
the benchmark as an indicator that a model is com-
pletely free from bias. A more responsible use of
the benchmark would be to compare scores before
and after debiasing initiatives in order to conclude
if the intervention successfully addressed some bi-
ases known to be present in a model. Furthermore,
FilBBQ evaluation results are also highly depen-
dent on a model’s QA performance; consequently,
models with suboptimal QA capacities may not be
accurately assessed by the benchmark. As such,
it would also be prudent to consider bias evalua-
tion findings from non-QA-centric benchmarks or
methods in order to gain a more holistic picture
of a model’s inherent biases. Finally, we repeat
warnings issued by previous works developing bias
tests: these datasets should not be used in train-
ing PLMs because doing so would invalidate the
results of future bias

a-
tion findings from non-QA-centric benchmarks or
methods in order to gain a more holistic picture
of a model’s inherent biases. Finally, we repeat
warnings issued by previous works developing bias
tests: these datasets should not be used in train-
ing PLMs because doing so would invalidate the
results of future bias evaluations.
8. Bibliographical References
AI Singapore. 2023. SEA-LION (Southeast Asian
Languages In One Network): A family of large
language models for Southeast Asia.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou,
Venkatesh Saligrama, and Adam T Kalai. 2016.
Man is to computer programmer as woman is
to homemaker? Debiasing word embeddings.
In Advances in Neural Information Processing
Systems, volume 29. Curran Associates, Inc.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli,
Russ Altman, Simran Arora, Sydney von Arx,
Michael S Bernstein, Jeannette Bohg, Antoine
Bosselut, Emma Brunskill, et al. 2021. On the op-
portunities and risks of foundation models. arXiv
preprint arXiv:2108.07258.
Aylin Caliskan, Joanna J. Bryson, and Arvind
Narayanan. 2017. Semantics derived automati-
cally from language corpora contain human-like
biases. Science, 356(6334):183–186.
Tanise Ceron, Neele Falk, Ana Barić, Dmitry Niko-
laev, and Sebastian Padó. 2024. Beyond prompt
brittleness: Evaluating the reliability and consis-
tency of political worldviews in LLMs. Transac-
tions of the Association for Computational Lin-
guistics, 12:1378–1400.
Jan Christian Blaise Cruz and Charibeth Cheng.
2022. Improving large-scale language models
and resources for Filipino. In Proceedings of
the Thirteenth Language Resources and Evalu-
ation Conference, pages 6548–6555, Marseille,
France. European Language Resources Associ-
ation.
Pieter Delobelle, Ewoenam Tokpo, Toon Calders,
and Bettina Berendt. 2022. Measuring fairness
with biased rulers: A comparative study on bias
metrics for pre-trained language models. In Pro-
ceedings of the 2022 Conference of the North
American Chapter of the

arseille,
France. European Language Resources Associ-
ation.
Pieter Delobelle, Ewoenam Tokpo, Toon Calders,
and Bettina Berendt. 2022. Measuring fairness
with biased rulers: A comparative study on bias
metrics for pre-trained language models. In Pro-
ceedings of the 2022 Conference of the North
American Chapter of the Association for Compu-
tational Linguistics: Human Language Technolo-
gies, pages 1693–1706, Seattle, United States.
Association for Computational Linguistics.
Vittoria Dentella, Fritz Günther, and Evelina
Leivada. 2023. Systematic testing of three
language models reveals low language accu-
racy, absence of response stability, and a yes-
response bias. Proceedings of the National
Academy of Sciences, 120(51):e2309583120.
Nina Evason. Filipino culture: Naming [online].
2025.
Virginia Felkner, Ho-Chun Herbert Chang, Eu-
gene Jang, and Jonathan May. 2023. Wino-
Queer: A community-in-the-loop benchmark for
anti-LGBTQ+ bias in large language models.
In Proceedings of the 61st Annual Meeting of
the Association for Computational Linguistics
(Volume 1: Long Papers), pages 9126–9140,
Toronto, Canada. Association for Computational
Linguistics.
Kathleen C. Fraser, Isar Nejadgholi, and Svetlana
Kiritchenko. 2021. Understanding and counter-
ing stereotypes: A computational approach to the
stereotype content model. In Proceedings of the

-- 9 of 12 --

59th Annual Meeting of the Association for Com-
putational Linguistics and the 11th International
Joint Conference on Natural Language Process-
ing (Volume 1: Long Papers), pages 600–616,
Online. Association for Computational Linguis-
tics.
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow,
Md Mehrab Tanjim, Sungchul Kim, Franck Der-
noncourt, Tong Yu, Ruiyi Zhang, and Nesreen K.
Ahmed. 2024. Bias and fairness in large lan-
guage models: A survey. Computational Lin-
guistics, 50(3):1097–1179.
Lance Calvin Lim Gamboa, Yue Feng, and Mark
Lee. 2025a. Social bias in multilingual language
models: A survey.
Lance Calvin Lim

im, Sungchul Kim, Franck Der-
noncourt, Tong Yu, Ruiyi Zhang, and Nesreen K.
Ahmed. 2024. Bias and fairness in large lan-
guage models: A survey. Computational Lin-
guistics, 50(3):1097–1179.
Lance Calvin Lim Gamboa, Yue Feng, and Mark
Lee. 2025a. Social bias in multilingual language
models: A survey.
Lance Calvin Lim Gamboa, Yue Feng, and Mark G.
Lee. 2025b. Bias attribution in Filipino language
models: Extending a bias interpretability metric
for application on agglutinative languages. In Pro-
ceedings of the 6th Workshop on Gender Bias in
Natural Language Processing (GeBNLP), pages
195–205, Vienna, Austria. Association for Com-
putational Linguistics.
Lance Calvin Lim Gamboa and Mark Lee. 2025.
Filipino benchmarks for measuring sexist and
homophobic bias in multilingual language mod-
els from Southeast Asia. In Proceedings of
the First Workshop on Language Models for
Low-Resource Languages, pages 123–134, Abu
Dhabi, United Arab Emirates. Association for
Computational Linguistics.
J. Neil C. Garcia. 1996. Philippine Gay Culture:
Binabae to Bakla, Silahis to MSM. Hong Kong
University Press.
Yufei Huang and Deyi Xiong. 2024. CBBQ: A
Chinese bias benchmark dataset curated with
human-AI collaboration for large language mod-
els. In Proceedings of the 2024 Joint Interna-
tional Conference on Computational Linguistics,
Language Resources and Evaluation (LREC-
COLING 2024), pages 2917–2929, Torino, Italia.
ELRA and ICCL.
Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Al-
ice Oh, and Hwaran Lee. 2024. KoBBQ: Korean
bias benchmark for question answering. Trans-
actions of the Association for Computational Lin-
guistics, 12:507–524.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Ka-
lika Bali, and Monojit Choudhury. 2020. The state
and fate of linguistic diversity and inclusion in the
NLP world. In Proceedings of the 58th Annual
Meeting of the Association for Computational Lin-
guistics, pages 6282–6293, Online. Association
for Computational Linguistics.
Masahiro Kaneko,

anty, Amar Budhiraja, Ka-
lika Bali, and Monojit Choudhury. 2020. The state
and fate of linguistic diversity and inclusion in the
NLP world. In Proceedings of the 58th Annual
Meeting of the Association for Computational Lin-
guistics, pages 6282–6293, Online. Association
for Computational Linguistics.
Masahiro Kaneko, Aizhan Imankulova, Danushka
Bollegala, and Naoaki Okazaki. 2022. Gender
bias in masked language models for multiple lan-
guages. In Proceedings of the 2022 Conference
of the North American Chapter of the Association
for Computational Linguistics: Human Language
Technologies, pages 2740–2750, Seattle, United
States. Association for Computational Linguis-
tics.
Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sab-
harwal, and Vivek Srikumar. 2020. UNQOVER-
ing stereotyping biases via underspecified ques-
tions. In Findings of the Association for Compu-
tational Linguistics: EMNLP 2020, pages 3475–
3489, Online. Association for Computational Lin-
guistics.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and
Samuel R. Bowman. 2020. CrowS-pairs: A
challenge dataset for measuring social biases
in masked language models. In Proceedings of
the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages
1953–1967, Online. Association for Computa-
tional Linguistics.
Ernesto III Nodado. Soaring beyond gender sport-
ing norms [online]. 2024.
Debora Nozza, Federico Bianchi, and Dirk Hovy.
2021. HONEST: Measuring hurtful sentence
completion in language models. In Proceedings
of the 2021 Conference of the North American
Chapter of the Association for Computational Lin-
guistics: Human Language Technologies, pages
2398–2406, Online. Association for Computa-
tional Linguistics.
Alicia Parrish, Angelica Chen, Nikita Nangia,
Vishakh Padmakumar, Jason Phang, Jana
Thompson, Phu Mon Htut, and Samuel Bowman.
2022. BBQ: A hand-built bias benchmark for
question answering. In Findings of the Associ-
ation for Computational Linguistics: ACL 2022,
pages

ine. Association for Computa-
tional Linguistics.
Alicia Parrish, Angelica Chen, Nikita Nangia,
Vishakh Padmakumar, Jason Phang, Jana
Thompson, Phu Mon Htut, and Samuel Bowman.
2022. BBQ: A hand-built bias benchmark for
question answering. In Findings of the Associ-
ation for Computational Linguistics: ACL 2022,
pages 2086–2105, Dublin, Ireland. Association
for Computational Linguistics.
Philippine Statistics Authority. Philippines’ most
common baby names of 2022 [online]. 2022.
Michael Prieler and Dave Centeno. 2025. Some
gender stereotypes persist in filipino tv ads: A
content analytic investigation of tv advertising in
2010 and 2020. Sex Roles, 91.
Shalaka Satheesh, Katrin Klug, Katharina Beckh,
Héctor Allende-Cid, Sebastian Houben, and
Teena Hassan. 2025. GG-BBQ: German gender
bias benchmark for question answering. In Pro-
ceedings of the 6th Workshop on Gender Bias in

-- 10 of 12 --

Natural Language Processing (GeBNLP), pages
137–148, Vienna, Austria. Association for Com-
putational Linguistics.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao,
Abu Awal Md Shoeb, Abubakar Abid, Adam
Fisch, Adam R. Brown, Adam Santoro, Aditya
Gupta, Adrià Garriga-Alonso, Agnieszka Kluska,
Aitor Lewkowycz, Akshat Agarwal, Alethea
Power, Alex Ray, Alex Warstadt, Alexander W.
Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Ali-
cia Parrish, Allen Nie, Aman Hussain, Amanda
Askell, Amanda Dsouza, Ambrose Slone, Ameet
Rahane, Anantharaman S. Iyer, Anders Johan
Andreassen, Andrea Madotto, Andrea Santilli,
Andreas Stuhlmüller, Andrew M. Dai, Andrew La,
Andrew Kyle Lampinen, Andy Zou, Angela Jiang,
Angelica Chen, Anh Vuong, Animesh Gupta,
Anna Gottardi, Antonio Norelli, Anu Venkatesh,
Arash Gholamidavoodi, Arfa Tabassum, Arul
Menezes, Arun Kirubarajan, Asher Mullokandov,
Ashish Sabharwal, Austin Herrick, Avia Efrat,
Aykut Erdem, Ayla Karakaş, B. Ryan Roberts,
Bao Sheng Loe, Barret Zoph, Bartłomiej Bo-
janowski, Batuhan Özyurt, Behnam Heday-
atnia, Behnam Neyshabur, Benjamin Inden,
Benno

Venkatesh,
Arash Gholamidavoodi, Arfa Tabassum, Arul
Menezes, Arun Kirubarajan, Asher Mullokandov,
Ashish Sabharwal, Austin Herrick, Avia Efrat,
Aykut Erdem, Ayla Karakaş, B. Ryan Roberts,
Bao Sheng Loe, Barret Zoph, Bartłomiej Bo-
janowski, Batuhan Özyurt, Behnam Heday-
atnia, Behnam Neyshabur, Benjamin Inden,
Benno Stein, Berk Ekmekci, Bill Yuchen Lin,
Blake Howald, Bryan Orinion, Cameron Diao,
Cameron Dour, Catherine Stinson, Cedrick Ar-
gueta, Cesar Ferri, Chandan Singh, Charles
Rathkopf, Chenlin Meng, Chitta Baral, Chiyu
Wu, Chris Callison-Burch, Christopher Waites,
Christian Voigt, Christopher D Manning, Christo-
pher Potts, Cindy Ramirez, Clara E. Rivera,
Clemencia Siro, Colin Raffel, Courtney Ashcraft,
Cristina Garbacea, Damien Sileo, Dan Gar-
rette, Dan Hendrycks, Dan Kilman, Dan Roth,
C. Daniel Freeman, Daniel Khashabi, Daniel
Levy, Daniel Moseguí González, Danielle Per-
szyk, Danny Hernandez, Danqi Chen, Daphne Ip-
polito, Dar Gilboa, David Dohan, David Drakard,
David Jurgens, Debajyoti Datta, Deep Ganguli,
Denis Emelin, Denis Kleyko, Deniz Yuret, Derek
Chen, Derek Tam, Dieuwke Hupkes, Diganta
Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi
Yang, Dong-Ho Lee, Dylan Schrader, Ekate-
rina Shutova, Ekin Dogus Cubuk, Elad Segal,
Eleanor Hagerman, Elizabeth Barnes, Eliza-
beth Donoway, Ellie Pavlick, Emanuele Rodolà,
Emma Lam, Eric Chu, Eric Tang, Erkut Erdem,
Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan
Jerzak, Ethan Kim, Eunice Engefu Manyasi,
Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh
Siar, Fernando Martínez-Plumed, Francesca
Happé, Francois Chollet, Frieda Rong, Gaurav
Mishra, Genta Indra Winata, Gerard de Melo,
Germàn Kruszewski, Giambattista Parascan-
dolo, Giorgio Mariani, Gloria Xinyue Wang, Gon-
zalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-
Ari, Hana Galijasevic, Hannah Kim, Hannah
Rashkin, Hannaneh Hajishirzi, Harsh Mehta,
Hayden Bogar, Henry Francis Anthony Shevlin,
Hinrich Schuetze, Hiromu Yakura, Hongming
Zhang, Hugh Mee Wong, Ian Ng, Isaac

battista Parascan-
dolo, Giorgio Mariani, Gloria Xinyue Wang, Gon-
zalo Jaimovitch-Lopez, Gregor Betz, Guy Gur-
Ari, Hana Galijasevic, Hannah Kim, Hannah
Rashkin, Hannaneh Hajishirzi, Harsh Mehta,
Hayden Bogar, Henry Francis Anthony Shevlin,
Hinrich Schuetze, Hiromu Yakura, Hongming
Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble,
Jaap Jumelet, Jack Geissinger, Jackson Kernion,
Jacob Hilton, Jaehoon Lee, Jaime Fernández
Fisac, James B Simon, James Koppel, James
Zheng, James Zou, Jan Kocon, Jana Thompson,
Janelle Wingfield, Jared Kaplan, Jarema Radom,
Jascha Sohl-Dickstein, Jason Phang, Jason Wei,
Jason Yosinski, Jekaterina Novikova, Jelle Boss-
cher, Jennifer Marsh, Jeremy Kim, Jeroen Taal,
Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Ji-
aming Song, Jillian Tang, Joan Waweru, John
Burden, John Miller, John U. Balis, Jonathan
Batchelder, Jonathan Berant, Jörg Frohberg, Jos
Rozen, Jose Hernandez-Orallo, Joseph Boude-
man, Joseph Guerr, Joseph Jones, Joshua B.
Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil
Kanclerz, Karen Livescu, Karl Krauth, Karthik
Gopalakrishnan, Katerina Ignatyeva, Katja Mark-
ert, Kaustubh Dhole, Kevin Gimpel, Kevin
Omondi, Kory Wallace Mathewson, Kristen Chi-
afullo, Ksenia Shkaruta, Kumar Shridhar, Kyle
McDonell, Kyle Richardson, Laria Reynolds, Leo
Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia
Contreras-Ochando, Louis-Philippe Morency,
Luca Moschella, Lucas Lam, Lucy Noble, Lud-
wig Schmidt, Luheng He, Luis Oliveros-Colón,
Luke Metz, Lütfi Kerem Senel, Maarten Bosma,
Maarten Sap, Maartje Ter Hoeve, Maheen Fa-
rooqi, Manaal Faruqui, Mantas Mazeika, Marco
Baturan, Marco Marelli, Marco Maru, Maria Jose
Ramirez-Quintana, Marie Tolkiehn, Mario Giu-
lianelli, Martha Lewis, Martin Potthast, Matthew L
Leavitt, Matthias Hagen, Mátyás Schubert, Med-
ina Orduna Baitemirova, Melody Arnaud, Melvin
McElrath, Michael Andrew Yee, Michael Co-
hen, Michael Gu, Michael Ivanitskiy, Michael
Starritt, Michael Strube, Michał Swędrowski,
Michele Bevilacqua, Michihiro

n, Mario Giu-
lianelli, Martha Lewis, Martin Potthast, Matthew L
Leavitt, Matthias Hagen, Mátyás Schubert, Med-
ina Orduna Baitemirova, Melody Arnaud, Melvin
McElrath, Michael Andrew Yee, Michael Co-
hen, Michael Gu, Michael Ivanitskiy, Michael
Starritt, Michael Strube, Michał Swędrowski,
Michele Bevilacqua, Michihiro Yasunaga, Mi-
hir Kale, Mike Cain, Mimee Xu, Mirac Suz-
gun, Mitch Walker, Mo Tiwari, Mohit Bansal,
Moin Aminnaseri, Mor Geva, Mozhdeh Gheini,
Mukund Varma T, Nanyun Peng, Nathan An-
drew Chi, Nayeon Lee, Neta Gur-Ari Krakover,
Nicholas Cameron, Nicholas Roberts, Nick Do-
iron, Nicole Martinez, Nikita Nangia, Niklas Deck-
ers, Niklas Muennighoff, Nitish Shirish Keskar,
Niveditha S. Iyer, Noah Constant, Noah Fiedel,
Nuan Wen, Oliver Zhang, Omar Agha, Omar
Elbaghdadi, Omer Levy, Owain Evans, Pablo
Antonio Moreno Casares, Parth Doshi, Pas-
cale Fung, Paul Pu Liang, Paul Vicol, Pegah
Alipoormolabashi, Peiyuan Liao, Percy Liang,
Peter W Chang, Peter Eckersley, Phu Mon
Htut, Pinyu Hwang, Piotr Miłkowski, Piyush

-- 11 of 12 --

Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu
Mei, Qing Lyu, Qinlang Chen, Rabin Ban-
jade, Rachel Etta Rudolph, Raefer Gabriel,
Rahel Habacker, Ramon Risco, Raphaël Mil-
lière, Rhythm Garg, Richard Barnes, Rif A.
Saurous, Riku Arakawa, Robbe Raymaekers,
Robert Frank, Rohan Sikand, Roman Novak,
Roman Sitelew, Ronan Le Bras, Rosanne Liu,
Rowan Jacobs, Rui Zhang, Russ Salakhutdinov,
Ryan Andrew Chi, Seungjae Ryan Lee, Ryan
Stovall, Ryan Teehan, Rylan Yang, Sahib Singh,
Saif M. Mohammad, Sajant Anand, Sam Dillavou,
Sam Shleifer, Sam Wiseman, Samuel Gruetter,
Samuel R. Bowman, Samuel Stern Schoenholz,
Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous,
Sarik Ghazarian, Sayan Ghosh, Sean Casey, Se-
bastian Bischoff, Sebastian Gehrmann, Sebas-
tian Schuster, Sepideh Sadeghi, Shadi Hamdan,
Sharon Zhou, Shashank Srivastava, Sherry Shi,
Shikhar Singh, Shima Asaadi, Shixiang Shane
Gu, Shubh Pachchigar, Shubham Toshniwal,
Shyam Upadhyay, Shyamolima

v Kwatra, Sarah A. Rous,
Sarik Ghazarian, Sayan Ghosh, Sean Casey, Se-
bastian Bischoff, Sebastian Gehrmann, Sebas-
tian Schuster, Sepideh Sadeghi, Shadi Hamdan,
Sharon Zhou, Shashank Srivastava, Sherry Shi,
Shikhar Singh, Shima Asaadi, Shixiang Shane
Gu, Shubh Pachchigar, Shubham Toshniwal,
Shyam Upadhyay, Shyamolima Shammie Deb-
nath, Siamak Shakeri, Simon Thormeyer, Si-
mone Melzi, Siva Reddy, Sneha Priscilla Makini,
Soo-Hwan Lee, Spencer Torene, Sriharsha Hat-
war, Stanislas Dehaene, Stefan Divic, Stefano
Ermon, Stella Biderman, Stephanie Lin, Stephen
Prasad, Steven Piantadosi, Stuart Shieber, Sum-
mer Misherghi, Svetlana Kiritchenko, Swaroop
Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu,
Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo
Desbordes, Theodore Rothschild, Thomas Phan,
Tianle Wang, Tiberius Nkinyili, Timo Schick,
Timofei Kornev, Titus Tunduny, Tobias Gersten-
berg, Trenton Chang, Trishala Neeraj, Tushar
Khot, Tyler Shultz, Uri Shaham, Vedant Misra,
Vera Demberg, Victoria Nyamai, Vikas Rau-
nak, Vinay Venkatesh Ramasesh, vinay uday
prabhu, Vishakh Padmakumar, Vivek Srikumar,
William Fedus, William Saunders, William Zhang,
Wout Vossen, Xiang Ren, Xiaoyu Tong, Xin-
ran Zhao, Xinyi Wu, Xudong Shen, Yadollah
Yaghoobzadeh, Yair Lakretz, Yangqiu Song,
Yasaman Bahri, Yejin Choi, Yichi Yang, Sophie
Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu-
fang Hou, Yuntao Bai, Zachary Seid, Zhuoye
Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and
Ziyi Wu. 2023. Beyond the imitation game: Quan-
tifying and extrapolating the capabilities of lan-
guage models. Transactions on Machine Learn-
ing Research. Featured Certification.
Vivienne Velez Valledor-Lukey. 2012. Pagkababae
at Pagkalalake (Femininity and Masculinity): De-
veloping a Filipino Gender Trait Inventory and
predicting self-esteem and sexism. Ph.D. thesis,
Syracuse University.
Gina Velasco. 2022. “That’s My Tomboy”: Queer
Filipinx diasporic transmasculinities. Alon: Jour-
nal for Filipinx American and

or-Lukey. 2012. Pagkababae
at Pagkalalake (Femininity and Masculinity): De-
veloping a Filipino Gender Trait Inventory and
predicting self-esteem and sexism. Ph.D. thesis,
Syracuse University.
Gina Velasco. 2022. “That’s My Tomboy”: Queer
Filipinx diasporic transmasculinities. Alon: Jour-
nal for Filipinx American and Diasporic Studies,
2(1):67–73.
Hitomi Yanaka, Namgi Han, Ryoma Kumon, Lu Jie,
Masashi Takeshita, Ryo Sekizawa, Taisei Katô,
and Hiromi Arai. 2025. JBBQ: Japanese bias
benchmark for analyzing social biases in large
language models. In Proceedings of the 6th
Workshop on Gender Bias in Natural Language
Processing (GeBNLP), pages 1–17, Vienna, Aus-
tria. Association for Computational Linguistics.
Jingfeng Yang, Haoming Jiang, Qingyu Yin, Dan-
qing Zhang, Bing Yin, and Diyi Yang. 2022. SE-
QZERO: Few-shot compositional semantic pars-
ing with sequential prompts and zero-shot mod-
els. In Findings of the Association for Compu-
tational Linguistics: NAACL 2022, pages 49–60,
Seattle, United States. Association for Computa-
tional Linguistics.
Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Ma-
hani Aljunied, Jianyu Wang, Chaoqun Liu, Yue
Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia,
Xin Li, and Lidong Bing. 2024. SeaLLMs 3: Open
foundation and chat multilingual large language
models for Southeast Asian languages.
Muitze Zulaika and Xabier Saralegi. 2025.
BasqBBQ: A QA benchmark for assessing social
biases in LLMs for Basque, a low-resource lan-
guage. In Proceedings of the 31st International
Conference on Computational Linguistics, pages
4753–4767, Abu Dhabi, UAE. Association for
Computational Linguistics.
9. Language Resource References

-- 12 of 12 --