Photo credit: www.technologyreview.com
“I hope that individuals leverage [SHADES] as a diagnostic tool to pinpoint where potential issues may exist within a model,” remarks Talat. “It serves as an insight into what might be lacking in a model, areas where confidence in its performance may wane, and whether it can be deemed accurate.”
In the development of a multilingual dataset, the research team gathered native and fluent speakers from diverse linguistic backgrounds such as Arabic, Chinese, and Dutch. They compiled a comprehensive list of stereotypes they could identify in their respective languages, which another native speaker subsequently verified. The speakers annotated each stereotype with information regarding the regions in which it was recognized, the target demographic, and the type of bias present.
Each stereotype was translated into English by the participants—a language common among all contributors—before additional translations into other languages were conducted. The speakers then identified whether the translated stereotypes were acknowledged in their language, culminating in 304 stereotypes related to aspects such as physical appearance, personal identity, and various social factors like occupation.
The team is scheduled to showcase their findings at the forthcoming annual conference of the Nations of the Americas chapter of the Association for Computational Linguistics in May.
“This represents a fascinating methodology,” states Myra Cheng, a PhD candidate at Stanford University specializing in social biases within AI. “The breadth of languages and cultures included captures the intricacies and subtleties of various biases.”
Mitchell expresses her aspiration for additional contributors to enrich SHADES with further languages, stereotypes, and regional insights, a resource that is publicly accessible. This collaborative effort aims to enhance the development of more effective language models in the future. “It has been an extensive collaborative initiative involving individuals dedicated to advancing technology for better outcomes,” she notes.
Source
www.technologyreview.com