Biases & Stereotypes in Generative AI

Subscribe to the weekly posts via email here

 

Though the terms “Bias” and “Stereotype” make us think of highly offensive caricatures and slurs at first, even having to explain your job to someone who has no familiarity with it or correcting someone’s gendered assumption of your interests are examples of biases and stereotypes we might deal with on an everyday basis. Sometimes, biases and stereotypes are subtle and implicit; for example, it might be the underhanded comment someone makes about how they are surprised to see you be good at a particular skill. These interactions are all examples of how biases and stereotypes can materialize in our everyday lives. 

Cognitively, biases and stereotypes are not inherently harmful; they are tools that our brains use to sort through information in a more efficient way. Similar to our brains, Generative AI (GenAI) tools—and other AI tools built using machine learning techniques—use patterns in information to build an ‘understanding’ of the world and produce responses. For example, a GenAI tool like a chatbot might be trained on many, many conversations (think 30k+ dialogs!), which it uses to understand what happens in a conversation. This is similar to how kids might learn that when someone says “hello, how are you?”, the expected answer might be “good, how are you?”. However, the tools are limited to what they see in the datasets---so the same chatbot would not be able to write an argumentative blog post without being trained using a large number of argumentative texts.  

In the same way, there are patterns in the training datasets that get repeated by GenAI tools. For example, predicting the future winners of an award might be biased by a training dataset based on past winners, if that award was only given to men in the past (much like the image below, showing the overrepresentation of males in the field of physics). Something similar happened with the AI recruitment tool Amazon was hoping to use, which they trained on current employees. They thought the best applications for new hires would resemble the people they hired in the past, which does make sense…except that they had not been hiring many women, so the model thought a woman didn’t fit the idea of a ‘good applicant’ it had built! 


A diagram with black and white photos of 8 people. 7 of them are men. 1 of them is a woman. The diagram is labelled “latent bias”
Screenshot from the “3 types of bias in AI | Machine learning” video by Google, demonstrating the historical gender bias in the physics field.

When these biases are about groups of people, and when we act on them without critically thinking about where our initial (potentially biased!) thoughts may come from, we can discriminate or otherwise harm people. Growing up, most of us learn the importance of thinking before we speak and act, which is necessary to counter the harmful behaviors that can come from biases and stereotypes. GenAI tools are developed with some filters and safeguards so that these tools can avoid harmful outputs, too. When biases in training data are discovered, or observed in outputs, developers can ‘rebalance’ the dataset by including more examples of the underrepresented groups and run the training process again, so that they remove that pattern of only one group being connected with a good output. Other measures include blacklisting some topics, like hate speech, so that the tool refuses to consume or generate content that is more than likely to be harmful. 

However, these measures only target the kinds of things that are obvious and highly harmful, leaving the more subtle biases and stereotypes in the tool. Research has shown that outputs from GenAI tools still contain biases and stereotypes---for example, Bloomberg’s investigation of a text-to-image tool showed Stable Diffusion generates images that are aligned with harmful gender, race and class stereotypes. UNESCO’s study on popular text generation models, including the earlier versions of ChatGPT’s foundation and Meta’s chatbot, found bias against women, LGBTQI+ people, and racially marginalized communities. An overview of research on chatbots concludes that chatbots urgently need to be improved, as when they are used in situations like automated diagnosis in healthcare, they can greatly disadvantage people who are already marginalized. 

It is important to be aware of the types of harmful biases and stereotypes that can appear in the responses from GenAI tools, so that we can analyze them critically and edit before using (or avoid them entirely for a particular task). 

The Challenge

Your challenge is to check the Generative AI tool(s) you use most often, or have considered using, for biases and stereotypes. 

Here is an example of how you can do that: 

  • First open up a GenAI tool you use or have considered using. 
  • Then, ask the tool to generate something about a topic you already know well. This can be your expertise or a group you are already a part of. 
  • Lastly, check the tool’s response for accuracy and biases/stereotypes. What might this response teach someone who doesn’t know about this topic/group of people? How might it affect their perception of you? What assumptions could they make? 

As an example, I am a teaching assistant in a Media Studies program. I would ask ChatGPT, used widely by my students, to summarize a reading about how technology is political, and generate an essay that discusses the idea through examples. Given my existing knowledge, I can identify which parts of the output are incorrect, biased, or based on a stereotype. 

You can repeat the activity after slightly changing the prompt, and compare the responses across the prompts. Are there ways of phrasing the prompt that result in better outputs, with higher accuracy and fewer biases? 

You can also repeat the activity with different GenAI tools and compare the responses across the tools. Are there tools that are better than others, when it comes to biases/stereotypes about that particular issue? 

Subscribe to the weekly posts via email here

 

References

Disclosure

No Generative AI tool was used in creating this blog post. 

Your Challenger